Data transmission method, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a data transmission method, electronic equipment and a storage medium, and the data transmission method comprises the steps: obtaining to-be-transmitted first communication data, the ratio of the memory access operation time consumption corresponding to the communication type of the first communication data to the communication duration being greater than a proportion threshold; according to the communication type of the first communication data, obtaining a corresponding vectorization memory access operator and a first default operator; determining a target communication operator used for processing the first communication data from the vectorization memory access operator and the first default operator; and if it is determined that the vectorized memory access operator is the target communication operator, transmitting at least part of data included in the first communication data from a first memory of a first computing node to a second memory of a second computing node according to a vectorized memory access mode through the vectorized memory access operator. According to the scheme, the efficiency of data transmission between the GPUs in the model training process can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of communication technologies, and in particular to a data transmission method, electronic device, and storage medium. Background Art

[0002] The Zero Redundancy Optimizer (ZeRO) is an optimization strategy for distributed training of large-scale models. It saves storage space by partitioning or distributing the optimizer state. When using ZeRO for distributed training, data must be transferred between different graphics processing units (GPUs).

[0003] Currently, when using ZeRO for distributed training, data exchange between different GPUs is implemented based on the communication operators provided by the collective communication library.

[0004] However, when transmitting data between different GPUs based on the communication operator provided by the collective communication library, if the starting address of the source data and the starting address of the target data are not 16-bytes aligned, the communication operator will access the memory by reading and writing 1 byte of data at a time. The memory access operation accounts for a large proportion of the communication time, resulting in low efficiency in data transmission between GPUs, which in turn affects the efficiency of model training. Summary of the Invention

[0005] In view of this, embodiments of the present application provide a data transmission method, an electronic device, and a storage medium to at least partially solve the above problems.

[0006] According to a first aspect of an embodiment of the present application, a data transmission method is provided, including: obtaining first communication data to be transmitted, wherein the ratio of the memory access operation time corresponding to the communication type of the first communication data to the communication duration is greater than a proportion threshold; according to the communication type of the first communication data, obtaining a corresponding vectorized memory access operator and a first default operator; determining a target communication operator for processing the first communication data from the vectorized memory access operator and the first default operator; if it is determined that the vectorized memory access operator is the target communication operator, then transmitting at least part of the data included in the first communication data from the first memory of the first computing node to the second memory of the second computing node in a vectorized memory access manner through the vectorized memory access operator.

[0007] According to the second aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the method described in the first aspect above.

[0008] According to a third aspect of the embodiments of the present application, a computer storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method described in the first aspect is implemented.

[0009] According to a fourth aspect of an embodiment of the present application, a computer program product is provided, comprising computer instructions, wherein the computer instructions instruct a computing device to execute the method described in the first aspect above.

[0010] It can be seen from the above technical solution that for the communication type in which the ratio of the memory access operation time to the communication duration is greater than the proportion threshold, a vectorized memory access operator and a default operator are constructed. The ratio of the memory access operation time to the communication duration corresponding to the communication type of the first communication data is greater than the proportion threshold, so the target communication operator for processing the first communication data can be determined from the corresponding vectorized memory access operator and the first default operator. If the vectorized memory access operator is determined as the target communication operator, then at least part of the data included in the first communication data can be transferred from the memory of the first computing node to the memory of the second computing node in accordance with the vectorized memory access method through the vectorized memory access operator. For the first communication data whose starting address in the memory is not aligned with NByte, the transmission of part of the data can be completed in accordance with the vectorized memory access method. Compared with the access method in which each access instruction accesses 1Byte of data, the transmission of part of the data in accordance with the vectorized memory access method can reduce the time consumed in transmitting the first communication data, thereby improving the efficiency of transmitting communication data between computing nodes. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0012] Figure 1 is a schematic diagram of an exemplary system used in one embodiment of the present application;

[0013] Figure 2 This is a flow chart of a data transmission method according to an embodiment of the present application;

[0014] Figure 3is a flow chart of a method for determining a target communication operator according to an embodiment of the present application;

[0015] Figure 4 is a flow chart of a communication data processing method according to an embodiment of the present application;

[0016] Figure 5 This is a schematic diagram of a data transmission process according to an embodiment of the present application;

[0017] Figure 6 is a schematic diagram of a data transmission process according to another embodiment of the present application;

[0018] Figure 7 is a schematic diagram of a data transmission process according to another embodiment of the present application;

[0019] Figure 8 is a schematic diagram of a data transmission device according to an embodiment of the present application;

[0020] Figure 9 It is a schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] The present application is described below based on examples, but the present application is not limited to these examples. In the detailed description of the present application below, certain specific details are described in detail. Those skilled in the art can fully understand the present application without describing these details. To avoid obscuring the essence of the present application, well-known methods, processes, and procedures are not described in detail. In addition, the drawings are not necessarily drawn to scale.

[0022] First, some nouns or terms that appear in the process of describing the embodiments of the present application are subject to the following explanations.

[0023] Communication operators: When performing distributed training on large models, data needs to be transmitted between different computing units (such as GPUs). Communication operators are functions used for communication between distributed computing units. For example, communication operators can implement broadcast, scatter, gather, all_gather, reduce, all_reduce, and reduce_scatter.

[0024] Vectorized memory access: Vectorized memory access is a way of accessing memory. Multiple data (such as multiple float data) can be requested through one instruction. Vectorized memory access can read more data with fewer instructions.

[0025] Vectorized memory access operator: A vectorized memory access operator is a communication operator that supports accessing memory in a vectorized manner. For example, a vectorized memory access operator can request 16 bytes of data with a single instruction.

[0026] Default operator: The default operator is a communication operator that accesses memory in a non-vectorized manner. The default operator requests 1 Byte of data through one instruction.

[0027] Zero Redundancy Optimizer: Zero Redundancy Optimizer (ZeRO) is an optimization strategy for distributed training of large-scale models. By splitting or distributing the optimizer state, it can save storage space.

[0028] Exemplary Systems

[0029] Figure 1 An exemplary system for a key generation method applicable to an embodiment of the present application is shown. Figure 1 As shown, the system may include a cloud service terminal 102, a communication network 104 and at least one user device 106. Figure 1 The example is a plurality of user devices 106. It should be noted that the solution of the embodiment of the present application can be applied to both the cloud service end 102 and the user device 106.

[0030] The cloud server 102 can be any suitable device for storing information, data, programs and / or any other suitable type of content, including but not limited to distributed storage system devices, server clusters, computing cloud server clusters, etc. In some embodiments, the cloud server 102 can perform any suitable function. For example, in some embodiments, the cloud server 102 can be used to perform distributed training of large-scale models through ZeRO. The cloud server 102 includes multiple GPUs, and distributed training of large-scale models is implemented based on multiple GPUs. During the model training process, data needs to be transmitted between different GPUs. ZeRO can implement communication between different GPUs through various types of communication operators.

[0031] The communication network 104 can be any suitable combination of one or more wired and / or wireless networks. For example, the communication network 104 can include any one or more of the following: the Internet, an intranet, a wide area network (WAN), a local area network (LAN), a wireless network, a digital subscriber line (DSL) network, a frame relay network, an asynchronous transfer mode (ATM) network, a virtual private network (VPN), and / or any other suitable communication network. The user device 106 can be connected to the communication network 104 via one or more communication links (e.g., communication link 112), and the communication network 104 can be linked to the cloud service end 102 via one or more communication links (e.g., communication link 114). The communication link can be any communication link suitable for transmitting data between the cloud service end 102 and the user device 106, such as a network link, a dial-up link, a wireless link, a hard-wired link, any other suitable communication link, or any suitable combination of such links.

[0032] The user device 106 may include any one or more user devices suitable for interaction. In some embodiments, upon receiving a model training request from the user device 106, the cloud service 102 may implement distributed model training using multiple GPUs based on a specified dataset and model architecture, and transfer data between different GPUs based on ZeRO. The user device 106 may include any suitable type of device, for example, a mobile device, a tablet computer, a laptop computer, a desktop computer, a wearable computer, a vehicle system, and / or any other suitable type of user device.

[0033] The embodiment of the present application mainly focuses on the process of data transmission between GPUs during the distributed training of large-scale models by the cloud server 102 using ZeRO. The process of data transmission between GPUs will be described in detail later.

[0034] Data transmission method

[0035] Based on the above system, an embodiment of the present application provides a data transmission method to realize data transmission between different computing units (such as GPUs). The data transmission method is described in detail below through multiple embodiments.

[0036] Figure 2 This is a flow chart of a data transmission method according to an embodiment of the present application. Figure 2 As shown, the data transmission method includes the following steps:

[0037] Step 201: Acquire first communication data to be transmitted.

[0038] The first communication data is the data that the first computing node needs to transmit to the second computing node. According to the communication requirements of the model distributed training process, the first computing node can transmit the first communication data to multiple computing nodes. The computing node is used to process input data or intermediate variables during the model distributed training process. The computing node can be a GPU. Different computing nodes have their own memory. When data is transmitted between computing nodes, the data to be transmitted is transferred from the memory of the source computing node to the memory of the destination computing node. For example, if GPU0 needs to send the first communication data to GPU1, the first communication data is stored in the memory of GPU0, and the first communication data needs to be transferred from the memory of GPU0 to the memory of GPU1.

[0039] The communication type of communication data indicates how the data will be transmitted. Communication types include broadcast, scatter, gather, all_gather, reduce, all_reduce, and reduce_scatter. Broadcast means distributing data on the root compute node to all other compute nodes. Scatter means scattering the data on the root compute node into data blocks of equal size, with each other compute node receiving one data block. Gather means concatenating the data blocks on other compute nodes, with the root compute node receiving this data. All_gather means that each compute node performs a gather operation, ensuring that each compute node receives data from all compute nodes. Reduce means performing a reduction operation (such as taking the maximum value or summing the values) on the data on each compute node and then writing the result to the root compute node. All_reduce means that each compute node acts as a root compute node and performs the aforementioned reduction operations. Scatter reduction means that a computing node divides its own data into data blocks of equal size, and other computing nodes perform reduction operations on the data obtained based on the index, that is, the scatter operation is performed first and then the reduction operation.

[0040] Different communication types have different memory access overheads, that is, different ratios of memory access operation time to communication duration. Memory access operation time refers to the time it takes to read data from source memory, and communication duration refers to the total time it takes to read data from source memory and write it to destination memory. For example, full-reduce operations have lower memory access overheads, while full-gather, scatter-reduce, and other operations have higher memory access overheads. The memory access overhead corresponding to the communication type of the first communication data is higher, that is, the ratio of the memory access operation time corresponding to the communication type of the first communication data to the communication time for transmitting the first communication data is greater than a ratio threshold, such as when the communication type of the first communication data is full-gather or scatter-reduce.

[0041] Step 202: According to the communication type of the first communication data, obtain a corresponding vectorized memory access operator and a first default operator.

[0042] If the ratio of the memory access operation time corresponding to the communication type to the communication duration is greater than the ratio threshold, a vectorized memory access operator and a default operator are pre-built for this communication type. The vectorized memory access operator can access data in the memory through vectorized memory access, and the default operator accesses data in the memory by accessing 1 Byte of data per access instruction.

[0043] For example, a default operator is constructed for the communication type of full reduction, a vectorized memory access operator and a default operator are constructed for the communication type of full aggregation, and a vectorized memory access operator and a default operator are constructed for the communication type of scattering reduction.

[0044] Because the ratio of the memory access operation time to the communication duration corresponding to the communication type of the first communication data is greater than the ratio threshold, a vectorized memory access operator and a default operator are pre-built for the communication type of the first communication data. Therefore, based on the communication type of the first communication data, a vectorized memory access operator and a default operator that can process the first communication data can be obtained. For ease of description, the default operator that can process the first communication data is defined as the first default operator.

[0045] For example, if full-gather processing needs to be performed on the first communication data, a vectorized memory access operator and a first default operator that can execute the full-gather operation are obtained.

[0046] Step 203: Determine a target communication operator for processing the first communication data from the vectorized memory access operator and the first default operator.

[0047] After obtaining the vectorized memory access operator and the first default operator corresponding to the communication type of the first communication data, the target communication operator can be determined from the vectorized memory access operator and the first default operator, and then the first communication data can be transferred from the memory of the first computing node to the memory of the second computing node through the target communication operator.

[0048] Step 204: If the vectorized memory access operator is determined to be the target communication operator, at least part of the data included in the first communication data is transferred from the first memory of the first computing node to the second memory of the second computing node according to the vectorized memory access method through the vectorized memory access operator.

[0049] After determining a target communication operator for processing the first communication data, the first communication data is transferred from the memory of the first computing node to the memory of the second computing node via the target communication operator. Since the vectorized memory access operator can access the memory via a vectorized memory access method, the vectorized memory access operator can access the memory in a manner such that each access instruction accesses N bytes of data, where N is a positive integer greater than 1, such as 16, 32, or 64.

[0050] The data transmission method of the embodiment of the present application is used to transmit data between different computing nodes. When transmitting the first communication data, the first computing node is the source computing node and the second computing node is the destination computing node, that is, the first communication data located at the first computing node needs to be transmitted to the second computing node. The first communication data can be located in the memory of the first computing node. When the first communication data is transmitted to the second computing node, the first communication data can be transmitted to the memory of the second computing node. For ease of description, the memory where the first communication data is located in the first computing node is defined as the first memory, and the memory for storing the first communication data in the second computing node is defined as the second memory.

[0051] After the vectorized memory access operator is determined as the target communication operator for processing the first communication data, part of the data included in the first communication data is accessed through the vectorized memory access operator in a vectorized memory access manner, so as to transfer this part of the data from the first memory to the second memory in a vectorized memory access manner.

[0052] During the distributed training of large models based on ZeRO, if the data in the memory is accessed in a vectorized memory access manner, the data is required to be aligned with the starting address NByte in the memory. However, only part of the first communication data may meet the NByte alignment of the starting address. This part of the data can be accessed in a vectorized memory access manner, while other data with non-NByte aligned starting addresses need to be accessed by accessing 1Byte of data per access instruction.

[0053] In an embodiment of the present application, for a communication type in which the ratio of the memory access operation time to the communication duration is greater than a percentage threshold, a vectorized memory access operator and a default operator are constructed. The ratio of the memory access operation time to the communication duration corresponding to the communication type of the first communication data is greater than the percentage threshold, so the target communication operator for processing the first communication data can be determined from the corresponding vectorized memory access operator and the first default operator. If the vectorized memory access operator is determined as the target communication operator, at least part of the data included in the first communication data can be transferred from the memory of the first computing node to the memory of the second computing node in accordance with the vectorized memory access method through the vectorized memory access operator. For the first communication data whose starting address in the memory is not aligned with NByte, the transmission of part of the data therein can be completed in accordance with the vectorized memory access method. Compared with the access method in which each access instruction accesses 1Byte of data, the transmission of part of the data therein in accordance with the vectorized memory access method can reduce the time consumed in transmitting the first communication data, thereby improving the efficiency of transmitting communication data between computing nodes.

[0054] In one possible implementation, if second communication data that needs to be transferred from a first memory to a second memory is obtained, and the ratio of the memory access operation time corresponding to the communication type of the second communication data to the communication duration is less than or equal to a ratio threshold, a second default operator for processing the second communication data is obtained based on the communication type of the second communication data, and the second communication data is transferred from the first memory to the second memory using the second default operator in a manner where each access instruction accesses 1 byte of data. The first memory is the memory of the first computing node, and the second memory is the memory of the second computing node.

[0055] If the ratio of the memory access operation duration to the communication duration corresponding to a communication type is less than or equal to the ratio threshold, a default operator is pre-built for that communication type instead of a vectorized memory access operator. For example, a default operator is built for the full protocol communication type.

[0056] Operators (vectorized memory access operators and / or default operators) need to be constructed separately for different communication types, and operators can perform operations of corresponding communication types. For example, a full reduction operator is constructed for the full protocol communication type, and the full reduction operator can perform full reduction operations. A full aggregation operator is constructed for the full aggregation communication type, and the full aggregation operator can perform full aggregation operations. For different communication types whose ratios of the corresponding memory access operation time to the communication duration are less than or equal to the proportion threshold, default operators are constructed respectively. Different default operators correspond to different communication types, and different default operators are used to implement different operations. The corresponding default operators can be obtained according to the communication type. For example, if communication type 1 and communication type 2 both meet the requirement that the ratio of the memory access operation time to the communication duration is less than or equal to the proportion threshold, then a default operator 1 is constructed for communication type 1, and a default operator 2 is constructed for communication type 2. The embodiment of the present application will not construct vectorized memory access operators for communication type 1 and communication type 2. Default operator 1 can be obtained according to communication type 1, and default operator 2 can be obtained according to communication type 2.

[0057] Because the ratio of the memory access operation time to the communication duration corresponding to the communication type of the second communication data is less than or equal to the ratio threshold, a default operator is pre-established for the communication type of the second communication data. Therefore, after obtaining the second communication data, a default operator capable of processing the second communication data can be obtained based on the communication type of the second communication data. For ease of distinction in description, the default operator capable of processing the second communication data is defined as the second default operator.

[0058] For example, if full reduction processing needs to be performed on the second communication data, a default operator that can perform a full reduction operation is obtained as a second default operator for processing the second communication data.

[0059] After determining the second default operator for processing the second communication data, the second communication data is transferred from the first memory to the second memory through the second default operator in a manner of accessing 1 Byte of data per access instruction.

[0060] In an embodiment of the present application, since the process of accessing memory by the vectorized memory access operator is more complex than that of the default operator and requires more computing resources, for communication types where the ratio of memory access operation time to communication duration is greater than the proportion threshold, processing the data of this communication type by the vectorized memory access operator can significantly improve the data transmission efficiency. However, for communication types where the ratio of memory access operation time to communication duration is less than or equal to the proportion threshold, processing the data of this communication type by the vectorized memory access operator does not significantly improve the data transmission efficiency and may even reduce the data transmission efficiency. Therefore, the first communication data is processed by the vectorized memory access operator to improve the overall efficiency of data transmission between GPUs, while the second communication data is processed by the default operator to ensure that the data transmission process occupies fewer computing resources.

[0061] In a possible implementation, the target communication operator for processing the first communication data may be determined based on the data volume of the first communication data and the communication performance of the vectorized memory access operator and the first default operator when processing the first communication data.

[0062] Figure 3 FIG. 1 is a flow chart of a method for determining a target communication operator according to an embodiment of the present application. Figure 3 As shown, the target communication operator determination method includes the following steps:

[0063] Step 301 : Determine whether the data volume of the first communication data is less than a data volume threshold; if so, execute step 302 ; otherwise, execute step 303 .

[0064] A data volume threshold is set in advance. If the data volume of the first communication data is greater than the data volume threshold, the benefit of processing the first communication data through the vectorized memory access operator is greater than the benefit of processing the first communication data through the first default operator. If the data volume of the first communication data is less than or equal to the data volume threshold, the benefit of processing the first communication data through the vectorized memory access operator is less than or equal to the benefit of processing the first communication data through the first default operator.

[0065] The benefit of using a communication operator to process communication data refers to the efficiency gain of using that operator on the overall model training process. For default operators and vectorized memory access operators of the same communication type, the default operator consumes fewer computing resources when executed than the vectorized memory access operator, but the data transmission efficiency of the default operator is lower than that of the vectorized memory access operator. Due to limited overall computing resources, the use of vectorized memory access operators can affect other processes during model training.

[0066] When the amount of communication data is less than the data amount threshold, the positive benefit brought by the computing resource occupation when processing the communication data through the default operator is greater than the negative benefit brought by the transmission efficiency, so the overall benefit of processing the communication data through the default operator is positive. The negative benefit brought by the computing resource occupation when processing the communication data through the vectorized memory access operator is greater than the positive benefit brought by the transmission efficiency, so the overall benefit of processing the communication data through the vectorized memory access operator is negative. Therefore, the benefit of processing the communication data through the default operator is greater than the benefit of processing the communication data through the vectorized memory access operator.

[0067] When the amount of communication data is greater than or equal to the quantity threshold, the positive benefit brought by the computing resource occupation when processing the communication data through the default operator is less than the negative benefit brought by the transmission efficiency, so the overall benefit of processing the communication data through the default operator is negative. When processing the communication data through the vectorized memory access operator, the negative benefit brought by the computing resource occupation is less than the positive benefit brought by the transmission efficiency, so the overall benefit of processing the communication data through the vectorized memory access operator is positive. Therefore, the benefit of processing the communication data through the vectorized memory access operator is greater than the benefit of processing the communication data through the default operator.

[0068] After receiving an instruction to transfer the first communication data from the first memory to the second memory, the data volume of the first communication data can be determined, and then the data volume of the first communication data can be compared with the data volume threshold. If the data volume of the first communication data is less than the data volume threshold, step 302 is executed; if the data volume of the first communication data is greater than or equal to the data volume threshold, step 303 is executed.

[0069] It should be noted that different communication types may correspond to different data volume thresholds. For example, the data volume threshold corresponding to full aggregation is S1, and the data volume threshold corresponding to scatter-reduce is S2, and S1 and S2 are different.

[0070] Step 302: Determine the first default operator as the target communication operator for processing the first communication data, and end the current process.

[0071] If the amount of the first communication data is less than the data amount threshold, the benefit of processing the first communication data using the vectorized memory access operator is less than the benefit of processing the first communication data using the first default operator. Therefore, the first default operator is determined as the target communication operator, and the first communication data is processed using the first default operator. When processing the first communication data using the first default operator, the first communication data is transferred from the first memory to the second memory in a manner such that each access instruction accesses 1 byte of data.

[0072] Compared to the first default operator, the vectorized memory access operator requires more computing resources to execute, and total computing resources are limited, so it affects other processes in the model training process. When the amount of first communication data is small, the benefits of using the vectorized memory access operator are not worth the loss of computing resources required by the vectorized memory access operator. Therefore, the first default operator is determined as the target communication operator.

[0073] Step 303 : Determine whether the communication performance of processing the first communication data using the vectorized memory access operator is lower than the communication performance of processing the first communication data using the first default operator. If so, execute step 302 ; otherwise, execute step 304 .

[0074] Different communication data has different data types and data structures. The communication performance of the vectorized memory access operator and the first default operator in processing the first communication data is affected by the data type and data structure of the first communication data. Communication performance can include multiple factors such as communication efficiency and communication stability.

[0075] After determining that the data volume of the first communication data is greater than the data volume threshold, the communication performance of processing the first communication data using the vectorized memory access operator and the communication performance of processing the first communication data using the first default operator can be predicted. If the communication performance of processing the first communication data using the vectorized memory access operator is less than the communication performance of processing the first communication data using the first default operator, then a greater benefit can be obtained by processing the first communication data using the first default operator, and the first default operator is determined as the target communication operator, and step 302 is executed accordingly. If the communication performance of processing the first communication data using the vectorized memory access operator is greater than or equal to the communication performance of processing the first communication data using the first default operator, then a greater benefit can be obtained by processing the first communication data using the vectorized memory access operator, and the vectorized memory access operator is determined as the target communication operator, and step 304 is executed accordingly.

[0076] Step 304: Determine the vectorized memory access operator as the target communication operator for processing the first communication data.

[0077] When it is determined that the data volume of the first communication data is greater than or equal to the data volume threshold, and the communication performance of processing the first communication data by the vectorized memory access operator is greater than or equal to the communication performance of processing the first communication data by the first default operator, the vectorized memory access operator is determined as the target communication operator.

[0078] In an embodiment of the present application, based on the data volume threshold and the communication performance of processing the first communication data respectively by the vectorized memory access operator and the first default operator, a target communication operator is selected from the vectorized memory access operator and the first default operator to ensure that the first target communication data is processed by the target communication operator, which can generate positive benefits for the model training process and thereby improve the efficiency of model training.

[0079] In one possible implementation, in order to predict the communication performance of a vectorized memory access operator and a first default operator in processing first communication data, respectively, the vectorized memory access operator and the first default operator can be used to process preamble data, respectively, where the preamble data is data transferred from a first memory to a second memory before the first communication data, and the preamble data corresponds to the same communication type as the first communication data. By processing the preamble data using the vectorized memory access operator, the communication performance of the first communication data processed by the vectorized memory access operator can be determined. By processing the preamble data using the first default operator, the communication performance of the first communication data processed by the first default operator can be determined.

[0080] During the model training process, multiple samples are needed to train the model. Different samples have similar data types and data formats, so the preamble data and the first communication data have similar data types and data formats, and the preamble data and the first communication data correspond to the same communication type, so the communication performance of the preamble data and the first communication data processed by the vectorized memory access operator are similar, and the communication performance of the preamble data and the first communication data processed by the first default operator are similar, so the communication performance of the first communication data processed by the vectorized memory access operator can be predicted based on the communication performance of the preamble data processed by the vectorized memory access operator, and the communication performance of the first communication data processed by the first default operator can be predicted based on the communication performance of the preamble data processed by the first default operator.

[0081] In an example, the communication performance of processing the first communication data by the vectorized memory access operator can be predicted based on the communication performance of processing multiple preceding data by the vectorized memory access operator. Correspondingly, the communication performance of processing the first communication data by the first default operator can be predicted based on the communication performance of processing multiple preceding data by the first default operator.

[0082] When the preamble data is processed respectively by the vectorized memory access operator and the first default operator, the same preamble data can be processed respectively by the vectorized memory access operator and the first default operator, or different preamble data can be processed respectively by the vectorized memory access operator and the first default operator. This embodiment of the present application does not limit this.

[0083] In an example, the first few iterations (iters) of model training process the data that needs to be transferred from the first memory to the second memory through the vectorized memory access operator and the first default operator, respectively, to determine the communication performance of processing the preceding data of the first communication data through the vectorized memory access operator, and the communication performance of processing the preceding data of the first communication data through the first default operator.

[0084] In an embodiment of the present application, the preamble data of the first communication data are processed respectively by the vectorized memory access operator and the first default operator to obtain the communication performance of the preamble data processed by the vectorized memory access operator and the first default operator respectively, and then the communication performance of the first communication data processed by the vectorized memory access operator is determined based on the communication performance of the preamble data processed by the vectorized memory access operator, and the communication performance of the first communication data processed by the first default operator is determined based on the communication performance of the preamble data processed by the first default operator, and then the target communication operator for processing the first communication data is selected based on the communication performance of the first communication data processed by the vectorized memory access operator and the communication performance of the first communication data processed by the first default operator, to ensure that when the first communication data is processed by the target communication operator, the data transmission efficiency and model training efficiency can be improved.

[0085] In one possible implementation, when the first communication data is processed by a vectorized memory access operator, the first communication data can be divided into blocks to divide the aligned blocks with NByte aligned starting addresses in the first memory, and then the aligned blocks can be transferred from the first memory to the second memory according to the vectorized memory access method.

[0086] Figure 4 FIG. 1 is a flow chart of a communication data processing method according to an embodiment of the present application. The data processing method can be implemented by a vectorized memory access operator. Figure 4 As shown, the communication data processing method includes the following steps:

[0087] Step 401: Perform block processing on first communication data in a first memory to obtain aligned blocks.

[0088] The starting address of the first communication data in the first memory is not fixed. In some cases, the starting address of the first communication data in the first memory is N-byte aligned, while in other cases, the starting address of the first communication data in the first memory is not N-byte aligned. When performing distributed training on the model based on ZeRO, the communication data is processed using vectorized memory access, which requires that the starting address of the communication data in the memory be N-byte aligned.

[0089] It should be noted that N-byte alignment of the data's starting address in memory means that the data's starting address in memory is an integer multiple of N bytes. For example, when N is 16, if the data's starting address in memory is an integer multiple of 16 bytes, such as 0, 16 bytes, 32 bytes, or 64 bytes, the data's starting address in memory is aligned to 16 bytes. If the data's starting address in memory is a non-integer multiple of 16 bytes, such as 1 byte, 2 bytes, 10 bytes, or 15 bytes, the data's starting address in memory is not aligned to 16 bytes.

[0090] To improve transmission efficiency of the first communication data through vectored memory access, the first communication data is divided into blocks, and alignment blocks are divided from the first communication data. Starting addresses of the alignment blocks in the first memory are aligned with N bytes. Then, at least a portion of the data included in the alignment blocks can be transferred from the first memory to the second memory through vectored memory access. N is an integer greater than 1, and the amount of data included in the alignment blocks is greater than or equal to N bytes.

[0091] Step 402: Transfer the first k×N bytes of data in the aligned block from the first memory to the second memory in a manner that each access instruction accesses N bytes of data.

[0092] When processing communication data using vectorized memory access, each access instruction accesses N bytes of data until the remaining data is less than N bytes. The amount of data in the alignment block may or may not be an integer multiple of N bytes. The amount of data included in the alignment block can be expressed as (k×N+M) bytes, where k is an integer greater than or equal to 1, and M is an integer less than N. Therefore, when processing the alignment block using vectorized memory access, the first k×N bytes of data in the alignment block are transferred from the first memory to the second memory in the manner in which each access instruction accesses N bytes of data.

[0093] In an embodiment of the present application, by dividing the first communication data into blocks, an alignment block with an NByte aligned starting address in the first memory is obtained, and then the first k×NByte data in the alignment block can be transferred from the first memory to the second memory in a manner that each access instruction accesses NByte data, thereby ensuring that the efficiency of the first communication data transmission is improved.

[0094] In one possible implementation, when the first communication data is processed in blocks, if the first communication data is aligned with the starting address NByte in the first memory, the first communication data in the first memory is divided into aligned blocks as a whole. At this time, the first k×NByte data in the aligned block are transferred from the first memory to the second memory in a manner that each access instruction accesses NByte data, and the last MByte data in the aligned block are transferred from the first memory to the second memory in a manner that each access instruction accesses 1Byte data. After that, the first communication data is completely transferred from the first memory to the second memory.

[0095] If the starting address of the first communication data in the first memory is not aligned with N bytes, the first communication data in the first memory is divided into a base block, an edge block, and an alignment block. The starting address of the base block in the first memory is the starting address of the first communication data in the first memory, the ending address of the base block in the first memory is adjacent to the starting address of the alignment block in the first memory, the ending address of the alignment block in the first memory is adjacent to the starting address of the edge block in the first memory, and the sum of the data size of the base block and the edge block is equal to N bytes.

[0096] The sum of the data volume of the basic block, edge block, and alignment block is equal to the data volume of the first communication data. The basic block, alignment block, and edge block have consecutive addresses in the first memory, that is, the next address after the end address of the basic block is the start address of the alignment block, and the next address after the end address of the alignment block is the start address of the edge block. The data volume of the basic block and edge block is less than NByte. The start address of the basic block is not NByte aligned. The start address of the edge block can be NByte aligned or not NByte aligned. The start address of the alignment block is NByte aligned.

[0097] In an embodiment of the present application, the starting address of the first communication data in the first memory is not NByte aligned, and several Byte data in the first communication data that deviate from the NByte alignment position are divided into basic blocks, so that the starting address of the alignment block located after the basic block is NByte aligned, and then the alignment block can be processed by vectorized memory access, so that the alignment block can be transferred from the first memory to the second memory at a higher rate, thereby improving the efficiency of transmitting the first communication data between GPUs.

[0098] In one possible implementation, after the first communication data is divided into a basic block, an alignment block, and an edge block, the last MByte data in the basic block, the edge block, and the alignment block can be transferred from the first memory to the second memory in a manner that each access instruction accesses 1Byte of data, and the first k×NByte data in the alignment block can be transferred from the first memory to the second memory in a manner that each access instruction accesses NByte of data.

[0099] In one example, the last MByte data in the basic block, edge block and alignment block can be transferred from the first memory to the second memory respectively through the first default operator in a manner that each access instruction accesses 1 Byte of data.

[0100] After the first k×N bytes of data in the alignment block are transferred from the first memory to the second memory in a manner that each access instruction accesses N bytes of data, the remaining M bytes of data in the alignment block remain, where M is an integer less than N and may be equal to 0. If M is not equal to 0, the remaining M bytes of data in the alignment block are transferred from the first memory to the second memory in a manner that each access instruction accesses 1 byte of data.

[0101] After the basic block, alignment block and edge block are transferred from the first memory to the second memory, the starting address of the basic block in the second memory is the starting address of the first communication data in the second memory, the ending address of the basic block in the second memory is adjacent to the starting address of the alignment block in the second memory, the ending address of the alignment block in the second memory is adjacent to the starting address of the edge block in the second memory, and the ending address of the edge block in the second memory is the ending address of the first communication data in the second memory, that is, the first communication data in the second memory is the same as the first communication data in the first memory.

[0102] It should be noted that, since the starting addresses of the basic blocks in the first memory and the second memory may be different, the starting addresses of the alignment blocks in the first memory are aligned by NByte, but the starting addresses of the alignment blocks in the second memory are not necessarily aligned by NByte.

[0103] In an embodiment of the present application, the last MByte data in the basic block, edge block and alignment block are respectively transferred from the first memory to the second memory in a manner that each access instruction accesses 1Byte of data, and the first k×NByte data in the alignment block are transferred from the first memory to the second memory in a manner that each access instruction accesses NByte of data, thereby completely transferring the first communication data from the first memory to the second memory, and ensuring that the first communication data in the second memory is the same as the first communication data in the first memory, thereby ensuring the integrity and correctness of the communication data transmission.

[0104] In one possible implementation, when the basic blocks, alignment blocks, and edge blocks are transferred from the first memory to the second memory, when data is read from the first memory and written to the second memory, the second memory may be occupied and the read data cannot be written to the second memory in time. Therefore, the basic blocks, alignment blocks, and edge blocks can be written to the first swap memory first, and then the basic blocks, alignment blocks, and edge blocks can be transferred from the first swap memory to the second memory when the second memory can be written.

[0105] Figure 5FIG. 1 is a schematic diagram of a data transmission process according to an embodiment of the present application. Figure 5 As shown, the first communication data located in the GPU0 memory is divided into a basic block 501, an alignment block 502, and an edge block 503. First, the basic block 501, the alignment block 502, and the edge block 503 are transferred from the GPU0 memory to the first swap memory 504. The starting address of the edge block 503 in the first swap memory 504 is the starting address of the first swap memory 504, the ending address of the edge block 503 in the first swap memory 504 is adjacent to the starting address of the basic block 501 in the first swap memory 504, and the ending address of the basic block 501 in the first swap memory 504 is adjacent to the starting address of the alignment block 502 in the first swap memory 504.

[0106] Figure 6 FIG. 1 is a schematic diagram of a data transmission process according to another embodiment of the present invention. Figure 6 As shown, the first communication data located in the memory of GPU0 is divided into a basic block 501, an alignment block 502, and an edge block 503. The basic block 501, the alignment block 502, and the edge block 503 are first transmitted to the first swap memory 504. The starting address of the basic block 501 in the first swap memory 504 is the starting address of the first swap memory 504, the ending address of the basic block 501 in the first swap memory 504 is adjacent to the starting address of the edge block 503 in the first swap memory 504, and the ending address of the edge block 503 in the first swap memory 504 is adjacent to the starting address of the alignment block 502 in the first swap memory 504.

[0107] like Figure 5 and Figure 6 As shown, since the sum of the data amounts of the basic block 501 and the edge block 503 is equal to NByte, and the starting address of the first swap memory 504 is aligned with NByte, the starting address of the alignment block 502 in the first swap memory 504 is aligned with NByte. Therefore, when the alignment block 502 is transferred from the first swap memory 504 to the GPU1 memory, the alignment block 502 can still be transferred from the first swap memory 504 to the GPU1 memory by vectorized memory access.

[0108] When transferring basic block 501 from GPU0 memory to first swap memory 504, basic block 501 is read from GPU0 memory using a 1-byte access per access instruction. When transferring edge block 503 from GPU0 memory to first swap memory 504, edge block 503 is read from GPU0 memory using a 1-byte access per access instruction. When transferring alignment block 502 from GPU0 memory to first swap memory 504, the first k×N bytes of alignment block 502 are read from GPU0 memory using a 1-byte access per access instruction, and the last M bytes of alignment block 502 are read from GPU0 memory using a 1-byte access per access instruction.

[0109] The process of transferring the basic block 501 from the first swap memory 504 to the GPU1 memory is similar to the process of transferring the basic block 501 from the GPU0 memory to the first swap memory 504. The process of transferring the edge block 503 from the first swap memory 504 to the GPU1 memory is similar to the process of transferring the edge block 503 from the GPU0 memory to the first swap memory 504. The process of transferring the alignment block 502 from the first swap memory 504 to the GPU1 memory is similar to the process of transferring the alignment block 502 from the GPU0 memory to the first swap memory 504, and will not be repeated here.

[0110] It should be noted that the first exchange memory may include the exchange memory of the first computing node and the exchange memory of the second computing node. When the first communication data is transferred from the first memory to the second memory, the first communication data is first transferred from the first memory to the exchange memory of the first computing node, then the first communication data is transferred from the exchange memory of the first computing node to the exchange memory of the second computing node, and then the first communication data is transferred from the exchange memory of the second computing node to the second memory. In the exchange memory of the first computing node and the exchange memory of the second computing node, the relative storage positions of the basic blocks, alignment blocks, and edge blocks are the same, as can be seen in Figure 5 or Figure 6 The storage method of the basic block 501, the alignment block 502 and the edge block 503 in the first exchange memory 504 is shown.

[0111] In an embodiment of the present application, after the first communication data is divided into a basic block, an alignment block, and an edge block, since the starting address of the alignment block in the first memory is aligned with NByte, the alignment block can be transferred from the first memory to the first exchange memory in accordance with the vectorized memory access method, and the sum of the data amount of the basic block and the edge block is equal to NByte, and the starting address of the first exchange memory is aligned with NByte, so the basic block and the edge block are placed before the alignment block in the first exchange memory, so that the starting address of the alignment block in the first exchange memory is still aligned with NByte, and then the alignment block can be transferred from the first exchange memory to the second memory in accordance with the vectorized memory access method. When transferring the alignment block from the first memory to the first exchange memory, and from the first exchange memory to the second memory, the alignment block can be transferred in accordance with the vectorized memory access method, thereby improving the efficiency of transmitting communication data between GPUs.

[0112] Because the second memory may be occupied when the first communication data is read from the first memory, the basic blocks, alignment blocks, and edge blocks are first transferred from the first memory to the first swap memory. When the second memory is ready for writing, the basic blocks, alignment blocks, and edge blocks are then transferred from the first swap memory to the second memory. This prevents the first communication data from being lost during transmission and ensures the reliability of communication data transmission. After the basic blocks, alignment blocks, and edge blocks are transferred to the second memory, they are stored in the order of basic blocks, alignment blocks, and edge blocks. This ensures that the first communication data transferred to the second memory is identical to the first communication data stored in the first memory, ensuring the correctness of communication data transmission.

[0113] In one possible implementation, during the distributed training of the model, the data of one GPU needs to be sent to multiple other GPUs. To this end, after the basic blocks, alignment blocks, and edge blocks are transferred from the first memory to the first exchange memory, the basic blocks, alignment blocks, and edge blocks can be transferred from the first exchange memory corresponding to the second computing node (GPU) to the second exchange memory corresponding to the third computing node (GPU), and then the basic blocks, alignment blocks, and edge blocks can be transferred from the second exchange memory to the third memory of the third computing node.

[0114] The first swap memory is a swap memory between the first computing node and the second computing node, and the second swap memory is a swap memory between the second computing node and the third computing node. The starting addresses of the first swap memory and the second swap memory are both N-byte aligned. In another embodiment, the basic block, alignment block, and edge block can also be directly transferred from the first swap memory to the third memory.

[0115] After the basic block, alignment block, and edge block are transferred from the first swap memory to the second swap memory, the relative storage positions of the basic block, alignment block, and edge block in the second swap memory are the same as the relative storage positions of the basic block, alignment block, and edge block in the first swap memory.

[0116] The starting address of the basic block in the third memory is the starting address of the first communication data in the third memory, the ending address of the basic block in the third memory is adjacent to the starting address of the alignment block in the third memory, and the ending address of the alignment block in the third memory is adjacent to the starting address of the edge block in the third memory.

[0117] like Figure 5 and Figure 6 As shown, after transferring the basic block 501, alignment block 502, and edge block 503 from GPU0 memory to the first swap memory 504, the basic block 501, alignment block 502, and edge block 503 are then transferred from the first swap memory 504 to the GPU1 memory, and then transferred from the first swap memory 504 to the second swap memory 505. The basic block 501, alignment block 502, and edge block 503 are then transferred from the second swap memory 505 to the GPU2 memory. The storage order of the basic block 501, alignment block 502, and edge block 503 in the first swap memory 504 and the second swap memory 505 is the same. The basic block 501, alignment block 502, and edge block 503 are stored sequentially in the GPU2 memory, so that the first communication data in the GPU2 memory is the same as the first communication data in the GPU0 memory.

[0118] In an embodiment of the present application, during the model training process, the first computing node needs to send the first communication data to multiple other computing nodes. After the first communication data in the first memory is transferred from the first memory to the first exchange memory, the first communication data in the first exchange memory is transferred to the second memory. At the same time, the first communication data in the first exchange memory can be transferred to the second exchange memory, and then the first communication data in the second exchange memory can be transferred to the third memory. Since the order of the basic blocks, alignment blocks, and edge blocks in the first exchange memory and the second exchange memory remains consistent, the data receiving logic of the second computing node and the third computing node can be reused to receive the first communication data from the first exchange memory and the second exchange memory respectively. While improving data communication efficiency, the line of sight logic of communication data transmission is simplified through data receiving logic multiplexing, and the development workload is reduced.

[0119] In one possible implementation, when the first communication data is processed in blocks, if the first communication data is aligned with the starting address NByte in the first memory, the first communication data in the first memory is divided into aligned blocks as a whole. At this time, the first k×NByte data in the aligned block are transferred from the first memory to the second memory in a manner that each access instruction accesses NByte data, and the last MByte data in the aligned block are transferred from the first memory to the second memory in a manner that each access instruction accesses 1Byte data. After that, the first communication data is completely transferred from the first memory to the second memory.

[0120] If the starting address of the first communication data in the first memory is not N-byte aligned, the first communication data in the first memory is divided into an offset block and an alignment block. The starting address of the offset block in the first memory is the starting address of the first communication data in the first memory, and the ending address of the offset block in the first memory is adjacent to the starting address of the alignment block in the first memory.

[0121] The offset block and alignment block have consecutive addresses in the first memory. That is, the next address after the end address of the offset block is the start address of the alignment block. If the offset block contains less than N bytes of data, the start address of the offset block in the first memory is not N-byte aligned, while the start address of the alignment block in the first memory is N-byte aligned. The end address of the alignment block in the first memory can be N-byte aligned or non-N-byte aligned.

[0122] In an embodiment of the present application, if the starting address of the first communication data in the first memory is aligned with NByte, the first communication data can be transferred from the first memory to the second memory in a vectored memory access manner, so the first communication data is divided into alignment blocks as a whole; if the starting address of the first communication data in the first memory is not aligned with NByte, several Byte data in the first communication data that deviate from the NByte alignment position are divided into offset blocks, so that the starting address of the alignment block located after the offset block is aligned with NByte, and then the alignment block can be processed in a vectored memory access manner, so that the alignment block can be transferred from the first memory to the second memory at a higher rate, thereby improving the efficiency of transmitting the first communication data between GPUs.

[0123] In one possible implementation, when the offset block and the alignment block are transferred from the first memory to the second memory, when data is read from the first memory and written to the second memory, the second memory may be occupied and the read data cannot be written to the second memory in time, so the offset block and the alignment block can be written to the second swap memory first, and then the offset block and the alignment block can be transferred from the second swap memory to the second memory when the second memory can be written.

[0124] Figure 7FIG. 1 is a schematic diagram of a data transmission process according to another embodiment of the present application. Figure 7 As shown, the first communication data located in the GPU0 memory is divided into an offset block 506 and an alignment block 502, and the offset block 506 and the alignment block 502 are first transferred from the GPU0 memory to the first swap memory 504. The starting address of the alignment block 502 in the first swap memory 504 is the starting address of the first swap memory 504, and the ending address of the alignment block 502 in the first swap memory 504 is adjacent to the starting address of the offset block 506 in the first swap memory 504.

[0125] like Figure 7 As shown, since the starting address of the first swap memory 504 is aligned with NByte, the starting address of the alignment block 502 in the first swap memory 504 is the starting address of the first swap memory 504, so the starting address of the alignment block 502 in the first swap memory 504 is aligned with NByte. Therefore, when the alignment block 502 is transferred from the first swap memory 504 to the GPU1 memory, the alignment block 502 can still be transferred from the first swap memory 504 to the GPU1 memory by vectorized memory access.

[0126] When transferring the offset block 506 from the GPU0 memory to the first swap memory 504, the offset block 506 is read from the GPU0 memory in a manner that each access instruction accesses 1 Byte of data. When transferring the alignment block 502 from the GPU0 memory to the first swap memory 504, the first k×N bytes of the alignment block 502 are read from the GPU0 memory in a manner that each access instruction accesses N Bytes of data, and the last M bytes of the alignment block 502 are read from the GPU0 memory in a manner that each access instruction accesses 1 Byte of data.

[0127] After transferring the offset block 506 and the alignment block 502 from the GPU0 memory to the first swap memory 504, the offset block 506 and the alignment block 502 are respectively transferred from the first swap memory 504 to the GPU1 memory, so that the starting address of the offset block 506 in the GPU1 memory is the starting address of the first communication data in the GPU1 memory, and the ending address of the offset block 506 in the GPU1 memory is adjacent to the starting address of the alignment block 502 in the GPU1 memory.

[0128] When transferring the offset block 506 from the first swap memory 504 to the GPU1 memory, the offset block 506 is read from the first swap memory 504 in a manner that each access instruction accesses 1 Byte of data. When transferring the alignment block 502 from the first swap memory 504 to the GPU1 memory, the first k×N bytes of the alignment block 502 are read from the first swap memory 504 in a manner that each access instruction accesses N Bytes of data, and the last M bytes of the alignment block 502 are read from the first swap memory 504 in a manner that each access instruction accesses 1 Byte of data.

[0129] It should be noted that the first exchange memory may include the exchange memory of the first computing node and the exchange memory of the second computing node. When the first communication data is transferred from the first memory to the second memory, the first communication data is first transferred from the first memory to the exchange memory of the first computing node, then the first communication data is transferred from the exchange memory of the first computing node to the exchange memory of the second computing node, and then the first communication data is transferred from the exchange memory of the second computing node to the second memory. In the exchange memory of the first computing node and the exchange memory of the second computing node, the relative storage positions of the offset block and the alignment block are the same, as can be seen in Figure 7 The storage manner of the offset block 506 and the alignment block 502 in the first swap memory 504 is shown.

[0130] In an embodiment of the present application, after the first communication data is divided into an offset block and an alignment block, since the starting address of the alignment block in the first memory is aligned with NByte, the alignment block can be transferred from the first memory to the first swap memory in accordance with a vectorized memory access method, and the starting address of the first swap memory is aligned with NByte, and the starting address of the alignment block in the first swap memory is the starting address of the first swap memory, so the starting address of the alignment block in the first swap memory is aligned with NByte, and then the alignment block can be transferred from the first swap memory to the second memory in accordance with a vectorized memory access method. When the alignment block is transferred from the first memory to the first swap memory, and when it is transferred from the first swap memory to the second memory, the alignment block can be transferred in accordance with the vectorized memory access method, thereby improving the efficiency of transmitting communication data between GPUs.

[0131] Because the second memory may be occupied when the first communication data is read from the first memory, the offset block and alignment block are first transferred from the first memory to the first swap memory. When the second memory is ready for writing, the offset block and alignment block are then transferred from the first swap memory to the second memory. This prevents the first communication data from being lost during transmission and ensures the reliability of communication data transmission. After transferring the offset block and alignment block to the second memory, they are stored in the order of the offset block and alignment block, ensuring that the first communication data transferred to the second memory is the same as the first communication data in the first memory, thus ensuring the correctness of communication data transmission.

[0132] In one possible implementation, during the distributed training of the model, the data of one GPU needs to be sent to multiple other GPUs. To this end, after transferring the offset block and the alignment block from the first memory to the first swap memory through the vectorized memory access operator, the offset block and the alignment block can be transferred from the first swap memory to the second swap memory, and then the offset block and the alignment block can be transferred from the second swap memory to the third memory of the third computing node.

[0133] The first swap memory is the swap memory between the first computing node and the second computing node, and the second swap memory is the swap memory between the second computing node and the third computing node. The starting addresses of the first swap memory and the second swap memory are both N-byte aligned.

[0134] The starting address of the alignment block in the second swap memory is the starting address of the first communication data in the second swap memory, and the ending address of the alignment block in the second swap memory is adjacent to the starting address of the offset block in the second swap memory. That is, the relative storage positions of the offset block and the alignment block in the second swap memory are the same as the relative storage positions of the offset block and the alignment block in the first swap memory. The starting address of the offset block in the third memory is the starting address of the first communication data in the third memory, and the ending address of the offset block in the third memory is adjacent to the starting address of the alignment block in the third memory.

[0135] like Figure 7 As shown, after the offset block 506 and the alignment block 502 are transferred from the GPU0 memory to the first swap memory 504, the offset block 506 and the alignment block 502 are transferred from the first swap memory 504 to the GPU1 memory, and the offset block 506 and the alignment block 502 are transferred from the first swap memory 504 to the second swap memory 505. Then, the offset block 506 and the alignment block 502 are transferred from the second swap memory 505 to the GPU2 memory. The offset block 506 and the alignment block 502 are sequentially stored in the GPU2 memory, so that the first communication data in the GPU2 memory is the same as the first communication data in the GPU0 memory.

[0136] In an embodiment of the present application, during the model training process, the first computing node needs to send the first communication data to multiple other computing nodes. After the first communication data in the first memory is transferred from the first memory to the first exchange memory, the first communication data in the first exchange memory is transferred to the second memory. At the same time, the first communication data in the first exchange memory can be transferred to the second exchange memory, and then the first communication data in the second exchange memory can be transferred to the third memory. Since the order of the offset block and the alignment block in the first exchange memory and the second exchange memory remains consistent, the data receiving logic of the second computing node and the third computing node can be reused to receive the first communication data from the first exchange memory and the second exchange memory respectively. While improving data communication efficiency, the line of sight logic of communication data transmission is simplified through data receiving logic multiplexing, and the development workload is reduced.

[0137] In one possible implementation, in order to process communication data through vectorized memory access operators, the communication primitives in the communication library can be modified to replace the memory access logic of underlying operations such as send / recv / recvCopySend / recvReduceSend, so that at most NByte of data at non-NByte aligned addresses cannot be vectored memory accessed.

[0138] It should be noted that during distributed model training, data typically requires sequential processing by multiple communication operators. To this end, a workflow (workload) can be constructed that sequentially includes multiple communication operators. Each communication operator in the workflow determines whether to use a vectorized access operator or a default operator based on the methods provided in the preceding embodiments. Two communication workflows are provided, supporting inplace and outplace modes, respectively, to meet different communication data processing requirements.

[0139] Data transmission device

[0140] Figure 8 Schematic diagram of a data transmission device according to an embodiment of the present application. Figure 8 As shown, the data transmission device 800 includes: an acquisition unit 801, a selection unit 802 and a processing unit 803;

[0141] The acquiring unit 801 is configured to acquire first communication data to be transmitted, wherein a ratio of a memory access operation time corresponding to a communication type of the first communication data to a communication duration is greater than a ratio threshold;

[0142] The selection unit 802 may obtain a corresponding vectorized memory access operator and a first default operator according to a communication type of the first communication data, and determine a target communication operator for processing the first communication data from the vectorized memory access operator and the first default operator;

[0143] When determining that the vectorized memory access operator is the target communication operator, the processing unit 803 may transfer at least part of the data included in the first communication data from the first memory of the first computing node to the second memory of the second computing node according to the vectorized memory access method through the vectorized memory access operator.

[0144] In an embodiment of the present application, for a communication type in which the ratio of the memory access operation time to the communication duration is greater than a percentage threshold, a vectorized memory access operator and a default operator are constructed. The ratio of the memory access operation time to the communication duration corresponding to the communication type of the first communication data obtained by the acquisition unit 801 is greater than the percentage threshold, so the target communication operator for processing the first communication data can be determined from the corresponding vectorized memory access operator and the first default operator. If the selection unit 802 determines the vectorized memory access operator as the target communication operator, the processing unit 803 can use the vectorized memory access operator to transfer at least part of the data included in the first communication data from the memory of the first computing node to the memory of the second computing node in accordance with the vectorized memory access method. For the first communication data whose starting address in the memory is not NByte aligned, the transmission of part of the data therein can be completed in accordance with the vectorized memory access method. Compared with the access method in which each access instruction accesses 1Byte of data, the transmission of part of the data therein in accordance with the vectorized memory access method can reduce the time consuming of transmitting the first communication data, thereby improving the efficiency of transmitting communication data between computing nodes.

[0145] It should be noted that the interaction between the various parts of the above-mentioned data transmission device is based on the same concept as the data transmission method embodiment. The specific content and beneficial effects can be found in the description of the above-mentioned data transmission method embodiment, and will not be repeated here.

[0146] electronic devices

[0147] Figure 9 This is a schematic block diagram of an electronic device provided in an embodiment of the present application. The specific embodiments of the present application do not limit the specific implementation of the electronic device. Figure 9 As shown, the electronic device may include: a processor (processor) 902, a communication interface (Communications Interface) 904, a memory (memory) 906, and a communication bus 908. Among them:

[0148] The processor 902 , the communication interface 904 , and the memory 906 communicate with each other via a communication bus 908 .

[0149] The communication interface 904 is used to communicate with other electronic devices or servers.

[0150] The processor 902 is configured to execute a program 910 , and specifically may execute the relevant steps in any of the aforementioned data transmission method embodiments.

[0151] Specifically, the program 910 may include program codes, which include computer operation instructions.

[0152] The processor 902 may be a CPU, an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs, or processors of different types, such as one or more CPUs and one or more ASICs.

[0153] RISC-V is an open-source instruction set architecture based on the principles of the Reduced Instruction Set (RISC). It can be applied to various fields, including microcontrollers and FPGA chips. Specifically, it has applications in areas such as IoT security, industrial control, mobile phones, and personal computers. Designed with small size, high speed, and low power consumption in mind, it is particularly suitable for modern computing devices such as warehouse-scale cloud computers, high-end mobile phones, and tiny embedded systems. With the rise of the artificial intelligence (AI) Internet of Things (AIoT), the RISC-V instruction set architecture has also received increasing attention and support, and is expected to become the next generation of widely used CPU architecture.

[0154] The computer operation instructions in the embodiments of the present application may be computer operation instructions based on the RISC-V instruction set architecture. Correspondingly, the processor 902 may be designed based on the RISC-V instruction set. Specifically, the processor chip in the electronic device provided in the embodiments of the present application may be a chip designed using the RISC-V instruction set. The chip may execute executable code based on the configured instructions, thereby implementing the data transmission method in the above embodiments.

[0155] The memory 906 is used to store the program 910. The memory 906 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0156] The program 910 may be specifically configured to enable the processor 902 to execute the data transmission method in any of the aforementioned embodiments.

[0157] The specific implementation of each step in program 910 can refer to the corresponding description of the corresponding steps and units in any of the aforementioned data transmission method embodiments, and will not be repeated here. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding process descriptions in the aforementioned method embodiments, and will not be repeated here.

[0158] Through the electronic device of the embodiment of the present application, for the communication type in which the ratio of the memory access operation time to the communication duration is greater than the proportion threshold, a vectorized memory access operator and a default operator are constructed. The ratio of the memory access operation time to the communication duration corresponding to the communication type of the first communication data is greater than the proportion threshold, so the target communication operator for processing the first communication data can be determined from the corresponding vectorized memory access operator and the first default operator. If the vectorized memory access operator is determined as the target communication operator, at least part of the data included in the first communication data can be transferred from the memory of the first computing node to the memory of the second computing node according to the vectorized memory access method through the vectorized memory access operator. For the first communication data whose starting address in the memory is not NByte aligned, the transmission of part of the data therein can be completed in accordance with the vectorized memory access method. Compared with the access method in which each access instruction accesses 1Byte of data, the transmission of part of the data therein in accordance with the vectorized memory access method can reduce the time consuming of transmitting the first communication data, thereby improving the efficiency of transmitting communication data between computing nodes.

[0159] Computer storage media

[0160] This application also provides a computer-readable storage medium storing instructions for causing a machine to execute the data transmission method described herein. Specifically, a system or device equipped with a storage medium can be provided, wherein the storage medium stores software program code that implements the functions of any of the above-described embodiments, and a computer (or CPU or MPU) of the system or device can read and execute the program code stored in the storage medium.

[0161] In this case, the program code read from the storage medium itself can realize the function of any one of the above embodiments, so the program code and the storage medium storing the program code constitute part of this application.

[0162] Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0163] Computer program product

[0164] An embodiment of the present application also provides a computer program product, including computer instructions, which instruct a computing device to perform any corresponding operation in the above-mentioned multiple method embodiments.

[0165] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0166] The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or can be implemented as software or computer code that can be stored in a recording medium (such as CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or can be implemented as computer code that is originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded via a network and will be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a special-purpose processor or programmable or special-purpose hardware (such as ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown here, the execution of the code converts the general-purpose computer into a special-purpose computer for executing the method shown here.

[0167] It should be noted that the user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to sample data used to train the model, data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0168] It should be understood that the various embodiments in this specification are described in a progressive manner. References to the same or similar parts of each embodiment can be made to the other embodiments. The embodiments focus on the differences from other embodiments. In particular, the method embodiments are generally similar to the methods described in the device and system embodiments, so their description is relatively simple. For relevant details, references to the descriptions of other embodiments can be made.

[0169] It should be understood that the foregoing description of this specification is based on specific embodiments. Other embodiments are within the scope of the claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0170] It should be understood that an element described herein in the singular or shown in the drawings as only one does not limit the number of the element to one. In addition, modules or elements described or shown herein as separate may be combined into a single module or element, and modules or elements described or shown herein as single may be split into multiple modules or elements.

[0171] It should also be understood that the terms and expressions used herein are for descriptive purposes only, and the one or more embodiments of this specification should not be limited to these terms and expressions. The use of these terms and expressions does not mean to exclude any equivalent features of the illustrations and descriptions (or portions thereof), and it should be recognized that various modifications that may exist should also be included in the scope of the claims. Other modifications, variations, and substitutions may also exist. Accordingly, the claims should be deemed to cover all such equivalents.

Claims

1. A data transmission method, comprising: Acquire first communication data to be transmitted, where a ratio of a memory access operation time corresponding to a communication type of the first communication data to a communication duration is greater than a ratio threshold; Acquire a corresponding vectorized memory access operator and a first default operator according to a communication type of the first communication data; Determining a target communication operator for processing the first communication data from the vectorized memory access operator and the first default operator; If it is determined that the vectorized memory access operator is the target communication operator, at least part of the data included in the first communication data is transmitted from the first memory of the first computing node to the second memory of the second computing node in a vectorized memory access manner through the vectorized memory access operator.

2. The method according to claim 1, further comprising: Acquire second communication data to be transmitted, where a ratio of a memory access operation time corresponding to a communication type of the second communication data to a communication duration is less than or equal to the proportion threshold; acquiring, according to a communication type of the second communication data, a second default operator for processing the second communication data; The second communication data is transferred from the first memory to the second memory through the second default operator in a manner in which each access instruction accesses 1 Byte of data.

3. The method according to claim 1, wherein The determining, from the vectorized memory access operator and the first default operator, a target communication operator for processing the first communication data includes: If the data volume of the first communication data is less than a data volume threshold, determining the first default operator as a target communication operator for processing the first communication data, so as to transfer the first communication data from the first memory to the second memory through the first default operator in a manner of accessing 1 Byte of data per access instruction; If the data volume of the first communication data is greater than or equal to the data volume threshold, and the communication performance of processing the first communication data by the vectorized memory access operator is lower than the communication performance of processing the first communication data by the first default operator, determining the first default operator as the target communication operator for processing the first communication data; If the data volume of the first communication data is greater than or equal to the data volume threshold, and the communication performance of processing the first communication data by the vectorized memory access operator is greater than or equal to the communication performance of processing the first communication data by the first default operator, then the vectorized memory access operator is determined as the target communication operator for processing the first communication data.

4. The method according to claim 3, further comprising: Processing preamble data by the vectorized memory access operator to determine communication performance of processing the first communication data by the vectorized memory access operator, where the preamble data is data transmitted from the first memory to the second memory before the first communication data, and the preamble data and the first communication data correspond to the same communication type; The preceding data is processed by the first default operator to determine the communication performance of the first communication data processed by the first default operator.

5. The method according to claim 1, wherein The method of transmitting, by the vectorized memory access operator, at least part of the data included in the first communication data from the first memory of the first computing node to the second memory of the second computing node in a vectorized memory access manner includes: The following operations are performed by the vectorized memory access operator: Performing block processing on the first communication data in the first memory to obtain alignment blocks, where the alignment blocks are aligned at a starting address N Byte in the first memory, where N is an integer greater than 1; According to the manner in which each access instruction accesses NByte data, the first k×NByte data in the alignment block is transferred from the first memory to the second memory. The amount of data included in the alignment block is (k×N+M)Byte, where k is an integer greater than or equal to 1, and M is an integer less than N.

6. The method according to claim 5, wherein: The block processing of the first communication data in the first memory to obtain aligned blocks includes: If the starting address NByte of the first communication data in the first memory is aligned, dividing the first communication data in the first memory into the aligned blocks as a whole; If the starting address of the first communication data in the first memory is not NByte aligned, the first communication data located in the first memory is divided into a basic block, an edge block and an alignment block, the starting address of the basic block in the first memory is the starting address of the first communication data in the first memory, the ending address of the basic block in the first memory is adjacent to the starting address of the alignment block in the first memory, the ending address of the alignment block in the first memory is adjacent to the starting address of the edge block in the first memory, and the sum of the data volume of the basic block and the edge block is equal to NByte.

7. The method according to claim 6, further comprising: According to the manner of accessing 1 Byte of data per access instruction, the last MByte data in the basic block, the edge block and the alignment block are respectively transferred from the first memory to the second memory, and the starting address of the basic block in the second memory is made the starting address of the first communication data in the second memory, the ending address of the basic block in the second memory is adjacent to the starting address of the alignment block in the second memory, and the ending address of the alignment block in the second memory is adjacent to the starting address of the edge block in the second memory.

8. The method according to claim 7, wherein: When transferring the basic block, the alignment block, and the edge block from the first memory to the second memory, Transferring the basic block, the alignment block, and the edge block from the first memory to the first swap memory, respectively, so that the starting address of the edge block in the first swap memory is the starting address of the first swap memory, the ending address of the edge block in the first swap memory is adjacent to the starting address of the basic block in the first swap memory, the ending address of the basic block in the first swap memory is adjacent to the starting address of the alignment block in the first swap memory, and the starting addresses of the first swap memory are aligned by N Byte; or, Transferring the basic block, the alignment block, and the edge block from the first memory to the first swap memory, respectively, so that the starting address of the basic block in the first swap memory is the starting address of the first swap memory, the ending address of the basic block in the first swap memory is adjacent to the starting address of the edge block in the first swap memory, the ending address of the edge block in the first swap memory is adjacent to the starting address of the alignment block in the first swap memory, and the starting addresses of the first swap memory are aligned by N Byte; The basic block, the alignment block, and the edge block are transferred from the first swap memory to the second memory, respectively.

9. The method according to claim 8, further comprising: transferring the basic block, the alignment block, and the edge block from the first swap memory to a second swap memory, and transferring the basic block, the alignment block, and the edge block from the second swap memory to a third memory of a third computing node; The relative positions of the basic block, the alignment block, and the edge block in the second swap memory are the same as the relative positions of the basic block, the alignment block, and the edge block in the first swap memory, and the start address of the second swap memory is aligned with NByte; The starting address of the basic block in the third memory is the starting address of the first communication data in the third memory, the ending address of the basic block in the third memory is adjacent to the starting address of the alignment block in the third memory, and the ending address of the alignment block in the third memory is adjacent to the starting address of the edge block in the third memory.

10. The method according to claim 5, wherein The block processing of the first communication data in the first memory to obtain aligned blocks includes: If the starting address NByte of the first communication data in the first memory is aligned, dividing the first communication data in the first memory into the aligned blocks as a whole; If the starting address of the first communication data in the first memory is not NByte aligned, the first communication data located in the first memory is divided into an offset block and an alignment block, the starting address of the offset block in the first memory is the starting address of the first communication data in the first memory, and the ending address of the offset block in the first memory is adjacent to the starting address of the alignment block in the first memory.

11. The method according to claim 10, further comprising: After dividing the first communication data into the offset block and the alignment block, the offset block and the alignment block are respectively transferred from the first memory to the first swap memory, so that the starting address of the alignment block in the first swap memory is the starting address of the first swap memory, the ending address of the alignment block in the first swap memory is adjacent to the starting address of the offset block in the first swap memory, the starting address of the first swap memory is aligned with N bytes, and the offset block is transferred from the first memory to the first swap memory in a manner in which each access instruction accesses 1 byte of data; The offset block and the alignment block are transferred from the first swap memory to the second memory respectively, so that the starting address of the offset block in the second memory is the starting address of the first communication data in the second memory, and the ending address of the offset block in the second memory is adjacent to the starting address of the alignment block in the second memory. The offset block is transferred from the first swap memory to the second memory in a manner that each access instruction accesses 1 Byte of data.

12. The method according to claim 11, further comprising: Using the vectorized memory access operator, the offset block and the alignment block are respectively transferred from the first swap memory to the second swap memory, and the offset block and the alignment block are respectively transferred from the second swap memory to the third memory of the third computing node; The starting address of the alignment block in the second swap memory is the starting address of the first communication data in the second swap memory, the ending address of the alignment block in the second swap memory is adjacent to the starting address of the offset block in the second swap memory, and the starting address of the second swap memory is aligned with NByte; The starting address of the offset block in the third memory is the starting address of the first communication data in the third memory, and the ending address of the offset block in the third memory is adjacent to the starting address of the alignment block in the third memory.

13. An electronic device comprising: Processor, memory, communication interface and communication bus, the processor, memory and communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to any one of the methods according to claims 1-12.

14. A computer storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

15. A computer program product comprising computer instructions, the computer instructions instructing a computing device to perform the method according to any one of claims 1 to 12.

Images