Model training method and system, and related device

Abstract

The present application provides a model training method and system, and a related device. The system comprises a plurality of nodes, and a plurality of network planes are established between the plurality of nodes by means of different communication modes. In this way, during distributed training, a node obtains partitioned data of a matrix parameter and trains a model on the basis of the partitioned data, if communication data that needs to be synchronized with other nodes is obtained, different network planes may be used for collective communication of communication data of different matrix parameters, so that collective communication of the plurality of matrix parameters can be executed in parallel, the number of collective communications is reduced, and the efficiency of collective communication is improved, thereby enhancing the efficiency of model training.

Description

A model training method, system, and related equipment

[0001] This application claims priority to Chinese Patent Application No. 202410309912.0, filed on March 15, 2024, with the China National Intellectual Property Administration, entitled “A Model Training Method, System and Related Equipment”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence (AI), and in particular to a model training method, system, and related equipment. Background Technology

[0003] As AI technology continues to develop, the training of AI models has begun to rely on greater computing power and storage capacity. A single node can no longer meet the training needs of AI models. Therefore, large-scale AI models are usually trained using distributed training. Distributed training involves dividing the model parameters and distributing them to multiple nodes for training. This not only reduces the computational pressure on a single node but also allows multiple nodes to train in parallel, improving training efficiency.

[0004] In distributed training, each node is responsible for updating a subset of the model's parameters. However, the goal of training is to train a complete model, so nodes need to communicate with each other, synchronizing their trained data with other nodes via ensemble communication. However, each ensemble communication consumes limited inter-node communication bandwidth. If the amount of data to be synchronized is large, multiple ensemble communications are necessary to achieve a single data synchronization. For example, before matrix multiplication of activation values ​​and weights, a first ensemble communication is needed to synchronize model weights, and a second ensemble communication is needed to synchronize activation values. Therefore, in distributed training, the number of ensemble communications is high, causing communication time to consume a significant portion of the model training time, thus impacting training efficiency. Summary of the Invention

[0005] This application provides a model training method, system, and related equipment to solve the problem that the large number of ensemble communications in distributed training leads to communication time occupying the majority of model training time and affecting model training efficiency.

[0006] Firstly, a model training method is provided. This method is applied to a training system comprising multiple nodes, which establish multiple network planes through different communication methods. The method includes the following steps: a first node obtains the mapping relationship between communication data and network planes. The first node can be any node among the multiple nodes. The first node obtains the segmented data of matrix parameters, where the matrix parameters are the multipliers in the matrix multiplication task during model training. The first node trains the model based on the segmented data to obtain the communication data corresponding to the matrix parameters. The communication data is the data to be transmitted by the first node to other nodes. The first node performs collective communication with other nodes based on the mapping relationship to obtain the communication data of the matrix parameters on each node. The communication data corresponding to different matrix parameters are collectively communicated using different network planes. The first node completes the matrix multiplication task based on the communication data of the matrix parameters on each node, thus completing the model training.

[0007] By implementing the method described in the first aspect, multiple network planes are established between multiple nodes in the training system through different communication methods. In this way, during distributed training, when a node obtains the segmented data of the matrix parameters and trains the model based on the segmented data, and obtains the communication data that needs to be synchronized with other nodes, the communication data of different matrix parameters can be used to perform aggregate communication using different network planes. This allows the aggregate communication of multiple matrix parameters to be executed in parallel, reducing the number of aggregate communication operations, improving the efficiency of aggregate communication, and thus improving the efficiency of model training.

[0008] In one possible implementation, the training system includes multiple communication groups, each communication group includes multiple nodes, the multiple nodes within a communication group establish a first network plane through a first communication method, and the communication groups establish a second network plane through a second communication method.

[0009] For example, each communication group can include two nodes. Each communication group can be a chip with a dual-die encapsulation architecture, which allows two identical or different dies to be contained in a single physical package; that is, one die is one node, and a dual-die encapsulation architecture constitutes a communication group. Nodes can also be computing units, such as graphics processing units (GPUs), central processing units (CPUs), tensor processing units (TPUs), data processing units (DPUs), or dedicated AI processing chips.

[0010] Furthermore, when each node in the training system is a die, and a communication group is a dual-die packaged chip, the first communication connection can be a communication connection used between the two dies, such as serial input / output (SIO). Two nodes within the communication group establish the first network plane via SIO. Multiple dual-die groups can establish communication connections via high-speed full-mesh interconnect bandwidth, and communication groups can establish a second network plane via high-speed full-mesh interconnect bandwidth. The high-speed full-mesh interconnect bandwidth connection can be Huawei Cloud Communication Service (HCCS). HCCS is a high-speed interconnect technology for cloud computing scenarios, providing high-speed, low-latency cloud communication services. In a full-mesh topology, HCCS can achieve direct connections between nodes while providing high bandwidth and reliability. High-speed full-mesh interconnect bandwidth can also be NVLink technology, which is a high-speed interconnect technology for direct communication between GPUs. It is suitable for high-performance computing and machine learning. In a full-mesh topology, NVLink can directly connect multiple GPUs to achieve efficient transmission. It should be understood that the above examples are for illustration purposes only, and this application does not limit the specific technology used for high-speed full-mesh interconnect bandwidth connection.

[0011] For another example, each communication group consists of four nodes. These nodes are interconnected via a first communication connection, forming a first network plane. These nodes can be computing units as described above, or computing devices (such as bare metal servers (BMS), virtual machines, or containers). The first communication connection can be a high-speed fullmesh interconnect bandwidth connection; please refer to the relevant descriptions above for details, which will not be repeated here. The communication groups are interconnected via a second communication connection, forming a second network plane. This second communication connection is implemented using a switch chip.

[0012] It should be understood that the above examples are for illustrative purposes only. This application does not limit the number of communication groups included in the training system, nor the number of nodes in each communication group, nor the specific connection methods of the first and second communication connections. It should be noted that in a specific implementation, the training system may also include more than two network planes. For example, three communication groups can form a large group, and multiple large groups can establish a third communication connection to form a third network plane. Of course, the connection method of the third communication connection is different from that of the first and second communication connections. This allows for parallel transmission based on three network planes during data synchronization. The specific number of network planes can be determined according to the actual business scenario, and this application does not impose any specific limitations.

[0013] The above implementation method, by establishing different network planes, allows for data synchronization during training when data needs to be communicated. This can be achieved by synchronizing some data through the first network plane and others through the second network plane, thereby reducing the number of data communications and training time.

[0014] In one possible implementation, the first node obtains the segmented data of the matrix parameters by: the first node performing at least one row segmentation and at least one column segmentation on the matrix parameters to obtain multiple segmented data of the matrix parameters, the number of segmented data being the same as the number of nodes in the training system; the multiple segmented data of the first node are distributed to other nodes, so that each node obtains one segmented data of the matrix parameters.

[0015] In practical implementation, matrix parameters refer to the two multipliers involved in matrix multiplication. Both multipliers are matrices, such as activation values ​​and weights. Activation values ​​are the values ​​calculated on neurons, including input data to the network and intermediate data during model training, such as the output data of the previous network layer and the result of the previous matrix multiplication. Weights refer to the model's weights. During model training, the weights are continuously adjusted to allow the model to learn the features and patterns in the data, thereby continuously improving model performance. Matrix parameters can also be bias terms, gradients, etc. It should be understood that model training includes multiple matrix multiplication tasks. The two matrix parameters in each matrix multiplication task can be segmented in the above way to obtain the segmented data of each matrix parameter, which is then distributed to various nodes. This allows a matrix multiplication task to be divided into multiple subtasks for parallel processing. Then, the calculation results of each subtask are aggregated through ensemble communication to obtain the calculation result of the matrix multiplication task.

[0016] For example, if the matrix parameter is 2×3, after one row partition, it becomes two 1×3 matrices; after three column partitions, it becomes six 1×1 matrices. As another example, if the matrix parameter is 4×4, after one row partition, it becomes two 2×4 matrices; after one column partition, it becomes four 2×2 matrices. In the above examples, an a×b matrix refers to a matrix with a rows and b columns. These examples are for illustrative purposes only and are not intended to impose specific limitations.

[0017] In the specific implementation, the first matrix parameter and the second matrix parameter participating in the same matrix multiplication operation are cut in the same way. That is, the number of row cuts of the first matrix parameter and the number of row cuts of the second matrix parameter are the same, and the number of column cuts of the first matrix parameter and the number of column cuts of the second matrix parameter are the same. In simple terms, if the first matrix parameter is the activation value and the second matrix parameter is the weight, and the activation value is split into 6 parts by 2 rows and 3 columns, then the weight also needs to be split into 6 parts. However, the weight can only be split by 3 columns and 2 rows. If the weight is split by 2 rows and 3 columns, although it can still be split into 6 parts, the number of rows in the split data of the activation value is different from the number of columns in the split data of the weight. This may cause a problem when the split data is distributed to each node for matrix multiplication, where the two split parameters cannot be multiplied because their inner axes are different. For example, if each row of split data x1 has 2 elements and each column of split data y1 has 3 elements, their inner axes are different, so they cannot be multiplied. Therefore, the two matrix parameters need to be split in the same way to avoid the problem of split data not being able to be multiplied.

[0018] The above implementation method, through at least one column split and at least one row split, can divide the matrix parameters into smaller blocks, distribute the matrix parameters more evenly among the nodes, help avoid some nodes being overloaded, and is more suitable for communication between multiple network planes.

[0019] In one possible implementation, the number of row cuts is determined by the number of communication groups, and the number of column cuts is determined by the number of nodes within a communication group; or, the number of row cuts is determined by the number of nodes within a communication group, and the number of column cuts is determined by the number of communication groups.

[0020] In the specific implementation, if there are m nodes in each communication group and n communication groups, then the number of row cuts is m-1, meaning the matrix parameters are divided into m parts by rows, and the number of column cuts is n-1, meaning the matrix parameters are further divided into m×n parts by columns. Alternatively, the number of row cuts is n-1, and the number of column cuts is m-1. Simply put, if the node size is 2×3 (3 communication groups, each with 2 nodes), then the number of row cuts is 1, and the number of column cuts is 2, meaning the matrix parameters are also cut into 2×3 parts. It should be noted that the other matrix parameter involved in the matrix multiplication also needs to be cut in the same way; for example, activation values ​​and weights are cut into 2×3 parts.

[0021] The above implementation determines the number of row and column cuts based on the number of communication groups and the number of nodes within each communication group. This ensures that when subsequent aggregated communication is performed along different network planes, the aggregated matrix obtained from the aggregated communication of each network plane can have complete rows or complete columns (the same number of columns as the original first matrix parameter, or the same number of rows as the original second matrix parameter). This ensures that each node uses the same inner axis of the aggregated matrix, enabling matrix multiplication operations and ensuring the correctness of the matrix multiplication task.

[0022] In one possible implementation, the matrix parameters include a first matrix parameter and a second matrix parameter, which are the left and right multipliers in the same matrix multiplication task. The communication data includes first communication data corresponding to the first matrix parameter and second communication data corresponding to the second matrix parameter. If the number of column cuts of the matrix parameter is determined according to the number of communication groups, the first communication data corresponds to the second network plane, and the second communication data corresponds to the first network plane. If the number of column cuts of the matrix parameter is determined according to the number of communication groups, the first communication data corresponds to the first network plane, and the second communication data corresponds to the second network plane.

[0023] It should be noted that, due to the aforementioned relationship between the mapping relationship and the splitting strategy, if the user configures the mapping relationship, the splitting strategy needs to flexibly adjust the number of column and row splits based on the mapping relationship; conversely, if the user configures the splitting strategy, the mapping relationship needs to be flexibly adjusted based on the splitting strategy. If both the mapping relationship and the splitting strategy are user-configured, the relationship between the two needs to be explained to the user, and any problems with the user's configuration should be promptly reported. If the user has not configured either the mapping relationship or the splitting strategy, the training system can use the default configuration to complete the training.

[0024] The above implementation, based on a segmentation strategy to determine the communication plane, ensures the correctness of the matrix multiplication task. It should be understood that during model training, after each ensemble communication, each node receives communication data from other nodes, then merges this data to obtain the merged result for each matrix parameter, and then performs matrix multiplication between the merged results. For example, multiple first communication data are merged into a third matrix parameter, and multiple second communication data are merged into a fourth matrix parameter, and then the third and fourth matrix parameters are multiplied. Matrix multiplication requires that the number of columns in the first multiplier is the same as the number of rows in the second multiplier, meaning the inner axes of the two multipliers are the same; otherwise, matrix multiplication cannot be performed. Therefore, the network plane used during ensemble communication needs to refer to the matrix parameter segmentation method to ensure that the number of columns in the merged third matrix parameter is the same as the number of rows in the fourth matrix parameter.

[0025] In one possible implementation, the first node completes the matrix multiplication task based on the communication data of the matrix parameters on each node, including: the first node concatenates the first communication data of the first matrix parameters on each node to obtain the third matrix parameters, and concatenates the second communication data of the second matrix parameters on each node to generate the fourth matrix parameters, wherein the number of columns of the third matrix parameters is the same as the number of columns of the first matrix parameters, and the number of rows of the fourth matrix parameters is the same as the number of rows of the second matrix parameters; the first node completes the matrix multiplication task based on the matrix multiplication result of the third matrix parameters and the fourth matrix parameters.

[0026] In simple terms, regardless of the segmentation strategy, during aggregated communication, the first communication data, when determining its communication plane, only needs to ensure that the aggregated communication results obtained through this network plane can be concatenated to obtain the width or number of columns H of the first matrix parameter. Similarly, the second communication data only needs to ensure that it can be concatenated through another network plane to obtain the height or number of rows H of the second matrix parameter. This allows matrix multiplication operations to proceed smoothly and avoids multiplication failures due to different inner axes. Therefore, during concatenation, it is necessary to determine whether to concatenate the multiple communication data obtained from aggregation horizontally or vertically, based on the segmentation and communication methods.

[0027] The above implementation method, based on the segmentation strategy and communication plane, determines how the aggregated communication data is aggregated, which further ensures the correctness of the matrix multiplication task. Since the first and second matrix parameters are the two parameters to be multiplied, and the number of columns in the first matrix parameter is the same as the number of rows in the second matrix parameter, ensuring that the number of columns in the third matrix parameter is the same as the number of columns in the first matrix parameter, and the number of rows in the fourth matrix parameter is the same as the number of rows in the second matrix parameter, ensures that the number of columns in the third matrix parameter is the same as the number of rows in the fourth matrix parameter. Following this idea, if the number of column segments for the first matrix parameter is determined by the number of communication groups, and the second communication method is inter-group communication, then when the first communication result of the first matrix parameter is aggregated, the second communication plane needs to be used for aggregated communication to ensure that the number of columns in the merged third matrix parameter is the same as the number of columns in the first matrix parameter. Similarly, if the number of column segments for the first matrix parameter is determined by the number of nodes within a communication group, the first communication plane needs to be used for aggregated communication to ensure that the number of columns in the merged third matrix parameter is the same as the number of columns in the first matrix parameter.

[0028] In one possible implementation, the first matrix parameter includes the activation value and the second matrix parameter includes the weight; or, the first matrix parameter includes the matrix multiplication result of the activation value and the first weight, and the second matrix parameter includes the second weight.

[0029] It should be noted that in the technical solution provided in this application, if there is a matrix multiplication operation task between other parameters in the distributed training process, the first matrix parameter and the second matrix parameter can also be other parameters. For example, the first matrix parameter can also be an attention parameter, and the second matrix parameter can be a weight or activation value. This application does not make specific limitations.

[0030] It should be understood that the scheme provided in this application is applicable to any matrix multiplication task in the distributed training process, including a single matrix multiplication task, such as the multiplication task between activation values ​​and weights, as well as multiple consecutive matrix multiplication tasks. Each matrix multiplication can be segmented and communicated in the manner provided above. In other words, the matrix parameters can also be temporary intermediate results generated during the calculation process or temporary input activation values. The scheme has wide applicability.

[0031] Taking a task involving two consecutive matrix multiplications as an example, suppose the activation value is multiplied by the first weight in the first matrix multiplication, and then multiplied by the second weight in the second matrix multiplication. We can first obtain the third matrix parameter corresponding to the activation value and the fourth matrix parameter corresponding to the first weight, as described above. After obtaining the first operation result between them, we use the first operation result as the first matrix parameter and the second weight as the second matrix parameter. After segmenting according to the description above, we determine the mapping relationship based on the segmentation strategy. Then, we simultaneously complete set communication according to their respective communication planes to obtain the third matrix parameter corresponding to the first operation result and the fourth matrix parameter corresponding to the second weight. Finally, based on the second operation result between them, we complete the task of two consecutive matrix multiplications.

[0032] Specifically, the second weight can be segmented to obtain segmented data, and the multiple segmentation results of the second weight can be distributed to various nodes. The description of this process can be found in the previous description of the segmentation and distribution of the first and second matrix parameters, which will not be repeated here. It should be noted that the segmentation method of the second weight is the same as that of the first and second matrix parameters, and the number of row and column segments is the same.

[0033] Optionally, the segmented data of the second weights is processed to obtain third communication data. Then, the first operation result is processed through aggregate communication along the second network plane. The third matrix parameters corresponding to the first operation result are obtained through aggregate communication of the third communication data via the first network plane to obtain the fourth matrix parameters corresponding to the second weights. Matrix multiplication is then performed on the two to obtain the second operation result. Since the two network planes use different communication methods, the aggregate communication of the two network planes can be performed simultaneously, improving the efficiency of aggregate communication during distributed training.

[0034] It should be noted that if the matrix multiplication task is a task performed by the model in the forward computation phase, the set communication includes global aggregation (allgather) communication; if the matrix multiplication task is a task performed by the model in the backpropagation phase, the set communication includes reduction (reduce scatter) communication. The splitting method and communication method for backpropagation will not be explained again here.

[0035] It should be understood that if distributed training is performed using traditional methods, two consecutive matrix multiplication operations require four set communication operations. The first set communication aggregates activation values, the second set communication aggregates the first weight, the third set communication aggregates the second weight, and the fourth set communication reduces the result of the matrix multiplication operation. With the training method provided in this application, two consecutive matrix multiplication operations only require two set communication operations. In the first set communication, the first communication data is processed through the second network plane, and the second communication data is processed through the first network plane. In the second set communication, the first operation result is processed through the second network plane, and the third communication data is processed through the first network plane. Compared with the traditional method, the technical solution provided in this application only requires two set communication operations to complete two matrix multiplication operations, improving communication efficiency and model training efficiency.

[0036] Secondly, a model training system is provided. This system is applied to a training system comprising multiple nodes. These nodes establish multiple network planes through different communication methods. The system includes: a configuration unit for acquiring the mapping relationship between communication data and the network planes, where the first node is any node among the multiple nodes; a processing unit for acquiring segmented data of matrix parameters, where the matrix parameters are multipliers in the matrix multiplication task during model training; a processing unit for training the model based on the segmented data to obtain communication data corresponding to the matrix parameters, where the communication data is the data to be transmitted from the first node to other nodes; a communication unit for performing aggregated communication with other nodes based on the mapping relationship to obtain the communication data of the matrix parameters on each node, wherein the communication data corresponding to different matrix parameters uses different network planes for aggregated communication; and a processing unit for completing the matrix multiplication task based on the communication data of the matrix parameters on each node, thereby completing the model training.

[0037] Implementing the system described in the second aspect, multiple network planes are established between multiple nodes in the training system through different communication methods. In this way, during distributed training, when a node obtains the segmented data of the matrix parameters and trains the model based on the segmented data, and obtains the communication data that needs to be synchronized with other nodes, the communication data of different matrix parameters can be used to conduct aggregate communication using different network planes. This allows the aggregate communication of multiple matrix parameters to be executed in parallel, reducing the number of aggregate communication operations, improving the efficiency of aggregate communication, and thus improving the efficiency of model training.

[0038] In one possible implementation, the training system includes multiple communication groups, each communication group includes multiple nodes, the multiple nodes within a communication group establish a first network plane through a first communication method, and the communication groups establish a second network plane through a second communication method.

[0039] In one possible implementation, the system includes a cutting unit; the cutting unit is used to perform at least one row cutting and at least one column cutting on the matrix parameters to obtain multiple cutting data of the matrix parameters, the number of cutting data being the same as the number of nodes in the training system; the cutting unit is used to distribute the multiple cutting data of the first node to other nodes, so that each node obtains one cutting data of the matrix parameters.

[0040] In one possible implementation, the number of row cuts is determined by the number of communication groups, and the number of column cuts is determined by the number of nodes within a communication group; or, the number of row cuts is determined by the number of nodes within a communication group, and the number of column cuts is determined by the number of communication groups.

[0041] In one possible implementation, the matrix parameters include a first matrix parameter and a second matrix parameter, which are the left and right multipliers in the same matrix multiplication task. The communication data includes first communication data corresponding to the first matrix parameter and second communication data corresponding to the second matrix parameter. If the number of column cuts of the matrix parameter is determined according to the number of communication groups, the first communication data corresponds to the second network plane, and the second communication data corresponds to the first network plane. If the number of column cuts of the matrix parameter is determined according to the number of communication groups, the first communication data corresponds to the first network plane, and the second communication data corresponds to the second network plane.

[0042] In one possible implementation, the processing unit is used to concatenate the first communication data of the first matrix parameter on each node to obtain the third matrix parameter, and to concatenate the second communication data of the second matrix parameter on each node to generate the fourth matrix parameter, wherein the number of columns of the third matrix parameter is the same as the number of columns of the first matrix parameter, and the number of rows of the fourth matrix parameter is the same as the number of rows of the second matrix parameter; the processing unit is used to complete the matrix multiplication task based on the matrix multiplication result of the third matrix parameter and the fourth matrix parameter.

[0043] In one possible implementation, the first matrix parameter includes the activation value and the second matrix parameter includes the weight; or, the first matrix parameter includes the matrix multiplication result of the activation value and the first weight, and the second matrix parameter includes the second weight.

[0044] In one possible implementation, if the matrix multiplication task is a task of the model in the forward computation phase, the set communication includes global aggregation (allgather) communication; if the matrix multiplication task is a task of the model in the backpropagation phase, the set communication includes reduction (reducescatter) communication.

[0045] Thirdly, a computing device is provided, the computing device including a processor and a memory, the memory for storing instructions and the processor for executing the instructions, such that the computing device implements the method described in the first aspect.

[0046] Fourthly, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and the instructions are executed by a computing device or a cluster of computing devices to implement the method described in the first aspect.

[0047] Fifthly, a computing device cluster is provided, the computing device cluster including at least one computing device, each computing device including a processor and a memory, the processor of the at least one computing device being configured to execute instructions stored in the memory of the at least one computing device, such that the computing device cluster implements the method described in the first aspect.

[0048] In a sixth aspect, a computer program product comprising instructions is provided, the computer program product including instructions capable of running on a computing device or stored in any available medium, and when the computer program product is run on a computing device or a cluster of computing devices, causing the computing device or cluster of computing devices to perform the method described in the first aspect. Attached Figure Description

[0049] Figure 1 is a structural diagram of a training system that implements distributed training through tensor parallelism;

[0050] Figure 2 is an architecture diagram of a training system provided in this application;

[0051] Figure 3 is an architecture diagram of another training system provided in this application;

[0052] Figure 4 is a schematic diagram of the structure of each node in a training system provided in this application;

[0053] Figure 5 is a flowchart illustrating the steps of a model training method provided in this application;

[0054] Figure 6 is an example diagram of matrix cutting provided in this application;

[0055] Figure 7 is an example diagram of a configuration interface provided in this application;

[0056] Figure 8 is a flowchart illustrating the steps of a training method provided in this application for processing matrix multiplication tasks.

[0057] Figure 9 is a flowchart illustrating the steps of a training method provided in this application for handling matrix multiplication tasks in an application scenario.

[0058] Figure 10 is a comparison chart of communication latency under different segmentation strategies in a model training method provided in this application;

[0059] Figure 11 is a schematic diagram of the structure of a computing device provided in this application. Detailed Implementation

[0060] The application scenarios involved in this application will be explained below.

[0061] With the continuous development of AI technology, the training of AI models has begun to rely on greater computing power and storage capacity. A single node is gradually becoming insufficient to meet the training needs of AI models. Therefore, large-scale AI models are typically trained using distributed training. Distributed training refers to assigning the training task of an AI model to multiple nodes for collaborative completion. Common algorithms include data parallelism, tensor parallelism, and pipeline parallelism. Data parallelism involves splitting the dataset and distributing it across multiple nodes for training; tensor parallelism involves distributing the model parameters across multiple nodes for training; and pipeline parallelism involves splitting the model according to network layers and distributing it across multiple nodes for training. Distributed training also includes many other algorithms, which will not be listed here.

[0062] For example, Figure 1 is a structural diagram of a training system that implements distributed training through tensor parallelism. This training system 100 may include multiple nodes (Figure 1 illustrates nodes 1 to 4 as examples). Each node can be a device with data processing capabilities, such as a graphics processing unit (GPU). A communication network exists between the nodes, enabling data communication between them. During model training, the model parameters are divided into four parts and distributed across the four nodes in Figure 1 for training.

[0063] Although each node is responsible for updating a portion of the model's parameters, the goal of training is to train a complete model. Therefore, nodes need to perform aggregate communication to synchronize their trained data with other nodes. Aggregate communication is a common communication method in distributed systems, involving global communication operations with all nodes in the system. For example, AllGather communication refers to aggregating data from each node onto each node, ensuring that each node includes data from all nodes. ReduceScatter communication refers to all nodes performing the same reduction operation and then distributing the results to each node. It should be understood that aggregate communication can include many other types, which will not be listed here.

[0064] Typically, in distributed training, the activation values ​​and weights obtained by each node during training need to be synchronized to other nodes so that each node can obtain the complete model's weights and activation values. Then, forward computation is performed to obtain the loss value, and backpropagation is performed based on the loss value to complete one round of model training. Both forward computation and backpropagation require matrix multiplication operations. The following describes the steps of matrix multiplication operations in the training system shown in Figure 1, using a specific application scenario.

[0065] For example, taking the Transformer model, in the forward pass of the model, the activation values ​​and weights are split and distributed to multiple nodes. The activation values ​​x and weights wa and wb need to be aggregated through allgather communication, and then matrix multiplication is performed. The activation value x is first multiplied by weight wa, and then by weight wb to obtain the result z. The result z is then split so that each node receives a portion of z for further training. In this process, after the activation values ​​and weights are split and distributed to multiple nodes, each node processes the activation values ​​and weights. Before matrix multiplication, as shown in Figure 1, suppose node 1 needs to synchronize the data to other nodes as weight Wa1 and activation value X1, node 2 as weight Wa2 and activation value X2, node 3 as weight Wa3 and activation value X3, and node 4 as weight Wa4 and activation value X4. To complete the synchronization of activation values ​​x and weights wa, due to the network bandwidth limitations between nodes 1 and 4, two allgather communications are required. Specifically, we can first complete allgather communication based on weight Wa, synchronizing Wa1 to Wa4 to each node. Then, we complete allgather communication based on activation value X, synchronizing X1 to X4 to each node. This requires two data communications to synchronize activation value X and weight Wa. Next, we perform matrix multiplication of activation value X and weight Wa to obtain the result Y. Then, we complete allgather communication based on weight Wb, obtaining the matrix multiplication result Z of weight Wb and Y. Finally, we distribute the matrix multiplication result Z to each node via reducescatter communication, ensuring each node receives a portion of the final output, thus completing the forward computation.

[0066] Therefore, in the example shown in Figure 1, training a model with a transformer architecture requires at least four rounds of ensemble communication for each forward computation. Backpropagation is a similar process, also requiring four rounds of ensemble communication. This results in multiple rounds of training, which severely impacts the efficiency of model training.

[0067] In summary, when training a model using the tensor parallel algorithm with distributed training, the data communication between nodes is limited by bandwidth, which means that each data synchronization requires multiple communications. This results in a large number of communication processes during distributed training, and the communication time may even occupy the majority of the model training time, thus affecting the efficiency of model training.

[0068] To address the issue of numerous communication processes during distributed training, which consume a significant portion of model training time and negatively impact training efficiency, this application provides a model training system 200. This system comprises multiple nodes, which establish multiple network planes through different communication methods. During distributed training, when a node obtains segmented matrix parameter data and trains the model based on this data, it can use different network planes to aggregate communication data for different matrix parameters when it needs to synchronize with other nodes. This allows for parallel execution of aggregate communication for multiple matrix parameters, reducing the number of aggregate communications, improving communication efficiency, and ultimately enhancing model training efficiency.

[0069] The possible architecture of the training system 200 provided in this application will be explained below with reference to Figures 2-4.

[0070] Figure 2 is an architecture diagram of a training system provided in this application. As shown in Figure 2, the training system 200A includes multiple communication groups 211, and each communication group includes multiple nodes. Figure 2 illustrates the example with 3 communication groups and 2 nodes in each communication group. In the specific implementation, this application does not limit the number of nodes in each communication group 211 or the number of communication groups 211 in the training system 200A.

[0071] Nodes can be computing units or computing devices with data processing capabilities. The computing unit can be a hardware accelerator implemented using technologies such as application-specific integrated circuits (ASICs) and field-programmable gate arrays (FPGAs). This hardware accelerator exhibits good performance when performing machine learning tasks, improving the speed and efficiency of neural network training. For example, a computing unit can specifically be a GPU, central processing unit (CPU), tensor processing unit (TPU), data processing unit (DPU), or dedicated AI processing chip, etc., without specific limitations in this application. The computing device can be a bare metal server (BMS), a virtual machine, or a container. A BMS refers to a general-purpose physical server, such as an ARM server or an x86 server; a virtual machine refers to a complete computer system simulated by software, possessing full hardware system functionality and running in a completely isolated environment. Any task that can be performed on a physical computer can also be performed on a virtual machine. When creating a virtual machine on a computing device, a portion of the physical machine's hard drive and memory capacity needs to be used as the virtual machine's hard drive and memory capacity. Each virtual machine has its own independent basic input / output system (BIOS), hard disk, and operating system, and can be operated like a physical machine. A container is a portable software unit that can combine an application and all its dependencies into a single software package. This package is not limited by the underlying host operating system, thus eliminating the need to build complex environments and simplifying the application development and deployment process.

[0072] The aforementioned nodes can be held by the user or provided to the user by the cloud service provider. When the nodes are provided to the user by the cloud service provider, the aforementioned nodes can also be referred to as instances. It should be understood that cloud service providers can provide users with GPU instances, TPU instances, virtual machine instances, or container instances specifically for machine learning, etc. Users can purchase usage rights of instances from cloud service providers according to their usage needs, and this application does not make specific limitations.

[0073] Optionally, multiple network planes can be established between multiple nodes using different communication methods. Multiple nodes using the same communication method to establish communication connections can form a single network plane. For example, Figure 2 illustrates two network planes. In the example shown in Figure 2, multiple nodes can be divided into multiple communication groups. Within each communication group, multiple nodes have a first communication connection, establishing a first network plane. Multiple communication groups also have a second communication connection, establishing a second network plane. The first and second communication connections are different communication channels. It should be understood that when the training system 200A performs distributed training, if the amount of data to be synchronized is large, some data can be synchronized through the first communication connection, and some data through the second communication connection, thereby reducing the number of data communications and training time.

[0074] For example, in the application scenario shown in Figure 1, the training system 200A provided in this application can realize allgather communication of weights W based on the first communication plane, and at the same time, realize allgather communication of activation values ​​X based on the second communication plane. The two data communications can be merged into one, and data synchronization at each stage can be realized in parallel, increasing the number of data communications and reducing training time.

[0075] Optionally, each communication group 211 in the training system 200A can be a chip with a dual-die encapsulation architecture, which allows two identical or different dies to be contained in a single physical package. Here, a die can be a computing unit as described above, such as a bare chip of a GPU, CPU, or TPU. Therefore, a die is a node, and a dual-die encapsulation architecture constitutes a communication group 211.

[0076] Furthermore, each node in the training system 200A is a die, and a communication group 211 is a dual-die packaged chip. Therefore, the first communication connection can be a connection used between the two dies, such as serial input / output (SIO). Two nodes within the communication group 211 establish a first network plane via SIO. Multiple dual-die groups can establish communication connections via high-speed full-mesh interconnect bandwidth, and communication groups 211 can establish a second network plane via high-speed full-mesh interconnect bandwidth. For example, the high-speed full-mesh interconnect bandwidth connection can be Huawei Cloud Communication Service (HCCS). HCCS is a high-speed interconnect technology for cloud computing scenarios, providing high-speed, low-latency cloud communication services. In a full-mesh topology, HCCS can achieve direct connections between nodes while providing high bandwidth and reliability. High-speed full-mesh interconnect bandwidth can also be NVLink technology, which is a high-speed interconnect technology for direct communication between GPUs. It is suitable for high-performance computing and machine learning. In a full-mesh topology, NVLink can directly connect multiple GPUs to achieve efficient transmission. It should be understood that the above examples are for illustration purposes only, and this application does not limit the specific technology used for high-speed full-mesh interconnect bandwidth connection.

[0077] It should be understood that Figure 2 illustrates a communication group with two nodes as an example. In a specific implementation, each communication group may include more nodes. For example, Figure 3 is an architecture diagram of another training system provided in this application. The training system 200B shown in Figure 3 includes two communication groups 211, and each communication group 211 includes four nodes.

[0078] The four nodes in each communication group 211 are interconnected via a first communication connection to form a first network plane. These nodes can be computing units or computing devices as described above. The first communication connection can be a high-speed full-mesh interconnect bandwidth connection; please refer to the relevant descriptions above for details, which will not be repeated here. The communication groups 211 are interconnected via a second communication connection to form a second network plane. This second communication connection is implemented using a switch chip (SW1 to SW4 in Figure 3 are the switch chips).

[0079] It should be understood that Figures 2 and 3 are for illustrative purposes only. This application does not limit the number of communication groups included in the training system, nor does it limit the number of nodes in each communication group, nor does it limit the specific connection method of the first communication connection and the second communication connection.

[0080] It should be noted that in the examples shown in Figures 2 and 3, the training system includes two network planes. In specific implementations, the training system can also include more than two network planes. For example, in the example shown in Figure 2, three communication groups can form a large group, and multiple large groups can establish a third communication connection to form a third network plane. Similarly, in the example shown in Figure 3, two communication groups can form a large group, and multiple large groups can establish a third communication connection to form a third network plane. Of course, the connection method of the third communication connection is different from that of the first and second communication connections. This allows for parallel transmission based on three network planes during data synchronization. The specific number of network planes can be determined according to the actual business scenario, and this application does not impose a specific limitation.

[0081] It should be noted that the training system shown in Figures 2 and 3 can be a supernode in a distributed training system. The distributed training system may include multiple supernodes as shown in Figures 2 and / or 3. One supernode can be used to train one model, and multiple supernodes can train the same model with different training data.

[0082] The structure of the training system provided in this application has been described above with reference to Figures 2 and 3, especially illustrating how multiple nodes in the training system form multiple network planes. The internal unit modules of each node in the training system will be explained below with reference to Figure 4.

[0083] Figure 4 is a schematic diagram of the structure of each node in a training system provided in this application. As shown in Figure 4, the training system may include a first node 410 and a second node 420. The first node 410 and the second node 420 can be any node in the embodiments of Figures 2 and 3. For example, as shown in Figure 4, the first node 410 and the second node 420 are nodes in the same communication group 211. Of course, in other embodiments, the first node 410 and the second node 420 may also be nodes in different communication groups 211. This application does not make specific limitations.

[0084] In this embodiment, the unit modules in each node are identical. For better explanation of the technical solution, Figure 4 refers to the node interacting with the user as the first node 410, and the other nodes as the second node 420. Specifically, the user can interact with any node in the training system and input configuration information.

[0085] The first node 410 includes a configuration unit 411, a configuration information database 412, a cutting unit 413, a processing unit 414, a first communication unit 415, and a second communication unit 416. The second node 420 includes a configuration unit 421, a configuration information database 422, a cutting unit 423, a processing unit 424, a first communication unit 425, and a second communication unit 426.

[0086] As mentioned above, a communication group can include two or more nodes. In the example shown in Figure 4, if communication group 211A includes more nodes, such as a third node, then the structure of the third node is the same as that of the first node 410 and the second node 420. Similarly, the training system can include two or more communication groups. In the example shown in Figure 4, the structures of other communication groups, such as communication group 211B, are the same as those of communication group 211A, and will not be elaborated upon here.

[0087] It should be understood that Figure 4 is an exemplary division method. The first node 410 and the second node 420 may also include more or fewer units. For example, the first communication unit and the second communication unit may be merged into a communication unit. For example, the node may also include a synchronization unit for synchronizing data in the configuration information database. This application does not make any specific limitations.

[0088] The following explanation is based on the functions of each unit module in the first node 410 and the second node 420.

[0089] The configuration unit 411 in the first node 410 is used to obtain configuration information and update the configuration information database 412 according to the configuration information.

[0090] In one embodiment, the configuration information can be configured by a user through client 300. Here, the user can be a developer training the AI ​​model. The configuration information includes the aforementioned mapping relationship, which includes the mapping relationship between communication data and the network plane. Here, the network plane refers to the network plane in the training system. For example, in training system 200A in Figure 2, the first network plane established by two nodes within a communication group via SIO, and the second network plane established between multiple communication groups via high-speed fullmesh interconnect bandwidth; or in Figure 3, the first network plane established by four nodes within a communication group via high-speed fullmesh interconnect bandwidth, and the second network plane established between multiple communication groups via a switch chip. Of course, the training system can also include more network planes; please refer to the relevant descriptions in the embodiments of Figures 2 and 3 for details, which will not be repeated here.

[0091] It's important to note that a network plane can correspond to one or more communication data sets. Communication data at different communication times can correspond to the same network plane, but communication data at the same communication time needs to correspond to different network planes for parallel communication. For example, during training, weights and activation values ​​are usually first communicated together. Each node calculates the gradient based on the weights and activation values ​​of all nodes, and then the gradients are communicated together. Therefore, weights and gradients can have a mapping relationship with the first network plane, and they can use the first network plane for communication at different communication times. However, activation values ​​can only have a mapping relationship with the second network plane. This is because activation values ​​and weights use different network planes for communication at the same time. In this way, communication data that needs to be communicated together at the same time can be transmitted in parallel through different network planes, which can improve the efficiency of communication and thus reduce training time.

[0092] In specific implementations, communication data refers to data that needs to be communicated collectively during model training, such as model weights, gradients, activation values, loss values, model state information, hyperparameters, attention parameters, etc., which are not specifically limited in this application. Collective communication refers to a global communication operation involving all nodes within the training system. It should be understood that the goal of collective communication is to pass messages throughout the entire training system, rather than simply conducting point-to-point communication between two nodes. Typically, set communication can include global reduction (allreduce), global aggregation (allgather), and reduction scatter (reducescatter). Allreduce refers to reducing the data on each node to obtain a reduced result, and then propagating the reduced result to each node so that each node receives the reduced result. Allgather refers to aggregating the data on each node to obtain an aggregated result, and then propagating the aggregated result to each node so that each node receives the aggregated result. Reducescatter refers to splitting the data on a node into multiple parts, sending each part to different nodes for local reduction to obtain local reduction results, and then propagating each node's local reduction result to other nodes so that each node receives the local reduction results from all nodes. It should be understood that set communication can also include more, and this application does not specifically limit it.

[0093] In one embodiment, the configuration information database 412 includes mapping relationships and a segmentation strategy. The segmentation strategy and mapping relationships can be user-configured, system defaults, or a combination of user configuration and system defaults; this application does not impose specific limitations. The segmentation strategy is used by the first node 410 to segment model data or training data into multiple parts for distributed training by multiple nodes.

[0094] In practice, the slicing strategy includes various types of parallel strategies, such as tensor parallel strategy, data parallel strategy, and pipeline parallel strategy. It should be understood that the model can use a combination of multiple slicing strategies to achieve distributed training during the training process. Therefore, users can configure multiple slicing strategies. The explanation of the above-mentioned multiple slicing strategies can be found in the previous content, and will not be repeated here.

[0095] It should be noted that when users configure splitting strategies, the training system can display multiple splitting strategies to users through the client for them to choose from, and save the user's selected splitting strategy. During the training process, the user-selected splitting strategy is used to split the model data or training data.

[0096] It should be understood that multiple matrix multiplication operations occur during AI model training. Under a data-parallel strategy, each node possesses a complete copy of the model. The training data is partitioned and distributed across nodes, allowing each node to perform independent computations using different subsets of data. Each node performs matrix multiplication independently, so the data-parallel strategy does not require additional consideration of whether data partitioning affects matrix multiplication. Similarly, under a pipelined parallel strategy, the model is divided into different parts according to network layers, and each part is trained on different nodes. Matrix multiplication within each network layer can also be performed independently, so pipelined parallelism also does not require additional consideration of whether data partitioning affects matrix multiplication. However, in the tensor parallel strategy, the model weights and activation values ​​are split into multiple parts and distributed to different nodes for computation. This results in a matrix multiplication operation (matrix multiplication of weights and activation values) being split into multiple parts. Therefore, the computation results of each node need to be merged and summarized through set communication to obtain a matrix multiplication result. Thus, tensor parallelism needs to consider the impact of data splitting on matrix multiplication. It requires careful handling of matrix multiplication splitting and set communication merging to avoid matrix operation errors caused by splitting and merging errors, which would affect the model training efficiency.

[0097] Therefore, this application optimizes the tensor parallel strategy in the segmentation strategy, enabling the segmented data to be reasonably distributed to various nodes for computation. After the computation results from each node are aggregated and communicated using different network planes, the obtained matrix multiplication results will not contain errors. The tensor parallel strategy provided in this application will be explained in detail below.

[0098] Optionally, the tensor parallel strategy includes: performing at least one row cut and at least one column cut on the matrix parameters to obtain cut data of the matrix parameters, where the number of cut data is the same as the number of nodes. Specifically, the number of row cuts is determined by the number of communication groups, and the number of column cuts is determined by the number of nodes within each communication group; or, the number of row cuts is determined by the number of nodes within each communication group, and the number of column cuts is determined by the number of communication groups. Row cutting refers to dividing the matrix horizontally into multiple parts along the direction of the matrix rows, and column cutting refers to dividing the matrix vertically into multiple parts along the direction of the matrix columns.

[0099] In practical implementation, matrix parameters refer to the two multipliers involved in matrix multiplication. Both multipliers are matrices, such as activation values ​​and weights. Activation values ​​are the values ​​calculated on neurons, including input data to the network and intermediate data during model training, such as the output data of the previous network layer and the result of the previous matrix multiplication. Weights refer to the model's weights. During model training, the weights are continuously adjusted to allow the model to learn the features and patterns in the data, thereby continuously improving model performance. Matrix parameters can also be bias terms, gradients, etc. It should be understood that model training includes multiple matrix multiplication tasks. The two matrix parameters in each matrix multiplication task can be segmented in the above way to obtain the segmented data of each matrix parameter, which is then distributed to various nodes. This allows a matrix multiplication task to be divided into multiple subtasks for parallel processing. Then, the calculation results of each subtask are aggregated through ensemble communication to obtain the calculation result of the matrix multiplication task. This application does not limit the specific content of the matrix parameters. In order to make this application better understandable, the following will use the matrix multiplication between activation values ​​and weights as an example, with the matrix parameters being activation values ​​and weights.

[0100] For example, if the matrix parameter is 2×3, after one row partition, it becomes two 1×3 matrices; after three column partitions, it becomes six 1×1 matrices. As another example, if the matrix parameter is 4×4, after one row partition, it becomes two 2×4 matrices; after one column partition, it becomes four 2×2 matrices. In the above examples, an a×b matrix refers to a matrix with a rows and b columns. These examples are for illustrative purposes only and are not intended to impose specific limitations.

[0101] In the specific implementation, the first matrix parameter and the second matrix parameter participating in the same matrix multiplication operation are cut in the same way. That is, the number of row cuts of the first matrix parameter and the number of row cuts of the second matrix parameter are the same, and the number of column cuts of the first matrix parameter and the number of column cuts of the second matrix parameter are the same. In simple terms, if the first matrix parameter is the activation value and the second matrix parameter is the weight, and the activation value is split into 6 parts by 2 rows and 3 columns, then the weight also needs to be split into 6 parts. However, the weight can only be split by 3 columns and 2 rows. If the weight is split by 2 rows and 3 columns, although it can still be split into 6 parts, the number of rows in the split data of the activation value is different from the number of columns in the split data of the weight. This may cause a problem when the split data is distributed to each node for matrix multiplication, where the two split data cannot be multiplied because their inner axes are different. For example, if each row of split data x1 has 2 elements and each column of split data y1 has 3 elements, their inner axes are different, so they cannot be multiplied. Therefore, the two matrix parameters need to be split in the same way to avoid the problem of split data not being able to be multiplied.

[0102] Furthermore, if each communication group has m nodes and the number of communication groups is n, then the number of row cuts is m-1, meaning the matrix parameters are divided into m parts by rows, and the number of column cuts is n-1, meaning the matrix parameters are further divided into m×n parts by columns. Alternatively, the number of row cuts is n-1, and the number of column cuts is m-1. Simply put, if the node size is 2×3 (3 communication groups, each with 2 nodes), then the number of row cuts is 1, and the number of column cuts is 2, meaning the matrix parameters are also cut into 2×3 parts. It should be noted that another matrix parameter involved in matrix multiplication also needs to be cut in the same way; for example, activation values ​​and weights are cut into 2×3 parts.

[0103] Optionally, the configuration information input by the user may also include parameter information that needs to be configured during model training. The configuration information database 412 may also include the aforementioned parameter information, which includes the structural parameters of the AI ​​model to be trained, as well as the scale parameters of the training system 200 (such as training system 200A or training system 200B). The structural parameters, scale parameters, and training parameters are explained below.

[0104] Optionally, the structural parameters of the AI ​​model may include the hidden layer dimension, the feedforward layer dimension, and the number and size of multi-head attention heads. The hidden layer dimension refers to the number or dimension of neurons in the hidden layer of the AI ​​model; the feedforward layer dimension refers to the number or dimension of neurons in the feedforward layer of the AI ​​model; and the number and size of multi-head attention heads refer to the number of independent attention heads used by the AI ​​model and the dimension of the sub-controls within each attention head, or the dimension of the weights learned within each attention head. Each attention head is responsible for focusing on a specific feature of the input data. Increasing the number of attention heads can increase the model's ability to capture different features in the input data, and larger attention heads allow the model to learn more complex features. It should be understood that the above examples are for illustrative purposes only and this application does not impose specific limitations.

[0105] Optionally, the scale parameters of the training system 200 may include parameters related to the training environment and hardware resources, such as the number of nodes in the training system 200, the specifications of each node, the number of network planes, the number of nodes in each network plane, and the network bandwidth. The number of nodes in the training system includes the number of nodes participating in training. The specifications of each node include the hardware specifications of each node, such as CPU parameters, GPU parameters, and memory parameters. The number of network planes refers to the number of network planes established by multiple nodes through different communication methods. Different network planes use different communication methods to connect; for example, training systems 200A and 200B shown in Figures 2 and 3 use two different communication methods to establish the first and second network planes. The number of nodes in each network plane and the network bandwidth refer to the number of nodes in each network plane and the bandwidth of communication between nodes. It should be understood that the above examples are for illustration, and the scale parameters may include more content, which is not specifically limited in this application.

[0106] Optionally, training parameters in the training process may include batch size, sequence length, optimizer, learning rate, activation function, and loss function. Batch size refers to the number of samples input for each model update; training data can be divided into multiple batches, with each batch containing a certain number of samples. Sequence length refers to the length of the sequence data expanded over time steps. The optimizer refers to how the model updates weights during training to minimize the loss function. The learning rate controls the magnitude of model parameter updates in each iteration. The activation function defines the output of each neuron in the model. The loss function measures the model's performance during training. Parallel strategies are methods for configuring model training to run in parallel across multiple nodes.

[0107] It should be understood that parameter information, in addition to the scale parameters of the training system, the training parameters of the training process, and the structural parameters of the AI ​​model mentioned above, may include more content, which will not be listed here. Configuration information, in addition to the cutting strategy, mapping relationship, and parameter information mentioned above, may also include more content, which will not be listed here.

[0108] Optionally, the configuration information database 412 in the first node 410 can synchronize mapping relationships with other nodes, so that the configuration information database on each node can obtain the mapping relationship. For example, the configuration information database 422 of the second node 420 can obtain the mapping relationship synchronized by the first node 410, so that the second node 420 can determine the network plane to be used for different communication data according to the mapping relationship during aggregated communication. In specific implementation, the first node 410 can complete the synchronization of the mapping relationship at a fixed time or a fixed frequency, which is not specifically limited in this application. Furthermore, other data in the configuration information database 412, such as the aforementioned segmentation strategy and parameter information, can also be synchronized to other nodes according to usage requirements, which is not specifically limited in this application.

[0109] Optionally, client 300 can be a client of an AI parallel training framework. A parallel training framework is a software tool designed to train AI models in parallel across multiple nodes. Commonly used parallel training frameworks include TensorFlow and PyTorh, and this application does not specifically limit them. In this embodiment, the user can generate configuration information by modifying the configuration file of the AI ​​parallel training framework.

[0110] Optionally, client 300 can be a client for an AI compiler. It should be understood that dedicated AI processing chips such as GPUs and TPUs provide AI compilers for users to configure, and users can generate configuration information through the AI ​​compiler's client.

[0111] Optionally, the client 300 can be a console in a cloud data center. This console can provide users with various communication data that need to be communicated during the training process, as well as multiple network planes existing in the training system. Users can select communication data and network planes from the configuration interface to establish a mapping relationship. It can also provide some training parameters, scale parameters, and structure parameters for users to select, thereby generating configuration information. This application does not specifically limit this.

[0112] It should be noted that the configuration unit 421 in the second node 420 has the same function as the configuration unit 411 in the first node 410. That is, the configuration unit 421 in the second node 420 can also receive configuration information sent by the user through the client 300 and store the configuration information in the configuration information database 422. However, in the application scenario shown in Figure 4, the first node 410 interacts with the client 300, so in the application scenario shown in Figure 4, the configuration unit 421 of the second node 420 does not participate in the work. However, in actual application scenarios, users can also interact with the second node 420 through the client 300 and input configuration information, which is not specifically limited in this application.

[0113] The cutting unit 413 is used to cut the matrix parameters according to the cutting strategy to obtain multiple cutting data of the matrix parameters.

[0114] In a specific implementation, the cutting unit 413 can complete the cutting of matrix parameters according to the tensor parallel strategy described above. For example, after cutting the first matrix parameter and the second matrix parameter that participate in the same matrix multiplication operation, multiple first cutting data of the first matrix parameter and multiple second cutting data of the second matrix parameter are obtained.

[0115] Optionally, the cutting unit 413 is further configured to distribute multiple cut data for each matrix parameter to each node. Referring to the foregoing, the number of cut data for each matrix parameter is the same as the number of nodes, so each node receives one cut data for one matrix parameter. Since multiple matrix parameters participate in the matrix multiplication operation, each node can receive multiple cut data for multiple matrix parameters. For example, if the first matrix parameter is the activation value and the second matrix parameter is the weight, then each node will receive cut data for the activation value and cut data for the weight. The above examples are for illustrative purposes only and are not intended to limit the scope of the application.

[0116] It should be noted that if the second node 420 receives the user's configuration information, it can also cut the matrix parameters through the cutting unit 423 to obtain multiple cut data of the matrix parameters, and then distribute them to other nodes. Examples will not be given here.

[0117] It should be noted that, as mentioned above, the slicing strategy includes various types of parallel strategies, such as tensor parallel strategy, data parallel strategy, and pipeline parallel strategy. If the training process uses a combination of multiple parallel strategies to train the model, the slicing unit 413 can slice the model parameters and training data according to multiple parallel strategies to obtain the sliced ​​data of the model parameters and training data, and then distribute it to each node. This will not be elaborated here.

[0118] The processing unit 414 is used to process the segmented data to obtain the communication data that needs to be used for collection communication.

[0119] Similarly, the processing units 414 on other nodes also need to process the received segmented data to obtain the communication data required for aggregate communication. For example, processing the first matrix parameters yields the first communication data, and processing the second matrix parameters yields the second communication data.

[0120] In specific implementation, the processing content of processing unit 414 can be determined according to the actual business scenario. Different models require different processing steps before ensemble communication during training, and the same model requires different processing steps before ensemble communication at different training stages. Therefore, this application does not limit the processing content of processing unit 414. For example, the processing content may include forward computation on the segmented data, where the communication data refers to the forward computation result. Alternatively, the processing content may include backpropagation on the segmented data, where the communication data refers to the gradient of the weights. The above examples are for illustrative purposes only and are not specifically limited in this application.

[0121] Optionally, the processing unit 414 can perform aggregated communication of communication data for different matrix parameters through different network planes. For example, the first communication data of the first matrix parameter can be aggregated using the first network plane, and the second communication data of the second matrix parameter can be aggregated using the second network plane; or, the first communication data can be aggregated using the second network plane, and the second matrix parameter can be aggregated using the first network plane. It should be understood that transmitting communication data of different matrix parameters through different network planes can improve the efficiency of aggregated communication. For example, during model training, after the activation values ​​and weights are distributed to each node after being split, each node performs forward calculation on the activation values ​​and weights, and then needs to aggregate the processed activation values ​​and weights through aggregated communication. If the training system shown in Figure 1 is used, the activation values ​​can only be aggregated first, and then the weights can be aggregated. However, using the training system 200 of this application, the activation values ​​and weights are aggregated using different network planes, which can greatly improve the efficiency of aggregated communication.

[0122] In specific implementation, processing unit 414 can send communication data to either the first communication unit 415 or the second communication unit 416 for aggregate communication based on the mapping relationship. As mentioned above, the mapping relationship includes the mapping between communication data and network planes. Communication data corresponding to the first network plane can be sent to the first communication unit 415 for aggregate communication, and communication data corresponding to the second network plane can be sent to the second communication unit 416 for aggregate communication. Since the two network planes use different communication methods, aggregate communication between the two network planes can be performed simultaneously, improving the efficiency of aggregate communication during distributed training.

[0123] Optionally, there is a correlation between the mapping relationship and the slicing strategy. The mapping relationship is determined based on the number of row slices and column slices of the matrix parameters. Specifically, if the number of row slices of the matrix parameters is determined based on the number of nodes in the communication group, and the number of column slices is determined based on the number of communication groups, then the first communication data is communicated collectively through the second network plane, and the second communication data is communicated collectively through the first network plane. Conversely, if the number of row slices of the matrix parameters is determined based on the number of communication groups, and the number of column slices is determined based on the number of nodes in the communication group, then the first communication data is communicated collectively through the first network plane, and the second communication data is communicated collectively through the second network plane.

[0124] It should be understood that during model training, after each ensemble communication, each node receives communication data from other nodes, then merges this data to obtain the merged result for each matrix parameter, and then performs matrix multiplication operations between the merged results. For example, multiple first communication data are merged into a third matrix parameter, and multiple second communication data are merged into a fourth matrix parameter, and then the third and fourth matrix parameters are multiplied. Matrix multiplication requires that the number of columns in the first multiplier is the same as the number of rows in the second multiplier, meaning the inner axes of the two multipliers are the same; otherwise, matrix multiplication cannot be performed. Therefore, the network plane used during ensemble communication needs to refer to the matrix parameter cutting method to ensure that the number of columns in the merged third matrix parameter is the same as the number of rows in the fourth matrix parameter.

[0125] It should be understood that since the first and second matrix parameters are two parameters that require matrix multiplication, and the number of columns in the first matrix parameter is the same as the number of rows in the second matrix parameter, ensuring that the number of columns in the third matrix parameter is the same as the number of columns in the first matrix parameter, and the number of rows in the fourth matrix parameter is the same as the number of rows in the second matrix parameter, will ensure that the number of columns in the third matrix parameter is the same as the number of rows in the fourth matrix parameter. Following this idea, if the number of column cuts in the first matrix parameter is determined by the number of communication groups, and the second communication method is inter-group communication, then when the first communication result of the first matrix parameter is used for aggregate communication, the second communication plane needs to be used for aggregate communication to ensure that the number of columns in the merged third matrix parameter is the same as the number of columns in the first matrix parameter. Similarly, if the number of column cuts in the first matrix parameter is determined by the number of nodes within a communication group, the first communication plane needs to be used for aggregate communication to ensure that the number of columns in the merged third matrix parameter is the same as the number of columns in the first matrix parameter.

[0126] It should be noted that after each node performs group communication and obtains multiple communication data from other nodes, when merging these multiple communication data, it is necessary to ensure that the number of columns in the merged third matrix parameter is the same as the number of rows in the fourth matrix parameter. Therefore, the multiple first communication data obtained through inter-group communication need to be horizontally concatenated to obtain the third matrix parameter, ensuring that the number of columns in the third matrix parameter is completely concatenated so that the number of columns in the third matrix parameter is the same as the number of columns in the first matrix parameter. Similarly, the multiple second communication data obtained through inter-group communication need to be vertically concatenated to obtain the fourth matrix parameter, ensuring that the number of rows in the fourth matrix parameter is completely concatenated so that the number of rows in the fourth matrix parameter is the same as the number of rows in the second matrix parameter. In this way, the number of columns in the final third matrix parameter is the same as the number of rows in the fourth matrix parameter, which allows for matrix multiplication.

[0127] For example, taking the training system 200 shown in Figure 2 as an example, the number of nodes m in a communication group is 2, the number of communication groups n is 3, the first communication plane is intra-group communication, and the second communication plane is inter-group communication. If the number of column cuts in the matrix parameters is n-1 = 2 times and the number of row cuts is m-1 = 1 time, then after each node obtains the first communication result, the first communication data needs to be merged into three first communication results using inter-group communication (second communication plane) to obtain the third matrix parameter. Since the first matrix parameter is cut into three columns twice, and the merging also merges three first communication results, the number of columns in the merged third matrix parameter is the same as the number of columns in the first matrix parameter. Similarly, the second matrix parameter needs to be merged into two second communication results using intra-group communication (first communication plane) to obtain the fourth matrix parameter. Since the second matrix parameter is cut into two rows once, and the merging also merges two second communication results, the number of rows in the merged fourth matrix parameter is the same as the number of rows in the second matrix parameter. The above examples are for illustration only, and this application does not impose specific limitations.

[0128] It should be noted that, due to the aforementioned relationship between the mapping relationship and the splitting strategy, if the user configures the mapping relationship, the splitting strategy needs to flexibly adjust the number of column and row splits based on the mapping relationship; conversely, if the user configures the splitting strategy, the mapping relationship needs to be flexibly adjusted based on the splitting strategy. If both the mapping relationship and the splitting strategy are user-configured, the relationship between the two needs to be indicated to the user, and any problems with the user's configuration should be promptly reported. If the user has not configured either the mapping relationship or the splitting strategy, the training system 200 can complete the training using the default configuration.

[0129] Optionally, the processing unit 414 is further configured to perform matrix multiplication on the third matrix parameters and the fourth matrix parameters obtained by the set communication to obtain a first operation result, and complete the matrix multiplication task between the first matrix parameters and the second matrix parameters based on the first operation result, thereby completing the training of the model.

[0130] Furthermore, during model training, there are often continuous matrix multiplication tasks. For example, as shown in Figure 1, after the activation value is multiplied by weight 1, it needs to be multiplied by weight 2. Therefore, for a single matrix multiplication task, the first matrix parameter includes the activation value, and the second matrix parameter includes the weight. For a multiple matrix multiplication task, the first matrix parameter includes the matrix multiplication result of the activation value and the first weight, and the second matrix parameter includes the second weight. In other words, the matrix parameter can also be a temporary intermediate result generated during the calculation or a temporary input activation value. This application does not limit this.

[0131] The following example uses two consecutive matrix multiplications. Assume the activation value is multiplied by the first weight in the first matrix multiplication, and then multiplied by the second weight in the second matrix multiplication. We can first obtain the third matrix parameter corresponding to the activation value and the fourth matrix parameter corresponding to the first weight as described above. After obtaining the first operation result between the two, we use the first operation result as the first matrix parameter and the second weight as the second matrix parameter. After segmenting according to the description above, we determine the mapping relationship according to the segmentation strategy. Then, we complete the set communication according to their respective communication planes to obtain the third matrix parameter corresponding to the first operation result and the fourth matrix parameter corresponding to the second weight. Finally, based on the second operation result between the two, we complete the two consecutive matrix multiplications.

[0132] Specifically, the cutting unit 413 can cut the second weight to obtain cut data, and distribute the multiple cut results of the second weight to each node. The description of this process can be referred to the previous description of the cutting and distribution of the first matrix parameter and the second matrix parameter, which will not be repeated here. It should be noted that the cutting method of the second weight is the same as that of the first matrix parameter and the second matrix parameter, and the number of row cutting and column cutting is the same.

[0133] Optionally, processing unit 414 processes the segmented data of the second weights to obtain third communication data, and then performs aggregate communication on the first operation result through second communication unit 416 or first communication unit 415. The third matrix parameters corresponding to the first operation result are obtained by performing aggregate communication on the third communication data through first communication unit 415 or second communication unit 416 to obtain the fourth matrix parameters corresponding to the second weights. Processing unit 414 performs matrix multiplication on the two to obtain the second operation result. Since the two network planes use different communication methods, the aggregate communication of the two network planes can be performed simultaneously, improving the efficiency of aggregate communication during distributed training.

[0134] It should be understood that if the training system shown in Figure 1 is used, two consecutive matrix multiplication operations require four ensemble communications. The first ensemble communication aggregates activation values, the second ensemble communication aggregates the first weight, the third ensemble communication aggregates the second weight, and the fourth ensemble communication reduces the result of the matrix multiplication operation. With the training system provided in this application, two consecutive matrix multiplication operations only require two ensemble communications. In the first ensemble communication, the first communication data is ensembled through the second network plane, and the second communication data is ensembled through the first network plane. In the second ensemble communication, the first operation result is ensembled through the second network plane, and the third communication data is ensembled through the first network plane. In this way, only two ensemble communications are needed to complete two matrix multiplications, improving communication efficiency and model training efficiency.

[0135] It should be noted that the process described above mainly refers to forward propagation. For backward propagation, the matrix parameters can also be divided in the same way, and then the data can be aggregated through different communication planes. Typically, allgather is used for aggregated communication in forward propagation, and reducescatter is used for aggregated communication in backward propagation.

[0136] It should be understood that in the architecture shown in Figure 1, the result of matrix multiplication needs to be split and distributed to each node. Therefore, in the architecture shown in Figure 1, the first three group communications are allgather, and the last group communication needs to use reducescatter. The splitting method provided in this application, combined with the multi-network plane architecture of the training system 200, can make multiple group communications use the same group communication method. For example, allgather is used for forward computation, and reducescatter is used for backward computation, which can improve training efficiency.

[0137] It should be noted that the matrix multiplication operation process described above is based on the processing unit 414, the first communication unit 415, and the second communication unit 416 as examples. In fact, during the matrix operation process, the processing flow of the processing units, the first communication unit, and the second communication unit of other nodes is the same as that of the first node 410. Each node performs the same processing process on the allocated segmented data and uses the same mapping relationship to complete the communication between different network planes. The descriptions of the processing unit 424, the first communication unit 425, and the second communication unit 416 can be found in the descriptions of the processing unit 414, the first communication unit 415, and the second communication unit 416, and will not be repeated here.

[0138] It should be noted that the above example uses the first matrix parameter as the activation value, the second matrix parameter as weight 1, and the second weight as weight 2. This scenario mainly applies to the training of transformer models. For other model training scenarios, a similar approach can be used to segment the matrix parameters and determine the network plane used for communication between different matrix parameters. For example, for neural networks using the attention module, if matrix multiplication operations are required between activation values, weights, and attention parameters during training, the attention parameters can also be segmented and communicated in a similar way. Examples will not be provided here.

[0139] Furthermore, the example above uses two matrix multiplication operations. In other model training, the number of matrix multiplications can be greater, and multiple different weights may be multiplied simultaneously. In all cases, the number of row and column cuts can be determined according to the description above, and the communication planes of different weights can be determined according to the cutting method. This fully utilizes the parallel communication of multiple communication planes to improve model training efficiency.

[0140] Furthermore, the above explanation uses two communication planes as an example, which is a two-layer interconnection network. There is one network plane between communication groups and one network plane within a communication group. If there are more than two network planes, such as three-layer network planes, multiple communication groups can also be considered as a large group. There is a third network plane between the large groups. In this case, according to the model architecture, three network planes can be used for aggregated communication, or any two network planes can be selected from the three network planes for aggregated communication. The specific choice can be determined according to the actual business scenario, and this application does not impose any specific limitations.

[0141] It should be noted that the allgather communication described above refers to aggregating communication data from multiple nodes so that each node has the aggregated communication result. In some different training platforms, allgather may also be called other names, such as allbond, but the processing flow is the same. Similarly, reducescatter communication may also have different names in different training platforms. This application does not limit the communication terminology.

[0142] In summary, this application provides a model training system comprising multiple nodes. These nodes establish multiple network planes through different communication methods. Thus, during distributed training, when a node obtains the segmented data of matrix parameters and trains the model based on the segmented data, and obtains communication data that needs to be synchronized with other nodes, the communication data of different matrix parameters can be used for aggregated communication through different network planes. This allows the aggregated communication of multiple matrix parameters to be executed in parallel, reducing the number of aggregated communications, improving the efficiency of aggregated communication, and thereby improving the efficiency of model training.

[0143] The training system provided in this application has been described in detail above. The model training method provided in this application will be explained below with reference to Figure 5.

[0144] Figure 5 is a schematic flowchart of a model training method provided in this application. This model training method can be applied to training system 200A or training system 200B as shown in Figures 2-4. As shown in Figure 5, the method may include the following steps:

[0145] S510: The first node 410 acquires the cutting strategy and mapping relationship. This step can be implemented by the configuration unit 411 in the embodiment of Figure 4.

[0146] In specific implementation, the cutting strategy and mapping relationship can be generated based on the configuration information sent by the user through the client 300, or it can be the default configuration of the training system 200. Alternatively, some cutting strategies and some mapping relationships can be configured by the user, while others can be automatically generated by the training system 200. This application does not impose any specific limitations.

[0147] In one possible implementation, the mapping relationship includes the mapping relationship between communication data and the network plane.

[0148] In this context, "network plane" refers to the network plane within the training system. For example, in training system 200A in Figure 2, it refers to the first network plane established by two nodes within a communication group via SIO, and the second network plane established between multiple communication groups via high-speed fullmesh interconnect bandwidth. Alternatively, in Figure 3, it refers to the first network plane established by four nodes within a communication group via high-speed fullmesh interconnect bandwidth, and the second network plane established between multiple communication groups via a switch chip. Of course, the training system can also include more network planes; please refer to the relevant descriptions in the embodiments of Figures 2 and 3 for details, which will not be repeated here.

[0149] The communication data refers to the data that needs to be communicated collectively during model training, such as model weights, gradients, activation values, loss values, hyperparameters, attention parameters, etc., which are not specifically limited in this application. Collective communication refers to a global communication operation involving all nodes within the training system. It should be understood that the goal of collective communication is to transmit messages throughout the entire training system, rather than just point-to-point communication between two nodes. Typically, collective communication may include allreduce, allgather, reducescatter, etc., as described in the embodiments of Figures 2-4 above, and will not be repeated here.

[0150] In this context, a network plane can correspond to one or more communication data. Communication data at different communication times can correspond to the same network plane, but communication data at the same communication time need to correspond to different network planes for parallel communication. In this way, communication data that needs to be aggregated at the same time can be transmitted in parallel through different network planes, which can improve the efficiency of aggregated communication and thus reduce the training time.

[0151] In one possible implementation, the slicing strategy includes various types of parallel strategies, such as tensor parallelism, data parallelism, and pipeline parallelism. It should be understood that the model can use a combination of multiple slicing strategies to achieve distributed training during the training process. Therefore, users can configure multiple slicing strategies. The explanation of the above-mentioned multiple slicing strategies can be found in the foregoing content, and will not be repeated here.

[0152] The tensor parallel strategy involves performing at least one row cut and at least one column cut on the matrix parameters to obtain cut data, where the number of cut data is the same as the number of nodes. Specifically, the number of row cuts is determined by the number of communication groups, and the number of column cuts is determined by the number of nodes within each communication group; alternatively, the number of row cuts is determined by the number of nodes within each communication group, and the number of column cuts is determined by the number of communication groups. A row cut refers to dividing the matrix horizontally along its rows, while a column cut refers to dividing the matrix vertically along its columns.

[0153] The tensor parallel strategy also includes ensuring that the first and second matrix parameters participating in the same matrix multiplication operation are sliced ​​in the same way. That is, the number of row slices for the first matrix parameter is the same as the number of row slices for the second matrix parameter, and the number of column slices for the first matrix parameter is the same as the number of column slices for the second matrix parameter. If the slicing method changes, when the sliced ​​data is distributed to various nodes for matrix multiplication, there may be a problem where the two sliced ​​data cannot be multiplied because their inner axes are different. Therefore, the two matrix parameters need to be sliced ​​in the same way to avoid the problem of sliced ​​data being unable to be multiplied.

[0154] Tensor parallelism also includes: if there are m nodes in each communication group and n communication groups, then the number of row cuts is m-1, meaning the matrix parameters are cut into m parts by rows, and the number of column cuts is n-1, meaning the matrix parameters are further cut into m×n parts by columns. Alternatively, the number of row cuts is n-1, and the number of column cuts is m-1.

[0155] For example, Figure 6 is an example diagram of matrix partitioning provided in this application. As shown in Figure 6, assuming the length of the first matrix parameter is H and the width is A, and the length of the second matrix parameter is B and the width is H, and the scale of the training system 200 is shown in Figure 2, that is, the number of communication groups n = 3, and the number of nodes in the communication group m = 2. If the number of row partitions is determined according to the number of nodes in the communication group, then the number of row partitions is 1, and the number of column partitions is 2, and the partitioning method of each matrix parameter is the same, that is, as shown in Figure 6, the first matrix parameter is partitioned into 6 parts, and the second matrix parameter is also partitioned into 6 parts. It should be understood that Figure 6 is for illustrative purposes only. If the number of row partitions is determined according to the number of communication groups, then the number of row partitions is 2, the number of column partitions is 1, and the partitioning method of the two matrix parameters is consistent. This application does not impose specific limitations.

[0156] For another example, if the scale of the training system 200 is as shown in Figure 3, that is, the number of communication groups n=2 and the number of nodes in the communication group m=4, if the number of row cuts is determined based on the number of nodes in the communication group, then the number of row cuts is 3 and the number of column cuts is 1; if the number of row cuts is determined based on the number of communication groups, then the number of row cuts is 1 and the number of column cuts is 3. The above examples are for illustration only, and this application does not impose specific limitations.

[0157] It should be noted that after determining the number of cuts, the matrix parameters can be cut evenly, or cut by row or column. For example, if there are 4 rows that need to be cut twice, then the first row can be cut once and the third row can be cut once. That is, it is not necessarily an even cut. The specific cutting method can be determined according to the actual business scenario. This application does not make specific limitations.

[0158] The above examples show that the technical solution provided in this application can flexibly determine the cutting method, but it is necessary to keep each matrix parameter cutting using the same cutting method. Furthermore, the cutting method is related to the number of communication groups or the number of nodes within the communication group, and it is necessary to ensure that the number of cut data is consistent with the number of nodes.

[0159] In one possible implementation, there is a correlation between the mapping relationship and the slicing strategy. The mapping relationship is determined based on the number of row slices and column slices of the matrix parameters. Specifically, if the number of row slices of the matrix parameters is determined based on the number of nodes in the communication group, and the number of column slices is determined based on the number of communication groups, then the first communication data is communicated collectively through the second network plane, and the second communication data is communicated collectively through the first network plane. Conversely, if the number of row slices of the matrix parameters is determined based on the number of communication groups, and the number of column slices is determined based on the number of nodes in the communication group, then the first communication data is communicated collectively through the first network plane, and the second communication data is communicated collectively through the second network plane.

[0160] It should be understood that during model training, after each ensemble communication, each node receives communication data from other nodes, then merges this data to obtain the merged result for each matrix parameter, and then performs matrix multiplication operations between the merged results. For example, multiple first communication data are merged into a third matrix parameter, and multiple second communication data are merged into a fourth matrix parameter, and then the third and fourth matrix parameters are multiplied. Matrix multiplication requires that the number of columns in the first multiplier is the same as the number of rows in the second multiplier, meaning the inner axes of the two multipliers are the same; otherwise, matrix multiplication cannot be performed. Therefore, the network plane used during ensemble communication needs to refer to the matrix parameter cutting method to ensure that the number of columns in the merged third matrix parameter is the same as the number of rows in the fourth matrix parameter.

[0161] It should be understood that since the first and second matrix parameters are two parameters that require matrix multiplication, and the number of columns in the first matrix parameter is the same as the number of rows in the second matrix parameter, ensuring that the number of columns in the third matrix parameter is the same as the number of columns in the first matrix parameter, and the number of rows in the fourth matrix parameter is the same as the number of rows in the second matrix parameter, ensures that the number of columns in the third matrix parameter is the same as the number of rows in the fourth matrix parameter. Following this idea, if the number of column cuts in the first matrix parameter is determined by the number of communication groups, and the second communication method is inter-group communication, then when the first communication result of the first matrix parameter is used for aggregate communication, the second communication plane needs to be used for aggregate communication to ensure that the number of columns in the merged third matrix parameter is the same as the number of columns in the first matrix parameter. Similarly, if the number of column cuts in the first matrix parameter is determined by the number of nodes within a communication group, the first communication plane needs to be used for aggregate communication to ensure that the number of columns in the merged third matrix parameter is the same as the number of columns in the first matrix parameter.

[0162] It should be noted that after each node performs group communication and obtains multiple communication data from other nodes, when merging these multiple communication data, it is necessary to ensure that the number of columns in the merged third matrix parameter is the same as the number of rows in the fourth matrix parameter. Therefore, the multiple first communication data obtained through inter-group communication need to be horizontally concatenated to obtain the third matrix parameter, ensuring that the number of columns in the third matrix parameter is completely concatenated so that the number of columns in the third matrix parameter is the same as the number of columns in the first matrix parameter. Similarly, the multiple second communication data obtained through inter-group communication need to be vertically concatenated to obtain the fourth matrix parameter, ensuring that the number of rows in the fourth matrix parameter is completely concatenated so that the number of rows in the fourth matrix parameter is the same as the number of rows in the second matrix parameter. In this way, the number of columns in the final third matrix parameter is the same as the number of rows in the fourth matrix parameter, which allows for matrix multiplication.

[0163] It should be noted that, due to the aforementioned relationship between the mapping relationship and the splitting strategy, if the user configures the mapping relationship, the splitting strategy needs to flexibly adjust the number of column and row splits based on the mapping relationship; conversely, if the user configures the splitting strategy, the mapping relationship needs to be flexibly adjusted based on the splitting strategy. If both the mapping relationship and the splitting strategy are user-configured, the relationship between the two needs to be indicated to the user, and any problems with the user's configuration should be promptly reported. If the user has not configured either the mapping relationship or the splitting strategy, the training system 200 can complete the training using the default configuration.

[0164] In one possible implementation, the first node 410 can also acquire parameter information, which can be system default configuration or input by the user through the client 300; this application does not impose specific limitations. The parameter information may include structural parameters of the AI ​​model, scale parameters of the training system 200, and training parameters that need to be configured during the training process. Structural parameters may include parameters related to the network structure of the AI ​​model, such as the size of the hidden layer dimension, the size of the feedforward layer dimension, and the number and size of multi-head attention. Scale parameters may include parameters related to the hardware resources of the training system 200, such as the number of nodes, node specifications, the number of network planes, the number of nodes in the network planes, and bandwidth. Training parameters may include parameters related to the model training process, such as batch size, sequence length, optimizer, learning rate, activation function, and loss function; the above examples are for illustrative purposes only and this application does not impose specific limitations.

[0165] Figure 7 is an example diagram of a configuration interface provided in this application. The configuration interface can be displayed to the user through the client 300 to obtain the configuration information input by the user, and then update the configuration information database according to the configuration information. As shown in Figure 7, the configuration interface may include a mapping relationship configuration area 710 and a cutting strategy configuration area 720.

[0166] The mapping configuration area 710 may include a communication data display area and a network plane display area. The system will display the configurable network plane and communication data in these two areas, and users can establish a connection between them. As mentioned above, the mapping relationship between communication data and the network plane can be configured by the user or generated by the system. Therefore, the client 300 can display the recommended mapping relationship to the user through this configuration interface, and the user can modify it according to their needs.

[0167] The slicing strategy configuration area 720 may include column slicing count configuration options and row slicing count configuration options, allowing users to configure the slicing strategy according to their needs. Of course, the slicing strategy configuration area may also include configuration areas for other parallel strategies, such as pipelined parallel strategies and data parallel strategies. Furthermore, the configuration interface may include more areas, such as configuration areas for training parameters, configuration areas for model structure parameters, and configuration areas for training system scale parameters. Figure 7 is an example of a configuration interface, and this application does not impose specific limitations on it.

[0168] As mentioned above, there is a correlation between the mapping relationship and the slicing strategy. The mapping relationship is determined based on the number of row and column slices in the matrix parameters. For example, if the number of row slices in the matrix parameters is determined based on the number of nodes in the communication group, then the first communication data will be aggregated through the second network plane. Therefore, when a user configures the data through the configuration interface shown in Figure 7, if the mapping relationship and slicing strategy selected by the user violate the above correlation, the client can prompt the user with a message indicating that the configuration is non-compliant. Alternatively, when the user configures the slicing strategy, the system generates the corresponding mapping relationship based on the above correlation and displays it in the mapping relationship configuration area 710. Similarly, when the user configures the mapping relationship, the system generates the corresponding slicing strategy based on the above correlation and displays it in the slicing strategy configuration area 720. This application does not impose any specific limitations.

[0169] It should be noted that Figure 7 is an example of a visual interface. In actual implementation, during the model training process, users can configure mapping relationships and splitting strategies through non-visual interfaces such as application programming interfaces (APIs) and command line interfaces (CLIs). This application does not impose any specific limitations.

[0170] S520: The first node 410 sends the mapping relationship to other nodes. The second node 420 receives and stores the mapping relationship. This step can be implemented by the configuration unit 411 and configuration unit 421 in the embodiment of Figure 4.

[0171] In the specific implementation, the first node 410 can synchronize the mapping relationship to other nodes, so that each node can complete the collective communication according to the stored mapping relationship. Different communication data use different network planes for collective communication, so that the communication data of multiple matrix parameters can be transmitted at the same time, improving the efficiency of collective communication and thus improving the efficiency of model training.

[0172] It should be noted that the first node 410 can also send the above-mentioned cutting strategy and parameter information to other nodes. The specific content to be synchronized can be determined according to the actual business needs, and this application does not make specific limitations.

[0173] S530: The first node 410 cuts each matrix parameter according to the cutting strategy to obtain the cutting data for each matrix parameter. This step can be implemented by the cutting unit 413 in the embodiment of Figure 4.

[0174] In a specific implementation, the first node 410 can complete the segmentation of matrix parameters according to the tensor parallel strategy described above. For example, after the first matrix parameter and the second matrix parameter participating in the same matrix multiplication operation are segmented, multiple first segmentation data of the first matrix parameter and multiple second segmentation data of the second matrix parameter are obtained.

[0175] As can be seen from the foregoing, the splitting strategy includes various types of parallel strategies, such as tensor parallel strategy, data parallel strategy, and pipeline parallel strategy. If multiple parallel strategies are used in combination to train the model during the training process, the first node 410 can split the model parameters and training data according to multiple parallel strategies to obtain the split data of the model parameters and training data, and then distribute it to each node. This will not be elaborated here.

[0176] S540: The first node 410 sends cutting data to other nodes. This step can be implemented by the cutting unit 413 in the embodiment of Figure 4.

[0177] As mentioned above, the number of data segments for each matrix parameter is the same as the number of nodes, so each node receives one data segment for one matrix parameter. Since multiple matrix parameters participate in matrix multiplication, each node can receive multiple data segments for multiple matrix parameters. For example, if the first matrix parameter is the activation value and the second matrix parameter is the weight, then each node will receive data segments for both the activation value and the weight. The above examples are for illustrative purposes only and are not intended to be specific.

[0178] S550A~S550B: The first node 410 and the second node 420 process the segmented data of each matrix parameter to obtain the communication data of each matrix parameter. This step can be implemented by the processing unit 414 of the first node 410 and the processing unit 424 of the second node 420 in the embodiment of Figure 4.

[0179] In simple terms, each node can process the allocated segmented data to generate communication data that needs to be transmitted to other nodes. Each segmented data point corresponds to one piece of communication data; for example, processing activation values ​​yields communication data for the activation values, and processing weights yields communication data for the weights. In specific implementations, the processing content for the segmented data can be determined based on the actual business scenario. Different models require different processing steps before ensemble communication during training, and even the same model requires different processing steps before ensemble communication at different training stages. Therefore, this application does not impose specific limitations. For example, the processing content may include forward computation on the segmented data, where the communication data refers to the result of the forward computation. Alternatively, the processing content may include backpropagation on the segmented data, where the communication data refers to the gradient of the weights. The above examples are for illustrative purposes only and are not intended to impose specific limitations.

[0180] S560: The first node 410 and the second node 420 complete the collective communication of each matrix parameter using multiple network planes according to the mapping relationship. This step can be implemented by the first communication unit 415 and the second communication unit 416 of the first node 410, and the first communication unit 425 and the second communication unit 426 of the second node 420 in the embodiment of Figure 4.

[0181] As mentioned above, the mapping relationship records the network planes corresponding to different communication data. At this time, according to the mapping relationship, different network planes can be used to communicate different communication data in parallel, so that the collection communication can be executed in parallel, and the collection communication of multiple communication data can be completed at the same time, thereby improving communication efficiency.

[0182] S570A~S570B: The first node 410 and the second node 420 perform matrix multiplication based on the set communication results, and complete model training based on the matrix multiplication results. This step can be implemented by the processing unit 414 of the first node 410 and the processing unit 424 of the second node 420 in the embodiment of Figure 4.

[0183] For example, Figure 8 is a flowchart illustrating the steps of a training method provided in this application for handling matrix multiplication tasks. Figure 8 uses the training system 200A shown in Figure 2 and the matrix parameters shown in Figure 6 as examples. The number of nodes in the communication group is m = 2, the number of communication groups is n = 3, the first communication plane is intra-group communication, and the second communication plane is inter-group communication. The number of column cuts of the matrix parameters is determined according to the number of communication groups, with n-1 = 2 column cuts and m-1 = 1 row cuts. The number of columns in the first matrix parameter is H, and the number of rows in the second matrix parameter is H. After the first and second matrix parameters are cut into 6 parts respectively, each node can be assigned one cut data of the first matrix parameter and one cut data of the second matrix parameter. In Figure 8, dark short bars represent the cut data of the first matrix parameter, and light long bars represent the cut data of the second matrix parameter. Node 11 is a node in the embodiment of Figure 2. After being assigned cut data, this node can process the cut data to generate corresponding communication data, and then use two different network plane sets to communicate the two cut data.

[0184] As shown in Figure 8, the segmented data of the second matrix parameter, i.e., the light-colored strips, is transmitted through the first network plane. The first network plane refers to intra-group communication. Therefore, nodes 12 and 11 in the same communication group exchange data. After the group communication, node 11 will obtain the communication data of all the second matrix parameters in the communication group, i.e., the two light-colored strips. Then, by vertical splicing, the fourth matrix parameter is obtained. Since the second matrix parameter is segmented into two parts by row segmentation, the height or number of rows H of the fourth matrix parameter obtained after vertical splicing of the two light-colored strips is the same as that of the second matrix parameter.

[0185] Similarly, the segmented data of the first matrix parameter, i.e., the dark short bars, is transmitted through the second network plane. The second network plane refers to inter-group communication. Therefore, nodes 21, 31 and 11 in different communication groups exchange data. After the aggregation communication, node 11 will obtain the communication data of all the first matrix parameters of different communication groups, i.e., 3 dark short bars. Then, by horizontal splicing, the third matrix parameter is obtained. Since the first matrix parameter is cut into 3 parts by column cutting during the segmentation, the length or column number H of the third matrix parameter obtained after horizontal splicing of the 3 dark short bars is the same as that of the first matrix parameter.

[0186] In this way, the number of columns H of the third matrix parameter obtained after set communication is the same as the number of rows H of the fourth matrix parameter. The inner axes of the two multipliers are the same, so matrix multiplication can be performed to obtain the matrix multiplication result. Based on the matrix multiplication result, the matrix multiplication result of the first matrix parameter and the second matrix parameter can be obtained.

[0187] It should be understood that Figure 8 is for illustrative purposes only. In fact, regardless of the cutting strategy, during aggregate communication, when determining its communication plane, the first communication data only needs to ensure that the aggregate communication results obtained through this network plane can be spliced ​​to obtain the width or number of columns H of the first matrix parameter. The second communication data only needs to ensure that it can be spliced ​​to obtain the height or number of rows H of the second matrix parameter through another network plane. This allows the matrix multiplication operation to proceed smoothly and avoids the failure of the multiplication task due to different inner axes.

[0188] It should be understood that during model training, the matrix multiplication result Y of the first matrix parameter and the second matrix parameter also needs to be distributed to each node for further calculation. Therefore, in the example shown in Figure 8, when using the training method provided in this application to perform matrix multiplication, each node can obtain one element of the matrix multiplication result Y, eliminating the distribution step. This can be completed through one allgather communication. If the traditional method is used for aggregate communication through a single network plane, at least two allgather communications (communicating the segmented data of the first matrix parameter and the segmented data of the second matrix parameter) and at least one reducescatter communication (distributing the matrix multiplication result) are required to complete one matrix multiplication task. The training method of this application can improve communication efficiency.

[0189] As mentioned above, the model training process may involve two consecutive matrix multiplication operations. Figure 8 illustrates a single matrix multiplication task. The following example, using Figure 9 as a case study, illustrates the steps and flow of a multi-matrix multiplication task.

[0190] Figure 9 is a flowchart illustrating the steps of a training method provided in this application for handling matrix multiplication tasks in an application scenario. As shown in Figure 9, the matrix multiplication task is assumed to be a matrix multiplication task between activation values ​​and first and second weights. Assuming the training system structure used in the application scenario shown in Figure 9 is the same as that shown in Figure 2, and the cutting method is the same as that shown in Figure 8, where the number of column cuts is determined based on the number of communication groups, and the number of row cuts is determined based on the number of nodes within the communication group, then the activation value, first weight, and second weight can all be cut into 6 parts and distributed to each node. Node 11 can be allocated 1 part of activation value data (represented by a gray bar), 1 part of first weight data (represented by a light-colored bar), and 1 part of second weight data (represented by a white bar).

[0191] After each node processes the assigned segmented data, it obtains the corresponding communication data and then begins the first aggregate communication. Since there are only two network planes, aggregate communication is performed simultaneously on the communication data of the activation values ​​and the first weight. The activation values ​​communicate between groups through the second network plane. With three communication groups, the communication data of the three activation values ​​is obtained. The width or number of columns of the third matrix obtained by horizontal concatenation is H. The communication data of the first weight communicates within the group through the first network plane. Since there are two nodes within the group, the communication data of the two first weights is obtained. The height or number of rows of the fourth matrix obtained by vertical concatenation is H, thus allowing matrix multiplication. At this time, the second weight is not aggregated during the first aggregate communication. A detailed description of the first aggregate communication can be found in the embodiment shown in Figure 8, and will not be repeated here.

[0192] After performing matrix multiplication of the third and fourth matrix parameters at each node to obtain the first result, each node includes the first result and the segmented data of the second weight. In Figure 9, the first result is represented by a black bar, and the segmented data of the second weight is represented by a white bar. Similar to the first set communication, in order to ensure that the inner axes of the two matrix parameters spliced ​​after set communication are the same, the first result is communicated between groups through the second network plane. The three communication groups obtain communication data of three activation values. The width or number of columns of the fifth matrix parameter obtained by horizontal splicing is H'. The communication data of the second weight is communicated within the group through the first network plane. There are two nodes within the group, so two communication data of the second weight are obtained. The height or number of rows of the sixth matrix parameter obtained by vertical splicing is H'. The inner axes of the two are the same, so matrix multiplication can be performed.

[0193] After performing matrix multiplication of the fifth and sixth matrix parameters at each node, each node can obtain the second operation result. After concatenating each second operation result, a complete result of matrix multiplication of the activation value, the first weight, and the second weight can be obtained. However, each node only needs to obtain a part of the complete result. Therefore, compared with the training system shown in Figure 1, Figure 1 requires four aggregate communications. The first three are allgather communications, and the last one requires reducescatter communication because the matrix multiplication result needs to be distributed to each node. The training method provided in this application, as shown in Figure 9, only requires two aggregate communications, both of which are allgather communications, which can reduce the number of communications and improve communication efficiency.

[0194] It should be noted that the example shown in Figure 9 is for illustrative purposes. In actual processing, the first and second matrix multiplications may include more steps, depending on the specific model training requirements. For example, taking the Transformer model, during the forward computation, the first matrix multiplication may include batch matrix multiplication and the application of an activation function, such as the GELU activation function, while the second matrix multiplication may also include batch matrix multiplication. In other model architectures, each matrix multiplication may have more steps. However, when ensemble communication is required, following the ideas provided in this application, the network plane is selected based on a segmentation strategy, ensuring that the inner axes of the two matrices concatenated after ensemble communication are the same. This not only improves communication efficiency during model training but also ensures that matrix multiplication operations are not affected.

[0195] It should be noted that Figure 9 uses two matrix multiplication operations as an example. In other model training, the number of matrix multiplications can be greater, and multiple different weights may be multiplied simultaneously. As described above, the number of row and column cuts can be determined, and the communication planes for different weights can be determined according to the cutting method. This makes full use of multiple communication planes to communicate in parallel and improves the efficiency of model training.

[0196] Furthermore, Figures 8 and 9 illustrate this using two communication planes as an example, which is a two-layer interconnection network. There is one network plane between communication groups and one network plane within a communication group. If there are more than two network planes, such as three-layer network planes, multiple communication groups can also be considered as a large group. There is a third network plane between the large groups. In this case, according to the model architecture, three network planes can be used for aggregated communication, or any two network planes can be selected from the three network planes for aggregated communication. The specific choice can be determined according to the actual business scenario, and this application does not impose any specific limitations.

[0197] As mentioned above, when segmenting matrix parameters, the number of column segments can be determined based on either the number of nodes within the communication group or the number of communication groups. In other words, there are at least two segmentation methods. In fact, the communication latency of these two segmentation methods differs in different application scenarios. Therefore, the segmentation method can be flexibly selected according to different application scenarios. The following uses the training system 200A shown in Figure 2 as an example, taking the matrix multiplication operation between the activation value and the first and second weights as an example, to determine the communication latency of the two different segmentation methods.

[0198] In the training system 200A shown in Figure 2, a dual-die co-packaged chip serves as a communication group. Therefore, the number of nodes (m) within a communication group is 2, and the number of communication groups is n. Consequently, the number of column cuts is 1, and the number of row cuts is n-1. The first cutting method involves m-1 column cuts and n-1 row cuts. The second cutting method involves n-1 column cuts and m-1 row cuts. The communication latency of the combined communication using the two cutting methods is shown in Figure 10. Figure 10 is a comparison chart of communication latency under different cutting strategies in a model training method provided in this application.

[0199] In this diagram, B refers to the batch size, S refers to the sequence length, H refers to the hidden dimension, and H' refers to the feedforward dimension. Furthermore, in Figure 9, all letters with subscripts 2 or sio refer to the computational cost after ensemble communication within the group (first network plane), and subscripts n or i refer to the computational cost after ensemble communication between groups (second network plane). For example, B2 refers to the batch size for ensemble communication in the first network plane, and Bn refers to the batch size for ensemble communication in the second network plane.

[0200] As shown in Figure 10, after calculating the communication latency corresponding to the two segmentation methods, the judgment conditions can be determined based on the relevant parameters in the communication latency. In actual business scenarios, the segmentation method with the lowest communication latency can be selected based on the judgment conditions it meets. For example, suppose net i =450Gbps, net sio =540Gbps, S=4096, B=2048, n=16, H=12288, H'=49152. Substituting these values, we can obtain the optimal cutting method as Scheme 2.

[0201] It should be understood that when users configure mapping relationships and splitting strategies, the system can display the communication latency of different schemes shown in Figure 10 to the user. The user can use this as a reference to select the splitting method most suitable for the current business scenario. Of course, the system can also generate default splitting strategies and mapping relationships based on the model's parameter information for the user to choose from. This application does not make any specific limitations.

[0202] It should be understood that the above example uses the training system shown in Figure 2. For the training system shown in Figure 3, the sio network plane is replaced by the switch network plane, and m=4. At this time, the communication delay of different schemes can also be inferred in the way shown in Figure 10. Similarly, other training systems can refer to the way shown in Figure 10 to infer the communication delay of the synchronization scheme for user reference. Examples will not be given here.

[0203] In summary, this application provides a model training system comprising multiple nodes. These nodes establish multiple network planes through different communication methods. Thus, during distributed training, when a node obtains the segmented data of matrix parameters and trains the model based on the segmented data, and obtains communication data that needs to be synchronized with other nodes, the communication data of different matrix parameters can be used for aggregated communication through different network planes. This allows the aggregated communication of multiple matrix parameters to be executed in parallel, reducing the number of aggregated communications, improving the efficiency of aggregated communication, and thereby improving the efficiency of model training.

[0204] The model training method and training system provided in this application have been described in detail above. The computing device provided in this application will be explained below with reference to Figure 11.

[0205] Figure 11 is a schematic diagram of a computing device provided in this application. The computing device 1100 can be the training system described above. Further, the computing device 1100 includes a processor 1101, a storage unit 1102, a storage medium 1103, and a communication interface 1104. The processor 1101, storage unit 1102, storage medium 1103, and communication interface 1104 communicate via a bus 1105, and also via wireless transmission or other means.

[0206] Processor 1101 comprises multiple general-purpose processors, such as a CPU. These general-purpose processors can be hardware chips, including application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or combinations thereof. The PLDs can be complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), generic array logic (GALs), data processing units (DPUs), systems-on-chips (SoCs), or any combination thereof. Processor 1101 executes various types of digital storage instructions, such as software or firmware programs stored in storage unit 1102, enabling computing device 1100 to provide a wide range of services.

[0207] In a specific implementation, as one embodiment, the processor 1101 includes one or more CPUs, such as CPU0 and CPU1 shown in FIG11.

[0208] In a specific implementation, as one embodiment, computing device 1100 also includes multiple processors, such as processors 1101 and 1106 shown in FIG. 11. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). Here, a processor refers to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0209] Storage unit 1102 is used to store program code, which is controlled by processor 1101 to execute the processing steps of the training system in any of the embodiments of Figures 1-11. The program code includes one or more software units. The aforementioned one or more software units are the configuration unit, processing unit, and communication unit in the embodiment of Figure 1. The configuration unit is used to obtain the mapping relationship between communication data and the network plane, specifically implementing steps S510-S520 and their optional steps in the embodiment of Figure 5. The processing unit obtains the segmentation data of the matrix parameters, trains the model based on the segmentation data, and obtains the communication data corresponding to the matrix parameters, specifically implementing steps S540-S550B and their optional steps in the embodiment of Figure 5. The communication unit is used to perform aggregated communication with other nodes based on the mapping relationship, obtaining the communication data of the matrix parameters on each node. The communication data can correspond to the first communication unit and the second communication unit in the embodiment of Figure 1, specifically implementing step S560 and its optional steps in the embodiment of Figure 5. The processing unit is also used to complete the matrix multiplication task based on the communication data of the matrix parameters on each node, completing the model training, specifically implementing steps S570A-S570B and their optional steps in the embodiment of Figure 5.

[0210] Storage unit 1102 includes read-only memory and random access memory, and provides instructions and data to processor 1101. Storage unit 1102 also includes non-volatile random access memory. Storage unit 1102 is volatile memory or non-volatile memory, or includes both. The non-volatile memory is read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory is random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are used, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It can also refer to hard disks, USB flash drives, flash memory, SD cards, Memory Sticks, etc., where hard disks include hard disk drives (HDDs), solid-state drives (SSDs), and mechanical hard disks (HDDs), etc., and this application does not specifically limit the types used.

[0211] Storage medium 1103 is a carrier for storing data, such as hard disk, USB flash drive, flash memory, SD card, memory stick, etc. The hard disk can be a hard disk drive (HDD), solid state disk (SSD), mechanical hard disk (HDD), etc. This application does not make specific limitations.

[0212] The communication interface 1104 is a wired interface (e.g., an Ethernet interface), an internal interface (e.g., a Peripheral Component Interconnect express (PCIe) bus interface), a wired interface (e.g., an Ethernet interface), or a wireless interface (e.g., a cellular network interface or a wireless LAN interface), used for communicating with other servers or units.

[0213] The 1105 bus is a Peripheral Component Interconnect Express (PCIe) bus, or an Extended Industry Standard Architecture (EISA) bus, Unified Bus (Ubus or UB), Compute Express Link (CXL), Cache Coherent Interconnect for Accelerators (CCIX), etc. The 1105 bus is divided into address bus, data bus, and control bus.

[0214] In addition to the data bus, bus 1105 also includes a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus 1105 in the diagram.

[0215] It should be noted that Figure 11 is merely one possible implementation of an embodiment of this application. In practical applications, the computing device 1100 may include more or fewer components, which is not limited here. For content not shown or described in the embodiments of this application, please refer to the relevant descriptions in the embodiments of Figures 1-10 above, which will not be repeated here.

[0216] This application embodiment also provides a computing device cluster, which can be the training system described above. The computing device cluster includes at least one computing device 1100. The storage units 1102 of one or more computing devices 1100 in the computing device cluster may store the same or different instructions for executing model training methods.

[0217] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to execute a model training method.

[0218] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., high-density digital video disc (DVD)), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute a model training method.

[0219] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. A computer program product includes a plurality of computer instructions. When the computer program instructions are loaded or executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0220] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A model training method, characterized in that, The method is applied to a training system, which includes multiple nodes. These nodes establish multiple network planes through different communication methods. The method includes: The first node obtains the mapping relationship between communication data and the network plane, and the first node is any node among the plurality of nodes; The first node acquires the segmentation data of the matrix parameters, where the matrix parameters are the multipliers in the matrix multiplication task during model training; The first node trains the model based on the segmented data to obtain the communication data corresponding to the matrix parameters. The communication data is the data to be transmitted from the first node to other nodes. The first node performs aggregated communication with the other nodes based on the mapping relationship to obtain the communication data of the matrix parameters on each node, wherein the communication data corresponding to different matrix parameters are aggregated through different network planes; The first node completes the matrix multiplication task based on the communication data between each node according to the matrix parameters, thereby completing the training of the model.

2. The method according to claim 1, characterized in that, The training system includes multiple communication groups, each communication group includes multiple nodes, and the multiple nodes in the communication group establish a first network plane through a first communication method, and the communication groups establish a second network plane through a second communication method.

3. The method according to claim 1 or 2, characterized in that, The first node obtains the segmentation data of the matrix parameters, including: The first node performs at least one row cut and at least one column cut on the matrix parameters to obtain multiple cut data of the matrix parameters, the number of cut data being the same as the number of nodes in the training system; The first node distributes the multiple cut data to other nodes, so that each node obtains one cut data of the matrix parameters.

4. The method according to claim 3, characterized in that, The number of row cuts is determined based on the number of communication groups, and the number of column cuts is determined based on the number of nodes within a communication group; or, the number of row cuts is determined based on the number of nodes within a communication group, and the number of column cuts is determined based on the number of communication groups.

5. The method according to any one of claims 2 to 4, characterized in that, The matrix parameters include a first matrix parameter and a second matrix parameter, which are the left multiplier and right multiplier in the same matrix multiplication task. The communication data includes first communication data corresponding to the first matrix parameter and second communication data corresponding to the second matrix parameter. If the number of column cuts in the matrix parameters is determined according to the number of communication groups, the first communication data corresponds to the second network plane, and the second communication data corresponds to the first network plane; If the number of column cuts in the matrix parameters is determined according to the number of communication groups, the first communication data corresponds to the first network plane, and the second communication data corresponds to the second network plane.

6. The method according to claim 5, characterized in that, The first node completes the matrix multiplication task based on the communication data between each node according to the matrix parameters, including: The first node concatenates the first communication data of the first matrix parameters on each node to obtain the third matrix parameters, and concatenates the second communication data of the second matrix parameters on each node to generate the fourth matrix parameters. The number of columns of the third matrix parameters is the same as the number of columns of the first matrix parameters, and the number of rows of the fourth matrix parameters is the same as the number of rows of the second matrix parameters. The first node completes the matrix multiplication task based on the matrix multiplication result of the third matrix parameter and the fourth matrix parameter.

7. The method according to claim 5 or 6, characterized in that, The first matrix parameters include activation values, and the second matrix parameters include weights; or, The first matrix parameter includes the matrix multiplication result of the activation value and the first weight, and the second matrix parameter includes the second weight.

8. The method according to any one of claims 1 to 7, characterized in that, If the matrix multiplication task is a task of the model in the forward computation stage, the set communication includes global aggregation allgather communication; If the matrix multiplication task is the task of the model during the backpropagation stage, the set communication includes reduce-scatter communication.

9. A model training system, characterized in that, The system is applied to a training system, which includes multiple nodes. These nodes establish multiple network planes through different communication methods. The system includes: A configuration unit is used to obtain the mapping relationship between communication data and the network plane, wherein the first node is any node among the plurality of nodes; The processing unit is used to acquire the segmented data of the matrix parameters, wherein the matrix parameters are the multipliers in the matrix multiplication task during model training; The processing unit is used to train the model based on the segmented data to obtain the communication data corresponding to the matrix parameters. The communication data is the data to be transmitted from the first node to other nodes. A communication unit is used to perform aggregate communication with the other nodes based on the mapping relationship, and to obtain the communication data of the matrix parameters on each node, wherein the communication data corresponding to different matrix parameters are aggregated using different network planes; The processing unit is used to complete the matrix multiplication task and train the model based on the communication data at each node according to the matrix parameters.

10. The system according to claim 9, characterized in that, The training system includes multiple communication groups, each communication group includes multiple nodes, and the multiple nodes in the communication group establish a first network plane through a first communication method, and the communication groups establish a second network plane through a second communication method.

11. The system according to claim 9 or 10, characterized in that, The system includes a cutting unit; The cutting unit is used to perform at least one row cutting and at least one column cutting on the matrix parameters to obtain multiple cutting data of the matrix parameters, wherein the number of cutting data is the same as the number of nodes in the training system; The cutting unit is used to distribute the multiple cutting data of the first node to other nodes, so that each node obtains one cutting data of the matrix parameters.

12. The system according to claim 11, characterized in that, The number of row cuts is determined based on the number of communication groups, and the number of column cuts is determined based on the number of nodes within a communication group; or, the number of row cuts is determined based on the number of nodes within a communication group, and the number of column cuts is determined based on the number of communication groups.

13. The system according to any one of claims 9 to 12, characterized in that, The matrix parameters include a first matrix parameter and a second matrix parameter, which are the left multiplier and right multiplier in the same matrix multiplication task. The communication data includes first communication data corresponding to the first matrix parameter and second communication data corresponding to the second matrix parameter. If the number of column cuts in the matrix parameters is determined according to the number of communication groups, the first communication data corresponds to the second network plane, and the second communication data corresponds to the first network plane; If the number of column cuts in the matrix parameters is determined according to the number of communication groups, the first communication data corresponds to the first network plane, and the second communication data corresponds to the second network plane.

14. The system according to claim 13, characterized in that, The processing unit is configured to concatenate the first communication data of the first matrix parameters on each node to obtain the third matrix parameters, and concatenate the second communication data of the second matrix parameters on each node to generate the fourth matrix parameters, wherein the number of columns of the third matrix parameters is the same as the number of columns of the first matrix parameters, and the number of rows of the fourth matrix parameters is the same as the number of rows of the second matrix parameters. The processing unit is used to complete the matrix multiplication task based on the matrix multiplication result of the third matrix parameter and the fourth matrix parameter.

15. The system according to claim 13 or 14, characterized in that, The first matrix parameters include activation values, and the second matrix parameters include weights; or, The first matrix parameter includes the matrix multiplication result of the activation value and the first weight, and the second matrix parameter includes the second weight.

16. The system according to any one of claims 9 to 15, characterized in that, If the matrix multiplication task is a task of the model in the forward computation stage, the set communication includes global aggregation allgather communication; If the matrix multiplication task is the task of the model during the backpropagation stage, the set communication includes reduce-scatter communication.

17. A computing device, characterized in that, The computing device includes a processor and a memory, the memory being used to store instructions and the processor being used to execute the instructions such that the computing device implements the method as claimed in any one of claims 1 to 8.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed by a computing device or a cluster of computing devices, implement the method as claimed in any one of claims 1 to 8.

Images