Model training method and system and related equipment

Abstract

The invention provides a model training method and system and related equipment, the system comprises a plurality of nodes, and a plurality of network planes are established among the plurality of nodes through different communication modes, so that when distributed training is carried out, one node obtains cutting data of matrix parameters, and a model is trained based on the cutting data; when the communication data needing to be synchronized to other nodes are obtained, the communication data of different matrix parameters can be subjected to set communication by using different network planes, so that the set communication of multiple matrix parameters can be executed in parallel, the number of times of set communication is reduced, the efficiency of set communication is improved, and the efficiency of model training is further improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence (AI), and in particular to a model training method, system, and related equipment. Background Art

[0002] With the continuous development of AI technology, the training of AI models has begun to rely on greater computing power and storage capabilities. A single node is gradually unable to meet the training needs of AI models. Therefore, ultra-large-scale AI models are usually completed using distributed training. Distributed training divides the model parameters and distributes them to multiple nodes for training. This not only reduces the computing pressure of a single node, but also allows multiple nodes to train in parallel, improving training efficiency.

[0003] During distributed training, each node is responsible for updating a portion of the model's parameters. However, the goal of training is to train a complete model, so nodes need to communicate with each other and synchronize their trained data with other nodes through collective communication. However, each collective communication requires the communication bandwidth between nodes, which is limited. If the amount of data to be synchronized is large, data synchronization can only be achieved through multiple collective communications. For example, before matrix multiplication of activation values ​​and weights, the first collective communication is required to synchronize the model weights, and the second collective communication is required to synchronize the activation values. Therefore, during distributed training, the number of collective communications is high, causing communication time to occupy the majority of the model training time, affecting the efficiency of model training. Summary of the Invention

[0004] The present application provides a model training method, system and related equipment for solving the problem that the number of collective communications in distributed training is large, resulting in communication time occupying the main time of model training and affecting the efficiency of model training.

[0005] In a first aspect, a model training method is provided, which is applied to a training system. The training system includes multiple nodes, and multiple network planes are established between the multiple nodes through different communication methods. The method includes the following steps: a first node obtains a mapping relationship between communication data and a network plane, the first node is any node among the multiple nodes, the first node obtains cutting data of matrix parameters, the matrix parameters are multipliers in the matrix multiplication task in model training, the first node trains the model based on the cutting data, obtains communication data corresponding to the matrix parameters, the communication data is data to be transmitted from the first node to other nodes, the first node performs collective communication with other nodes based on the mapping relationship, and obtains communication data of the matrix parameters on each node, wherein the communication data corresponding to different matrix parameters use different network planes for collective communication, and the first node completes the matrix multiplication task according to the communication data of the matrix parameters on each node to complete the training of the model.

[0006] By implementing the method described in the first aspect, multiple network planes are established between multiple nodes in the training system through different communication methods. In this way, when performing distributed training, a node obtains the cutting data of the matrix parameters and trains the model based on the cutting data. When obtaining the communication data that needs to be synchronized to other nodes, the communication data of different matrix parameters can be collectively communicated using different network planes, so that the collective communication of multiple matrix parameters can be executed in parallel, reducing the number of collective communications, improving the efficiency of collective communication, and thereby improving the efficiency of model training.

[0007] In one possible implementation, the training system includes multiple communication groups, each communication group includes multiple nodes, a first network plane is established between the multiple nodes in the communication group through a first communication method, and a second network plane is established between the communication groups through a second communication method.

[0008] For example, each communication group can include two nodes. Each communication group can be a chip with a dual-die package architecture, which allows two identical or different dies to be contained in a single physical package. In other words, one die is a node, and a dual-die package architecture constitutes a communication group. Nodes can also be computing units, such as graphics processing units (GPUs), central processing units (CPUs), tensor processing units (TPUs), data processing units (DPUs), or dedicated AI processing chips.

[0009] Furthermore, when each node in the training system is a die and a communication group is a dual-die packaged chip, the first communication connection can be the communication connection used between the dual dies, such as serial input / output (SIO), and the two nodes within the communication group establish a first network plane through SIO. Communication connections can be established between multiple groups of dual dies through high-speed full mesh interconnection bandwidth, and a second network plane is established between communication groups through high-speed full mesh interconnection bandwidth connections. The high-speed full mesh interconnection bandwidth connection can be Huawei Cloud Communication Service (HCCS), which is a high-speed interconnection technology used in cloud computing scenarios. It provides high-speed, low-latency intra-cloud communication services. In a full mesh topology, HCCS can achieve direct connection between nodes while providing high bandwidth and reliability. The high-speed fullmesh interconnection bandwidth can also be NVLink technology, which is a high-speed interconnection technology for direct communication between GPUs. It is suitable for high-performance computing and machine learning. In a full mesh topology, NVLink can directly connect multiple GPUs to achieve efficient transmission. It should be understood that the above examples are for illustration only. This application does not limit the specific technology used for high-speed full mesh interconnection bandwidth connection.

[0010] For another example, each communication group includes 4 nodes. The nodes in the communication group are interconnected through a first communication connection to form a first network plane. The nodes here can be the computing units mentioned above, or computing devices (such as bare metal servers (BMS), virtual machines or containers). The first communication connection can be a high-speed fullmesh interconnection bandwidth connection. For details, please refer to the relevant description of the above content, which will not be repeated here. The communication groups are interconnected through a second communication connection to form a second network plane. Among them, the second communication connection is implemented through a switch chip.

[0011] It should be understood that the above examples are for illustration purposes only, and this application does not limit the number of communication groups included in the training system, nor does it limit the number of nodes in each communication group, nor does it limit the specific connection method of the first communication connection and the second communication connection. It should be noted that, in a specific implementation, the training system may also include more than two network planes. For example, three communication groups can be used as a large group, and a third communication connection can be established between multiple large groups to form a third network plane. Of course, the connection method of the third communication connection is different from that of the first communication connection and the second communication connection. In this way, when data is synchronized, parallel transmission can be achieved based on three network planes. The number of network planes can be determined according to the actual business scenario, and this application does not make any specific restrictions.

[0012] The above implementation method establishes different network planes. When collective communication is required during training, part of the data can be synchronized through the first network plane and part of the data can be synchronized through the second network plane, thereby reducing the number of data communications and reducing training time.

[0013] In one possible implementation, the first node obtains the cutting data of the matrix parameters, including: the first node performs at least one row cutting and at least one column cutting on the matrix parameters to obtain multiple cutting data of the matrix parameters, and the number of cutting data is the same as the number of nodes in the training system; the first node distributes the multiple cutting data to other nodes, so that each node obtains one cutting data of the matrix parameters.

[0014] In a specific implementation, matrix parameters refer to the two multipliers involved in a matrix multiplication operation. Both multipliers are matrices, such as activations and weights. Activations refer to the values ​​calculated on neurons, including input data fed into the network, as well as intermediate data during model training, such as the output data of the previous network layer and the results of the previous matrix multiplication operation. Weights refer to the weights of the model. During model training, the weights are continuously adjusted to enable the model to learn the features and patterns in the data, continuously improving model performance. Matrix parameters can also be biases, gradients, and so on. It should be understood that the model training process includes multiple matrix multiplication tasks. The two matrix parameters in each matrix multiplication task can be segmented using the above method to obtain segmented data for each matrix parameter, which is then distributed to each node. This allows a matrix multiplication task to be segmented into multiple subtasks for parallel processing. The computational results of each subtask are then aggregated through collective communication to obtain the computational result of the matrix multiplication task.

[0015] For example, if the matrix parameter is a 2×3 matrix, after one row split, it becomes two 1×3 matrices, and after three column splits, it becomes six 1×1 matrices. For another example, if the matrix parameter is a 4×4 matrix, after one row split, it becomes two 2×4 matrices, and after one column split, it becomes four 2×2 matrices. In the above example, an a×b matrix refers to a matrix with a rows and b columns. The above example is for illustrative purposes only and is not specifically limited in this application.

[0016] In a specific implementation, the first matrix parameter and the second matrix parameter participating in the same matrix multiplication operation are cut in the same manner, that is, the number of row cuts of the first matrix parameter is the same as the number of row cuts of the second matrix parameter, and the number of column cuts of the first matrix parameter is the same as the number of column cuts of the second matrix parameter. Simply put, if the first matrix parameter is the activation value and the second matrix parameter is the weight, if the activation value has 2 row cuts and 3 column cuts, the activation value is cut into 6 parts, then the weight also needs to be cut into 6 parts, but the weight can only be cut according to the number of column cuts of 3 and the number of row cuts of 2; if the weight is cut according to the number of row cuts of 2 and the number of column cuts of 3, although the weight can also be cut into 6 parts, due to the change in the cutting method, the number of rows of the activation value cutting data is different from the number of columns of the weight cutting data, so when the cutting data is distributed to each node for matrix multiplication, there may be a problem that the two cutting parameters involved in the multiplication cannot be multiplied due to their different internal axes. For example, each row of the cutting data x1 is 2 elements, and each column of the cutting data y1 is 3 elements. The two have different internal axes and cannot be multiplied, so the two matrix parameters need to be cut according to the same cutting method to avoid the problem that the cutting data cannot be multiplied.

[0017] The above implementation method can cut the matrix parameters into smaller blocks by at least one column cut and at least one row cut, and distribute the matrix parameters more evenly to each node, which helps to avoid the situation where some nodes are overloaded and is more suitable for communication between multiple network planes.

[0018] In one possible implementation, the number of row cuts is determined based on the number of communication groups, and the number of column cuts is determined based on the number of nodes in the communication groups; or, the number of row cuts is determined based on the number of nodes in the communication groups, and the number of column cuts is determined based on the number of communication groups.

[0019] In the specific implementation, if the number of nodes in each communication group is m and the number of communication groups is n, then the number of row cuts is m-1, that is, the matrix parameters are cut into m parts by row, and the number of column cuts is n-1, that is, the matrix parameters are further cut into m×n parts by column. Alternatively, the number of row cuts is n-1, and the number of column cuts is m-1. Simply put, if the node scale is 2×3, that is, there are 3 communication groups, each of which includes 2 nodes, then the number of row cuts is 1, and the number of column cuts is 2, that is, the matrix parameters are also cut according to the scale of 2×3. It should be noted that the other matrix parameters involved in the matrix multiplication also need to be cut according to the same cutting method, for example, the activation values ​​and weights are both cut according to the 2×3 cutting method.

[0020] The above implementation method determines the number of row cuts and column cuts based on the number of communication groups and the number of nodes in the communication groups, so that when subsequent collective communications are performed along different network planes, the aggregated matrix obtained by collective communications on each network plane can obtain complete rows or complete columns (the same number of columns as the original first matrix parameters, or the same number of rows as the original second matrix parameters). This ensures that each node uses the same internal axis of the aggregated matrix, can perform matrix multiplication operations, and ensures the correctness of the matrix multiplication task.

[0021] In one possible implementation, the matrix parameters include a first matrix parameter and a second matrix parameter, the first matrix parameter and the second matrix parameter are the left multiplier and the right multiplier in the same matrix multiplication task, and the communication data includes the first communication data corresponding to the first matrix parameter and the second communication data corresponding to the second matrix parameter; if the number of column cuts of the matrix parameters is determined according to the number of communication groups, the first communication data corresponds to the second network plane, and the second communication data corresponds to the first network plane; if the number of column cuts of the matrix parameters is determined according to the number of communication groups, the first communication data corresponds to the first network plane, and the second communication data corresponds to the second network plane.

[0022] It should be noted that due to the aforementioned association between mappings and cutting strategies, if the user configures a mapping, the cutting strategy must flexibly adjust the number of column and row cuts based on the mapping. Similarly, if the user configures a cutting strategy, the mapping must be flexibly adjusted based on the cutting strategy. If both the mapping and cutting strategies are user-configured, the user must be informed of the association between the two, and any issues with the user configuration must be promptly notified. If neither the mapping nor the cutting strategy are configured by the user, the training system can complete training using the default configuration.

[0023] The above implementation method determines the communication plane based on the cutting strategy, which can ensure the correctness of the matrix multiplication task. It should be understood that during the model training process, after each collective communication, each node will obtain the communication data of other nodes, and then merge the communication data to obtain the merged result corresponding to each matrix parameter, and then perform matrix multiplication between the merged results. For example, multiple first communication data are merged into a third matrix parameter, and multiple second communication data are merged into a fourth matrix parameter, and then the third matrix parameter and the fourth matrix parameter are multiplied. The matrix multiplication operation needs to ensure that the number of columns of the first multiplier is the same as the number of rows of the second multiplier, that is, the inner axes of the two multipliers are the same, otherwise the matrix multiplication operation cannot be performed. Therefore, the network plane used for collective communication needs to refer to the matrix parameter cutting method to ensure that the number of columns of the merged third matrix parameter is the same as the number of rows of the fourth matrix parameter.

[0024] In one possible implementation, the first node completes the matrix multiplication task according to the communication data of the matrix parameters on each node, including: the first node splices the first communication data of the first matrix parameters on each node to obtain a third matrix parameter, and splices the second communication data of the second matrix parameters on each node to generate a fourth matrix parameter, wherein the number of columns of the third matrix parameter is the same as the number of columns of the first matrix parameter, and the number of rows of the fourth matrix parameter is the same as the number of rows of the second matrix parameter; the first node completes the matrix multiplication task according to the matrix multiplication result of the third matrix parameter and the fourth matrix parameter.

[0025] Simply put, regardless of the splitting strategy, when performing aggregate communication, the first communication data only needs to ensure that the aggregate communication results obtained through this network plane can be spliced ​​to the width or number of columns H of the first matrix parameter when determining its communication plane. The second communication data only needs to ensure that the height or number of rows H of the second matrix parameter can be spliced ​​through another network plane. This ensures that matrix multiplication operations can proceed smoothly and avoids multiplication task failures due to different internal axes. Therefore, when splicing, it is necessary to determine whether to splice the multiple communication data obtained by aggregation horizontally or vertically based on the splitting method and communication method.

[0026] The above implementation method, based on the cutting strategy and the communication plane, determines how the aggregated communication data is aggregated, which can further ensure the correctness of the matrix multiplication task. Since the first matrix parameter and the second matrix parameter are the two parameters that need to be matrix multiplied, the number of columns of the first matrix parameter is the same as the number of rows of the second matrix parameter, so by ensuring that the number of columns of the third matrix parameter is the same as the number of columns of the first matrix parameter, and the number of rows of the fourth matrix parameter is the same as the number of rows of the second matrix parameter, it can be ensured that the number of columns of the third matrix parameter is the same as the number of rows of the fourth matrix parameter. With this idea in mind, if the number of column cuts of the first matrix parameter is determined according to the number of communication groups, and the second communication method is inter-group communication between communication groups, then when the first communication result of the first matrix parameter performs collective communication, it is necessary to use the second communication plane for collective communication so that the number of columns of the merged third matrix parameter is the same as the number of columns of the first matrix parameter. Similarly, if the number of column cuts of the first matrix parameter is determined according to the number of nodes in the communication group, it is necessary to use the first communication plane for collective communication so that the number of columns of the merged third matrix parameter is the same as the number of columns of the first matrix parameter.

[0027] In one possible implementation, the first matrix parameter includes the activation value, and the second matrix parameter includes the weight; or, the first matrix parameter includes the matrix multiplication result of the activation value and the first weight, and the second matrix parameter includes the second weight.

[0028] It should be noted that in the technical solution provided by this application, if there are matrix multiplication operations between other parameters in the distributed training process, the first matrix parameter and the second matrix parameter may also be other parameters. For example, the first matrix parameter may also be an attention parameter, and the second matrix parameter may be a weight or activation value. This application does not make any specific limitations.

[0029] It should be understood that the solution provided in this application is applicable to any matrix multiplication task in the distributed training process, including a single matrix multiplication task, such as the multiplication task between the activation value and the weight, and multiple consecutive matrix multiplication tasks. Each matrix multiplication can be cut and communicated in the manner provided above. In other words, the matrix parameters can also be temporary intermediate results generated during the calculation process or temporary input activation values. The solution has a wide range of applicability.

[0030] Taking two consecutive matrix multiplication tasks as an example, assuming that the activation value performs the first matrix multiplication with the first weight, and then performs the second matrix multiplication with the second weight, the third matrix parameter corresponding to the activation value and the fourth matrix parameter corresponding to the first weight can be obtained according to the above description. After obtaining the first operation result between the two, the first operation result is used as the first matrix parameter, and the second weight is used as the second matrix parameter. After cutting according to the above description, the mapping relationship is determined according to the cutting strategy, and then the collective communication is completed according to their respective corresponding communication planes at the same time, and the third matrix parameter corresponding to the first operation result and the fourth matrix parameter corresponding to the second weight are obtained. Then, according to the second operation result between the two, the two consecutive matrix multiplication tasks are completed.

[0031] Specifically, the second weight can be sliced ​​to obtain sliced ​​data, and the multiple slice results of the second weight can be distributed to each node. The description of this process can refer to the aforementioned description of the slicing and distribution of the first and second matrix parameters, and will not be repeated here. It should be noted that the slicing method of the second weight is the same as the slicing method of the first and second matrix parameters, and the number of row and column slicing is the same.

[0032] Optionally, the cut data of the second weight is processed to obtain third communication data, and then the first operation result is collectively communicated along the second network plane. The third matrix parameters corresponding to the first operation result are collectively communicated on the third communication data through the first network plane to obtain fourth matrix parameters corresponding to the second weight. Matrix multiplication is performed on the two to obtain the second operation result. Because the two network planes use different communication methods, collective communication of the two network planes can be carried out simultaneously, improving the efficiency of collective communication during distributed training.

[0033] It should be noted that if the matrix multiplication task is a task in the forward calculation phase of the model, collective communication includes global aggregation allgather communication; if the matrix multiplication task is a task in the backward propagation phase of the model, collective communication includes specification scatter reduce scatter communication. The cutting method and communication method for backward propagation will not be repeated here.

[0034] It should be understood that if the traditional method is used for distributed training, two consecutive matrix multiplication operations require four collective communications. The first collective communication aggregates the activation value, the second collective communication aggregates the first weight, the third collective communication aggregates the second weight, and the fourth collective communication specifies the result of the matrix multiplication. Through the training method provided in this application, two consecutive matrix multiplication operations only require two collective communications. During the first collective communication, the first communication data is collectively communicated through the second network plane, and the second communication data is collectively communicated through the first network plane; during the second collective communication, the first operation result is collectively communicated through the second network plane, and the third communication data is collectively communicated through the first network plane. Compared with the traditional method, the technical solution provided by this application only requires two collective communications to complete two matrix multiplications, thereby improving communication efficiency and model training efficiency.

[0035] In a second aspect, a model training system is provided, which is applied to a training system. The training system includes multiple nodes, and multiple network planes are established between the multiple nodes through different communication methods. The system includes: a configuration unit, which is used to obtain a mapping relationship between communication data and a network plane, and the first node is any node among the multiple nodes; a processing unit, which is used to obtain cutting data of matrix parameters, and the matrix parameters are multipliers in the matrix multiplication task in model training; the processing unit, which is used to train the model based on the cutting data, and obtain communication data corresponding to the matrix parameters, and the communication data is data to be transmitted from the first node to other nodes; a communication unit, which is used to perform collective communication with other nodes based on the mapping relationship, and obtain communication data of the matrix parameters on each node, wherein the communication data corresponding to different matrix parameters use different network planes for collective communication; the processing unit, which is used to complete the matrix multiplication task according to the communication data of the matrix parameters on each node, and complete the training of the model.

[0036] Implement the system described in the second aspect, and establish multiple network planes between multiple nodes in the training system through different communication methods. In this way, when performing distributed training, a node obtains the cutting data of the matrix parameters and trains the model based on the cutting data. When obtaining the communication data that needs to be synchronized to other nodes, the communication data of different matrix parameters can be collectively communicated using different network planes, so that the collective communication of multiple matrix parameters can be executed in parallel, reducing the number of collective communications, improving the efficiency of collective communication, and thereby improving the efficiency of model training.

[0037] In one possible implementation, the training system includes multiple communication groups, each communication group includes multiple nodes, a first network plane is established between the multiple nodes in the communication group through a first communication method, and a second network plane is established between the communication groups through a second communication method.

[0038] In one possible implementation, the system includes a cutting unit; the cutting unit is used to perform at least one row cutting and at least one column cutting on the matrix parameters to obtain multiple cutting data of the matrix parameters, and the number of cutting data is the same as the number of nodes of the training system; the cutting unit is used to distribute the multiple cutting data of the first node to other nodes, so that each node obtains one cutting data of the matrix parameters.

[0039] In one possible implementation, the number of row cuts is determined based on the number of communication groups, and the number of column cuts is determined based on the number of nodes in the communication groups; or, the number of row cuts is determined based on the number of nodes in the communication groups, and the number of column cuts is determined based on the number of communication groups.

[0040] In one possible implementation, the matrix parameters include a first matrix parameter and a second matrix parameter, the first matrix parameter and the second matrix parameter are the left multiplier and the right multiplier in the same matrix multiplication task, and the communication data includes the first communication data corresponding to the first matrix parameter and the second communication data corresponding to the second matrix parameter; if the number of column cuts of the matrix parameters is determined according to the number of communication groups, the first communication data corresponds to the second network plane, and the second communication data corresponds to the first network plane; if the number of column cuts of the matrix parameters is determined according to the number of communication groups, the first communication data corresponds to the first network plane, and the second communication data corresponds to the second network plane.

[0041] In one possible implementation, the processing unit is used to splice the first communication data of the first matrix parameters on each node to obtain third matrix parameters, and to splice the second communication data of the second matrix parameters on each node to generate fourth matrix parameters, wherein the number of columns of the third matrix parameters is the same as the number of columns of the first matrix parameters, and the number of rows of the fourth matrix parameters is the same as the number of rows of the second matrix parameters; and the processing unit is used to complete the matrix multiplication task according to the matrix multiplication result of the third matrix parameters and the fourth matrix parameters.

[0042] In one possible implementation, the first matrix parameter includes the activation value, and the second matrix parameter includes the weight; or, the first matrix parameter includes the matrix multiplication result of the activation value and the first weight, and the second matrix parameter includes the second weight.

[0043] In one possible implementation, if the matrix multiplication task is a task of the model in the forward calculation phase, the collective communication includes global aggregation allgather communication; if the matrix multiplication task is a task of the model in the backward propagation phase, the collective communication includes reduction scatter communication.

[0044] In a third aspect, a computing device is provided, which includes a processor and a memory, wherein the memory is used to store instructions and the processor is used to execute instructions, so that the computing device implements the method described in the first aspect.

[0045] In a fourth aspect, a computer-readable storage medium is provided, in which instructions are stored. When the instructions are executed by a computing device or a computing device cluster, the method described in the first aspect is implemented.

[0046] In a fifth aspect, a computing device cluster is provided, which includes at least one computing device, each of the at least one computing device includes a processor and a memory, and the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster implements the method described in the first aspect.

[0047] In a sixth aspect, a computer program product comprising instructions is provided, wherein the computer program product includes instructions that can be run on a computing device or stored in software or program products in any available medium, and when the computer program product is run on a computing device or a computing device cluster, the computing device or computing device cluster executes the method described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a structural diagram of a training system that implements distributed training through tensor parallelism;

[0049] Figure 2 This is an architectural diagram of a training system provided by this application;

[0050] Figure 3 This is an architectural diagram of another training system provided by this application;

[0051] Figure 4 This is a schematic diagram of the structure of each node in a training system provided by this application;

[0052] Figure 5 This is a flowchart of the steps of a model training method provided by this application;

[0053] Figure 6 This is an example diagram of matrix cutting provided by this application;

[0054] Figure 7 This is an example diagram of a configuration interface provided by this application;

[0055] Figure 8 This is an example diagram of the steps of a training method provided in this application for processing matrix multiplication tasks;

[0056] Figure 9This is an example diagram of the steps of a training method provided by this application to process a matrix multiplication task in an application scenario;

[0057] Figure 10 This is a comparison chart of communication delays under different cutting strategies in a model training method provided by this application;

[0058] Figure 11 This is a structural diagram of a computing device provided by this application. DETAILED DESCRIPTION

[0059] The application scenarios involved in this application are explained below.

[0060] With the continuous development of AI technology, the training of AI models has begun to rely on greater computing power and storage capabilities. A single node is gradually unable to meet the training needs of AI models. Therefore, ultra-large-scale AI models are usually completed using distributed training. Distributed training refers to assigning AI model training tasks to multiple nodes for collaborative completion. Common algorithms include data parallelism, tensor parallelism, and pipeline parallelism. Data parallelism refers to splitting the dataset and distributing it to multiple nodes for training. Tensor parallelism refers to distributing model parameters to multiple nodes for training. Pipeline parallelism refers to splitting the model according to network layers and distributing it to multiple nodes for training. Distributed training also includes many other algorithms, which are not listed here.

[0061] For example, Figure 1 This is a structural diagram of a training system that implements distributed training through tensor parallelism. The training system 100 may include multiple nodes ( Figure 1 Taking nodes 1 to 4 as an example, each node can be a device with data processing capabilities, such as a graphics processing unit (GPU). There is a communication network between the nodes, which enables data communication between the nodes. During the model training process, the model parameters will be divided into 4 parts and distributed to Figure 1 Training is performed on 4 nodes in .

[0062] Although each node is responsible for updating part of the model's parameters, the goal of training is to train a complete model, so collective communication is required between nodes to synchronize their trained data to other nodes. Among them, collective communication is a common communication method for distributed systems. It is a global communication operation in which all nodes in the system participate. For example, AllGather communication refers to aggregating the data of each node on each node so that each node includes the data of all nodes. For example, ReduceScatter communication refers to all nodes performing the same reduction operation and then distributing the operation results to each node. It should be understood that collective communication can also include more types, which are not illustrated here one by one.

[0063] Generally, during distributed training, the activation values ​​and weights obtained from each node training need to be synchronized to other nodes so that each node can obtain the weights and activation values ​​of the complete model. Then, forward calculation is performed to obtain the loss value, and then back propagation is performed based on the loss value to complete a round of model training. Both forward calculation and back propagation require matrix multiplication operations. The following is an example of a specific application scenario. Figure 1 The step flow of the training system performing matrix multiplication operations is described.

[0064] For example, taking the transformer model as an example, in the forward operation of the model, the activation value and weight are split and distributed to multiple nodes. The activation value x and weights wa and wb need to be aggregated through collective communication, and then matrix multiplication is performed. The activation value x is first matrix multiplied with the weight wa, and then matrix multiplication is performed with the weight wb to obtain the result z. The result z is then split so that each node obtains a part of the result z for subsequent further training. In this step process, after the activation value and weight are split and distributed to multiple nodes, each node will process the activation value and weight before performing matrix multiplication, such as Figure 1As shown in the figure, suppose node 1 needs to synchronize the weight Wa1 and activation value X1 with other nodes, node 2 needs the weight Wa2 and activation value X2, node 3 needs the weight Wa3 and activation value X3, and node 4 needs the weight Wa4 and activation value X4. To synchronize the activation value x and the weight Wa, two allgather communications are required due to the limited network bandwidth between nodes 1 to 4. Specifically, allgather communication based on the weight Wa can be completed first to synchronize Wa1 to Wa4 with each node. Then, allgather communication based on the activation value X can be completed again to synchronize X1 to X4 with each node. In other words, two data communications are required to synchronize the activation value X and the weight Wa. Matrix multiplication of the activation value X and the weight Wa is then performed to obtain the result Y. Next, the allgather communication based on the weight Wb is completed first, and then the matrix multiplication result Z of the weight Wb and Y is obtained. Then, the matrix multiplication result Z is distributed to each node through reducescatter communication, so that each node obtains part of the final output and completes the forward calculation.

[0065] therefore, Figure 1 The example shown trains a transformer-based model. Each forward computation requires at least four collective communications. Backward propagation is a similar process, also requiring four collective communications. This multiple rounds of training severely impacts the efficiency of model training.

[0066] In summary, when training a model using the tensor parallel algorithm for distributed training, data communication between nodes is limited by bandwidth, requiring multiple communications for each data synchronization. This results in a large number of communication processes in the distributed training process, and even the communication time occupies the majority of the model training time, affecting the efficiency of model training.

[0067] In order to solve the problem that a large number of communication processes occur during distributed training, resulting in communication time occupying the main time of model training and affecting the efficiency of model training, the present application provides a model training system. The training system 200 includes multiple nodes, and multiple network planes are established between multiple nodes through different communication methods. In this way, when performing distributed training, a node obtains the cutting data of the matrix parameters and trains the model based on the cutting data. When obtaining the communication data that needs to be synchronized to other nodes, the communication data of different matrix parameters can be collectively communicated using different network planes, so that the collective communication of multiple matrix parameters can be executed in parallel, reducing the number of collective communications, improving the efficiency of collective communications, and thereby improving the efficiency of model training.

[0068] The following combination Figure 2-Figure 4 The possible architecture of the training system 200 provided in this application is explained.

[0069] Figure 2 This is an architecture diagram of a training system provided by this application, such as Figure 2 As shown, the training system 200A includes multiple communication groups 211, each communication group includes multiple nodes. Figure 2 An example is given of three communication groups, each of which includes two nodes. In a specific implementation, this application does not limit the number of nodes in each communication group 211 or the number of communication groups 211 in the training system 200A.

[0070] A node can be a computing unit or computing device with data processing capabilities. Among them, the computing unit can be a hardware accelerator implemented based on technologies such as application-specific integrated circuit (ASIC) and field-programmable gate array (FPGA). The hardware accelerator has good performance when performing machine learning tasks and can improve the speed and efficiency of neural network training. For example, the computing unit can be a GPU, a central processing unit (CPU), a tensor processing unit (TPU), a data processing unit (DPU) or a dedicated AI processing chip, etc., which is not specifically limited in this application. The computing device can be a bare metal server (BMS), a virtual machine or a container. BMS refers to a general physical server, such as an ARM server or an X86 server; a virtual machine refers to a complete computer system with complete hardware system functions simulated by software and running in a completely isolated environment. Any work that can be completed in a physical computer can be implemented in a virtual machine. When creating a virtual machine in a computing device, part of the hard disk and memory capacity of the physical machine needs to be used as the hard disk and memory capacity of the virtual machine. Each virtual machine has an independent basic input / output system (BIOS), hard disk, and operating system, and can be operated like a physical machine. A container is a portable software unit that can combine an application and all its dependencies into a single software package. This package is not restricted by the underlying host operating system, eliminating the need to build a complex environment and simplifying the process from application development to deployment.

[0071] The above nodes can be owned by the user or provided to the user by a cloud service provider. In the case where the node is provided to the user by a cloud service provider, the above node can also be referred to as an instance. It should be understood that the cloud service provider can provide users with GPU instances, TPU instances, virtual machine instances, or container instances specifically for machine learning. Users can purchase instance usage rights from the cloud service provider based on their usage needs. This application does not specifically limit this.

[0072] Optionally, multiple nodes establish multiple network planes through different communication methods, and multiple nodes that establish communication connections using the same communication method can form a network plane. Figure 2 Take two network planes as an example to illustrate. Figure 2 In the example shown, multiple nodes can be divided into multiple communication groups. A first communication connection exists between multiple nodes within each communication group, establishing a first network plane. Second communication connections exist between multiple communication groups, establishing a second network plane. The first and second communication connections are different communication channels. It should be understood that when training system 200A performs distributed training, if a large amount of data needs to be synchronized, some data can be synchronized via the first communication connection and some via the second communication connection, thereby reducing the number of data communications and shortening training time.

[0073] For example, Figure 1 The application scenario shown uses the training system 200A provided in this application, which can realize the allgather communication of the weight W based on the first communication plane. At the same time, the allgather communication of the activation value X is realized based on the second communication plane. The two data communications can be merged into one, and the data synchronization of each stage can be realized in parallel, thereby increasing the number of data communications and reducing the training time.

[0074] Optionally, each communication group 211 in the training system 200A can be a chip with a dual-die package architecture, which allows two identical or different dies to be included in one physical package at the same time. The die here can be the computing unit mentioned above, such as the bare chip of GPU, CPU, TPU, etc. Therefore, one die is a node, and a dual-die package architecture constitutes a communication group 211.

[0075] Furthermore, each node in the training system 200A is a die, and a communication group 211 is a dual-die packaged chip. The first communication connection can be a communication connection used between the dual dies, such as a serial input / output (SIO). The two nodes within the communication group 211 establish a first network plane through the SIO. Multiple groups of dual dies can establish communication connections via high-speed full mesh interconnection bandwidth, and the communication groups 211 establish a second network plane via high-speed full mesh interconnection bandwidth connections. Exemplarily, the high-speed full mesh interconnection bandwidth connection can be Huawei Cloud Communication Service (HCCS). HCCS is a high-speed interconnection technology used in cloud computing scenarios that provides high-speed, low-latency intra-cloud communication services. In a full mesh topology, HCCS can achieve direct connections between nodes while providing high bandwidth and reliability. The high-speed fullmesh interconnection bandwidth can also be NVLink technology, which is a high-speed interconnection technology for direct communication between GPUs. It is suitable for high-performance computing and machine learning. In a full mesh topology, NVLink can directly connect multiple GPUs to achieve efficient transmission. It should be understood that the above examples are for illustration only. This application does not limit the specific technology used for high-speed full mesh interconnection bandwidth connection.

[0076] It should be understood that Figure 2 In the example above, a communication group including two nodes is used for explanation. In a specific implementation, each communication group may include a larger number of nodes. Figure 3 This is an architecture diagram of another training system provided by this application. Figure 3 The training system 200B shown includes two communication groups 211, and each communication group 211 includes four nodes.

[0077] The four nodes in each communication group 211 are interconnected through a first communication connection to form a first network plane. The nodes here can be the computing units or computing devices mentioned above. The first communication connection can be a high-speed fullmesh interconnection bandwidth connection. For details, please refer to the relevant description of the above content and will not be repeated here. The communication groups 211 are interconnected through a second communication connection to form a second network plane. The second communication connection is implemented by a switch chip ( Figure 3 SW1~SW4 in the figure are switch chips).

[0078] It should be understood that Figure 2 and Figure 3For illustration, the present application does not limit the number of communication groups included in the training system, nor does it limit the number of nodes in each communication group, nor does it limit the specific connection method of the first communication connection and the second communication connection.

[0079] It should be noted that Figure 2 and Figure 3 In the examples shown, the training system includes two network planes. In a specific implementation, the training system may also include more than two network planes, such as Figure 2 In the example shown, three communication groups can be used as a large group, and a third communication connection can be established between multiple large groups to form a third network plane. Figure 3 In the example shown, two communication groups can be used as a large group, and a third communication connection can be established between multiple large groups to form a third network plane. Of course, the connection method of the third communication connection is different from that of the first and second communication connections. In this way, when synchronizing data, parallel transmission can be achieved based on three network planes. The specific number of network planes can be determined based on the actual business scenario and is not specifically limited in this application.

[0080] It should be noted that Figure 2 and Figure 3 The training system shown may be a super node in a distributed training system, which may include multiple Figure 2 and / or Figure 3 As shown in the super node, one super node can be used to train one model, and multiple super nodes can train the same model with different training data.

[0081] Combined with the above Figure 2 and Figure 3 The structure of the training system provided by this application is described, especially how multiple nodes in the training system form multiple network planes. Figure 4 , explaining the internal unit modules of each node in the training system.

[0082] Figure 4 This is a schematic diagram of the structure of each node in a training system provided by this application, such as Figure 4 As shown, the training system may include a first node 410 and a second node 420. The first node 410 and the second node 420 may be Figure 2 and Figure 3 Any node in the embodiment, such as Figure 4 As shown, the first node 410 and the second node 420 are nodes in the same communication group 211. Of course, in other embodiments, the first node 410 and the second node 420 may also be nodes in different communication groups 211, which is not specifically limited in this application.

[0083] In the embodiment of the present application, the unit modules in each node are the same. Figure 4 To better explain the technical solution of the present application, the node that interacts with the user is referred to as the first node 410, and the other nodes are referred to as the second node 420. In a specific implementation, the user can interact with any node in the training system and input configuration information.

[0084] The first node 410 includes a configuration unit 411, a configuration information database 412, a cutting unit 413, a processing unit 414, a first communication unit 415, and a second communication unit 416. The second node 420 includes a configuration unit 421, a configuration information database 422, a cutting unit 423, a processing unit 424, a first communication unit 425, and a second communication unit 426.

[0085] As can be seen from the above, a communication group may include two or more nodes. Figure 4 In the example shown, if the communication group 211A includes more nodes, such as a third node, the structure of the third node is the same as that of the first node 410 and the second node 420. Similarly, the training system may include two or more communication groups. Figure 4 In the example shown, the structures of other communication groups such as the communication group 211B are the same as that of the communication group 211A, and are not repeated here.

[0086] It should be understood that Figure 4 This is an exemplary division method. The first node 410 and the second node 420 may also include more or fewer units. For example, the first communication unit and the second communication unit may be combined into a communication unit. For example, the node may also include a synchronization unit for synchronizing data in the configuration information database. This application does not make specific limitations.

[0087] The following is an explanation based on the functions of each unit module in the first node 410 and the second node 420.

[0088] The configuration unit 411 in the first node 410 is used to obtain configuration information and update the configuration information database 412 according to the configuration information.

[0089] In one embodiment, the configuration information may be configured by a user through the client 300. The user here may be a developer who trains an AI model. The configuration information includes the above-mentioned mapping relationship, which includes the mapping relationship between communication data and the network plane. The network plane here refers to the network plane in the training system, such as Figure 2 In the training system 200A, two nodes in a communication group establish a first network plane through SIO, and a second network plane is established between multiple communication groups through high-speed fullmesh interconnection bandwidth, or Figure 3 In the example, the four nodes in the communication group establish a first network plane through high-speed fullmesh interconnection bandwidth, and a second network plane is established between multiple communication groups through switch chips. Of course, the training system can also include more network planes, which can be found in the following examples. Figure 2 and Figure 3 The relevant descriptions in the embodiments are not repeated here.

[0090] It should be noted that a network plane can correspond to one or more communication data, and communication data at different communication times can correspond to the same network plane, but communication data at the same communication time needs to correspond to different network planes for parallel communication. For example, during the training process, weights and activation values ​​are usually communicated collectively first, and each node calculates the gradient based on the weights and activation values ​​on all nodes, and then the gradient is communicated collectively. Therefore, the weights and gradients can be mapped to the first network plane, and the two can use the first network plane to achieve collective communication at different communication times, but the activation value can only be mapped to the second network plane. This is because the activation value and weight use different network planes to achieve collective communication at the same time. In this way, the communication data that needs to be communicated collectively at the same time is transmitted in parallel through different network planes, which can improve the efficiency of collective communication and thus reduce the training time.

[0091] In the specific implementation, communication data refers to the data that needs to be collectively communicated during the model training process, such as model weights, gradients, activation values, loss values, model state information, hyperparameters, attention parameters, etc., which are not specifically limited in this application. Among them, collective communication refers to the global communication operation in which all nodes in the training system participate. It should be understood that the goal of collective communication is to transmit messages throughout the training system, rather than just point-to-point communication between two nodes. Generally, collective communication may include global reduction (allreduce), global aggregation (allgather), reduction scatter (reducescatter), etc., wherein allreduce refers to reducing the data on each node to obtain a reduction result, and then propagating the reduction result to each node so that each node obtains the reduction result; allgather refers to aggregating the data on each node to obtain an aggregation result, and then propagating the aggregation result to each node so that each node obtains the aggregation result; reducescatter refers to cutting the data on a node into multiple parts, each part is sent to a different node for local reduction to obtain a local reduction result, and then propagating the local reduction result of each node to other nodes so that each node obtains the local reduction result of all nodes. It should be understood that collective communication may also include more content, which is not specifically limited in this application.

[0092] In one embodiment, the configuration information database 412 includes mapping relationships and cutting strategies. The cutting strategies and mapping relationships may be user-configured, system-defaulted, or partially user-configured and partially system-defaulted, which are not specifically limited in this application. The cutting strategies are used by the first node 410 to cut model data or training data into multiple parts for distributed training by multiple nodes.

[0093] In the specific implementation, the cutting strategy includes various types of parallel strategies, such as tensor parallel strategy, data parallel strategy, and pipeline parallel strategy. It should be understood that the model can use a mixture of multiple cutting strategies to achieve distributed training during the training process, so the user can configure multiple cutting strategies. The explanation of the above-mentioned multiple cutting strategies can be referred to the above content, which will not be repeated here.

[0094] It should be noted that when the user configures the cutting strategy, the training system can display multiple cutting strategies to the user through the client for the user to choose from, and save the cutting strategy selected by the user. During the training process, the model data or training data is cut using the cutting strategy selected by the user.

[0095] It should be understood that during AI model training, multiple matrix multiplication operations are performed. Under the data parallel strategy, each node has a complete copy of the model. The training data is split and distributed to each node, allowing each node to perform independent calculations using different data subsets. Each node will perform matrix multiplication independently, so the data parallel strategy does not need to consider whether the matrix multiplication after data splitting will have an impact. Similarly, under the pipeline parallel strategy, the model is divided into different parts according to the network layer, and each part is trained on different nodes. The matrix multiplication of each network layer can also be calculated independently, so pipeline parallelism does not need to consider whether the matrix multiplication after data splitting will have an impact. However, under the tensor parallel strategy, the model weights and activation values ​​are cut into multiple parts and then distributed to different nodes for calculation, so that a matrix multiplication operation (matrix multiplication operation of weights and activation values) is cut into multiple parts. Therefore, the calculation results of each node need to be merged and summarized through collective communication to obtain a matrix multiplication operation result. Therefore, tensor parallelism needs to additionally consider the impact of data cutting on matrix multiplication, and needs to carefully handle the cutting of matrix multiplication and the merging of collective communication to avoid matrix operation errors due to cutting and merging errors, which affect the model training efficiency.

[0096] Therefore, this application optimizes the tensor parallel strategy in the cutting strategy so that the cut data can be reasonably distributed to each node for calculation. After the calculation results of each node are collectively communicated using different network planes, the matrix multiplication results obtained will not be incorrect. The tensor parallel strategy provided by this application is explained in detail below.

[0097] Optionally, the tensor parallel strategy includes: performing at least one row cut and at least one column cut on the matrix parameters to obtain cut data of the matrix parameters, where the number of cut data is the same as the number of nodes. In a specific implementation, the number of row cuts is determined by the number of communication groups, and the number of column cuts is determined by the number of nodes in the communication group; or, the number of row cuts is determined by the number of nodes in the communication group, and the number of column cuts is determined by the number of communication groups. Row cutting refers to cutting the matrix horizontally into multiple parts along the direction of the matrix rows, and column cutting refers to cutting the matrix vertically into multiple parts along the direction of the matrix columns.

[0098] In a specific implementation, matrix parameters refer to the two multipliers involved in a matrix multiplication operation. Both multipliers are matrices, such as activations and weights. Activations refer to the values ​​calculated on neurons, including input data fed into the network, as well as intermediate data during model training, such as the output data of the previous network layer and the results of the previous matrix multiplication operation. Weights refer to the weights of the model. During model training, the weights are continuously adjusted to enable the model to learn the features and patterns in the data, continuously improving model performance. Matrix parameters can also be biases, gradients, and so on. It should be understood that the model training process includes multiple matrix multiplication tasks. The two matrix parameters in each matrix multiplication task can be segmented using the above method to obtain segmented data for each matrix parameter, which is then distributed to each node. This allows a matrix multiplication task to be segmented into multiple subtasks for parallel processing. The computational results of each subtask are then aggregated through collective communication to obtain the computational result of the matrix multiplication task. This application does not limit the specific content of the matrix parameters. In order to make this application better understood, the following will take the matrix multiplication between activation values ​​and weights as an example, and the matrix parameters are activation values ​​and weights as an example for illustration.

[0099] For example, if the matrix parameter is a 2×3 matrix, after one row split, it becomes two 1×3 matrices, and after three column splits, it becomes six 1×1 matrices. For another example, if the matrix parameter is a 4×4 matrix, after one row split, it becomes two 2×4 matrices, and after one column split, it becomes four 2×2 matrices. In the above example, an a×b matrix refers to a matrix with a rows and b columns. The above example is for illustrative purposes only and is not specifically limited in this application.

[0100] In a specific implementation, the first matrix parameter and the second matrix parameter participating in the same matrix multiplication operation are cut in the same manner, that is, the number of row cuts of the first matrix parameter is the same as the number of row cuts of the second matrix parameter, and the number of column cuts of the first matrix parameter is the same as the number of column cuts of the second matrix parameter. Simply put, if the first matrix parameter is the activation value and the second matrix parameter is the weight, if the number of row cuts of the activation value is 2 and the number of column cuts is 3, the activation value is cut into 6 parts, then the weight also needs to be cut into 6 parts, but the weight can only be cut according to the number of column cuts of 3 and the number of row cuts of 2; if the weight is cut according to the number of row cuts of 2 and the number of column cuts of 3, although the weight can also be cut into 6 parts, due to the change in the cutting method, the number of rows of the activation value cutting data is different from the number of columns of the weight cutting data, so when the cutting data is distributed to each node for matrix multiplication, there may be a problem that the two cutting data involved in the multiplication cannot be multiplied due to the different internal axes. For example, each row of the cutting data x1 is 2 elements, and each column of the cutting data y1 is 3 elements. The internal axes of the two are different and cannot be multiplied, so the two matrix parameters need to be cut according to the same cutting method to avoid the problem that the cutting data cannot be multiplied.

[0101] Furthermore, if the number of nodes in each communication group is m and the number of communication groups is n, then the number of row cuts is m-1, that is, the matrix parameters are cut into m parts by row, and the number of column cuts is n-1, that is, the matrix parameters are further cut into m×n parts by column. Alternatively, the number of row cuts is n-1, and the number of column cuts is m-1. Simply put, if the node scale is 2×3, that is, there are 3 communication groups, each of which includes 2 nodes, then the number of row cuts is 1, and the number of column cuts is 2, that is, the matrix parameters are also cut according to the scale of 2×3. It should be noted that the other matrix parameters involved in matrix multiplication also need to be cut according to the same cutting method, for example, the activation values ​​and weights are both cut according to the 2×3 cutting method.

[0102] Optionally, the configuration information input by the user may also include parameter information required for model training. Configuration information database 412 may also include the aforementioned parameter information, wherein the parameter information includes the structural parameters of the AI ​​model to be trained and the scale parameters of training system 200 (e.g., training system 200A or training system 200B). The aforementioned structural parameters, scale parameters, and training parameters are explained below.

[0103] Optionally, the structural parameters of the AI ​​model may include hidden layer dimension, feedforward layer dimension, multi-head attention number and size, etc., wherein the hidden layer dimension refers to the number or dimension of the hidden layer neurons of the AI ​​model, the feedforward layer dimension refers to the number or dimension of the feedforward layer neurons of the AI ​​model, and the multi-head attention number and size refers to the number of independent attention heads used by the AI ​​model and the dimension of the sub-controls in each attention, or the dimension of the weights learned in each attention, wherein the attention head is responsible for paying attention to a certain feature of the input data. Increasing the number of attention heads can increase the model's ability to capture different features in the input data, and larger attention heads allow the model to learn more complex features. It should be understood that the above examples are for illustration only and this application does not make specific limitations.

[0104] Optionally, the scale parameters of the training system 200 may include parameters related to the training environment and hardware resources, such as the number of nodes of the training system 200, the specification parameters of each node, the number of network planes, the number of nodes in the network planes, and the network bandwidth. The number of nodes in the training system includes the number of nodes participating in the training. The specification parameters of each node include the hardware specification parameters of each node, such as CPU parameters, GPU parameters, memory parameters, etc. The number of network planes refers to the number of network planes established by multiple nodes through different communication methods, wherein different network planes use different communication connections, such as Figure 2 and Figure 3 Training systems 200A and 200B are shown as establishing a first network plane and a second network plane using two different communication methods. The number of nodes and network bandwidth in a network plane refer to the number of nodes in each network plane and the bandwidth for communication between nodes. It should be understood that the above examples are for illustrative purposes only, and scale parameters may include more content, which is not specifically limited in this application.

[0105] Optionally, training parameters in the training process may include batch size, sequence length, optimizer, learning rate, activation function, loss function, etc. Batch size refers to the number of samples input each time the model is updated. The training data can be divided into multiple batches for training, with each batch containing a certain number of samples. Sequence length refers to the length of the time step when the model processes sequence data. The optimizer refers to how the model updates weights during training to minimize the loss function. The learning rate is used to control the update amplitude of the model parameters in each iteration. The activation function defines the output of each neuron in the model. The loss function is used to measure the performance of the model during training. The parallel strategy is a method of configuring model training to train in parallel on multiple nodes.

[0106] It should be understood that parameter information includes not only the scale parameters of the training system, the training parameters of the training process, and the structural parameters of the AI ​​model, but also more content, which is not illustrated here one by one. Configuration information includes not only the cutting strategy, mapping relationship, and parameter information, but also more content, which is not illustrated here one by one.

[0107] Optionally, the configuration information database 412 in the first node 410 can synchronize the mapping relationship to other nodes so that the configuration information database on each node can obtain the mapping relationship. For example, the configuration information database 422 of the second node 420 can obtain the mapping relationship synchronized by the first node 410, so that the second node 420 can determine the network plane to be used for different communication data according to the mapping relationship when communicating in a group. In a specific implementation, the first node 410 can complete the synchronization of the mapping relationship at a fixed time or a fixed frequency, which is not specifically limited in this application. In addition, other data in the configuration information database 412, such as the above-mentioned cutting strategy and parameter information, can also be determined whether to be synchronized to other nodes based on usage requirements, which is not specifically limited in this application.

[0108] Optionally, client 300 may be a client of an AI parallel training framework. A parallel training framework is a software tool designed to train AI models in parallel across multiple nodes. Commonly used parallel training frameworks include TensorFlow, PyTorch, etc., which are not specifically limited in this application. In an embodiment of this application, a user may generate configuration information by modifying a configuration file of the AI ​​parallel training framework.

[0109] Optionally, client 300 may be a client of an AI compiler. It should be understood that dedicated AI processing chips such as GPUs and TPUs will provide an AI compiler for user configuration, and users can generate configuration information through the AI ​​compiler client.

[0110] Optionally, the client 300 can be a console of a cloud data center, which can provide users with a variety of communication data that need to be collectively communicated during the training process, as well as multiple network planes existing in the training system. Users can select communication data and network planes from the configuration interface to establish a mapping relationship. It can also provide some training parameters, scale parameters and structural parameters for users to select, and then generate configuration information. This application does not specifically limit this.

[0111] It should be noted that the configuration unit 421 in the second node 420 has the same function as the configuration unit 411 in the first node 410, that is, the configuration unit 421 in the second node 420 can also receive the configuration information sent by the user through the client 300 and store the configuration information in the configuration information database 422, but Figure 4 The application scenario shown is that the first node 410 interacts with the client 300, so Figure 4 In the application scenario shown, the configuration unit 421 of the second node 420 does not participate in the work. However, in actual application scenarios, the user can also interact with the second node 420 through the client 300 to input configuration information, which is not specifically limited in this application.

[0112] The cutting unit 413 is used to cut the matrix parameters according to the cutting strategy to obtain a plurality of cutting data of the matrix parameters.

[0113] In a specific implementation, the cutting unit 413 can complete the cutting of matrix parameters according to the above-mentioned tensor parallel strategy. For example, after cutting the first matrix parameter and the second matrix parameter participating in the same matrix multiplication operation, multiple first cutting data of the first matrix parameter and multiple second cutting data of the second matrix parameter are obtained.

[0114] Optionally, the cutting unit 413 is further configured to distribute multiple cutting data for each matrix parameter to each node. Referring to the foregoing content, it can be seen that the number of cutting data for each matrix parameter is the same as the number of nodes, so each node obtains one cutting data for each matrix parameter. There are multiple matrix parameters involved in the matrix multiplication operation, so each node can obtain multiple cutting data for multiple matrix parameters. For example, if the first matrix parameter is an activation value and the second matrix parameter is a weight, then each node will obtain cutting data for the activation value and cutting data for the weight. The above examples are for illustration only and are not specifically limited in this application.

[0115] It should be noted that, if the second node 420 receives the user's configuration information, it can also cut the matrix parameters through the cutting unit 423 to obtain multiple cut data of the matrix parameters, and then distribute them to other nodes. Examples are not given here one by one.

[0116] It should be noted that, referring to the above content, the cutting strategy includes many different types of parallel strategies, such as tensor parallel strategy, data parallel strategy and pipeline parallel strategy. If the training process uses a combination of multiple parallel strategies to train the model, the cutting unit 413 can cut the model parameters and training data according to multiple parallel strategies, obtain the cut data of the model parameters and training data, and then distribute it to each node. It will not be elaborated here.

[0117] The processing unit 414 is used to process the cutting data to obtain communication data required for collective communication.

[0118] Similarly, the processing units 414 on other nodes also need to process the received segmentation data to obtain communication data required for collective communication, for example, first communication data is obtained after processing the first matrix parameters, and second communication data is obtained after processing the second matrix parameters.

[0119] In a specific implementation, the processing content of the processing unit 414 can be determined according to the actual business scenario. Different models require different processing steps before collective communication during training. The same model requires different processing steps before collective communication at different training stages. Therefore, this application does not limit the processing content of the processing unit 414. For example, the processing content may include forward calculation of the cutting data, and the communication data refers to the forward calculation result. For example, the processing content may include back propagation of the cutting data, and the communication data refers to the gradient of the weight. The above examples are for illustration only and are not specifically limited in this application.

[0120] Optionally, the processing unit 414 can collectively communicate the communication data of different matrix parameters through different network planes, such as using the first network plane to collectively communicate the first communication data of the first matrix parameter and the second network plane to collectively communicate the second communication data of the second matrix parameter, or using the second network plane to collectively communicate the first communication data and the first network plane to collectively communicate the second matrix parameter. It should be understood that transmitting the communication data of different matrix parameters through different network planes can improve the efficiency of collective communication. For example, during model training, after the activation values ​​and weights are distributed to each node after cutting, each node performs forward calculations on the activation values ​​and weights, and then needs to aggregate the processed activation values ​​and weights through collective communication. If through Figure 1 The training system shown can only aggregate communication activation values ​​first and then aggregate communication weights. However, when using the training system 200 of the present application, activation values ​​and weights use different network planes for aggregate communication, which can greatly improve the efficiency of aggregate communication.

[0121] In a specific implementation, the processing unit 414 can send the communication data to the first communication unit 415 or the second communication unit 416 for collective communication based on the mapping relationship. Referring to the foregoing, it can be seen that the mapping relationship includes the mapping relationship between communication data and network planes. Communication data corresponding to the first network plane can be sent to the first communication unit 415 for collective communication, and communication data corresponding to the second network plane can be sent to the second communication unit 416 for collective communication. Because the two network planes use different communication methods, collective communication of the two network planes can be carried out simultaneously, improving the efficiency of collective communication during distributed training.

[0122] Optionally, a correlation exists between the mapping relationship and the cutting strategy, and the mapping relationship is determined based on the number of row cuts and the number of column cuts of the matrix parameters. Specifically, if the number of row cuts of the matrix parameters is determined based on the number of nodes in the communication group, and the number of column cuts is determined based on the number of communication groups, then the first communication data is collectively communicated via the second network plane, and the second communication data is communicated via the first network plane. Conversely, if the number of row cuts of the matrix parameters is determined based on the number of communication groups, and the number of column cuts is determined based on the number of nodes in the communication group, then the first communication data is collectively communicated via the first network plane, and the second communication data is collectively communicated via the second network plane.

[0123] It should be understood that during the model training process, after each collective communication, each node will obtain the communication data of other nodes, and then merge the communication data to obtain the merged result corresponding to each matrix parameter, and then perform matrix multiplication between the merged results. For example, multiple first communication data are merged into a third matrix parameter, and multiple second communication data are merged into a fourth matrix parameter, and then the third matrix parameter and the fourth matrix parameter are multiplied. The matrix multiplication operation needs to ensure that the number of columns of the first multiplier is the same as the number of rows of the second multiplier, that is, the inner axes of the two multipliers are the same, otherwise the matrix multiplication operation cannot be performed. Therefore, the network plane used for collective communication needs to refer to the method of cutting the matrix parameters to ensure that the number of columns of the merged third matrix parameter is the same as the number of rows of the fourth matrix parameter.

[0124] It should be understood that since the first matrix parameter and the second matrix parameter are two parameters that need to be matrix multiplied, the number of columns of the first matrix parameter is the same as the number of rows of the second matrix parameter. Therefore, by ensuring that the number of columns of the third matrix parameter is the same as the number of columns of the first matrix parameter, and the number of rows of the fourth matrix parameter is the same as the number of rows of the second matrix parameter, it is possible to ensure that the number of columns of the third matrix parameter is the same as the number of rows of the fourth matrix parameter. With this idea in mind, if the number of column cuts of the first matrix parameter is determined according to the number of communication groups, and the second communication method is inter-group communication between communication groups, then when the first communication result of the first matrix parameter is collectively communicated, it is necessary to use the second communication plane for collective communication so that the number of columns of the merged third matrix parameter is the same as the number of columns of the first matrix parameter. Similarly, if the number of column cuts of the first matrix parameter is determined according to the number of nodes in the communication group, it is necessary to use the first communication plane for collective communication so that the number of columns of the merged third matrix parameter is the same as the number of columns of the first matrix parameter.

[0125] It should be noted that after each node performs collective communication, it obtains multiple communication data on other nodes. When merging the multiple communication data, it is necessary to ensure that the number of columns of the merged third matrix parameter is the same as the number of rows of the fourth matrix parameter. Therefore, the multiple first communication data obtained through inter-group communication need to be horizontally spliced ​​to obtain the third matrix parameter, ensuring that the number of columns of the third matrix parameter is completely spliced, so that the number of columns of the third matrix parameter is the same as the number of columns of the first matrix parameter. The multiple second communication data obtained through inter-group communication are vertically spliced ​​to obtain the fourth matrix parameter, ensuring that the number of rows of the fourth matrix parameter is completely spliced, so that the number of rows of the fourth matrix parameter is the same as the number of rows of the second matrix parameter. In this way, the number of columns of the third matrix parameter finally obtained is the same as the number of rows of the fourth matrix parameter, and matrix multiplication can be performed.

[0126] For example, Figure 2 Taking the training system 200 shown as an example, the number of nodes m in the communication group is 2, the number of communication groups n is 3, the first communication plane is intra-group communication, and the second communication plane is inter-group communication. If the number of column cuts of the matrix parameter is n-1=2 times and the number of row cuts is m-1=1 time, then after each node obtains the first communication result, the first communication data needs to use inter-group communication (second communication plane) to merge the three first communication results to obtain the third matrix parameter. Since the first matrix parameter column is cut twice and divided into three parts, the merge also merges the three first communication results, so the number of columns of the merged third matrix parameter is the same as the number of columns of the first matrix parameter. Similarly, the second matrix parameter needs to use intra-group communication (first communication plane) to merge the two second communication results to obtain the fourth matrix parameter. Since the second matrix parameter row is cut once and divided into two parts, the merge also merges the two second communication results, so the number of rows of the merged fourth matrix parameter is the same as the number of rows of the second matrix parameter. The above examples are for illustration only and are not specifically limited in this application.

[0127] It should be noted that due to the aforementioned association between the mapping relationship and the cutting strategy, if the user configures the mapping relationship, the cutting strategy needs to flexibly adjust the number of column and row cuts based on the mapping relationship. If the user configures the cutting strategy, the mapping relationship needs to be flexibly adjusted based on the cutting strategy. If both the mapping relationship and the cutting strategy are user-configured, the association between the two needs to be indicated to the user, and any problems with the user configuration need to be promptly prompted. If neither the mapping relationship nor the cutting strategy is configured by the user, the training system 200 can use the default configuration to complete the training.

[0128] Optionally, the processing unit 414 is also used to perform matrix multiplication operation on the third matrix parameters and the fourth matrix parameters obtained by collective communication to obtain a first operation result, and complete the matrix multiplication task between the first matrix parameters and the second matrix parameters based on the first operation result, thereby completing the training of the model.

[0129] Furthermore, during the model training process, there are usually continuous matrix multiplication operations, such as Figure 1 As shown, after the activation value is matrix multiplied with weight 1, it also needs to be matrix multiplied with weight 2. Therefore, for a single matrix multiplication task, the first matrix parameter includes the activation value, and the second matrix parameter includes the weight. For multiple matrix multiplication tasks, the first matrix parameter includes the matrix multiplication result of the activation value and the first weight, and the second matrix parameter includes the second weight. In other words, the matrix parameter can also be a temporary intermediate result generated during the calculation process or a temporarily input activation value, which is not limited in this application.

[0130] Taking two consecutive matrix multiplication tasks as an example, it is assumed that the activation value is first matrix multiplied with the first weight, and then second matrix multiplication is performed with the second weight. At this time, the third matrix parameter corresponding to the activation value and the fourth matrix parameter corresponding to the first weight can be obtained according to the above description. After obtaining the first operation result between the two, the first operation result is used as the first matrix parameter, and the second weight is used as the second matrix parameter. After cutting according to the above description, the mapping relationship is determined according to the cutting strategy. Then, the collective communication is completed according to their respective corresponding communication planes at the same time, and the third matrix parameter corresponding to the first operation result and the fourth matrix parameter corresponding to the second weight are obtained. Then, based on the second operation result between the two, the two consecutive matrix multiplication tasks are completed.

[0131] Specifically, the cutting unit 413 can cut the second weight to obtain cutting data and distribute the multiple cutting results of the second weight to each node. The description of this process can refer to the description of cutting and distributing the first matrix parameters and the second matrix parameters in the above content, and will not be repeated here. It should be noted that the cutting method of the second weight is the same as the cutting method of the first matrix parameters and the second matrix parameters, and the number of row cutting and column cutting is the same.

[0132] Optionally, the processing unit 414 processes the cut data of the second weight to obtain third communication data, and then performs collective communication on the first operation result through the second communication unit 416 or the first communication unit 415, and the third matrix parameter corresponding to the first operation result is obtained by collective communication on the third communication data through the first communication unit 415 or the second communication unit 416 to obtain the fourth matrix parameter corresponding to the second weight. The processing unit 414 performs a matrix multiplication operation on the two to obtain a second operation result. Since the two network planes use different communication methods, the collective communication of the two network planes can be carried out simultaneously, thereby improving the efficiency of collective communication during distributed training.

[0133] It should be understood that if you use Figure 1In the training system shown, two consecutive matrix multiplication operations require four collective communications. The first collective communication aggregates the activation value, the second collective communication aggregates the first weight, the third collective communication aggregates the second weight, and the fourth collective communication specifies the result of the matrix multiplication. Through the training system provided by the present application, two consecutive matrix multiplication operations only require two collective communications. During the first collective communication, the first communication data is collectively communicated through the second network plane, and the second communication data is collectively communicated through the first network plane; during the second collective communication, the first operation result is collectively communicated through the second network plane, and the third communication data is collectively communicated through the first network plane. In this way, only two collective communications are needed to complete the two matrix multiplications, thereby improving communication efficiency and model training efficiency.

[0134] It should be noted that the process described above is mainly for forward propagation. For backward propagation, the matrix parameters can also be cut in the same way, and then collective communication is performed through different communication planes. Usually, allgather is used for collective communication of forward propagation, and reducescatter is used for collective communication of backward propagation.

[0135] It should be understood that Figure 1 In the architecture shown, the result of matrix multiplication needs to be split and distributed to each node, so Figure 1 When the architecture shown performs matrix multiplication operations, the first three collective communications are all allgather, and the last collective communication needs to use reducescatter. The cutting method provided in this application, combined with the multi-network plane architecture of the training system 200, can enable multiple collective communications to use the same collective communication method, such as using allgather for forward calculations and reducescatter for reverse calculations, which can improve training efficiency.

[0136] It should be noted that the matrix multiplication operation process in the above text is described using the processing unit 414, the first communication unit 415 and the second communication unit 416 as an example. In fact, during the matrix operation process, the processing flow of the processing unit, the first communication unit and the second communication unit of other nodes is the same as that of the first node 410. Each node performs the same processing process on the allocated cutting data and uses the same mapping relationship to complete the communication of different network planes. The description of the processing unit 424, the first communication unit 425 and the second communication unit 416 can refer to the description of the processing unit 414, the first communication unit 415 and the second communication unit 416, and will not be repeated here.

[0137] It should be noted that the above example uses the first matrix parameter as the activation value, the second matrix parameter as weight 1, and the second weight as weight 2. This scenario is mainly for the transformer model training scenario. For other model training scenarios, a similar method can be used to cut the matrix parameters and determine the network plane used for collective communication of different matrix parameters. For example, for a neural network using an attention module, during training, if matrix multiplication operations between activation values, weights, and attention parameters are required, the attention parameters can also be cut and communicated in a similar manner. Examples are not given here one by one.

[0138] In addition, the above example uses two matrix multiplication operations. In other model training, the number of matrix multiplications can be greater, and multiple matrix multiplications with different weights may be performed simultaneously. The number of row and column splits can be determined according to the description above, and communication planes with different weights can be determined based on the split method. This can make full use of multiple communication planes for parallel communication and improve model training efficiency.

[0139] Furthermore, the above is explained using two communication planes as an example, that is, a two-layer Internet network, with one network plane between communication groups and one network plane within a communication group. If there are more than two network planes, such as a three-layer network plane, multiple communication groups can also be treated as a large group, and there is a third network plane between the large groups. At this time, according to the model architecture, three network planes can be used for collective communication, or any two network planes can be selected from the three network planes for collective communication. The specific details can be determined according to the actual business scenario, and this application does not make specific limitations.

[0140] It should be noted that the allgather communication in the above description refers to the aggregation of communication data on multiple nodes so that each node has the aggregated communication results. In some different training platforms, allgather also has other aliases, such as allbond, but the processing flow is the same. Similarly, reducescatter communication also has different aliases in different training platforms. This application does not limit the communication nouns.

[0141] In summary, the present application provides a model training system, which includes multiple nodes, and multiple network planes are established between the multiple nodes through different communication methods. In this way, when performing distributed training, a node obtains the cutting data of the matrix parameters and trains the model based on the cutting data. When obtaining the communication data that needs to be synchronized to other nodes, the communication data of different matrix parameters can be collectively communicated using different network planes, so that the collective communication of multiple matrix parameters can be executed in parallel, reducing the number of collective communications, improving the efficiency of collective communications, and thereby improving the efficiency of model training.

[0142] The training system provided by this application is described in detail above. Figure 5 The model training method provided in this application is explained.

[0143] Figure 5 This is a flow chart of the steps of a model training method provided by this application. The model training method can be applied to Figure 2-Figure 4 In the training system 200A or the training system 200B shown in FIG. Figure 5 As shown, the method may include the following steps:

[0144] S510: The first node 410 obtains the cutting strategy and mapping relationship. This step can be performed by Figure 4 The configuration unit 411 in the embodiment is implemented.

[0145] In a specific implementation, the cutting strategy and mapping relationship may be generated based on the configuration information sent by the user through the client 300, or may be configured by default by the training system 200, or part of the cutting strategy and part of the mapping relationship may be configured by the user and part may be automatically generated by the training system 200. This application does not make any specific limitations.

[0146] In a possible implementation, the mapping relationship includes a mapping relationship between communication data and a network plane.

[0147] Among them, the network plane refers to the network plane in the training system, such as Figure 2 In the training system 200A, two nodes in a communication group establish a first network plane through SIO, and a second network plane is established between multiple communication groups through high-speed fullmesh interconnection bandwidth, or Figure 3 In the example, the four nodes in the communication group establish a first network plane through high-speed fullmesh interconnection bandwidth, and a second network plane is established between multiple communication groups through switch chips. Of course, the training system can also include more network planes, which can be found in the following examples. Figure 2 and Figure 3 The relevant descriptions in the embodiments are not repeated here.

[0148] Among them, communication data refers to the data that needs to be collectively communicated during the model training process, such as model weights, gradients, activation values, loss values, hyperparameters, attention parameters, etc., which are not specifically limited in this application. Among them, collective communication refers to the global communication operation in which all nodes in the training system participate. It should be understood that the goal of collective communication is to transmit messages throughout the training system, rather than just point-to-point communication between two nodes. Generally, collective communication may include allreduce, allgather, reducescatter, etc., and you can refer to the aforementioned Figure 2-Figure 4 The relevant descriptions in the embodiments are not repeated here.

[0149] Among them, one network plane can correspond to one or more communication data, and communication data at different communication times can correspond to the same network plane, but communication data at the same communication time needs to correspond to different network planes for parallel communication. In this way, the communication data that need to be collectively communicated at the same time are transmitted in parallel through different network planes, which can improve the efficiency of collective communication and thus reduce the training time.

[0150] In one possible implementation, the cutting strategy includes multiple different types of parallel strategies, such as tensor parallel strategy, data parallel strategy, and pipeline parallel strategy. It should be understood that the model can use a mixture of multiple cutting strategies to achieve distributed training during the training process, so the user can configure multiple cutting strategies. The explanation of the above-mentioned multiple cutting strategies can be referred to the above content and will not be repeated here.

[0151] The tensor parallel strategy includes performing at least one row split and at least one column split on the matrix parameters to obtain split data for the matrix parameters, where the number of split data is equal to the number of nodes. In a specific implementation, the number of row splits is determined by the number of communication groups, and the number of column splits is determined by the number of nodes within a communication group; alternatively, the number of row splits is determined by the number of nodes within a communication group, and the number of column splits is determined by the number of communication groups. Row splitting refers to splitting a matrix horizontally into multiple parts along its rows, while column splitting refers to splitting a matrix vertically into multiple parts along its columns.

[0152] The tensor parallel strategy also includes: the first matrix parameter and the second matrix parameter participating in the same matrix multiplication operation are cut in the same way, that is, the number of row cuts of the first matrix parameter is the same as the number of row cuts of the second matrix parameter, and the number of column cuts of the first matrix parameter is the same as the number of column cuts of the second matrix parameter. If the cutting method changes, when the cut data is distributed to each node for matrix multiplication, the problem of multiplication may occur due to the different internal axes of the two cut data participating in the multiplication. Therefore, the two matrix parameters need to be cut according to the same cutting method to avoid the problem of the cut data being unable to be multiplied.

[0153] Tensor parallel strategies also include: if the number of nodes in each communication group is m and the number of communication groups is n, then the number of row splits is m-1, that is, the matrix parameters are split into m parts by row, and the number of column splits is n-1, that is, the matrix parameters are further split into m×n parts by column. Alternatively, the number of row splits is n-1, and the number of column splits is m-1.

[0154] For example, Figure 6 This is a matrix cutting example diagram provided by this application, such as Figure 6 As shown, assuming that the length of the first matrix parameter is H and the width is A, and the length of the second matrix parameter is B and the width is H, the scale of the training system 200 is as follows Figure 2 As shown, the number of communication groups n = 3, the number of nodes in the communication group m = 2, if the number of row cuts is determined according to the number of nodes in the communication group, then the number of row cuts is 1, the number of column cuts is 2, and the cutting method of each matrix parameter is the same, that is, Figure 6 The first matrix parameter is cut into 6 parts, and the second matrix parameter is also cut into 6 parts. It should be understood that Figure 6 For example, if the number of row cuts is determined according to the number of communication groups, then the number of row cuts is 2, the number of column cuts is 1, and the cutting methods of the two matrix parameters remain consistent, which is not specifically limited in this application.

[0155] For another example, if the scale of the training system 200 is Figure 3 As shown, the number of communication groups n = 2, the number of nodes in the communication group m = 4, if the number of row cuts is determined according to the number of nodes in the communication group, then the number of row cuts is 3, and the number of column cuts is 1; if the number of row cuts is determined according to the number of communication groups, then the number of row cuts is 1, and the number of column cuts is 3. The above examples are for illustration only and are not specifically limited in this application.

[0156] It should be noted that after determining the number of cuts, the matrix parameters can be cut evenly, or cut by rows or columns. For example, if there are 4 rows that need to be cut twice, then the first row can be cut once and the third row can be cut once. In other words, it is not necessarily an average cut. The specific cutting method can be determined according to the actual business scenario, and this application does not make specific limitations.

[0157] The above examples show that the technical solution provided by this application can flexibly determine the cutting method, but it is necessary to keep each matrix parameter cut using the same cutting method, and the cutting method is related to the number of communication groups or the number of nodes in the communication group, and it is necessary to ensure that the number of cut data is consistent with the number of nodes.

[0158] In one possible implementation, a mapping relationship is associated with a cutting strategy, and the mapping relationship is determined based on the number of row cuts and column cuts of the matrix parameters. Specifically, if the number of row cuts of the matrix parameters is determined based on the number of nodes in the communication group, and the number of column cuts is determined based on the number of communication groups, then the first communication data is collectively communicated via the second network plane, and the second communication data is communicated via the first network plane. Conversely, if the number of row cuts of the matrix parameters is determined based on the number of communication groups, and the number of column cuts is determined based on the number of nodes in the communication group, then the first communication data is collectively communicated via the first network plane, and the second communication data is collectively communicated via the second network plane.

[0159] It should be understood that during the model training process, after each collective communication, each node will obtain the communication data of other nodes, and then merge the communication data to obtain the merged result corresponding to each matrix parameter, and then perform matrix multiplication between the merged results. For example, multiple first communication data are merged into a third matrix parameter, and multiple second communication data are merged into a fourth matrix parameter, and then the third matrix parameter and the fourth matrix parameter are multiplied. The matrix multiplication operation needs to ensure that the number of columns of the first multiplier is the same as the number of rows of the second multiplier, that is, the inner axes of the two multipliers are the same, otherwise the matrix multiplication operation cannot be performed. Therefore, the network plane used for collective communication needs to refer to the method of cutting the matrix parameters to ensure that the number of columns of the merged third matrix parameter is the same as the number of rows of the fourth matrix parameter.

[0160] It should be understood that since the first matrix parameter and the second matrix parameter are two parameters that need to be matrix multiplied, the number of columns of the first matrix parameter is the same as the number of rows of the second matrix parameter. Therefore, by ensuring that the number of columns of the third matrix parameter is the same as the number of columns of the first matrix parameter, and the number of rows of the fourth matrix parameter is the same as the number of rows of the second matrix parameter, the number of columns of the third matrix parameter and the number of rows of the fourth matrix parameter can be ensured to be the same. With this idea in mind, if the number of column cuts of the first matrix parameter is determined by the number of communication groups, and the second communication method is inter-group communication between communication groups, then when the first communication result of the first matrix parameter is collectively communicated, it is necessary to use the second communication plane for collective communication so that the number of columns of the merged third matrix parameter is the same as the number of columns of the first matrix parameter. Similarly, if the number of column cuts of the first matrix parameter is determined by the number of nodes in the communication group, it is necessary to use the first communication plane for collective communication so that the number of columns of the merged third matrix parameter is the same as the number of columns of the first matrix parameter.

[0161] It should be noted that after each node performs collective communication, it obtains multiple communication data on other nodes. When merging the multiple communication data, it is necessary to ensure that the number of columns of the merged third matrix parameter is the same as the number of rows of the fourth matrix parameter. Therefore, the multiple first communication data obtained through inter-group communication need to be horizontally spliced ​​to obtain the third matrix parameter, ensuring that the number of columns of the third matrix parameter is completely spliced, so that the number of columns of the third matrix parameter is the same as the number of columns of the first matrix parameter. The multiple second communication data obtained through inter-group communication are vertically spliced ​​to obtain the fourth matrix parameter, ensuring that the number of rows of the fourth matrix parameter is completely spliced, so that the number of rows of the fourth matrix parameter is the same as the number of rows of the second matrix parameter. In this way, the number of columns of the third matrix parameter finally obtained is the same as the number of rows of the fourth matrix parameter, and matrix multiplication can be performed.

[0162] It should be noted that due to the aforementioned association between the mapping relationship and the cutting strategy, if the user configures the mapping relationship, the cutting strategy needs to flexibly adjust the number of column and row cuts based on the mapping relationship. If the user configures the cutting strategy, the mapping relationship needs to be flexibly adjusted based on the cutting strategy. If both the mapping relationship and the cutting strategy are user-configured, the association between the two needs to be indicated to the user, and any problems with the user configuration need to be promptly prompted. If neither the mapping relationship nor the cutting strategy is configured by the user, the training system 200 can use the default configuration to complete the training.

[0163] In one possible implementation, the first node 410 may also obtain parameter information, which may be the system default configuration or may be input by the user through the client 300, and this application does not make any specific limitations. The parameter information may include the structural parameters of the AI ​​model, and may also include the scale parameters of the training system 200, and may also include the training parameters that need to be configured in the training process. Among them, the structural parameters may include the hidden layer dimension size, the feedforward layer dimension size, the number and size of multi-head attention, and other parameters related to the network structure of the AI ​​model. The scale parameters may include the number of nodes, the specification parameters of the nodes, the number of network planes, the number of nodes in the network plane, and the bandwidth, and other parameters related to the hardware resources of the training system 200. The training parameters may include batch data size, sequence length, optimizer, learning rate, activation function, loss function, and other parameters related to the model training process. The above examples are for illustration only and this application does not make any specific limitations.

[0164] Figure 7 This is an example diagram of a configuration interface provided by the present application. The configuration interface can be displayed to the user through the client 300, and the configuration information input by the user is obtained, and then the configuration information database is updated according to the configuration information. Figure 7 As shown, the configuration interface may include a mapping relationship configuration area 710 and a cutting strategy configuration area 720 .

[0165] The mapping relationship configuration area 710 may include a communication data display area and a network plane display area. The system displays the configurable network planes and communication data in these two areas, and the user can establish a connection between them. As previously mentioned, the mapping relationship between communication data and network planes can be user-configured or system-generated. Therefore, the client 300 can display recommended mapping relationships to the user through this configuration interface, and the user can modify them according to their needs.

[0166] The cutting strategy configuration area 720 may include column cutting times configuration options and row cutting times configuration options. Users can configure the cutting strategy according to their needs. Of course, the cutting strategy configuration area may also include other parallel strategy configuration areas, such as pipeline parallel strategy, data parallel strategy, etc., and the configuration interface may also include more areas, such as training parameter configuration area, model structure parameter configuration area, training system scale parameter configuration area, etc. Figure 7 This is an example of a configuration interface and is not specifically limited in this application.

[0167] As can be seen from the above content, there is a correlation between the mapping relationship and the cutting strategy. The mapping relationship is determined based on the number of row cutting times and column cutting times of the matrix parameters. For example, if the number of row cutting times of the matrix parameters is determined based on the number of nodes in the communication group, then the first communication data is collectively communicated through the second network plane. Figure 7 When configuring the configuration interface shown, if the mapping relationship and cutting strategy selected by the user violate the above association, the client can prompt the user with a message that the configuration is non-compliant, or, when the user configures the cutting strategy, the system generates a corresponding mapping relationship based on the above association and displays it in the mapping relationship configuration area 710. Similarly, when the user configures the mapping relationship, the system generates a corresponding cutting strategy based on the above association and displays it in the cutting strategy configuration area 720. This application does not make specific limitations.

[0168] It should be noted that Figure 7 This is an example of a visual interface. In the specific implementation, during the model training process, users can configure mapping relationships and cutting strategies through non-visual interfaces such as the application programming interface (API) and command line interface (CLI). This application does not make specific limitations.

[0169] S520: The first node 410 sends the mapping relationship to other nodes. The second node 420 receives and stores the mapping relationship. Figure 4 The configuration unit 411 and the configuration unit 421 in the embodiment are implemented.

[0170] In a specific implementation, the first node 410 can synchronize the mapping relationship to other nodes, so that each node can complete collective communication based on the stored mapping relationship. Different communication data use different network planes for collective communication, so that the communication data of multiple matrix parameters can be transmitted simultaneously, thereby improving the efficiency of collective communication and thus improving the efficiency of model training.

[0171] It should be noted that the first node 410 may also send the above-mentioned cutting strategy and parameter information to other nodes. The specific synchronization content may be determined according to actual business needs, and this application does not make any specific limitations.

[0172] S530: The first node 410 cuts each matrix parameter according to the cutting strategy to obtain the cutting data of each matrix parameter. Figure 4 The cutting unit 413 in the embodiment is implemented.

[0173] In a specific implementation, the first node 410 can complete the cutting of matrix parameters according to the above-mentioned tensor parallel strategy. For example, after cutting the first matrix parameter and the second matrix parameter participating in the same matrix multiplication operation, multiple first cutting data of the first matrix parameter and multiple second cutting data of the second matrix parameter are obtained.

[0174] Referring to the above content, it can be seen that the cutting strategy includes many different types of parallel strategies, such as tensor parallel strategy, data parallel strategy and pipeline parallel strategy. If the training process uses a combination of multiple parallel strategies to train the model, the first node 410 can cut the model parameters and training data according to multiple parallel strategies, obtain the cut data of the model parameters and training data, and then distribute it to each node. I will not elaborate on it here.

[0175] S540: The first node 410 sends the cutting data to other nodes. This step can be done by Figure 4 The cutting unit 413 in the embodiment is implemented.

[0176] As can be seen from the above content, the number of cut data for each matrix parameter is the same as the number of nodes, so each node obtains one cut data for each matrix parameter. There are multiple matrix parameters involved in the matrix multiplication operation, so each node can obtain multiple cut data for multiple matrix parameters. For example, the first matrix parameter is the activation value and the second matrix parameter is the weight. Then each node will obtain the cut data for the activation value and the cut data for the weight. The above examples are for illustrative purposes only and are not specifically limited in this application.

[0177] S550A-S550B: The first node 410 and the second node 420 process the cutting data of each matrix parameter to obtain the communication data of each matrix parameter. Figure 4 The processing unit 414 of the first node 410 and the processing unit 424 of the second node 420 in the embodiment are implemented.

[0178] In simple terms, each node can process the assigned cutting data and generate communication data that needs to be transmitted to other nodes, wherein one cutting data corresponds to one communication data, such as the communication data of the activation value obtained after the activation value is processed, and the communication data of the weight is obtained after the weight is processed. In the specific implementation, the processing content when processing the cutting data can be determined according to the actual business scenario. Different models require different processing steps before collective communication during the training process. The same model requires different processing steps before collective communication at different training stages. Therefore, this application does not make specific limitations. For example, the processing content may include forward calculation of the cutting data, and the communication data at this time refers to the forward calculation result. For example, the processing content may include back propagation of the cutting data, and the communication data refers to the gradient of the weight. The above examples are for illustration only and are not specifically limited in this application.

[0179] S560: The first node 410 and the second node 420 use multiple network planes to complete the collective communication of each matrix parameter according to the mapping relationship. Figure 4 In the embodiment, the first communication unit 415 and the second communication unit 416 of the first node 410 and the first communication unit 425 and the second communication unit 426 of the second node 420 are implemented.

[0180] Referring to the above content, it can be seen that the mapping relationship records the network planes corresponding to different communication data. At this time, different network planes can be used to communicate different communication data in parallel according to the mapping relationship, so that collective communication can be executed in parallel, and the collective communication of multiple communication data can be completed at the same time, thereby improving communication efficiency.

[0181] S570A-S570B: The first node 410 and the second node 420 perform matrix multiplication based on the collective communication result, and complete model training based on the matrix multiplication result. This step can be performed by Figure 4 The processing unit 414 of the first node 410 and the processing unit 424 of the second node 420 in the embodiment are implemented.

[0182] For example, Figure 8 This is an example diagram of the steps of a training method provided in this application for processing matrix multiplication tasks. Figure 8 by Figure 2 The training system 200A shown, Figure 6Taking the matrix parameters shown as an example, the number of nodes m in the communication group is 2, the number of communication groups n is 3, the first communication plane is intra-group communication, and the second communication plane is inter-group communication. The number of column cuts of the matrix parameters is determined according to the number of communication groups, the number of column cuts is n-1=2 times, the number of row cuts is m-1=1 times, the number of columns of the first matrix parameter is H, and the number of rows of the second matrix parameter is H. After the first matrix parameter and the second matrix parameter are cut into 6 parts respectively, each node can be allocated a cut data of the first matrix parameter and a cut data of the second matrix parameter. Figure 8 The dark short bars refer to the cutting data of the first matrix parameter, the light long bars refer to the cutting data of the second matrix parameter, and node 11 is Figure 2 A node in the embodiment, after being assigned cutting data, can process the cutting data to generate corresponding communication data, and then use two different network plane sets to communicate the two cutting data.

[0183] like Figure 8 As shown, the cutting data of the second matrix parameters, that is, the light-colored strips, are transmitted through the first network plane. The first network plane refers to intra-group communication, so the nodes 12 and 11 in the same communication group exchange data. After the collective communication, the node 11 will obtain the communication data of all the second matrix parameters in the communication group, that is, two light-colored strips, and then obtain the fourth matrix parameters by vertical splicing. Since the second matrix parameters are cut into two parts by row cutting during cutting, the height or number of rows H of the fourth matrix parameters obtained after vertical splicing of the two light-colored strips is the same as the second matrix parameters.

[0184] Similarly, the cut data of the first matrix parameters, that is, the dark short strips, are transmitted through the second network plane. The second network plane refers to inter-group communication, so nodes 21, node 31, and node 11 in different communication groups exchange data. After the collective communication, node 11 will obtain the communication data of all first matrix parameters of different communication groups, that is, three dark short strips, and then obtain the third matrix parameters through horizontal splicing. Since the first matrix parameters are cut into three parts by column cutting during cutting, the length or number of columns H of the third matrix parameters obtained after horizontal splicing of the three dark short strips is the same as the first matrix parameters.

[0185] In this way, the number of columns H of the third matrix parameter obtained after collective communication is the same as the number of rows H of the fourth matrix parameter, and the inner axes of the two multipliers are the same. Matrix multiplication operations can be performed to obtain a matrix multiplication result, and based on the matrix multiplication result, the matrix multiplication result of the first matrix parameter and the second matrix parameter can be obtained.

[0186] It should be understood that Figure 8For example, in fact, no matter what cutting strategy is used, during collective communication, when the first communication data determines its communication plane, it only needs to ensure that the collective communication result obtained through this network plane can splice out the width or number of columns H of the first matrix parameter, and the second communication data only needs to ensure that the height or number of rows H of the second matrix parameter can be spliced ​​out through another network plane, so that the matrix multiplication operation can proceed smoothly and avoid the failure of the multiplication task due to different internal axes.

[0187] It should be understood that during the model training process, the matrix multiplication result Y of the first matrix parameter and the second matrix parameter also needs to be distributed to each node for further calculation, so Figure 8 In the example shown, the training method provided in the present application is used to perform matrix multiplication operations. Each node can obtain an element of the matrix multiplication result Y, eliminating the distribution step and completing it through a single allgather communication. If collective communication is performed through a single network plane in the traditional way, at least 2 allgather communications (communication of the cut data of the first matrix parameters and the cut data of the second matrix parameters) and at least one reducescatter communication (distribution of the results of the matrix multiplication) are required to complete a matrix multiplication task. The training method of the present application can improve communication efficiency.

[0188] As mentioned above, during the model training process, there will be two consecutive matrix multiplication operations. Figure 8 A single matrix multiplication task is used as an example. Figure 9 Taking a specific application scenario as an example, the steps and processes of multiple matrix multiplication tasks are explained.

[0189] Figure 9 This is a flowchart of a training method provided by this application for processing matrix multiplication tasks in an application scenario, such as Figure 9 As shown, it is assumed that the matrix multiplication task is a matrix multiplication task between the activation value and the first weight and the second weight. Figure 9 The training system structure used in the application scenario shown is Figure 2 The structure shown, and the cutting method is Figure 8 The cutting method shown, that is, the number of column cuttings is determined based on the number of communication groups, and the number of row cuttings is determined based on the number of nodes in the communication group, then the activation value, the first weight and the second weight can all be cut into 6 parts and then distributed to each node, among which node 11 can be assigned 1 activation value cutting data, replaced by a gray short bar, 1 first weight cutting data, replaced by a light-colored long bar, and 1 second weight cutting data, replaced by a white short bar.

[0190] After each node processes the assigned cutting data, it obtains the corresponding communication data and then starts the first collective communication. Since there are only two network planes, the communication data of the activation value and the first weight are collectively communicated at the same time. The activation value communicates between groups through the second network plane. There are three communication groups, so the communication data of three activation values ​​is obtained. The third matrix parameter width or number of columns obtained by horizontal splicing is H. The communication data of the first weight is communicated within the group through the first network plane. There are two nodes in the group, so the communication data of two first weights is obtained. The fourth matrix parameter height or number of rows obtained by vertical splicing is H, so the two can perform matrix multiplication. At this time, the first collective communication does not perform collective communication on the second weight. For a detailed description of the first collective communication, please refer to Figure 8 The embodiments are not repeated here.

[0191] After performing a matrix multiplication operation of the third matrix parameter and the fourth matrix parameter at each node to obtain a first operation result, each node includes cutting data of the first operation result and the second weight. Figure 9 In the figure, the first operation result is represented by a black short bar, and the cut data of the second weight is a white short bar. Similar to the first collective communication, in order to ensure that the inner axes of the two matrix parameters spliced ​​after the collective communication are the same, the first operation result is communicated between groups through the second network plane. The three communication groups obtain communication data of three activation values, and the width or number of columns of the fifth matrix parameter obtained by horizontal splicing is H'. The communication data of the second weight is communicated within the group through the first network plane. There are two nodes in the group, so two communication data of the second weight are obtained. The height or number of rows of the sixth matrix parameter obtained by vertical splicing is H'. The inner axes of the two are the same, and matrix multiplication can be performed.

[0192] After each node performs the matrix multiplication operation of the fifth matrix parameter and the sixth matrix parameter, each node can obtain the second operation result. After each second operation result is spliced, the complete result of the matrix multiplication operation of the activation value, the first weight and the second weight can be obtained. However, each node just needs to obtain a part of the complete result, so compared to Figure 1 The training system shown, Figure 1 Four collective communications are required, the first three of which are allgather communications, and the last one requires reducescatter communication because the matrix multiplication result needs to be distributed to each node. The training method provided by this application, such as Figure 9 As shown, only two collective communications are required, both of which are allgather communications, which can reduce the number of communications and improve communication efficiency.

[0193] It should be noted that Figure 9The example shown is for illustration. In the actual processing process, the first matrix multiplication and the second matrix multiplication may include more steps, which can be determined according to the actual model training requirements. For example, taking the transformer model as an example, in the forward calculation process, the first matrix multiplication may include batch matrix multiplication (batchmatmul) and application of an activation function, such as the gelu activation function, and the second matrix multiplication may include batchmatmul. Under other model architectures, each matrix multiplication can have more step processes, but when collective communication is required, according to the idea provided in this application, the network plane is selected according to the cutting strategy so that the inner axes of the two matrices spliced ​​after the collective communication are the same, so that not only the communication efficiency can be improved during model training, but also the matrix multiplication operation will not be affected.

[0194] Need to explain, Figure 9 Taking two matrix multiplication operations as an example, in other model training, the number of matrix multiplications can be greater, and multiple matrix multiplications with different weights may be performed simultaneously. The above description can be used to determine the number of row and column cuts, and the communication planes with different weights can be determined based on the cut method. This can make full use of multiple communication planes for parallel communication and improve model training efficiency.

[0195] Further, Figure 8 and Figure 9 The explanation is given using two communication planes as an example, that is, a two-layer Internet network, with one network plane between communication groups and one network plane within a communication group. If there are more than two network planes, such as a three-layer network plane, multiple communication groups can also be treated as a large group, and there is a third network plane between the large groups. At this time, according to the model architecture, three network planes can be used for collective communication, or any two network planes can be selected from the three network planes for collective communication. The specific details can be determined according to the actual business scenario, and this application does not make any specific limitations.

[0196] As can be seen from the above content, when cutting the matrix parameters, the number of column cuts can be determined based on the number of nodes in the communication group, or based on the number of communication groups. In other words, there are at least two cutting methods. In fact, the communication delays of these two cutting methods are different in different application scenarios. Therefore, the cutting method can be flexibly selected according to different application scenarios. Figure 2 Taking the training system 200A shown as an example, taking the matrix multiplication operation of the activation value with the first weight and the second weight as an example, the communication delays of two different cutting methods are determined respectively.

[0197] Figure 2In the training system 200A shown, a dual-die packaged chip serves as a communication group. Therefore, the number of nodes in a communication group, m, is 2, and the number of communication groups is n. Therefore, the number of column cuts is 1, and the number of row cuts is n-1. The first cut method is: the number of column cuts is m-1, and the number of row cuts is n-1. The second cut method is: the number of column cuts is n-1, and the number of row cuts is m-1. The communication delay of the collective communication for the two cut methods can be expressed as follows: Figure 10 shown. Figure 10 This is a comparison chart of communication delays under different cutting strategies in a model training method provided in this application.

[0198] Here, B refers to the batch size, S refers to the sequence length, H refers to the hidden dimension, and H' refers to the feedforward dimension. Figure 9 All subscripts 2 or sio refer to the related computational amount after collective communication of intra-group communication (first network plane), and subscripts n or i refer to the related computational amount after collective communication of inter-group communication (second network plane). For example, B2 refers to the batch processing specification of collective communication of the first network plane, and Bn refers to the batch processing specification of collective communication of the second network plane.

[0199] like Figure 10 As shown, after calculating the communication delays corresponding to the two cutting methods, the judgment conditions can be determined according to the relevant parameters in the communication delay. The actual business scenario can select the cutting method with the lowest communication delay according to the judgment conditions it meets. For example, assuming that net i =450Gbps,net sio =540 Gbps, S=4096, B=2048, n=16, H=12288, H'=49152. After substituting, we can obtain the best cutting method, which is Solution 2.

[0200] It should be understood that when the user configures the mapping relationship and cutting strategy, the system can display the Figure 10 The communication delays of different schemes shown can be used as a reference by users to select the most suitable cutting method for the current business scenario. Of course, the system can also generate default cutting strategies and mapping relationships for users to choose based on the parameter information of the model. This application does not make specific limitations.

[0201] It should be understood that the above Figure 2 The training system shown in the figure is used as an example to illustrate the Figure 3 For the training system shown in the figure, the sio network plane is replaced by the switch network plane, m=4, and it can also be passed Figure 10 The communication delay of different schemes can be inferred in the same way as shown. Other training systems can also refer to Figure 10 The communication delay of the synchronization scheme is inferred in the manner shown for user reference, and examples are not given here one by one.

[0202] In summary, the present application provides a model training system, which includes multiple nodes, and multiple network planes are established between the multiple nodes through different communication methods. In this way, when performing distributed training, a node obtains the cutting data of the matrix parameters and trains the model based on the cutting data. When obtaining the communication data that needs to be synchronized to other nodes, the communication data of different matrix parameters can be collectively communicated using different network planes, so that the collective communication of multiple matrix parameters can be executed in parallel, reducing the number of collective communications, improving the efficiency of collective communications, and thereby improving the efficiency of model training.

[0203] The above describes in detail the model training method and training system provided by this application. Figure 11 The computing device provided in this application is explained.

[0204] Figure 11 11 is a schematic diagram of the structure of a computing device provided in this application. The computing device 1100 may be the training system described above. Furthermore, the computing device 1100 includes a processor 1101, a storage unit 1102, a storage medium 1103, and a communication interface 1104. The processor 1101, storage unit 1102, storage medium 1103, and communication interface 1104 communicate via a bus 1105, and may also communicate via other means such as wireless transmission.

[0205] Processor 1101 is composed of multiple general-purpose processors, such as CPUs. The general-purpose processor can be a hardware chip, which is an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD is a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a system on chip (SoC), or any combination thereof. Processor 1101 executes various types of digital storage instructions, such as software or firmware programs stored in storage unit 1102, which enables computing device 1100 to provide a wide variety of services.

[0206] In a specific implementation, as an embodiment, the processor 1101 includes one or more CPUs, such as Figure 11 CPU0 and CPU1 are shown in the figure.

[0207] In a specific implementation, as an embodiment, the computing device 1100 also includes multiple processors, such as Figure 11 1 and 1106. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor here refers to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0208] The storage unit 1102 is used to store program codes, and is controlled by the processor 1101 to execute the above Figures 1-11 The processing steps of the training system in any embodiment. The program code includes one or more software units. The above one or more software units are Figure 1 The configuration unit, processing unit and communication unit in the embodiment, wherein the configuration unit is used to obtain the mapping relationship between the communication data and the network plane, specifically for implementing Figure 5 S510-S520 and optional steps in the embodiment; the processing unit user obtains the cutting data of the matrix parameters, trains the model based on the cutting data, and obtains the communication data corresponding to the matrix parameters, specifically for implementing Figure 5S540-S550B and optional steps in the embodiment; the communication unit is used to communicate with other nodes based on the mapping relationship, obtain the communication data of the matrix parameters on each node, and the communication data can correspond to Figure 1 The first communication unit and the second communication unit in the embodiment are specifically used to implement Figure 5 S560 and its optional steps in the embodiment; the processing unit is also used to complete the matrix multiplication task according to the communication data of the matrix parameters on each node, complete the training of the model, specifically for realizing Figure 5 S570A-S570B and optional steps in the embodiment.

[0209] The storage unit 1102 includes a read-only memory and a random access memory, and provides instructions and data to the processor 1101. The storage unit 1102 also includes a non-volatile random access memory. The storage unit 1102 is a volatile memory or a non-volatile memory, or includes both volatile and non-volatile memories. Among them, the non-volatile memory is a read-only memory (ROM), a programmable ROM (PROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), or a flash memory. The volatile memory is a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are used, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DR RAM). A hard disk, a universal serial bus (USB), a flash memory, a secure digital memory card (SD card), a memory stick, etc., and a hard disk can be a hard disk drive (HDD), a solid state drive (SSD), a mechanical hard disk (HDD), etc., which is not specifically limited in this application.

[0210] The storage medium 1103 is a carrier for storing data, such as a hard disk, a USB flash drive (universal serialbus), a flash memory, an SD card (secure digital memory card, SD card), a memory stick, etc. The hard disk can be a hard disk drive (HDD), a solid state disk (SSD), a mechanical hard disk (HDD), etc., and this application does not make specific limitations.

[0211] The communication interface 1104 is a wired interface (such as an Ethernet interface), an internal interface (such as a high-speed serial computer expansion bus (Peripheral Component Interconnect express, PCIe) bus interface), a wired interface (such as an Ethernet interface) or a wireless interface (such as a cellular network interface or a wireless local area network interface) for communicating with other servers or units.

[0212] Bus 1105 is a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a unified bus (UBus or UB), a Compute Express Link (CXL), or a Cache Coherent Interconnect for Accelerators (CCIX). Bus 1105 is divided into an address bus, a data bus, and a control bus.

[0213] In addition to the data bus, the bus 1105 also includes a power bus, a control bus, a status signal bus, etc. However, for the sake of clarity, various buses are labeled as bus 1105 in the figure.

[0214] Need to explain, Figure 11 This is only one possible implementation of the embodiment of the present application. In actual applications, the computing device 1100 may also include more or fewer components, which is not limited here. For details not shown or described in the embodiment of the present application, please refer to the aforementioned Figures 1-10 The relevant explanations in the embodiments will not be repeated here.

[0215] The present application also provides a computing device cluster, which may be the training system described above, and includes at least one computing device 1100. The storage unit 1102 in one or more computing devices 1100 in the computing device cluster may store the same or different instructions for executing the model training method.

[0216] The present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the model training method.

[0217] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a high-density digital video disc (DVD)), or a semiconductor medium (for example, a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the model training method.

[0218] The above embodiments can be implemented in whole or in part through software, hardware, firmware, or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes multiple computer instructions. When the computer program instructions are loaded or executed on a computer, the processes or functions according to the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium.

[0219] The above are only specific embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent repairs or replacements within the technical scope disclosed in this application, and such repairs or replacements should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A model training method, characterized in that: The method is applied to a training system, the training system including multiple nodes, multiple network planes being established between the multiple nodes through different communication modes, and the method including: A first node acquires a mapping relationship between communication data and a network plane, where the first node is any node among the multiple nodes; The first node obtains cutting data of a matrix parameter, where the matrix parameter is a multiplier in a matrix multiplication task in model training; The first node trains the model based on the cutting data to obtain communication data corresponding to the matrix parameters, where the communication data is data to be transmitted by the first node to other nodes; The first node performs collective communication with the other nodes based on the mapping relationship to obtain communication data of the matrix parameter on each node, wherein communication data corresponding to different matrix parameters use different network planes for collective communication; The first node completes the matrix multiplication task according to the communication data of the matrix parameters on each node, thereby completing the training of the model.

2. The method according to claim 1, characterized in that The training system includes multiple communication groups, each communication group includes multiple nodes, a first network plane is established between the multiple nodes in the communication group through a first communication method, and a second network plane is established between the communication groups through a second communication method.

3. The method according to claim 1 or 2, characterized in that The first node obtains the cutting data of the matrix parameters including: The first node performs at least one row cut and at least one column cut on the matrix parameter to obtain a plurality of cut data of the matrix parameter, where the number of the cut data is the same as the number of nodes in the training system; The first node distributes the plurality of cutting data to other nodes, so that each node obtains one cutting data of the matrix parameter.

4. The method according to claim 3, characterized in that The number of row cuts is determined according to the number of communication groups, and the number of column cuts is determined according to the number of nodes in the communication group; or, the number of row cuts is determined according to the number of nodes in the communication group, and the number of column cuts is determined according to the number of communication groups.

5. The method according to any one of claims 2 to 4, characterized in that: The matrix parameters include a first matrix parameter and a second matrix parameter, the first matrix parameter and the second matrix parameter are a left multiplier and a right multiplier in the same matrix multiplication task, and the communication data includes first communication data corresponding to the first matrix parameter and second communication data corresponding to the second matrix parameter; If the number of column cuts of the matrix parameter is determined according to the number of the communication groups, the first communication data corresponds to the second network plane, and the second communication data corresponds to the first network plane; If the number of column cuts of the matrix parameter is determined according to the number of the communication groups, the first communication data corresponds to the first network plane, and the second communication data corresponds to the second network plane.

6. The method according to claim 5, characterized in that The first node completing the matrix multiplication task according to the communication data of the matrix parameter on each node includes: The first node concatenates the first communication data of the first matrix parameters on each node to obtain a third matrix parameter, and concatenates the second communication data of the second matrix parameters on each node to generate a fourth matrix parameter, wherein the number of columns of the third matrix parameter is the same as the number of columns of the first matrix parameter, and the number of rows of the fourth matrix parameter is the same as the number of rows of the second matrix parameter; The first node completes the matrix multiplication task according to a matrix multiplication result of the third matrix parameter and the fourth matrix parameter.

7. The method according to claim 5 or 6, characterized in that The first matrix parameters include activation values, and the second matrix parameters include weights; or, The first matrix parameter includes a matrix multiplication result of the activation value and the first weight, and the second matrix parameter includes the second weight.

8. The method according to any one of claims 1 to 7, characterized in that: If the matrix multiplication task is a task of the model in the forward calculation phase, the collective communication includes global aggregation allgather communication; If the matrix multiplication task is a task of the model in the back-propagation stage, the collective communication includes reduce-scatter communication.

9. A model training system, characterized in that: The system is applied to a training system, the training system including multiple nodes, multiple network planes being established between the multiple nodes through different communication modes, and the system including: a configuration unit, configured to obtain a mapping relationship between communication data and a network plane, wherein the first node is any node among the plurality of nodes; a processing unit for obtaining cut data of matrix parameters, wherein the matrix parameters are multipliers in a matrix multiplication task in model training; The processing unit is configured to train a model based on the cutting data to obtain communication data corresponding to the matrix parameters, where the communication data is data to be transmitted from the first node to other nodes; a communication unit, configured to perform collective communication with the other nodes based on the mapping relationship, and obtain communication data of the matrix parameters on each node, wherein communication data corresponding to different matrix parameters use different network planes for collective communication; The processing unit is used to complete the matrix multiplication task according to the communication data of the matrix parameters on each node, thereby completing the training of the model.

10. The system according to claim 9, characterized in that The training system includes multiple communication groups, each communication group includes multiple nodes, a first network plane is established between the multiple nodes in the communication group through a first communication method, and a second network plane is established between the communication groups through a second communication method.

11. The system according to claim 9 or 10, characterized in that The system includes a cutting unit; The cutting unit is used to perform at least one row cutting and at least one column cutting on the matrix parameter to obtain a plurality of cutting data of the matrix parameter, where the number of the cutting data is the same as the number of nodes of the training system; The cutting unit is configured to distribute the plurality of cutting data of the first node to other nodes, so that each node obtains one cutting data of the matrix parameter.

12. The system according to claim 11, characterized in that The number of row cuts is determined according to the number of communication groups, and the number of column cuts is determined according to the number of nodes in the communication group; or, the number of row cuts is determined according to the number of nodes in the communication group, and the number of column cuts is determined according to the number of communication groups.

13. The system according to any one of claims 9 to 12, characterized in that: The matrix parameters include a first matrix parameter and a second matrix parameter, the first matrix parameter and the second matrix parameter are a left multiplier and a right multiplier in the same matrix multiplication task, and the communication data includes first communication data corresponding to the first matrix parameter and second communication data corresponding to the second matrix parameter; If the number of column cuts of the matrix parameter is determined according to the number of the communication groups, the first communication data corresponds to the second network plane, and the second communication data corresponds to the first network plane; If the number of column cuts of the matrix parameter is determined according to the number of the communication groups, the first communication data corresponds to the first network plane, and the second communication data corresponds to the second network plane.

14. The system according to claim 13, wherein: The processing unit is configured to concatenate the first communication data of the first matrix parameters on each node to obtain third matrix parameters, and concatenate the second communication data of the second matrix parameters on each node to generate fourth matrix parameters, wherein the number of columns of the third matrix parameters is the same as the number of columns of the first matrix parameters, and the number of rows of the fourth matrix parameters is the same as the number of rows of the second matrix parameters; The processing unit is used to complete the matrix multiplication task according to the matrix multiplication result of the third matrix parameter and the fourth matrix parameter.

15. The system according to claim 13 or 14, characterized in that The first matrix parameters include activation values, and the second matrix parameters include weights; or, The first matrix parameter includes a matrix multiplication result of the activation value and the first weight, and the second matrix parameter includes the second weight.

16. The system according to any one of claims 9 to 15, characterized in that If the matrix multiplication task is a task of the model in the forward calculation phase, the collective communication includes global aggregation allgather communication; If the matrix multiplication task is a task of the model in the back-propagation stage, the collective communication includes reduce-scatter communication.

17. A computing device, characterized in that The computing device includes a processor and a memory, wherein the memory is used to store instructions, and the processor is used to execute the instructions, so that the computing device implements the method according to any one of claims 1 to 8.

18. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, and when the instructions are executed by a computing device or a computing device cluster, the method according to any one of claims 1 to 8 is implemented.

Images