Model training method and device, electronic equipment, storage medium and computer program product

Abstract

The embodiment of the invention provides a model training method, a model training device, electronic equipment, a chip, a storage medium and a computer program product, and the method comprises the steps: training a first model through a plurality of computing power islands, and obtaining a plurality of first weight parameters; the plurality of computing power islands are in one-to-one correspondence with the plurality of first weight parameters; wherein the computing power island comprises one or more artificial intelligence (AI) acceleration cards; the AI acceleration cards in one computing power island belong to the same manufacturer; and updating the parameters of the first model based on the plurality of first weight parameters.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of large model technology, and specifically to a model training method, a model training device, an electronic device, a chip, a storage medium, and a computer program product. Background Art

[0002] The development and training of large language models requires powerful computing power as a necessary support. The construction of large-scale computing facilities requires a large number of artificial intelligence (AI) accelerator cards, which are generally rented from cloud service providers or purchased independently. Cloud service provider procurement or independent procurement is often implemented in batches. Due to the rapid update and iteration speed of AI accelerator cards, the entire computing facility center must have heterogeneous AI accelerator cards coexisting. The computing power, inter-card bandwidth, video memory bandwidth, driver, runtime, and inter-card communication protocol between heterogeneous AI accelerator cards are often different, which brings great difficulties to the scheduling and execution of the deep learning framework required for large model training, and the inter-card collective communication operations. Summary of the invention

[0003] Embodiments of the present application provide a model training method, a model training device, an electronic device, a chip, a storage medium, and a computer program product.

[0004] The model training method provided in the embodiment of the present application includes:

[0005] A first model is trained by multiple computing power islands to obtain multiple first weight parameters; the multiple computing power islands correspond to the multiple first weight parameters one by one; wherein the computing power islands include: one or more artificial intelligence AI accelerator cards; the AI ​​accelerator cards in one computing power island belong to the same manufacturer;

[0006] Based on the plurality of first weight parameters, parameters of the first model are updated.

[0007] The model training device provided in the embodiment of the present application includes:

[0008] A processing unit: used to train a first model through a plurality of computing power islands to obtain a plurality of first weight parameters; the plurality of computing power islands correspond one to one with the plurality of first weight parameters; wherein the computing power islands include: one or more artificial intelligence AI accelerator cards; the AI ​​accelerator cards in a computing power island belong to the same manufacturer;

[0009] The processing unit is further configured to update the parameters of the first model based on the multiple first weight parameters.

[0010] The electronic device provided in an embodiment of the present application includes: a processor and a memory, the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute any model training method provided in the embodiment of the present application.

[0011] The chip provided in the embodiment of the present application includes: a processor, which is used to call and run a computer program from a memory, so that a device equipped with the chip executes any model training method provided in the embodiment of the present application.

[0012] The storage medium provided in the embodiments of the present application is used to store a computer program, and the computer program enables a computer to execute any model training method provided in the embodiments of the present application.

[0013] The computer program product provided in the embodiments of the present application includes a computer program, which, when executed by a processor, implements any model training method provided in the embodiments of the present application.

[0014] Through the model training method provided in the embodiment of the present application, AI accelerator cards from the same manufacturer are arranged in the same computing power island, and the first model is trained collaboratively through multiple computing power islands. The deep learning framework API, driver, protocol stack and runtime environment of the AI ​​accelerator cards from the same manufacturer are often the same or compatible, which can shield the differences in inter-card communication protocols, drivers, and runtime libraries of AI accelerator card products from different manufacturers to the greatest extent, thereby facilitating the scheduling and operation of the upper-level deep learning framework program, better coordinating the use of relevant computing power resources, and realizing distributed training of large models. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0016] Figure 1 A schematic diagram of the implementation flow of the model training method provided in the embodiment of the present application;

[0017] Figure 2 A schematic diagram of the network provided in the embodiment of the present application;

[0018] Figure 3 A schematic diagram of the networking communication protocol provided in the embodiment of the present application;

[0019] Figure 4 A schematic structural diagram of a model training device 400 provided in an embodiment of the present application;

[0020] Figure 5 A schematic structural diagram of an electronic device provided in an embodiment of the present application;

[0021] Figure 6 A schematic structural diagram of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0023] It should be noted that in the embodiments of the present application, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the embodiments of the present application, the character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0024] In the description of the embodiments of the present application, the term "corresponding" may indicate a direct or indirect correspondence between two items, or an association relationship between the two items, or a relationship between indication and being indicated, configuration and being configured, and the like.

[0025] To facilitate understanding of the technical solutions of the embodiments of the present application, the relevant technologies of the embodiments of the present application are described below. The following related technologies can be arbitrarily combined with the technical solutions of the embodiments of the present application as optional solutions, and they all belong to the protection scope of the embodiments of the present application.

[0026] The development and training of large language models requires powerful computing power as necessary support. According to incomplete statistics, the development of large language models requires tens of thousands or even more high-end AI accelerator cards for related training. The computing facilities of the above scale are generally rented from cloud service providers or purchased independently; cloud service providers purchase or purchase independently and are often built in batches. With the acceleration of product upgrades and iterations of AI accelerator card manufacturers, they are bound to face the upgrade of product models of AI accelerator cards purchased in different batches, which is bound to bring a new problem, namely: in order to protect expensive investments and introduce competition, the entire computing power facility must have the reality of coexistence of heterogeneous AI accelerator cards. Heterogeneous AI accelerator cards refer to different models of AI accelerator cards from the same manufacturer, or AI accelerator cards from different manufacturers; generally speaking, the computing power, inter-card bandwidth, and video memory bandwidth between heterogeneous AI accelerator cards are often different; and for AI accelerator cards from different manufacturers, the card drivers, runtime libraries, and inter-card communication protocols are also different. Heterogeneous AI accelerator cards bring many challenges to large model training, such as: how to effectively utilize AI accelerator card resources, how to implement communication protocols, runtime library compatibility, etc.

[0027] For computing power infrastructure, there are usually different models of AI accelerator cards from different domestic and foreign manufacturers. These different AI accelerator cards from different manufacturers are incompatible in inter-card communication protocols, inter-card communication network technologies, drivers, protocol stacks, and runtime environments. The significant differences between these heterogeneous AI accelerator cards have brought great difficulties to the scheduling and execution of deep learning frameworks and inter-card collective communication operations required for large model training. How to effectively train large models is a new and urgent problem that needs to be solved.

[0028] Because the number of parameters in large models is huge, and has now reached a trillion parameter scale, the memory space required for AI accelerator cards of this scale will reach the TeraBytes level; for this level of memory demand, obviously no AI accelerator card can meet it, so we can only adopt a distributed training strategy and rely on the collective resources of the entire computing power facility to complete the corresponding training. On the other hand, in order to make full use of the computing power FLOPS of various heterogeneous AI accelerator cards and maximize the shielding of differences in inter-card bandwidth, inter-card communication protocols, underlying libraries, protocol stacks, and runtime environments of heterogeneous AI accelerator cards from different manufacturers at home and abroad, it is necessary to build a new large model distributed training strategy.

[0029] Figure 1 A schematic diagram of the implementation flow of the model training method provided in the embodiment of the present application, such as Figure 1 As shown, the embodiment of the present application provides a model training method, which includes the following steps:

[0030] Step 101: Train a first model through multiple computing power islands to obtain multiple first weight parameters; the multiple computing power islands correspond one-to-one to the multiple first weight parameters; wherein the computing power islands include: one or more artificial intelligence AI acceleration cards; the AI ​​acceleration cards in one computing power island belong to the same manufacturer.

[0031] In the embodiment of the present application, the first model may specifically be a large language model or other models, and the embodiment of the present application does not limit this.

[0032] In the embodiment of the present application, the computing power island includes one or more artificial intelligence AI acceleration cards, which is a computing power cluster. In actual application, the computing power island can also be called by other names, such as computing partitions in data centers, high-performance computing clusters, computing resource pools, etc. The embodiment of the present application does not limit this.

[0033] AI accelerator cards from the same manufacturer are arranged in the same computing power island, and the first model is trained collaboratively through multiple computing power islands. The deep learning framework API, driver, protocol stack, and runtime environment of AI accelerator cards from the same manufacturer are often the same or compatible, which can shield the differences in inter-card communication protocols, drivers, and runtime libraries of AI accelerator card products from different manufacturers to the greatest extent.

[0034] In the embodiment of the present application, within the computing power island, one or more AI accelerator cards can be further grouped according to the model to form one or more computing power groups. AI accelerator cards of the same model are in the same group, which shields the differences in computing power, inter-card bandwidth, and video memory bandwidth of AI accelerator cards of different models, making the implementation of inter-card communication simple and reliable. Here, the computing power group includes one or more AI accelerator cards of the same model, which is a computing power cluster. In actual application, the computing power group can also be called by other names, such as Pod, high-performance computing cluster, computing resource pool, etc., which is not limited by the embodiment of the present application.

[0035] refer to Figure 2 , Figure 2 The network diagram provided in the embodiment of the present application is as follows: Figure 2 As shown, the network includes multiple computing islands, each computing island includes multiple Pods, each pod includes multiple AI servers, and the AI ​​server is configured with one or more AI accelerator cards. In the same pod, the AI ​​accelerator cards of different AI servers can communicate with each other through a short-distance high-speed network. Each Pod is configured with a secondary server, and the AI ​​accelerator card in the pod can communicate with the secondary server configured in the pod through a medium- and short-distance high-speed network. The secondary servers configured in different pods can communicate with each other through a medium- and short-distance high-speed network. Each computing island is configured with a primary server, and the secondary server in the computing island can communicate with the primary server configured in the computing island through a medium- and short-distance high-speed network. The primary servers configured in each computing island can communicate with each other through a medium- and long-distance high-speed network. In this way, a computing network is formed.

[0036] In the embodiment of the present application, for a computing power infrastructure including different heterogeneous types of AI accelerator cards, the AI ​​servers are divided into different logical subsets according to the manufacturers and models of the AI ​​accelerator cards contained therein. Servers with AI accelerator cards of the same manufacturer and the same model constitute a logical subset, forming a Pod; the AI ​​servers in the same Pod are interconnected through a 10Gbps~25Gbps~100Gbps~200Gbps optical fiber network. In actual application, the network bandwidth depends on the inter-card communication network technology supported by the AI ​​accelerator card product; several Pods formed by AI accelerator cards of the same manufacturer constitute a computing power island; a computing power island contains AI accelerator cards of the same manufacturer, and different Pods in the computing power island respectively contain AI accelerator cards of different models from the same manufacturer; if there are M AI accelerator cards from different manufacturers in the computing power infrastructure, it contains M computing power islands, each Each Pod is equipped with a secondary server; each computing island is equipped with a primary server; the secondary servers within the computing island and the primary parameter coordination server of the computing island can be interconnected through a 10Gbps-100Gbps high-speed optical fiber network; the primary servers between the computing islands are interconnected through a 10Gbps-100Gbps high-speed optical fiber network. Considering that the AI ​​accelerator card server equipped with an AI accelerator card consumes a lot of power, a computing infrastructure that supports large-scale model training with hundreds of billions or trillions of parameters usually contains tens of thousands or even more AI accelerator cards. Therefore, the power consumption of the entire computing infrastructure is very huge. Therefore, each computing island can be set up on different floors of the data center, or even in data centers in different locations; the same computing island can be set up in the same data center, and a single Pod can be set up in the same room.

[0037] Based on this, in an optional implementation of the present application, each of the multiple computing power islands is configured with a primary server and one or more secondary servers; one or more secondary servers in a computing power island correspond one to one with one or more computing power groups; wherein,

[0038] The primary server is used to communicate with the secondary server in the computing island corresponding to the primary server and the primary servers in other computing islands;

[0039] The secondary server is used to communicate with the AI ​​acceleration card in the computing power group corresponding to the secondary server and to communicate with other secondary servers in the computing power island to which the computing power group corresponding to the secondary server belongs.

[0040] In the embodiment of the present application, the primary server and the secondary server may adopt parameter coordination servers or other types of servers, and the embodiment of the present application does not limit this.

[0041] In an embodiment of the present application, a data parallel training method can be adopted between multiple computing power islands. The data parallel method divides the entire training data set into several training data subsets, each sub-training data set is assigned to a computing power island, each computing power island has a complete model copy, and each computing power island is trained on the assigned sub-training data set; after the model is back-propagated, the gradient parameters of the model will be ALL-Reduce operated between all related computing power islands to obtain the final model parameters. For example, multiple computing power islands have a complete copy of the first model structure; at the same time, the training data is allocated according to the resources of each computing power island, the number of computing power islands and other indicators. The data set is divided into several training data subsets and allocated to each computing island in the multiple computing islands; here, the resources include AI accelerator card computing resources, inter-card communication bandwidth, card memory bandwidth, CPU computing resources of the artificial intelligence server, host memory capacity, etc.; each computing island trains the first model based on its own allocated training data subset. When all training data subsets are calculated, the first-level server in each computing island has some weight parameters of the first model trained by its training data subset, that is, the first weight parameters; then, all the first-level servers of the multiple computing islands perform ALL-Reduce operations of the message passing interface (MPI) of their respective first weight parameters; after the ALL_Reduce operation is completed, all the first-level servers of the multiple computing islands will obtain the complete weight parameter set of the first model for all training data sets, and update the parameters of the first model according to the complete weight parameter set, thereby finally completing the training task of the first model. The data parallel training method can reduce the amount of communication between cards and improve the training efficiency.

[0042] In an embodiment of the present application, on the basis of the data parallel model, an expert parallel training mechanism can also be added. The expert parallel training method is to replace the feedforward network layer in the large language model with a mixture of experts (Mixture of Expert, MoE) layer to achieve sparse activation of the large language model training, thereby achieving support for large model training with larger model structure, larger parameter scale and larger training data set on the basis of the same computing power, thereby improving the quality of the large model; the MoE layer is composed of a gating network and several experts. The gating network determines to which expert each Token of the training set is sent to complete precise training. Exemplarily, on the basis of data parallel training between computing power islands, each of the multiple computing power islands contains several unique experts, and different experts activate the weight parameters of the model to be trained corresponding to different input training data Tokens, thereby activating the weight parameters of the previous training full model to be trained as the weight parameters of the training specific model to be trained, that is, changing the dense operation to the sparse operation. During the training process, the tokens in the training data subset allocated to each computing island are first sent to which expert through the expert parallel method, and the gating network located in each computing island decides which expert to send them to. The expert decides which weight parameters to be trained are activated by the input tokens. During the entire training process, the experts of each computing island perform MPI ALL-to-ALL operations with each other to realize data sharing among experts, so that each expert can use sufficient data training.

[0043] Based on this, in an optional implementation manner of the present application, training the first model through multiple computing power islands includes:

[0044] The first model is trained by using the multiple computing power islands in a dual parallel mode of data parallelism and expert parallelism.

[0045] In an embodiment of the present application, within a computing power island, the first model can be distributedly trained by one or more computing power groups. After the training of each computing power group is completed, the second weight parameters after the training of each computing power group are aggregated to obtain the first weight parameters of the first model trained by the computing power island.

[0046] Based on this, in an optional implementation of the present application, for each computing power island in the multiple computing power islands, the AI ​​accelerator cards in the computing power islands form one or more computing power groups, and the models of the AI ​​accelerator cards in a computing power group are the same. The first model is trained by multiple computing power islands to obtain multiple first weight parameters, including:

[0047] For each computing power island among the multiple computing power islands, training the first model through one or more computing power groups in the computing power island to obtain one or more second weight parameters;

[0048] Based on one or more second weight parameters corresponding to each of the multiple computing power islands, a first weight parameter corresponding to each of the multiple computing power islands is obtained.

[0049] In an embodiment of the present application, in a computing power island, multiple computing power groups can be used to train the first model in a pipeline parallel mode. Pipeline parallelism is to divide the model into several model blocks by layer, and each model block is handed over to a computing power group for training. Exemplarily, each computing power island has a complete copy of the first model structure. According to the idea of ​​pipeline parallelism, according to the resources of each computing power group in the computing power island, the number of computing power groups and other indicators, the first model structure is divided into several parallel executable model blocks according to the standard of parallel execution. Here, the resources of the computing power group include AI accelerator card computing power resources, inter-card communication bandwidth, card memory bandwidth, CPU computing resources of the artificial intelligence server, host memory capacity, etc.; each model block is assigned to a computing power group in the computing power island for execution, and the resources required for training the model block are matched according to the resource status of each computing power group to make full use of the resources of each computing power group, while ensuring a shorter training time for the model block; each computing power group performs corresponding model block training for the training data subset allocated to the computing power island to which it belongs; the third weight parameter obtained by the training of each computing power group is MPI-trained between all AI accelerator cards in the computing power group. ALL-Reduce operation is performed to obtain the second weight parameter corresponding to the computing power group; thereafter, the AI ​​server where the AI ​​accelerator card with Local_Rank=0 in the computing power group is located synchronizes or peer-to-peer the second weight parameter after the ALL_Reduce execution is completed to the secondary server in the computing power group; when all computing power groups in the computing power island have completed the training tasks of their assigned model blocks, the second weight parameters of all secondary servers in the computing power island are ALL_Reduce operated to obtain the first weight parameter, so that all secondary servers have the first weight parameter of the first model for the sub-training data set; at the same time, the first weight parameter is synchronized or peer-to-peer to the primary server of the computing power island by the secondary server No. 0 in the computing power island.

[0050] Based on this, in an optional implementation manner of the present application, the training of the first model by one or more computing power groups in the computing power island to obtain one or more second weight parameters includes:

[0051] For each of the one or more computing power groups in a computing power island, training the first model through one or more AI accelerator cards in the computing power group to obtain one or more third weight parameters;

[0052] Based on one or more third weight parameters corresponding to each computing power group in one or more computing power groups in a computing power island, a second weight parameter corresponding to each computing power group in one or more computing power groups in a computing power island is obtained.

[0053] In an embodiment of the present application, within the computing power group, the tensor parallel mode can be used to train the first model. Tensor parallelism is to divide a tensor into N blocks along a specific dimension. Each AI accelerator card holds 1 / N of the entire tensor without affecting the correctness of the calculation graph. Considering that tensor parallelism will generate frequent and large-scale All-Reduce operation requirements between accelerator cards, the inter-card communication volume is the largest compared to other parallel training methods; in an embodiment of the present application, the AI ​​accelerator cards in a Pod are AI accelerator cards of the same model, and the computing power, inter-card communication protocol, driver, and runtime library of each accelerator card are the same. The inter-card Internet network rate PCIe4.0 is 32GB / s, and NVLINK2.0 is 300GB / s, the Internet rate between cards is the highest and the bandwidth is the largest. The operation scheduling of the large model training framework is unified and convenient. Therefore, the tensor parallel training mechanism can be used in each computing power group to ensure the training efficiency. For example, after the pipeline parallel division of several model blocks of the first model structure in each computing power group is performed in each computing power island, the tensor parallel method is used for the model block in each computing power group, and the main tensor is selected and divided into several sub-tensors along a specific dimension. Each sub-tensor is assigned to an AI accelerator card in the computing power group; then, multiple AI accelerator cards in the same computing power group perform corresponding training through tensor parallelism. During the training process, frequent and large-scale MPI ALL-Reduce operations will be performed between the AI ​​accelerator cards. Finally, each AI accelerator card in the computing power group will have a copy of the parameter weight of the model block, corresponding to the second weight parameter; finally, the second weight parameter of the AI ​​accelerator card with Local_Rank=0 in the computing power group is synchronized or peer-to-peer to the secondary server corresponding to the computing power group.

[0054] Based on this, in an optional implementation manner of the present application, training the first model by one or more computing power groups in the computing power island includes:

[0055] Using one or more computing groups in the computing island, training the first model in a pipeline parallel mode;

[0056] The step of training the first model by using one or more AI acceleration cards in the computing power group includes:

[0057] The first model is trained in tensor parallel mode using one or more AI accelerator cards in the computing power group.

[0058] Step 102: Based on the multiple first weight parameters, update the parameters of the first model.

[0059] Taking into account that AI accelerator card products from different manufacturers at home and abroad support different inter-card communication protocols, this application introduces a server, where the server includes a primary server and a secondary server. On the one hand, the server supports the communication protocol of the AI ​​accelerator card with which it communicates, and on the other hand, it supports another standard communication protocol, so that the servers can communicate with each other. The inter-card communication protocol of the AI ​​accelerator card and the communication protocol of the server itself serve as the back-end protocol for implementing the collective communication MPI operation. With the help of the server, large-model distributed training based on heterogeneous AI accelerator card clusters from different manufacturers can be realized.

[0060] The secondary server located in a separate computing group supports the communication protocols and collective communications library (CCL) that are compatible or supported by the AI ​​accelerator card in the computing group. When the training task of each computing group is completed, the secondary server synchronizes or peers with the artificial intelligence server where the AI ​​accelerator card with Local_Rank=0 in the computing group to which it belongs, and the second weight parameter trained by the computing group is synchronized or peered. In this way, the secondary server has the second weight parameter trained by the computing group. In actual application, the second weight parameter can also be synchronized or peered to the secondary server through other AI accelerator cards.

[0061] The first-level server located in a separate computing power island, on the one hand, supports the communication protocol and collective communication library that are compatible or supported by the AI ​​accelerator card in the computing power island where it is located; on the other hand, a unified communication protocol and collective communication library are used between the first-level servers of all computing power islands. When all computing power groups in a computing power island complete the training task, the second weight parameters between all secondary servers in the computing power island are ALL_Reduce operated, so that all secondary servers finally have the first weight parameter of the large model to be trained for the training data subset that the computing power island is responsible for; at the same time, the first weight parameter is synchronized or peer-to-peer to the first-level server of the computing power island by the second-level server No. 0 in the computing power island. In actual application, other second-level servers can also be used to synchronize or peer-to-peer to the first-level server of the computing power island. The embodiment of the present application does not limit this; when the training tasks of each computing power island in the entire computing power infrastructure are completed, all first-level servers perform ALL-Reduce operations on their respective first weight parameters, so that all first-level servers finally have the complete weight parameter set of the large model for all training data sets, thereby finally completing the training task of the entire large model. like Figure 3 As shown, Figure 3A schematic diagram of the networking communication protocol provided in the embodiment of the present application, in which the AI ​​servers in the same computing power group communicate with each other through the same manufacturer's communication protocol, the AI ​​servers in the same computing power group communicate with the secondary servers corresponding to the computing power group through the same manufacturer's communication protocol, the secondary servers in the same computing power island communicate with the primary servers corresponding to the computing power island through the same manufacturer's communication protocol, and the primary servers between different computing power islands communicate through a unified third-party communication protocol.

[0062] Based on this, in an optional implementation of the present application, the second weight parameter corresponding to each computing power group in one or more computing power groups in a computing power island is obtained based on one or more third weight parameters corresponding to each computing power group in one or more computing power groups in a computing power island, including:

[0063] Perform an ALL_Reduce operation on one or more third weight parameters corresponding to each computing power group in one or more computing power groups in a computing power island to obtain a second weight parameter corresponding to each computing power group;

[0064] The obtaining, based on one or more second weight parameters corresponding to each of the plurality of computing power islands, a first weight parameter corresponding to each of the plurality of computing power islands, comprises:

[0065] Performing an ALL_Reduce operation on one or more second weight parameters corresponding to each of the multiple computing power islands to obtain a first weight parameter corresponding to each of the multiple computing power islands;

[0066] Based on the plurality of first weight parameters, updating the parameters of the first model comprises:

[0067] An ALL_Reduce operation is performed on the multiple first weight parameters to obtain a complete weight parameter set, and the parameters of the first model are updated according to the complete weight parameter set.

[0068] The model training method provided in the embodiment of the present application proposes a computing power networking method for a computing power cluster including AI accelerator card products from multiple manufacturers at home and abroad, in which the computing power of AI accelerator cards of the same model from the same manufacturer is networked into a logical computing power group. Multiple computing power groups composed of AI accelerator card products from the same manufacturer form a logical computing power island, and AI accelerator card products from different manufacturers constitute multiple computing power islands. Based on the above computing power division method, the computing power, video memory capacity, video memory bandwidth, inter-card communication protocol, inter-card communication network technology, driver, and runtime library differences of AI accelerator card products from different manufacturers can be shielded to the greatest extent, thereby facilitating upper-level deep learning. The scheduling and operation of the learning framework program can be coordinated and utilized better for the relevant computing resources. By setting up a secondary server in the computing group and a primary server in the computing island, on the one hand, the collective communication needs between heterogeneous AI accelerator cards from different manufacturers between different computing islands are realized, and on the other hand, the scalability of the cluster scale of various types of heterogeneous AI accelerator cards is improved, and the maintainability of the entire computing infrastructure is enhanced. In addition, by comprehensively adopting a variety of parallel training methods, the computing resources of different heterogeneous AI accelerator cards can be utilized as much as possible, the accuracy convergence speed of large models can be accelerated, and the training time of large models can be reduced. At the same time, it can also meet the training of large models with larger parameter scales.

[0069] The present application embodiment also provides a model training device 400, referring to Figure 4 The model training device 400 in this embodiment includes:

[0070] A processing unit: used to train a first model through a plurality of computing power islands to obtain a plurality of first weight parameters; the plurality of computing power islands correspond to the plurality of first weight parameters one by one; wherein the computing power islands include: one or more artificial intelligence AI accelerator cards; the AI ​​accelerator cards in a computing power island belong to the same manufacturer;

[0071] The processing unit is further configured to update the parameters of the first model based on the multiple first weight parameters.

[0072] In an embodiment of the present application, for each computing power island in the multiple computing power islands, the AI ​​accelerator cards in the computing power island form one or more computing power groups, and the models of the AI ​​accelerator cards in a computing power group are the same; the processing unit is used to train the first model for each computing power island in the multiple computing power islands through one or more computing power groups in the computing power island to obtain one or more second weight parameters; based on the one or more second weight parameters corresponding to each computing power island in the multiple computing power islands, obtain the first weight parameter corresponding to each computing power island in the multiple computing power islands.

[0073] In an embodiment of the present application, the processing unit is used to train the first model for each of the one or more computing power groups in a computing power island through one or more AI acceleration cards in the computing power group to obtain one or more third weight parameters; based on the one or more third weight parameters corresponding to each of the one or more computing power groups in a computing power island, obtain the second weight parameters corresponding to each of the one or more computing power groups in a computing power island.

[0074] In an embodiment of the present application, the processing unit is used to perform an ALL_Reduce operation on one or more third weight parameters corresponding to each computing power group in one or more computing power groups within a computing power island to obtain a second weight parameter corresponding to each computing power group; perform an ALL_Reduce operation on one or more second weight parameters corresponding to each computing power island in the multiple computing power islands to obtain a first weight parameter corresponding to each computing power island in the multiple computing power islands; perform an ALL_Reduce operation on the multiple first weight parameters to obtain a complete weight parameter set, and update the parameters of the first model according to the complete weight parameter set.

[0075] In an embodiment of the present application, each of the multiple computing power islands is configured with a primary server and one or more secondary servers; one or more secondary servers in a computing power island correspond one-to-one to one or more computing power groups; wherein the primary server is used to communicate with the secondary server in the computing power island corresponding to the primary server and to communicate with the primary servers in other computing power islands; the secondary server is used to communicate with the AI ​​acceleration card in the computing power group corresponding to the secondary server and to communicate with other secondary servers in the computing power island to which the computing power group corresponding to the secondary server belongs.

[0076] In an embodiment of the present application, the processing unit is used to train the first model through the multiple computing power islands in a dual parallel mode of data parallelism and expert parallelism.

[0077] In an embodiment of the present application, the processing unit is used to train the first model in a pipeline parallel mode through one or more computing power groups in the computing power island; and to train the first model in a tensor parallel mode through one or more AI acceleration cards in the computing power group.

[0078] Those skilled in the art should understand that Figure 4 The implementation functions of each unit in the model training device 400 shown can be understood by referring to the relevant description of the aforementioned method. Figure 4 The functions of each unit in the model training device 400 shown can be implemented by a program running on a processor, or by a specific logic circuit.

[0079] Figure 5It is a schematic structural diagram of an electronic device 500 provided in an embodiment of the present application. Figure 5 The electronic device 500 shown includes a processor 510, which can call and run a computer program from a memory to implement the method in the embodiment of the present application.

[0080] Alternatively, if Figure 5 As shown, the electronic device 500 may further include a memory 520. The processor 510 may call and run a computer program from the memory 520 to implement the method in the embodiment of the present application.

[0081] The memory 520 may be a separate device independent of the processor 510 , or may be integrated into the processor 510 .

[0082] Alternatively, if Figure 5 As shown, the electronic device 500 may further include a transceiver 530, and the processor 510 may control the transceiver 530 to communicate with other devices, specifically, may send information or data to other devices, or receive information or data sent by other devices.

[0083] The transceiver 530 may include a transmitter and a receiver. The transceiver 530 may further include an antenna, and the number of the antennas may be one or more.

[0084] The electronic device 500 may specifically be the model training device 400 of the embodiment of the present application, and the electronic device 500 may implement the corresponding processes implemented by the model training device 400 in each method of the embodiment of the present application, which will not be described in detail here for the sake of brevity.

[0085] Illustratively, an embodiment of the present application further provides a computer program product, including a computer program, which can be executed by the processor 510 of the electronic device 500 to complete the steps of any of the aforementioned methods.

[0086] Figure 6 It is a schematic structural diagram of the chip of an embodiment of the present application. Figure 6 The chip 600 shown includes a processor 610, which can call and run a computer program from a memory to implement the method in the embodiment of the present application.

[0087] Alternatively, if Figure 6 As shown, the chip 600 may further include a memory 620. The processor 610 may call and run a computer program from the memory 620 to implement the method in the embodiment of the present application.

[0088] The memory 620 may be a separate device independent of the processor 610 , or may be integrated into the processor 610 .

[0089] Optionally, the chip 600 may further include an input interface 630. The processor 610 may control the input interface 630 to communicate with other devices or chips, and specifically, may obtain information or data sent by other devices or chips.

[0090] Optionally, the chip 600 may further include an output interface 640. The processor 610 may control the output interface 640 to communicate with other devices or chips, and specifically, may output information or data to other devices or chips.

[0091] The chip can be applied to the electronic device 500 in the embodiment of the present application, and the chip can implement the corresponding processes implemented by the electronic device 500 in the various methods of the embodiment of the present application, which will not be described in detail here for the sake of brevity.

[0092] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0093] It should be understood that the processor of the embodiment of the present application may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by the hardware integrated logic circuit or software instructions in the processor. The above processor can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor are combined and performed. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, and other mature storage media in the art. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0094] It can be understood that the memory in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0095] It should be understood that the above-mentioned memory is exemplary but not restrictive. For example, the memory in the embodiments of the present application may also be static random access memory (static RAM, SRAM), dynamic random access memory (dynamic RAM, DRAM), synchronous dynamic random access memory (synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (synch link DRAM, SLDRAM) and direct memory bus random access memory (Direct Rambus RAM, DR RAM), etc. That is to say, the memory in the embodiments of the present application is intended to include but not limited to these and any other suitable types of memory.

[0096] The embodiment of the present application further provides a storage medium for storing a computer program. The storage medium can be applied to the electronic device 500 in the embodiment of the present application, and the computer program enables the computer to execute the corresponding processes implemented by the electronic device 500 in each method of the embodiment of the present application, which will not be described in detail for the sake of brevity.

[0097] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0098] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0099] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0100] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0101] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0102] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or an electronic device 500, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0103] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A model training method, characterized in that: include: Training a first model through multiple computing power islands to obtain multiple first weight parameters; The multiple computing power islands correspond to the multiple first weight parameters one by one; wherein the computing power island includes: one or more artificial intelligence AI acceleration cards; the AI ​​acceleration cards in one computing power island belong to the same manufacturer; Based on the plurality of first weight parameters, parameters of the first model are updated.

2. The method according to claim 1, characterized in that For each computing power island in the multiple computing power islands, the AI ​​accelerator cards in the computing power islands form one or more computing power groups, and the models of the AI ​​accelerator cards in a computing power group are the same; the first model is trained by the multiple computing power islands to obtain multiple first weight parameters, including: For each computing power island among the multiple computing power islands, training the first model through one or more computing power groups in the computing power island to obtain one or more second weight parameters; Based on one or more second weight parameters corresponding to each of the multiple computing power islands, a first weight parameter corresponding to each of the multiple computing power islands is obtained.

3. The method according to claim 2, characterized in that The step of training the first model by using one or more computing power groups in the computing power island to obtain one or more second weight parameters includes: For each of the one or more computing power groups in a computing power island, training the first model through one or more AI accelerator cards in the computing power group to obtain one or more third weight parameters; Based on one or more third weight parameters corresponding to each computing power group in one or more computing power groups in a computing power island, a second weight parameter corresponding to each computing power group in one or more computing power groups in a computing power island is obtained.

4. The method according to claim 3, characterized in that The obtaining, based on one or more third weight parameters corresponding to each computing power group in one or more computing power groups in a computing power island, a second weight parameter corresponding to each computing power group in one or more computing power groups in a computing power island includes: Perform an ALL_Reduce operation on one or more third weight parameters corresponding to each computing power group in one or more computing power groups in a computing power island to obtain a second weight parameter corresponding to each computing power group; The obtaining, based on one or more second weight parameters corresponding to each of the plurality of computing power islands, a first weight parameter corresponding to each of the plurality of computing power islands, comprises: Performing an ALL_Reduce operation on one or more second weight parameters corresponding to each of the multiple computing power islands to obtain a first weight parameter corresponding to each of the multiple computing power islands; Based on the plurality of first weight parameters, updating the parameters of the first model comprises: An ALL_Reduce operation is performed on the multiple first weight parameters to obtain a complete weight parameter set, and the parameters of the first model are updated according to the complete weight parameter set.

5. The method according to any one of claims 2 to 4, characterized in that: Each of the plurality of computing power islands is configured with a primary server and one or more secondary servers; one or more secondary servers in a computing power island correspond one to one with one or more computing power groups; wherein, The primary server is used to communicate with the secondary server in the computing island corresponding to the primary server and the primary servers in other computing islands; The secondary server is used to communicate with the AI ​​acceleration card in the computing power group corresponding to the secondary server and to communicate with other secondary servers in the computing power island to which the computing power group corresponding to the secondary server belongs.

6. The method according to claim 5, characterized in that The step of training the first model by using a plurality of computing power islands includes: The first model is trained by using the multiple computing power islands in a dual parallel mode of data parallelism and expert parallelism.

7. The method according to claim 6, characterized in that The step of training the first model by using one or more computing power groups in the computing power island includes: Using one or more computing groups in the computing island, training the first model in a pipeline parallel mode; The step of training the first model by using one or more AI acceleration cards in the computing power group includes: The first model is trained in tensor parallel mode using one or more AI accelerator cards in the computing power group.

8. A model training device, characterized in that: include: Processing unit: used for training the first model through multiple computing power islands to obtain multiple first weight parameters; The multiple computing power islands correspond to the multiple first weight parameters one by one; wherein the computing power island includes: one or more artificial intelligence AI acceleration cards; the AI ​​acceleration cards in one computing power island belong to the same manufacturer; The processing unit is further configured to update the parameters of the first model based on the multiple first weight parameters.

9. An electronic device, characterized in that: include: A processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory to execute the model training method as described in any one of claims 1 to 7.

10. A chip, characterized in that: include: A processor, used to call and run a computer program from a memory so that a device equipped with the chip executes the model training method described in any one of claims 1 to 7.

11. A storage medium, characterized in that: Used to store a computer program, wherein the computer program enables a computer to execute the model training method as described in any one of claims 1 to 7.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the model training method according to any one of claims 1 to 7.

Images