A model training method based on a cloud management platform and the cloud management platform

Abstract

This application discloses a model training method and a cloud management platform based on a cloud management platform, which can improve model training efficiency. The method includes: a tenant sending a model training request for a specific model to the cloud management platform, whereby the tenant sets the number of queries, keys, and values ​​for the model. Based on this request, the cloud management platform determines multiple computing nodes. If the number of computing nodes exceeds the number of keys, the cloud management platform decomposes the model into multiple sub-models and deploys them on these computing nodes. These computing nodes can use these sub-models to process training data, obtaining the queries, subkeys, and subvalues. Then, these computing nodes can fuse the subkeys and subvalues ​​to obtain the keys and values. Subsequently, these computing nodes can use the queries, keys, and values ​​to complete the subsequent training of the sub-models.

Description

Technical Field

[0001] This application relates to the field of cloud technology, and in particular to a model training method based on a cloud management platform and a cloud management platform. Background Technology

[0002] With the rapid development of cloud technology, more and more tenants are choosing distributed training clusters provided by cloud vendors to train neural network models. Generally, a distributed training cluster can contain multiple computing nodes, and cloud vendors can utilize the resources provided by these computing nodes to complete the training tasks of tenants' models.

[0003] In related technologies, if the model that the tenant needs to train is a model built on the grouped-query attention (GQA) mechanism, the cloud provider can allow multiple computing nodes to train the model in a tensor parallelism (TP) manner. That is, the model is decomposed into multiple sub-models and these sub-models are deployed on multiple computing nodes respectively, so that these multiple computing nodes can use training data to train these multiple sub-models, thereby obtaining the trained model.

[0004] In the above process, since the grouped query attention mechanism can include the number of queries, keys and values ​​set by the tenant, the number of keys and values ​​must be greater than or equal to the number of multiple computing nodes. However, the number of keys and values ​​is often small, which makes it impossible to use a large number of computing nodes to complete the model training, resulting in low model training efficiency. Summary of the Invention

[0005] This application provides a model training method and a cloud management platform based on a cloud management platform. The cloud management platform can utilize a larger number of computing nodes to complete the model training process, which can improve the model training efficiency to a certain extent.

[0006] The first aspect of this application provides a model training method based on a cloud management platform. The cloud management platform implementing this method can manage infrastructure that provides cloud services to tenants. This infrastructure includes a cluster of computing nodes, which contains a certain number of computing nodes. The method includes:

[0007] When a tenant has a model to be trained, the tenant can input a model training request for that model into the model training interface provided by the cloud management platform. Therefore, the cloud management platform can receive the model training request sent by the tenant through the model training interface. The model training request is used to indicate the model, which is built based on the grouped query attention mechanism. The grouped query attention mechanism is used to indicate the number of multiple queries, the number of multiple keys, and the number of multiple values.

[0008] Next, the cloud management platform can select multiple computing nodes from the computing node cluster that can be used to train the model based on the model training request. If the number of these multiple queries is greater than or equal to the number of these multiple computing nodes, and the number of these multiple keys and the number of these multiple values ​​are less than the number of these multiple computing nodes, the cloud management platform can decompose the model into multiple sub-models and deploy these multiple sub-models on these multiple computing nodes respectively.

[0009] Then, the cloud management platform can trigger these multiple computing nodes to process the training data using these multiple sub-models, thereby obtaining multiple queries, multiple subkeys, and multiple subvalues. Based on this, these multiple computing nodes can fuse these multiple subkeys and multiple subvalues ​​to obtain multiple keys and multiple values. Finally, these multiple computing nodes can use these multiple queries, these multiple keys, and these multiple values ​​to continue training these multiple sub-models, thereby obtaining the trained model.

[0010] As can be seen from the above method, even if the number of keys and values ​​is less than the number of computing nodes, the cloud management platform can still decompose the model to be trained into a corresponding number of sub-models and distribute them across these computing nodes. Although this means that when these computing nodes process training data using these sub-models, they cannot directly obtain the keys and values, but instead directly obtain subkeys and subvalues, the cloud management platform can control these computing nodes to merge the subkeys into keys and the subvalues ​​into values. This ensures that, while maintaining data accuracy, the computing nodes can accurately train the multiple sub-models using these keys and values. Therefore, it is evident that the cloud management platform can utilize a larger number of computing nodes (more than the number of keys and values) to complete model training, which can improve model training efficiency to some extent.

[0011] In one possible implementation, the number of queries is an integer multiple of the number of keys, the number of keys is equal to the number of values, and the number of subkeys, subvalues, and compute nodes are equal. In the aforementioned implementation, the number of keys and values ​​set by the tenant is typically equal, and the number of queries is usually an integer multiple of the number of keys. For example, a tenant might set the number of queries to 4, and the number of keys and values ​​to 2 each. Furthermore, since the model is divided into multiple sub-models, one sub-model is typically deployed on a compute node, and a sub-model may have at least one query, one subkey, and one subvalue; therefore, the number of subkeys, the number of subvalues, and the number of compute nodes are equal.

[0012] In one possible implementation, the model includes a query matrix, a key matrix, and a value matrix. Multiple sub-models include multiple sub-query matrices decomposed from the query matrix, multiple sub-key matrices decomposed from the key matrix, and multiple sub-value matrices decomposed from the value matrix. The multiple sub-query matrices are used by multiple computing nodes to process training data, resulting in multiple queries. The multiple sub-key matrices are used by multiple computing nodes to process training data, resulting in multiple subkeys. The multiple sub-value matrices are used by multiple computing nodes to process training data, resulting in multiple sub-values. In this implementation, since the model includes a query matrix, a key matrix, and a value matrix, the multiple sub-models decomposed from this model can include multiple sub-query matrices decomposed from the query matrix, multiple sub-key matrices decomposed from the key matrix, and multiple sub-value matrices decomposed from the value matrix. Therefore, the cloud management platform can deploy these multiple sub-models on multiple computing nodes. For any one of these computing nodes, the computing node can use the sub-query matrices, sub-key matrices, and sub-value matrices contained in its deployed sub-model to process the training data, thereby obtaining at least one query, one subkey, and one sub-value. In this way, these multiple computing nodes can obtain a total of multiple queries, multiple subkeys, and multiple subvalues. In subsequent iterations, multiple subkeys can be merged into multiple keys, and multiple subvalues ​​can be merged into multiple values ​​to ensure the correctness of the data and thus ensure the accurate completion of model training.

[0013] In one possible implementation, the model training request is further used to instruct the tenant to set the training strategy for the model as tensor parallelism, and the tenant allows the cloud management platform to determine the degree of parallelism of tensor parallelism. Based on the model training request, the cloud management platform selects multiple computing nodes from the computing node cluster, including: the cloud management platform determines the degree of parallelism based on the number of multiple queries, the number of multiple keys, and the number of multiple values, wherein the number of multiple queries is greater than or equal to the degree of parallelism, and the number of multiple keys and the number of multiple values ​​are less than the degree of parallelism; the cloud management platform selects multiple computing nodes from the computing node cluster whose number is equal to the degree of parallelism. In the aforementioned implementation, since the model training request for the model is also used to instruct the tenant to set the training strategy for the model as tensor parallelism and the tenant allows the cloud management platform to determine the degree of parallelism of tensor parallelism independently, the cloud management platform can determine the number of multiple queries, the number of multiple keys, and the number of multiple values ​​from the model based on the model training request, and refer to the number of multiple queries, the number of multiple keys, and the number of multiple values ​​to accurately determine the degree of parallelism. Once the degree of parallelism is determined, the cloud management platform can select multiple computing nodes from the computing node cluster, with the number equal to the degree of parallelism, to efficiently complete the training of the model.

[0014] In one possible implementation, the end time of multiple computing nodes acquiring multiple subkeys precedes the start time of multiple computing nodes acquiring multiple subvalues; the end time of multiple computing nodes acquiring multiple subvalues ​​precedes the start time of multiple computing nodes acquiring multiple queries; the end time of multiple computing nodes fusing multiple subkeys precedes the start time of multiple computing nodes fusing multiple subvalues; and the end time of multiple computing nodes fusing multiple subvalues ​​precedes the end time of multiple computing nodes acquiring multiple queries. In this approach, since the query, subkey, and subvalue acquisition operations are located in the computation phase of the computing nodes, and the subkey and subvalue fusion operations are located in the communication phase between the computing nodes, the computation phase and the communication phase can be executed in parallel. Therefore, the cloud management platform can control the multiple computing nodes to have a certain degree of overlap in the execution of acquisition and fusion operations, thereby reducing the time cost required for model training.

[0015] In one possible implementation, the end time of multiple computing nodes acquiring multiple sub-values ​​precedes the start time of multiple computing nodes acquiring multiple sub-keys; the end time of multiple computing nodes acquiring multiple sub-keys precedes the start time of multiple computing nodes acquiring multiple queries; the end time of multiple computing nodes fusing multiple sub-values ​​precedes the start time of multiple computing nodes fusing multiple sub-keys; and the end time of multiple computing nodes fusing multiple sub-keys precedes the end time of multiple computing nodes acquiring multiple queries. In this approach, since the query, sub-key, and sub-value acquisition operations are located in the computation phase of the computing nodes, and the sub-key and sub-value fusion operations are located in the communication phase between the computing nodes, the computation phase and the communication phase can be executed in parallel. Therefore, the cloud management platform can control the multiple computing nodes to have a certain degree of overlap in the execution of acquisition and fusion operations, thereby reducing the time cost required for model training.

[0016] In one possible implementation, the cloud management platform instructs multiple computing nodes to train multiple sub-models based on multiple queries, multiple keys, and multiple values ​​to obtain the trained model. This process includes: the cloud management platform instructing multiple computing nodes to process the multiple queries, multiple keys, and multiple values ​​using the multiple sub-models to obtain the processing results of the training data; the cloud management platform instructing multiple computing nodes to obtain the gradients of the multiple queries, the gradients of the multiple keys, and the gradients of the multiple values ​​based on the processing results and the actual processing results of the training data; the cloud management platform instructing multiple computing nodes to decompose the gradients of the multiple keys to obtain the gradients of the multiple subkeys, and to decompose the gradients of the multiple values ​​to obtain the gradients of the multiple subvalues; the cloud management platform instructing multiple computing nodes to obtain the gradients of the multiple subquery matrices based on the gradients of the multiple queries, the gradients of the multiple subkey matrices based on the gradients of the multiple subkeys, and the gradients of the multiple subvalue matrices based on the gradients of the multiple subvalues; and the cloud management platform instructing multiple computing nodes to update the multiple subquery matrices, the multiple subkey matrices, and the multiple subvalue matrices based on the gradients of the multiple subquery matrices, the multiple subkey matrices, and the multiple subvalue matrices to obtain the trained model. In the aforementioned implementation, after obtaining the multiple queries, keys, and values, the multiple computing nodes can further process these queries, keys, and values ​​using multiple sub-models to obtain the processing results of the training data. Next, based on the processing results of the training data and the actual processing results of the training data, the multiple computing nodes can obtain the gradients of the multiple queries, keys, and values. Then, the multiple computing nodes decompose the gradients of the keys and values ​​to obtain the gradients of the subkeys and subvalues. Subsequently, based on the gradients of the queries, subkeys, and subvalues, the multiple computing nodes can obtain the gradients of the subquery matrices, subkey matrices, and subvalue matrices. Finally, based on the gradients of the subquery matrices, subkey matrices, and subvalue matrices, the multiple computing nodes update the subquery matrices, subkey matrices, and subvalue matrices to obtain the trained model.

[0017] In one possible implementation, the completion time of multiple computing nodes acquiring the gradients of multiple subquery matrices precedes the start time of multiple computing nodes acquiring the gradients of multiple key matrices; the completion time of multiple computing nodes acquiring the gradients of multiple key matrices precedes the start time of multiple computing nodes acquiring the gradients of multiple value matrices; the completion time of multiple computing nodes decomposing the gradients of multiple keys precedes the start time of multiple computing nodes decomposing the gradients of multiple values; and the completion time of multiple computing nodes decomposing the gradients of multiple values ​​precedes the completion time of multiple computing nodes acquiring the gradients of multiple value matrices. In the aforementioned implementation, since the acquisition operations of the gradients of the subquery matrices, subkey matrices, and value matrices are located in the computation phase of the computing nodes, and the decomposition operations of the gradients of the keys and values ​​are located in the communication phase between the computing nodes, the computation phase of the computing nodes and the communication phase can be executed in parallel. Therefore, the cloud management platform can control the multiple computing nodes to have a certain degree of overlap when performing acquisition and decomposition operations, thereby reducing the time cost required for model training.

[0018] In one possible implementation, the completion time of multiple computing nodes acquiring the gradients of multiple subquery matrices precedes the start time of multiple computing nodes acquiring the gradients of multiple subvalue matrices; the completion time of multiple computing nodes acquiring the gradients of multiple subvalue matrices precedes the start time of multiple computing nodes acquiring the gradients of multiple subkey matrices; the completion time of multiple computing nodes decomposing the gradients of multiple values ​​precedes the start time of multiple computing nodes decomposing the gradients of multiple keys; and the completion time of multiple computing nodes decomposing the gradients of multiple keys precedes the completion time of multiple computing nodes acquiring the gradients of multiple subkey matrices. In the aforementioned implementation, since the acquisition operations of the gradients of the subquery matrices, subkey matrices, and subvalue matrices are located in the computation phase of the computing nodes, and the decomposition operations of the gradients of the keys and values ​​are located in the communication phase between the computing nodes, the computation phase of the computing nodes and the communication phase can be executed in parallel. Therefore, the cloud management platform can control the multiple computing nodes to have a certain degree of overlap when performing acquisition and decomposition operations, thereby reducing the time cost required for model training.

[0019] A second aspect of this application provides a cloud management platform for managing infrastructure providing cloud services. The infrastructure includes a cluster of computing nodes. The cloud management platform includes: a receiving module for receiving a model training request sent by a tenant, wherein the model training request indicates a model to be trained, the model is constructed based on a grouped query attention mechanism, and the grouped query attention mechanism indicates the number of queries, the number of keys, and the number of values; a selection module for selecting multiple computing nodes from the cluster based on the model training request; a processing module for decomposing the model into multiple sub-models, deploying the multiple sub-models on the multiple computing nodes, and notifying the multiple computing nodes to process the training data using the multiple sub-models to obtain multiple queries, multiple sub-keys, and multiple sub-values ​​when the number of queries is greater than or equal to the number of computing nodes, and the number of keys and the number of values ​​are less than the number of computing nodes; a first notification module for notifying the multiple computing nodes to fuse the multiple sub-keys to obtain multiple keys, and to fuse the multiple sub-values ​​to obtain multiple values; and a second notification module for notifying the multiple computing nodes to train the multiple sub-models based on the multiple queries, multiple keys, and multiple values ​​to obtain a trained model.

[0020] In one possible implementation, the number of queries is an integer multiple of the number of keys, the number of keys is equal to the number of values, and the number of subkeys, the number of subvalues, and the number of compute nodes are equal.

[0021] In one possible implementation, the model includes a query matrix, a key matrix, and a value matrix. Multiple sub-models include multiple sub-query matrices obtained by decomposing the query matrix, multiple sub-key matrices obtained by decomposing the key matrix, and multiple sub-value matrices obtained by decomposing the value matrix. The multiple sub-query matrices are used by multiple computing nodes to process the training data and obtain multiple queries. The multiple sub-key matrices are used by multiple computing nodes to process the training data and obtain multiple sub-keys. The multiple sub-value matrices are used by multiple computing nodes to process the training data and obtain multiple sub-values.

[0022] In one possible implementation, the model training request is also used to instruct the tenant to set the training strategy for the model as tensor parallelism, and the tenant allows the cloud management platform to determine the degree of parallelism of tensor parallelism; the selection module is used to determine the degree of parallelism based on the number of multiple queries, the number of multiple keys, and the number of multiple values, wherein the number of multiple queries is greater than or equal to the degree of parallelism, and the number of multiple keys and the number of multiple values ​​are less than the degree of parallelism; the cloud management platform selects multiple computing nodes from the computing node cluster whose number is equal to the degree of parallelism.

[0023] In one possible implementation, the end time of multiple computing nodes acquiring multiple subkeys precedes the start time of multiple computing nodes acquiring multiple subvalues, the end time of multiple computing nodes acquiring multiple subvalues ​​precedes the start time of multiple computing nodes acquiring multiple queries, the end time of multiple computing nodes merging multiple subkeys precedes the start time of multiple computing nodes merging multiple subvalues, and the end time of multiple computing nodes merging multiple subvalues ​​precedes the end time of multiple computing nodes acquiring multiple queries.

[0024] In one possible implementation, the end time of multiple computing nodes acquiring multiple sub-values ​​precedes the start time of multiple computing nodes acquiring multiple sub-keys, the end time of multiple computing nodes acquiring multiple sub-keys precedes the start time of multiple computing nodes acquiring multiple queries, the end time of multiple computing nodes merging multiple sub-values ​​precedes the start time of multiple computing nodes merging multiple sub-keys, and the end time of multiple computing nodes merging multiple sub-keys precedes the end time of multiple computing nodes acquiring multiple queries.

[0025] In one possible implementation, the second notification module is configured to: notify multiple computing nodes to process multiple queries, multiple keys, and multiple values ​​using multiple sub-models to obtain the processing results of the training data; notify multiple computing nodes to obtain the gradients of multiple queries, multiple keys, and multiple values ​​based on the processing results and the actual processing results of the training data; notify multiple computing nodes to decompose the gradients of multiple keys to obtain the gradients of multiple subkeys, and decompose the gradients of multiple values ​​to obtain the gradients of multiple subvalues; notify multiple computing nodes to obtain the gradients of multiple subquery matrices based on the gradients of multiple queries, obtain the gradients of multiple subkey matrices based on the gradients of multiple subkeys, and obtain the gradients of multiple subvalue matrices based on the gradients of multiple subvalues; and notify multiple computing nodes to update the multiple subquery matrices, multiple subkey matrices, and multiple subvalue matrices based on the gradients of the multiple subquery matrices, the multiple subkey matrices, and the multiple subvalue matrices to obtain the trained model.

[0026] In one possible implementation, the end time of multiple computing nodes acquiring the gradients of multiple subquery matrices precedes the start time of multiple computing nodes acquiring the gradients of multiple subkey matrices; the end time of multiple computing nodes acquiring the gradients of multiple subkey matrices precedes the start time of multiple computing nodes acquiring the gradients of multiple subvalue matrices; the end time of multiple computing nodes decomposing the gradients of multiple keys precedes the start time of multiple computing nodes decomposing the gradients of multiple values; and the end time of multiple computing nodes decomposing the gradients of multiple values ​​precedes the end time of multiple computing nodes acquiring the gradients of multiple subvalue matrices.

[0027] In one possible implementation, the end time of multiple computing nodes acquiring the gradients of multiple subquery matrices precedes the start time of multiple computing nodes acquiring the gradients of multiple subvalue matrices, the end time of multiple computing nodes acquiring the gradients of multiple subvalue matrices precedes the start time of multiple computing nodes acquiring the gradients of multiple subkey matrices, the end time of multiple computing nodes decomposing the gradients of multiple values ​​precedes the start time of multiple computing nodes decomposing the gradients of multiple keys, and the end time of multiple computing nodes decomposing the gradients of multiple keys precedes the end time of multiple computing nodes acquiring the gradients of multiple subkey matrices.

[0028] A third aspect of this application provides a computing device cluster, the computing device cluster including at least one computing device, each computing device including a processor and a memory: the memory is used to store instructions; the processor is used to cause the computing device cluster to perform the method described in the first aspect or any possible implementation of the first aspect according to the instructions.

[0029] A fourth aspect of this application provides a computer storage medium storing one or more instructions that, when executed by one or more computers, cause the one or more computers to perform the method described in the first aspect or any possible implementation of the first aspect.

[0030] A fifth aspect of this application provides a computer program product storing instructions that, when executed by a computer, cause the computer to perform the method described in the first aspect or any possible implementation of the first aspect.

[0031] In this embodiment, when a tenant needs to train a model, the tenant can send a model training request to the cloud management platform. Since the model training request indicates the model to be trained, and the model is built based on a grouped query attention mechanism, the tenant can set the number of queries, keys, and values ​​within this mechanism. Based on this request, the cloud management platform can select multiple computing nodes from the computing node cluster. If the number of computing nodes is less than or equal to the number of queries, but greater than the number of keys and values, the cloud management platform can decompose the model into multiple sub-models and deploy these sub-models on the computing nodes respectively. This allows the computing nodes to process the training data using these sub-models, thereby obtaining the queries, subkeys, and subvalues. Then, the computing nodes can fuse the subkeys and subvalues ​​to obtain the keys and values. Subsequently, the computing nodes can use these queries, keys, and values ​​to train the sub-models, resulting in the trained model. In the aforementioned process, even if the number of keys and values ​​is less than the number of computing nodes, the cloud management platform can still decompose the model to be trained into a corresponding number of sub-models and distribute them across these computing nodes. Although this means that when these computing nodes process training data using these sub-models, they cannot directly obtain the keys and values, but instead directly obtain subkeys and subvalues, the cloud management platform can control these computing nodes to merge the subkeys into keys and the subvalues ​​into values. This ensures that, while maintaining data accuracy, the computing nodes can accurately train the multiple sub-models using these keys and values. Therefore, it is evident that the cloud management platform can utilize a larger number of computing nodes (more than the number of keys and values) to complete model training, which can improve model training efficiency to some extent. Attached Figure Description

[0032] Figure 1 A schematic diagram of the structure of the cloud service system provided in the embodiments of this application;

[0033] Figure 2 A schematic diagram of the structure of the model provided in the embodiments of this application;

[0034] Figure 3 A schematic flowchart of a model training method based on a cloud management platform provided in an embodiment of this application;

[0035] Figure 4 A schematic diagram of a tenant interface provided in an embodiment of this application;

[0036] Figure 5 A schematic diagram illustrating the calculation process of the grouped query attention layer provided in this application embodiment;

[0037] Figure 6 Another schematic diagram illustrating the calculation process of the grouped query attention layer provided in the embodiments of this application;

[0038] Figure 7 Another schematic diagram illustrating the calculation process of the grouped query attention layer provided in the embodiments of this application;

[0039] Figure 8 A schematic diagram of gradient calculation provided for an embodiment of this application;

[0040] Figure 9 Another schematic diagram of gradient calculation provided for an embodiment of this application;

[0041] Figure 10 A schematic diagram of the structure of the cloud management platform provided in the embodiments of this application;

[0042] Figure 11 A schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0043] Figure 12 A schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0044] Figure 13 This is a schematic diagram illustrating the network connection of computer devices in a computer cluster provided in an embodiment of this application. Detailed Implementation

[0045] This application provides a model training method and a cloud management platform based on a cloud management platform. The cloud management platform can utilize a larger number of computing nodes to complete the model training process, which can improve the model training efficiency to a certain extent.

[0046] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0047] With the rapid development of cloud technology, more and more tenants are choosing distributed training clusters provided by cloud vendors to train neural network models. Generally, a distributed training cluster can contain multiple computing nodes, and cloud vendors can utilize the resources provided by these computing nodes to complete the training tasks of tenants' models.

[0048] In related technologies, if the model that the tenant needs to train is a model built on a grouped query attention mechanism, the cloud provider can allow multiple computing nodes to train the model in a tensor parallel manner, that is, to decompose the model into multiple sub-models and deploy these sub-models on multiple computing nodes respectively, so that these multiple computing nodes can use training data to train these multiple sub-models, thereby obtaining the trained model.

[0049] In the above process, because the grouped query attention mechanism can include the number of queries, keys, and values ​​set by the tenant, the number of keys and values ​​must be greater than or equal to the number of compute nodes. However, the number of keys and values ​​is often small, which means that a large number of compute nodes cannot be used to complete the model training, resulting in low model training efficiency. For example, if a tenant sets the number of queries to 4 and the number of keys and values ​​to 2 in the grouped query attention mechanism of this model, the cloud provider can only select 2 compute nodes to complete the training of this model, and cannot use more compute nodes such as 4 or 8 compute nodes to complete the training of this model, resulting in low model training efficiency.

[0050] To address the aforementioned issues, this application provides a model training method based on a cloud management platform, which can be implemented through a cloud service system. Figure 1 A schematic diagram of the cloud service system provided in the embodiments of this application is shown below. Figure 1 As shown, a cloud service system includes the infrastructure that provides cloud services and a cloud management platform that manages this infrastructure. The cloud management platform and the infrastructure are described separately below:

[0051] A cloud management platform can centrally manage the infrastructure of the entire cloud service system (for example, selecting multiple compute nodes from a compute node cluster to meet the tenant's needs according to the tenant's instructions, and having these compute nodes train the tenant's model to fulfill the tenant's model training requirements, etc.). The cloud management platform can also be open to tenants outside the cloud service system and respond to their requests. For example, the cloud management platform can provide various interfaces such as login and model training interfaces for tenant clients (e.g., the terminal device used by the tenant or the browser on that device, etc.) to access. Specifically, the cloud management platform can authenticate a tenant's client through the login interface, allowing the tenant's client to log in to the cloud management platform after successful authentication. For example, the cloud management platform can also allow a tenant's client to send a model training request to the cloud management platform through a model training interface. Since the model training request indicates the tenant's model to be trained, and this model is built based on a grouped query attention mechanism, where the tenant can set the number of queries (i.e., the number of query heads), the number of keys (i.e., the number of key heads), and the number of values ​​(i.e., the number of value heads) within the grouped query attention mechanism, the cloud management platform can select multiple compute nodes from the compute node cluster. If the number of these compute nodes is less than or equal to the number of queries, but greater than the number of keys and the number of values, the cloud management platform can decompose the model into multiple sub-models and deploy these sub-models on these compute nodes respectively. Since each compute node deploys one sub-model, it can use the sub-model deployed on itself to process the training data, thereby obtaining one query, one subkey, and one subvalue. Therefore, these multiple compute nodes can obtain a total of these queries, subkeys, and subvalues. Then, these multiple computing nodes can fuse these multiple subkeys and multiple subvalues ​​to obtain these multiple keys and multiple values. Subsequently, these multiple computing nodes can use these multiple queries, these multiple keys, and these multiple values ​​to train these multiple sub-models, thereby obtaining multiple trained sub-models, i.e., the trained model.

[0052] The infrastructure includes a cluster of compute nodes that provide cloud services to the tenant. This cluster may contain a large number of compute nodes, each of which may occupy a certain amount of physical resources. For example, any one of these compute nodes may occupy any one or any combination of at least one of the physical server's computing resources (e.g., a central processing unit (CPU) and a graphics processing unit (GPU), etc.), at least one of the physical server's storage resources (e.g., memory and disk), and at least one of the physical server's network resources (e.g., a network interface card, etc.), without any limitation.

[0053] It is worth noting that, such as Figure 2 As shown ( Figure 2 (This is a schematic diagram of the model provided in an embodiment of this application). The model that the tenant needs to train is usually a neural network model, which may contain multiple neural network layers, including a group query attention layer and other neural network layers. This embodiment of the application focuses on the group query attention layer, which will be described below. The group query attention layer may contain three weight matrices: a query matrix, a key matrix, and a value matrix. Since the group query attention layer is a neural network layer built on a group query attention mechanism, the tenant can pre-set the number of queries, the number of keys, and the number of values ​​in the group query attention mechanism. Therefore, by processing the training data using these three weight matrices, these multiple queries, these multiple keys, and these multiple values ​​can be obtained.

[0054] During the training process of this model, due to the use of a tensor parallel training strategy, the cloud management platform can decompose the model into multiple sub-models. Each sub-model can contain sub-neural network layers derived from the decomposition of the model's neural network layers. Therefore, this sub-model can contain a sub-group query attention layer (i.e., a part of the group query attention layer). This sub-group query attention layer contains one of the sub-query matrices obtained from the decomposition of the query matrix, one of the sub-key matrices obtained from the decomposition of the key matrix, and one of the sub-value matrices obtained from the decomposition of the value matrix. The cloud management platform can then deploy these sub-models on multiple computing nodes. For any one of these computing nodes, it can use its deployed sub-model—containing the sub-query matrix, sub-key matrix, and sub-value matrix—to process the training data, thereby obtaining at least one query, one sub-key, and one sub-value. The other computing nodes do the same, thus ultimately obtaining multiple queries, multiple sub-keys, and multiple sub-values. Then, these multiple computing nodes can merge these multiple subkeys and subvalues ​​to obtain these multiple keys and values. In this way, after obtaining these multiple queries, keys, and values, these multiple computing nodes can continue to use these multiple sub-models to further process these queries, keys, and values ​​to complete the training of these multiple sub-models, thereby obtaining the trained model.

[0055] Furthermore, for the multiple computing nodes serving the tenant, these computing nodes can be presented in various forms. For example, these multiple computing nodes can be physical servers selected by the cloud management platform (containing computing resources, storage resources, and network resources of a certain specification); they can also be bare metal servers selected by the cloud management platform (containing computing resources, storage resources, and network resources of a certain specification); they can also be virtual machines (VMs) created by the cloud management platform on physical servers or bare metal servers using virtualization technology; they can also be containers (Docker) created by the cloud management platform on physical servers or bare metal servers using virtualization technology; they can also be micro virtual machines (microVMs) created by the cloud management platform on physical servers or bare metal servers using virtualization technology, and so on.

[0056] Furthermore, for multiple computing nodes serving a tenant, these computing nodes can be set up at the same site or different sites. These sites can be presented in various forms, such as regions, availability zones, data centers (DCs), rooms, physical server groups (also known as racks), and physical servers, etc.

[0057] Based on the aforementioned cloud service system, when a tenant needs to train a model, the tenant can send a model training request to the cloud management platform. This request indicates the model to be trained, and the model is built based on a grouped query attention mechanism. Within this mechanism, the tenant can set the number of queries, keys, and values. Based on this request, the cloud management platform can select multiple compute nodes from the compute node cluster. If the number of compute nodes is less than or equal to the number of queries, but greater than the number of keys and values, the cloud management platform can decompose the model into multiple sub-models and deploy these sub-models on the compute nodes. Since each compute node deploys one sub-model, it can use the deployed sub-model to process the training data, obtaining one query, one subkey, and one subvalue. Therefore, the compute nodes can obtain a total of these queries, subkeys, and subvalues. Then, the compute nodes can fuse these subkeys and subvalues ​​to obtain the final keys and values. Subsequently, these multiple computing nodes can use these multiple queries, keys, and values ​​to train these multiple sub-models, thereby obtaining the trained model. In the aforementioned process, even if the number of keys and values ​​is less than the number of computing nodes, the cloud management platform can still decompose the model to be trained into a corresponding number of sub-models and distribute them across these computing nodes. Although this may result in the computing nodes directly obtaining multiple subkeys and subvalues ​​instead of the keys and values ​​when processing training data using these sub-models, the cloud management platform can control the computing nodes to merge the subkeys into a single key and the subvalues ​​into a single value. This ensures that, while maintaining data accuracy, the computing nodes can accurately train the multiple sub-models using these keys and values. Therefore, it is evident that the cloud management platform can utilize a larger number of computing nodes (more than the number of keys and values) to complete model training, which can improve model training efficiency to some extent. To further understand the workflow of the cloud management platform, the following section combines... Figure 3 This workflow will be described in further detail. Figure 3 A flowchart illustrating the model training method based on a cloud management platform provided in this application embodiment is shown below. Figure 3 As shown, this method can be achieved through, as Figure 1 The illustrated cloud service system implementation includes infrastructure that provides cloud services to tenants and a cloud management platform that manages this infrastructure, which includes a cluster of compute nodes created by the cloud management platform. The method includes:

[0058] 301. The cloud management platform receives a model training request sent by a tenant. The model training request is used to indicate the model to be trained. The model is built based on a grouped query attention mechanism, which is used to indicate the number of multiple queries, the number of multiple keys, and the number of multiple values.

[0059] In this embodiment, when a tenant needs to train a model, the cloud management platform can provide the tenant with a model training interface (e.g., the model training section on the tenant's interface). The tenant can then input their model training request into the model training interface via their client. In this way, the cloud management platform can receive the model training request sent by the tenant's client through the model training interface. The model training request indicates the tenant's model to be trained, which is built based on a grouped query attention mechanism. This grouped query attention mechanism indicates the number of queries, keys, and values ​​set by the tenant.

[0060] Specifically, the number of queries set by the tenant is an integer multiple of the number of keys, and the number of keys is usually equal to the number of values.

[0061] More specifically, the model training request can not only indicate the tenant's model to be trained, but also indicate that the training strategy set by the tenant for that model is tensor parallelism, and that the tenant allows the cloud management platform to determine (extend) the parallelism of tensor parallelism. In other words, after receiving the model training request, the cloud management platform can parse it to obtain the model to be trained, the training strategy for that model, and the operational permissions granted by the tenant to the cloud management platform to extend the parallelism of the training strategy. In this way, the cloud management platform can combine the model and training strategy information to determine the parallelism of the training strategy itself in subsequent operations.

[0062] For example, such as Figure 4 As shown ( Figure 4 (This is a schematic diagram of a tenant interface provided in an embodiment of this application). Assume a tenant has a model to be trained, which is a transformer. The transformer includes a grouped query attention layer, built on a grouped query attention mechanism. The tenant sets the number of queries Q to 4, the number of keys K to 2, and the number of values ​​V to 2 in this mechanism. Next, the tenant can log in to the cloud management platform, which provides a tenant interface. This interface includes a model training section, where the tenant can input the following information: the model to be trained is a transformer, the training strategy for this model is tensor parallelism, and the tenant authorizes the cloud management platform to extend the parallelism of tensor parallelism. In this way, the model training section can generate a model training request based on the tenant's input and send the request to the cloud management platform.

[0063] 302. The cloud management platform selects multiple computing nodes from the computing node cluster based on the model training request.

[0064] Upon receiving the model training request, the cloud management platform can determine that the tenant needs to train the model based on the request. Therefore, the cloud management platform can select multiple computing nodes from the computing node cluster that can be used to train the model.

[0065] Specifically, the cloud management platform can select multiple computing nodes in the following ways:

[0066] Since the model training request indicates the tenant's model to be trained, the training strategy set by the tenant for the model as tensor parallelism, and the tenant allows the cloud management platform to determine the degree of parallelism for tensor parallelism, the cloud management platform can determine the number of queries, the number of keys, and the number of values ​​from the model based on the model training request. Because the number of queries is an integer multiple of the number of keys, and the number of keys equals the number of values, the cloud management platform can use the number of queries, keys, and values ​​as constraints to determine the degree of parallelism. It should be noted that the number of queries is usually greater than or equal to the degree of parallelism, while the number of keys and values ​​is usually less than the degree of parallelism.

[0067] Once the degree of parallelism is determined, the cloud management platform can select multiple computing nodes from the computing node cluster, with the number equal to the degree of parallelism, to use these multiple computing nodes to complete the training of the model in subsequent processes.

[0068] Continuing with the example above, after receiving the model training request, the cloud management platform can determine that the model to be trained is a transformer, the training strategy for the transformer is tensor parallelism, and the tenant allows the cloud management platform to scale the parallelism of tensor parallelism independently. Therefore, the cloud management platform can determine from the transformer that the number of Q is 4, the number of K is 2, and the number of V is 2, and determine the parallelism of tensor parallelism to be 4.

[0069] Then, assuming there are n computing nodes in the computing node cluster, the cloud management platform can select 4 computing nodes from these n computing nodes, namely computing node 1, computing node 2, computing node 3 and computing node 4.

[0070] 303. When the number of multiple queries is greater than or equal to the number of multiple computing nodes, and the number of multiple keys and multiple values ​​is less than the number of multiple computing nodes, the cloud management platform decomposes the model into multiple sub-models, deploys the multiple sub-models on multiple computing nodes, and notifies the multiple computing nodes to use the multiple sub-models to process the training data, thereby obtaining multiple queries, multiple subkeys, and multiple subvalues.

[0071] 304. The cloud management platform notifies multiple computing nodes to merge multiple subkeys to obtain multiple keys, and merges multiple subvalues ​​to obtain multiple values.

[0072] 305. The cloud management platform notifies multiple computing nodes to train multiple sub-models based on multiple queries, multiple keys, and multiple values ​​to obtain the trained model.

[0073] After determining the multiple computing nodes, if the number of queries is greater than or equal to the number of computing nodes, and the number of keys and values ​​is less than the number of computing nodes, the cloud management platform can decompose the model into multiple sub-models and deploy these sub-models on the computing nodes respectively. Since each computing node deploys one sub-model, the cloud management platform can trigger that computing node to process the training data using the sub-model deployed on it, thereby obtaining at least one query, one subkey, and one subvalue. Therefore, the multiple computing nodes can obtain a total of multiple queries, multiple subkeys, and multiple subvalues ​​(since each computing node can obtain at least one query, one subkey, and one subvalue, the number of subkeys and subvalues ​​is usually equal to the number of computing nodes, and the number of queries is usually greater than or equal to the number of computing nodes). Then, the multiple computing nodes can fuse these subkeys and subvalues ​​to obtain the keys and values. Subsequently, these multiple computing nodes can use these multiple queries, these multiple keys, and these multiple values ​​to train these multiple sub-models, thereby obtaining the trained model.

[0074] Specifically, since the model comprises a query matrix, a key matrix, and a value matrix, any one of these sub-models can contain one of the sub-query matrices obtained from the decomposition of the query matrix, one of the sub-key matrices obtained from the decomposition of the key matrix, and one of the sub-value matrices obtained from the decomposition of the value matrix. The cloud management platform can then deploy these sub-models on multiple computing nodes. Each computing node can use its own deployed sub-model—containing the sub-query matrix, sub-key matrix, and sub-value matrix—to process the training data, thereby obtaining at least one query, one sub-key, and one sub-value. The other computing nodes can perform similar operations, thus ultimately obtaining multiple queries, multiple sub-keys, and multiple sub-values. Subsequently, these multiple computing nodes can fuse these multiple sub-keys and multiple sub-values ​​to obtain multiple keys and multiple values. In this way, after obtaining these multiple queries, keys, and values, these multiple computing nodes can continue to use these multiple sub-models to further process these multiple queries, keys, and values, thereby obtaining the processing results of the training data. The processing results of the training data can then be used to complete the training of these multiple sub-models, thus obtaining the trained model.

[0075] Continuing with the example above, suppose the transformer's grouped query attention layer contains a query matrix W. Q Key matrix W K and the value matrix W V After selecting compute nodes 1 to 4, the cloud management platform can divide the transformer into sub-models 1 to 4. Each of sub-models 1 to 4 contains one-quarter of the original grouped query attention layer. Therefore, this one-quarter grouped query attention layer of sub-model 1 can contain elements from W... Q The subquery matrix W obtained by partitioning Q1 By W K The subkey matrix W obtained by partitioning K1 By W V The subkey matrix W obtained by partitioning V1 Sub-model 4 may contain W Q The subquery matrix W obtained by partitioning Q4 By W K The subkey matrix W obtained by partitioning K4 By W V The subkey matrix W obtained by partitioning V4 .

[0076] Next, as Figure 5 and Figure 6 As shown ( Figure 5This is a schematic diagram illustrating the calculation process of the grouped query attention layer provided in an embodiment of this application. Figure 6 (This is another schematic diagram illustrating the computation process of the grouped query attention layer provided in this application embodiment). The cloud management platform can deploy sub-model 1 on computing node 1, sub-model 2 on computing node 2, sub-model 3 on computing node 3, and sub-model 4 on computing node 4. Since the operations performed by computing nodes 1 to 4 are similar, computing node 1 will be used as an example for illustration below. After the deployment of the sub-models is completed, the cloud management platform can trigger computing node 1 to input training data into sub-model 1. The quarter-grouped query attention layer in sub-model 1 can first perform the following acquisition operations:

[0077] Q1 = Matmul(W Q1 ,x)

[0078] K 1-1 =Matmul(W K1 ,x)

[0079] V 1-1 =Matmul(W V1 ,x) (1)

[0080] In the above formula, x represents the input to the quarter-group query attention layer, which can be either the training data input to sub-model 1 or the result obtained after some processing of the training data (i.e., in sub-model 1, there are other layers before this quarter-group query attention layer, etc.). Q1 is a query obtained by computing node 1, and K... 1-1 To compute one subkey of node 1, V 1-1 This is the sub-value obtained by computing node 1.

[0081] Similarly, Q2,K can also be obtained by computing node 2. 1-2 V 1-2 Q3,K can also be obtained by computing node 3. 2-1 V 2-1 Q4,K can also be obtained by computing node 4. 2-2 V 2-2 .

[0082] Next, compute node 1 can work with compute node 2 to perform the following fusion operations:

[0083] K1 = AllGather(K 1-1 ,K 1-2 )

[0084] V1 = AllGather(V 1-1 V 1-2 (2)

[0085] In the above formula, AllGather is calculated as a fusion (also known as full acquisition or full collection) operation, and K1 refers to the value of node 1 in the calculation of K. 1-1 and K 1-2 The resulting key, V1, refers to the key obtained by computing node 1. 1-1 and V 1-2 The single value obtained by fusion.

[0086] Similarly, compute node 2 can also obtain K1, V1, compute node 3 can also obtain K2, V2, and compute node 4 can also obtain K2, V2. Thus, compute nodes 1 to 4 can obtain 4 queries, 2 keys, and 2 values, namely Q1, Q2, Q3, Q4, K1, K2, V1, V2.

[0087] Then, compute node 1 can trigger the quarter-grouped query attention layer in sub-model 1 to continue performing the following operations:

[0088] ο 1-1 =scaleDotAttention(Q1,K1,V1)

[0089]

[0090] In the above formula, The weight matrix in sub-model 1, ο 1-2 The output of this quarter-grouped query attention layer in sub-model 1 can be used as the input to the remaining quarter-grouped layers after this quarter-grouped query attention layer, so that the remaining quarter-grouped layers can continue to focus on the query attention layer. 1-2 The data is then processed to obtain one-quarter of the training data.

[0091] Similarly, computation nodes 2 through 4 can also obtain the remaining three-quarters of the processing results of the training data. These results are then concatenated with the one-quarter processing result obtained by computation node 1 to obtain the final processing result of the training data. Therefore, these four computation nodes can use this processing result to complete the subsequent training of sub-models 1 through 4, thereby obtaining the trained transformer.

[0092] More specifically, these multiple computing nodes can save time costs in the following ways:

[0093] Since the query, subkey, and subvalue retrieval operations are located within the computation phase of the compute node, and the subkey and subvalue fusion operation is located in the communication phase between compute nodes, and the computation phase and communication phase of the compute node can be executed in parallel, the cloud management platform can control these multiple compute nodes to satisfy one of the following two conditions when performing retrieval and fusion operations:

[0094] (1) The end time of these multiple computing nodes acquiring these multiple subkeys is earlier than the start time of these multiple computing nodes acquiring these multiple subvalues. The end time of these multiple computing nodes acquiring these multiple subvalues ​​is earlier than the start time of these multiple computing nodes acquiring these multiple queries. The end time of these multiple computing nodes merging these multiple subkeys is earlier than the start time of these multiple computing nodes merging these multiple subvalues. The end time of these multiple computing nodes merging these multiple subvalues ​​is earlier than the end time of these multiple computing nodes acquiring these multiple queries.

[0095] (2) The end time of these multiple computing nodes acquiring these multiple sub-values ​​is earlier than the start time of these multiple computing nodes acquiring these multiple sub-keys, the end time of these multiple computing nodes acquiring these multiple sub-keys is earlier than the start time of these multiple computing nodes acquiring these multiple queries, the end time of these multiple computing nodes merging these multiple sub-values ​​is earlier than the start time of these multiple computing nodes merging these multiple sub-keys, and the end time of these multiple computing nodes merging these multiple sub-keys is earlier than the end time of these multiple computing nodes acquiring these multiple queries.

[0096] As in the example above, such as Figure 7 As shown ( Figure 7 This is another schematic diagram illustrating the calculation process of the grouped query attention layer provided in the embodiments of this application. Figure 7 Is Figure 5 Based on the data obtained from the previous drawing, for computation node 1, computation node 1 can first let sub-model 1 start to obtain K. 1-1 In obtaining K 1-1 After completion, computation node 1 can then allow sub-model 1 to begin acquiring V. 1-1 In obtaining K 1-1 Or obtain V 1-1 During the process, compute node 1 can also cooperate with compute node 2 to start executing K. 1-1 and K 1-2 The fusion. In obtaining V 1-1 After completion, computation node 1 can then allow sub-model 1 to begin acquiring Q1. In K 1-1 and K 1-2 After obtaining K1 through fusion, compute node 1 can continue to work with compute node 2 to start executing V. 1-1 and V 1-2 In the fusion process, computing node 1 needs to strictly control V. 1-1 and V 1-2 The fusion was completed before the end of Q1.

[0097] More specifically, the cloud management platform enables these multiple computing nodes to complete model training:

[0098] After obtaining the multiple queries, keys, and values, these multiple computing nodes can further process these queries, keys, and values ​​using the multiple sub-models to obtain the processing results of the training data. Since the actual processing results of the training data are known, the cloud management platform can trigger these multiple computing nodes to calculate a loss using both the processing results of the training data and the actual processing results. This loss describes the difference between the processing results of the training data and the actual processing results. Then, for any one of these computing nodes, it can use this loss to calculate the gradient of at least one query, the gradient of one key, and the gradient of one value obtained previously. The same process is repeated for the remaining computing nodes. Therefore, these multiple computing nodes ultimately obtain the gradients of the multiple queries, the gradients of the multiple keys, and the gradients of the multiple values. Then, these multiple computing nodes can decompose the gradients of the multiple keys to obtain the gradients of the multiple subkeys, and decompose the gradients of the multiple values ​​to obtain the gradients of the multiple subvalues. Thus, for any one of these computing nodes, it can obtain the gradient of at least one query, the gradient of a subkey, and the gradient of a subvalue, as previously obtained. The same applies to the other computing nodes. Subsequently, for any one of these computing nodes, it can use the gradient of the previously obtained query to obtain the gradient of the subquery matrix contained in the submodel deployed on that node, use the gradient of the previously obtained subkeys to obtain the gradient of the subkey matrix contained in the submodel deployed on that node, and use the gradient of the previously obtained subvalues ​​to obtain the gradient of the subvalue matrix contained in the submodel deployed on that node. The same applies to the other computing nodes. Therefore, these multiple computing nodes can ultimately obtain the gradients of the multiple subquery matrices, the gradients of the multiple subkey matrices, and the gradients of the multiple subvalue matrices, respectively. Finally, for any one of these computing nodes, the computing node can use the gradients of the subquery matrix, subkey matrix, and subvalue matrix contained in the submodel deployed on the computing node to update the subquery matrix, subkey matrix, and subvalue matrix contained in the submodel deployed on the computing node. The same applies to the other computing nodes, so that these multiple computing nodes can obtain the trained model.

[0099] As in the example above, such as Figure 8 As shown ( Figure 8 A schematic diagram of gradient calculation provided in an embodiment of this application. Figure 8 Is Figure 5Based on the results obtained from the training data processing, computation nodes 1 to 4 can use the processing results of the training data and the actual processing results of the training data to calculate a loss. For computation node 1, it can use this loss to obtain the gradient g1 of the output of the quarter-group query attention layer in sub-model 1, and perform the following operations on g1:

[0100] ο g1 =BackwardMatmul(W o1 ,ο 1-2 ,g1)

[0101] Q g1 ,K g1 V g1 =BackwardscaleDotAttention(Q1,K1,V1) (4)

[0102] In the above formula, Q g1 To calculate the gradient of Q1 obtained at node 1, K g1 Let V be the gradient of K1. g1 Let V1 be the gradient.

[0103] Similarly, calculating node 2 can also yield the gradients Q of Q2, K1, and V1. g2 ,K g2 V g2 By calculating node 3, the gradients Q of Q3, K2, and V2 can be obtained. g3 ,K g3 V g3 Calculating node 4 yields the gradients Q of Q4, K2, and V2. g4 ,K g4 V g4 .

[0104] Next, compute node 1 can also perform the following operations:

[0105] K g1-1 =ReduceGather(K) g1 ,K g2 )

[0106] V g1-1 =ReduceGather(V) g1 V g2 (5)

[0107] In the above formula, ReduceGather is the decomposition operation, and K g1-1 For compute node 1, compute node 2 is combined with K g1 and K g2 K obtained by decomposition 1-1 The gradient of Vg1-1 For compute node 1 and compute node 2, V g1 and V g2 V obtained by decomposition 1-1 The gradient.

[0108] Similarly, K can also be obtained by computing node 2. 1-2 V 1-2 gradient K g1-2 V g1-2 K can be obtained by computing node 3. 2-1 V 2-1 gradient K g2-1 V g2-1 K can be obtained by computing node 4. 2-2 V 2-2 gradient K g2-2 V g2-2 .

[0109] Then, compute node 1 can also perform the following operations:

[0110]

[0111] In the above formula, W obtained for computing node 1 Q1 gradient, For W K1 gradient, For W V1 The gradient.

[0112] Similarly, W can also be obtained by computing node 2. Q2 W K2 W V2 gradient ...and W can also be obtained from compute node 4. Q4 W K4 W V4 gradient

[0113] Subsequently, computing node 1 can utilize For W Q1 W K1 W V1 To update, ... compute node 4 can utilize For W Q4 W K4 W V4 Update the transformer to obtain the trained transformer.

[0114] More specifically, these multiple computing nodes can save time costs in the following ways:

[0115] Since the operations of obtaining the gradients of the subquery matrix, the subkey matrix, and the subvalue matrix are located within the computation phase of the computing node, and the decomposition operations of the gradients of the keys and values ​​are located within the communication phase between computing nodes, and the computation phase of the computing node and the communication node can be executed in parallel, the cloud management platform can control these multiple computing nodes to satisfy one of the following two conditions when performing the acquisition and decomposition operations:

[0116] (1) The end time of the multiple computing nodes obtaining the gradients of the multiple subquery matrices is earlier than the start time of the multiple computing nodes obtaining the gradients of the multiple key matrices. The end time of the multiple computing nodes obtaining the gradients of the multiple key matrices is earlier than the start time of the multiple computing nodes obtaining the gradients of the multiple subvalue matrices. The end time of the multiple computing nodes decomposing the gradients of the multiple keys is earlier than the start time of the multiple computing nodes decomposing the gradients of the multiple values. The end time of the multiple computing nodes decomposing the gradients of the multiple values ​​is earlier than the end time of the multiple computing nodes obtaining the gradients of the multiple subvalue matrices.

[0117] (2) The end time of the multiple computing nodes obtaining the gradients of the multiple subquery matrices is earlier than the start time of the multiple computing nodes obtaining the gradients of the multiple subvalue matrices. The end time of the multiple computing nodes obtaining the gradients of the multiple subvalue matrices is earlier than the start time of the multiple computing nodes obtaining the gradients of the multiple subkey matrices. The end time of the multiple computing nodes decomposing the gradients of the multiple values ​​is earlier than the start time of the multiple computing nodes decomposing the gradients of the multiple keys. The end time of the multiple computing nodes decomposing the gradients of the multiple keys is earlier than the end time of the multiple computing nodes obtaining the gradients of the multiple subkey matrices.

[0118] As in the example above, such as Figure 9 As shown ( Figure 9 This is another schematic diagram of gradient calculation provided in an embodiment of this application. Figure 9 Is Figure 8 Based on the data obtained from the above, for compute node 1, compute node 1 can start by acquiring... In acquiring After completion, compute node 1 can continue to acquire data. In acquiring Or obtain During the process, compute node 1 can also cooperate with compute node 2 to start executing K. g1 and K g2 Decomposition. In obtaining After completion, compute node 1 can continue to acquire data. In K g1 and K g2 The decomposition yields K g1-1 Afterwards, compute node 1 can continue to work with compute node 2 to begin executing V.g1 and V g2 In the decomposition, computation node 1 needs to strictly control V. g1 and V g2 Decomposition in obtaining Complete before the end.

[0119] In this embodiment, when a tenant needs to train a model, the tenant can send a model training request to the cloud management platform. Since the model training request indicates the model to be trained, and the model is built based on a grouped query attention mechanism, the tenant can set the number of queries, keys, and values ​​within this mechanism. Based on this request, the cloud management platform can select multiple computing nodes from the computing node cluster. If the number of computing nodes is less than or equal to the number of queries, but greater than the number of keys and values, the cloud management platform can decompose the model into multiple sub-models and deploy these sub-models on the computing nodes respectively. This allows the computing nodes to process the training data using these sub-models, thereby obtaining the queries, subkeys, and subvalues. Then, the computing nodes can fuse the subkeys and subvalues ​​to obtain the keys and values. Subsequently, the computing nodes can use these queries, keys, and values ​​to train the sub-models, resulting in the trained model. In the aforementioned process, even if the number of keys and values ​​is less than the number of computing nodes, the cloud management platform can still decompose the model to be trained into a corresponding number of sub-models and distribute them across these computing nodes. Although this means that when these computing nodes process training data using these sub-models, they cannot directly obtain the keys and values, but instead directly obtain subkeys and subvalues, the cloud management platform can control these computing nodes to merge the subkeys into keys and the subvalues ​​into values. This ensures that, while maintaining data accuracy, the computing nodes can accurately train the multiple sub-models using these keys and values. Therefore, it is evident that the cloud management platform can utilize a larger number of computing nodes (more than the number of keys and values) to complete model training, which can improve model training efficiency to some extent.

[0120] The above is a detailed description of the model training method based on a cloud management platform provided in the embodiments of this application. The cloud management platform provided in the embodiments of this application will be introduced below. Figure 10 This application provides a schematic diagram of the structure of a cloud management platform. The cloud management platform is used to manage the infrastructure that provides cloud services. The infrastructure includes a cluster of computing nodes. The cloud management platform includes:

[0121] The receiving module 1001 is used to receive a model training request sent by a tenant. The model training request indicates the model to be trained, which is built based on a grouped query attention mechanism. This grouped query attention mechanism indicates the number of queries, keys, and values. For example, the receiving module 1001 is used to implement... Figure 3 Step 301 in the illustrated embodiment.

[0122] Selection module 1002 is used to select multiple computing nodes from the computing node cluster based on model training requests; for example, selection module 1002 is used to implement Figure 3 Step 302 in the illustrated embodiment.

[0123] Processing module 1003 is used to decompose the model into multiple sub-models when the number of multiple queries is greater than or equal to the number of multiple computing nodes, and the number of multiple keys and multiple values ​​is less than the number of multiple computing nodes. It then deploys these sub-models on multiple computing nodes and instructs the multiple computing nodes to process the training data using these sub-models, resulting in multiple queries, multiple subkeys, and multiple subvalues. For example, processing module 1003 is used to implement... Figure 3 Step 303 in the illustrated embodiment.

[0124] The first notification module 1004 is used to notify multiple computing nodes to merge multiple subkeys to obtain multiple keys, and to merge multiple subvalues ​​to obtain multiple values; for example, the first notification module 1004 is used to implement Figure 3 Step 304 in the illustrated embodiment.

[0125] The second notification module 1005 is used to notify multiple computing nodes to train multiple sub-models based on multiple queries, multiple keys, and multiple values, thereby obtaining the trained model. For example, the second notification module 1005 is used to implement... Figure 3 Step 305 in the illustrated embodiment.

[0126] In one possible implementation, the number of queries is an integer multiple of the number of keys, the number of keys is equal to the number of values, and the number of subkeys, the number of subvalues, and the number of compute nodes are equal.

[0127] In one possible implementation, the model includes a query matrix, a key matrix, and a value matrix. Multiple sub-models include multiple sub-query matrices obtained by decomposing the query matrix, multiple sub-key matrices obtained by decomposing the key matrix, and multiple sub-value matrices obtained by decomposing the value matrix. The multiple sub-query matrices are used by multiple computing nodes to process the training data and obtain multiple queries. The multiple sub-key matrices are used by multiple computing nodes to process the training data and obtain multiple sub-keys. The multiple sub-value matrices are used by multiple computing nodes to process the training data and obtain multiple sub-values.

[0128] In one possible implementation, the model training request is also used to instruct the tenant to set the training strategy for the model as tensor parallelism, and the tenant allows the cloud management platform to determine the degree of parallelism of tensor parallelism; the selection module is used to determine the degree of parallelism based on the number of multiple queries, the number of multiple keys, and the number of multiple values, wherein the number of multiple queries is greater than or equal to the degree of parallelism, and the number of multiple keys and the number of multiple values ​​are less than the degree of parallelism; the cloud management platform selects multiple computing nodes from the computing node cluster whose number is equal to the degree of parallelism.

[0129] In one possible implementation, the end time of multiple computing nodes acquiring multiple subkeys precedes the start time of multiple computing nodes acquiring multiple subvalues, the end time of multiple computing nodes acquiring multiple subvalues ​​precedes the start time of multiple computing nodes acquiring multiple queries, the end time of multiple computing nodes merging multiple subkeys precedes the start time of multiple computing nodes merging multiple subvalues, and the end time of multiple computing nodes merging multiple subvalues ​​precedes the end time of multiple computing nodes acquiring multiple queries.

[0130] In one possible implementation, the end time of multiple computing nodes acquiring multiple sub-values ​​precedes the start time of multiple computing nodes acquiring multiple sub-keys, the end time of multiple computing nodes acquiring multiple sub-keys precedes the start time of multiple computing nodes acquiring multiple queries, the end time of multiple computing nodes merging multiple sub-values ​​precedes the start time of multiple computing nodes merging multiple sub-keys, and the end time of multiple computing nodes merging multiple sub-keys precedes the end time of multiple computing nodes acquiring multiple queries.

[0131] In one possible implementation, the second notification module is configured to: notify multiple computing nodes to process multiple queries, multiple keys, and multiple values ​​using multiple sub-models to obtain the processing results of the training data; notify multiple computing nodes to obtain the gradients of multiple queries, multiple keys, and multiple values ​​based on the processing results and the actual processing results of the training data; notify multiple computing nodes to decompose the gradients of multiple keys to obtain the gradients of multiple subkeys, and decompose the gradients of multiple values ​​to obtain the gradients of multiple subvalues; notify multiple computing nodes to obtain the gradients of multiple subquery matrices based on the gradients of multiple queries, obtain the gradients of multiple subkey matrices based on the gradients of multiple subkeys, and obtain the gradients of multiple subvalue matrices based on the gradients of multiple subvalues; and notify multiple computing nodes to update the multiple subquery matrices, multiple subkey matrices, and multiple subvalue matrices based on the gradients of the multiple subquery matrices, the multiple subkey matrices, and the multiple subvalue matrices to obtain the trained model.

[0132] In one possible implementation, the end time of multiple computing nodes acquiring the gradients of multiple subquery matrices precedes the start time of multiple computing nodes acquiring the gradients of multiple subkey matrices; the end time of multiple computing nodes acquiring the gradients of multiple subkey matrices precedes the start time of multiple computing nodes acquiring the gradients of multiple subvalue matrices; the end time of multiple computing nodes decomposing the gradients of multiple keys precedes the start time of multiple computing nodes decomposing the gradients of multiple values; and the end time of multiple computing nodes decomposing the gradients of multiple values ​​precedes the end time of multiple computing nodes acquiring the gradients of multiple subvalue matrices.

[0133] In one possible implementation, the end time of multiple computing nodes acquiring the gradients of multiple subquery matrices precedes the start time of multiple computing nodes acquiring the gradients of multiple subvalue matrices, the end time of multiple computing nodes acquiring the gradients of multiple subvalue matrices precedes the start time of multiple computing nodes acquiring the gradients of multiple subkey matrices, the end time of multiple computing nodes decomposing the gradients of multiple values ​​precedes the start time of multiple computing nodes decomposing the gradients of multiple keys, and the end time of multiple computing nodes decomposing the gradients of multiple keys precedes the end time of multiple computing nodes acquiring the gradients of multiple subkey matrices.

[0134] It should be noted that the information interaction and implementation process between the modules / units of the above-mentioned device are based on the same concept as the method embodiments of this application, and the resulting technical effects are the same as those of the method embodiments of this application. For details, please refer to the description in the method embodiments shown above in the embodiments of this application, and will not be repeated here.

[0135] Please see Figure 11 , Figure 11 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Figure 11As shown, the computing device 1100 (which can be used to present the aforementioned cloud management platform) includes: a processor 1101, a memory 1102, a communication interface 1103, and a bus 1104. The processor 1101, memory 1102, and communication interface 1103 are coupled via the bus (not shown in the figure). The memory 1102 stores instructions. When the execution instructions in the memory 1102 are executed, the computing device 1100 executes the method performed by the cloud management platform in the above method embodiment.

[0136] The computing device 1100 may be one or more integrated circuits configured to implement the methods described above, such as: one or more application-specific integrated circuits (ASICs), or one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs), or a combination of at least two of these forms of integrated circuits. Furthermore, when the units in the device can be implemented in the form of a processing element scheduler, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor capable of calling programs. Alternatively, these units may be integrated together and implemented as a system-on-a-chip (SOC).

[0137] The processor 1101 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0138] The memory 1102 can be volatile memory or non-volatile memory, or may include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0139] The memory 1102 stores executable program code, and the processor 1101 executes the executable program code to implement the functions of the aforementioned receiving module, selection module, processing module, first notification module, and second notification module, thereby realizing the aforementioned model training method based on the cloud management platform. That is, the memory 1102 stores instructions for executing the aforementioned model training method based on the cloud management platform.

[0140] The communication interface 1103 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1100 and other devices or communication networks.

[0141] In addition to the data bus, the 1104 bus can also include a power bus, a control bus, and a status signal bus. The bus can be a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL) bus, a Cache Coherent Interconnect for Accelerators (CCIX) bus, etc. The bus can be divided into address bus, data bus, and control bus.

[0142] Please see Figure 12 , Figure 12 This is a schematic diagram of a computing device cluster provided in an embodiment of this application. Figure 12 As shown, the computing device cluster 1200 includes at least one computing device 1100.

[0143] like Figure 12 As shown, the computing device cluster 1200 includes at least one computing device 1100. The memory 1102 of one or more computing devices 1100 in the computing device cluster 1200 may store the same instructions for executing the above-described model training method based on the cloud management platform.

[0144] In some possible implementations, the memory 1102 of one or more computing devices 1100 in the computing device cluster 1200 may also store partial instructions for executing the aforementioned model training method based on the cloud management platform. In other words, a combination of one or more computing devices 1100 can jointly execute the aforementioned model training method based on the cloud management platform.

[0145] It should be noted that the memory 1102 in different computing devices 1100 within the computing device cluster 1200 can store different instructions, which are used to execute certain functions of the aforementioned cloud management platform. That is, the instructions stored in the memory 1102 of different computing devices 1100 can implement the functions of one or more modules, such as the receiving module, selection module, processing module, first notification module, and second notification module.

[0146] In some possible implementations, one or more computing devices 1100 in the computing device cluster 1200 can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc.

[0147] Please see Figure 13 , Figure 13 This is a schematic diagram illustrating the network connection of computer devices in a computer cluster provided in an embodiment of this application. Figure 13 As shown, the two computing devices 1100A and 1100B are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device.

[0148] In one possible implementation, the memory in computing device 1100A stores instructions for performing the functions of modules such as the receiving module. Meanwhile, the memory in computing device 1100B stores instructions for performing the functions of modules such as the selection module, processing module, first notification module, and second notification module.

[0149] It should be understood that Figure 13 The functions of computing device 1100A shown can also be performed by multiple computing devices. Similarly, the functions of computing device 1100B can also be performed by multiple computing devices.

[0150] This application also relates to a computer storage medium storing a program for signal processing, which, when run on a computer, causes the computer to perform actions such as... Figure 3 The steps performed by the cloud management platform in the illustrated embodiment.

[0151] This application also relates to a computer program product that stores instructions that, when executed by a computer, cause the computer to perform actions such as... Figure 3 The steps performed by the cloud management platform in the illustrated embodiment.

[0152] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0153] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0154] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0155] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0156] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A model training method based on a cloud management platform, characterized in that, The cloud management platform is used to manage the infrastructure that provides cloud services, the infrastructure including a cluster of computing nodes, and the method includes: The cloud management platform receives a model training request sent by a tenant. The model training request is used to indicate the model to be trained. The model is built based on a grouped query attention mechanism, which is used to indicate the number of multiple queries, the number of multiple keys, and the number of multiple values. Based on the model training request, the cloud management platform selects multiple computing nodes from the computing node cluster. If the number of multiple queries is greater than or equal to the number of multiple computing nodes, and the number of multiple keys and the number of multiple values ​​are less than the number of multiple computing nodes, the cloud management platform decomposes the model into multiple sub-models, deploys the multiple sub-models on the multiple computing nodes, and notifies the multiple computing nodes to use the multiple sub-models to process the training data to obtain the multiple queries, multiple subkeys, and multiple subvalues. The cloud management platform notifies the multiple computing nodes to merge the multiple subkeys to obtain the multiple keys, and to merge the multiple subvalues ​​to obtain the multiple values; The cloud management platform notifies the multiple computing nodes to train the multiple sub-models based on the multiple queries, the multiple keys, and the multiple values, thereby obtaining the trained model.

2. The method according to claim 1, characterized in that, The number of queries is an integer multiple of the number of keys, the number of keys is equal to the number of values, and the number of subkeys, the number of subvalues, and the number of compute nodes are equal.

3. The method according to claim 1 or 2, characterized in that, The model includes a query matrix, a key matrix, and a value matrix. The multiple sub-models include multiple sub-query matrices obtained by decomposing the query matrix, multiple sub-key matrices obtained by decomposing the key matrix, and multiple sub-value matrices obtained by decomposing the value matrix. The multiple sub-query matrices are used by the multiple computing nodes to process the training data and obtain the multiple queries; The plurality of subkey matrices are used by the plurality of computing nodes to process the training data and obtain the plurality of subkeys; The plurality of sub-value matrices are used by the plurality of computing nodes to process the training data and obtain the plurality of sub-values.

4. The method according to any one of claims 1 to 3, characterized in that, The model training request is also used to instruct the tenant to set the training strategy for the model as tensor parallelism, and the tenant to allow the cloud management platform to determine the degree of parallelism of the tensor parallelism; Based on the model training request, the cloud management platform selects multiple computing nodes from the computing node cluster, including: The cloud management platform determines the parallelism based on the number of queries, the number of keys, and the number of values, wherein the number of queries is greater than or equal to the parallelism, and the number of keys and the number of values ​​are less than the parallelism. The cloud management platform selects multiple computing nodes from the computing node cluster, with the number equal to the parallelism.

5. The method according to any one of claims 1 to 4, characterized in that, The end time of the multiple computing nodes acquiring the multiple subkeys is earlier than the start time of the multiple computing nodes acquiring the multiple subvalues. The end time of the multiple computing nodes acquiring the multiple subvalues ​​is earlier than the start time of the multiple computing nodes acquiring the multiple queries. The end time of the multiple computing nodes merging the multiple subkeys is earlier than the start time of the multiple computing nodes merging the multiple subvalues. The end time of the multiple computing nodes merging the multiple subvalues ​​is earlier than the end time of the multiple computing nodes acquiring the multiple queries.

6. The method according to any one of claims 1 to 4, characterized in that, The end time of the multiple computing nodes obtaining the multiple sub-values ​​is earlier than the start time of the multiple computing nodes obtaining the multiple sub-keys. The end time of the multiple computing nodes obtaining the multiple sub-keys is earlier than the start time of the multiple computing nodes obtaining the multiple queries. The end time of the multiple computing nodes merging the multiple sub-values ​​is earlier than the start time of the multiple computing nodes merging the multiple sub-keys. The end time of the multiple computing nodes merging the multiple sub-keys is earlier than the end time of the multiple computing nodes obtaining the multiple queries.

7. The method according to claim 3, characterized in that, The cloud management platform notifies the multiple computing nodes to train the multiple sub-models based on the multiple queries, the multiple keys, and the multiple values, resulting in trained models including: The cloud management platform notifies the multiple computing nodes to use the multiple sub-models to process the multiple queries, the multiple keys, and the multiple values ​​to obtain the processing results of the training data; The cloud management platform notifies the multiple computing nodes to obtain the gradients of the multiple queries, the gradients of the multiple keys, and the gradients of the multiple values ​​based on the processing results and the actual processing results of the training data. The cloud management platform instructs the multiple computing nodes to decompose the gradients of the multiple keys to obtain the gradients of the multiple subkeys, and to decompose the gradients of the multiple values ​​to obtain the gradients of the multiple subvalues. The cloud management platform notifies the multiple computing nodes to obtain the gradients of the multiple sub-query matrices based on the gradients of the multiple queries, to obtain the gradients of the multiple sub-key matrices based on the gradients of the multiple sub-keys, and to obtain the gradients of the multiple sub-value matrices based on the gradients of the multiple sub-values. The cloud management platform notifies the multiple computing nodes to update the multiple subquery matrices, the multiple subkey matrices, and the multiple subvalue matrices based on the gradients of the multiple subquery matrices, the multiple subkey matrices, and the multiple subvalue matrices, so as to obtain the trained model.

8. The method according to claim 7, characterized in that, The completion time of the plurality of computing nodes obtaining the gradients of the plurality of subquery matrices is earlier than the start time of the plurality of computing nodes obtaining the gradients of the plurality of subkey matrices. The completion time of the plurality of computing nodes obtaining the gradients of the plurality of subkey matrices is earlier than the start time of the plurality of computing nodes obtaining the gradients of the plurality of subvalue matrices. The completion time of the plurality of computing nodes decomposing the gradients of the plurality of keys is earlier than the start time of the plurality of computing nodes decomposing the gradients of the plurality of values. The completion time of the plurality of computing nodes decomposing the gradients of the plurality of values ​​is earlier than the completion time of the plurality of computing nodes obtaining the gradients of the plurality of subvalue matrices.

9. The method according to claim 7, characterized in that, The completion time of the plurality of computing nodes obtaining the gradients of the plurality of subquery matrices is earlier than the start time of the plurality of computing nodes obtaining the gradients of the plurality of subvalue matrices. The completion time of the plurality of computing nodes obtaining the gradients of the plurality of subvalue matrices is earlier than the start time of the plurality of computing nodes obtaining the gradients of the plurality of subkey matrices. The completion time of the plurality of computing nodes decomposing the gradients of the plurality of values ​​is earlier than the start time of the plurality of computing nodes decomposing the gradients of the plurality of keys. The completion time of the plurality of computing nodes decomposing the gradients of the plurality of keys is earlier than the completion time of the plurality of computing nodes obtaining the gradients of the plurality of subkey matrices.

10. A cloud management platform, characterized in that, The cloud management platform is used to manage the infrastructure that provides cloud services, the infrastructure including a cluster of computing nodes, and the cloud management platform includes: The receiving module is used to receive a model training request sent by a tenant, wherein the model training request is used to indicate the model to be trained, the model is built based on a grouped query attention mechanism, and the grouped query attention mechanism is used to indicate the number of multiple queries, the number of multiple keys, and the number of multiple values; The selection module is used to select multiple computing nodes from the computing node cluster based on the model training request. The processing module is configured to, when the number of multiple queries is greater than or equal to the number of multiple computing nodes, and the number of multiple keys and the number of multiple values ​​are less than the number of multiple computing nodes, decompose the model into multiple sub-models, deploy the multiple sub-models on the multiple computing nodes, and notify the multiple computing nodes to use the multiple sub-models to process the training data to obtain the multiple queries, multiple subkeys, and multiple subvalues. The first notification module is used to notify the multiple computing nodes to merge the multiple subkeys to obtain the multiple keys, and to merge the multiple subvalues ​​to obtain the multiple values; The second notification module is used to notify the multiple computing nodes to train the multiple sub-models based on the multiple queries, the multiple keys, and the multiple values ​​to obtain the trained model.

11. The cloud management platform according to claim 10, characterized in that, The number of queries is an integer multiple of the number of keys, the number of keys is equal to the number of values, and the number of subkeys, the number of subvalues, and the number of compute nodes are equal.

12. The cloud management platform according to claim 10 or 11, characterized in that, The model includes a query matrix, a key matrix, and a value matrix. The multiple sub-models include multiple sub-query matrices obtained by decomposing the query matrix, multiple sub-key matrices obtained by decomposing the key matrix, and multiple sub-value matrices obtained by decomposing the value matrix. The multiple sub-query matrices are used by the multiple computing nodes to process the training data and obtain the multiple queries; The plurality of subkey matrices are used by the plurality of computing nodes to process the training data and obtain the plurality of subkeys; The plurality of sub-value matrices are used by the plurality of computing nodes to process the training data and obtain the plurality of sub-values.

13. The cloud management platform according to any one of claims 10 to 12, characterized in that, The model training request is also used to instruct the tenant to set the training strategy for the model as tensor parallelism, and the tenant to allow the cloud management platform to determine the degree of parallelism of the tensor parallelism; The selection module is used to determine the parallelism based on the number of multiple queries, the number of multiple keys, and the number of multiple values, wherein the number of multiple queries is greater than or equal to the parallelism, and the number of multiple keys and the number of multiple values ​​are less than the parallelism. The cloud management platform selects multiple computing nodes from the computing node cluster, with the number equal to the parallelism.

14. The cloud management platform according to any one of claims 10 to 13, characterized in that, The end time of the multiple computing nodes acquiring the multiple subkeys is earlier than the start time of the multiple computing nodes acquiring the multiple subvalues. The end time of the multiple computing nodes acquiring the multiple subvalues ​​is earlier than the start time of the multiple computing nodes acquiring the multiple queries. The end time of the multiple computing nodes merging the multiple subkeys is earlier than the start time of the multiple computing nodes merging the multiple subvalues. The end time of the multiple computing nodes merging the multiple subvalues ​​is earlier than the end time of the multiple computing nodes acquiring the multiple queries.

15. The cloud management platform according to any one of claims 10 to 13, characterized in that, The end time of the multiple computing nodes obtaining the multiple sub-values ​​is earlier than the start time of the multiple computing nodes obtaining the multiple sub-keys. The end time of the multiple computing nodes obtaining the multiple sub-keys is earlier than the start time of the multiple computing nodes obtaining the multiple queries. The end time of the multiple computing nodes merging the multiple sub-values ​​is earlier than the start time of the multiple computing nodes merging the multiple sub-keys. The end time of the multiple computing nodes merging the multiple sub-keys is earlier than the end time of the multiple computing nodes obtaining the multiple queries.

16. The cloud management platform according to claim 12, characterized in that, The second notification module is used for: The multiple computing nodes are notified to use the multiple sub-models to process the multiple queries, the multiple keys, and the multiple values ​​to obtain the processing results of the training data; The multiple computing nodes are notified to obtain the gradients of the multiple queries, the gradients of the multiple keys, and the gradients of the multiple values ​​based on the processing results and the actual processing results of the training data. The multiple computing nodes are notified to decompose the gradients of the multiple keys to obtain the gradients of the multiple sub-keys, and to decompose the gradients of the multiple values ​​to obtain the gradients of the multiple sub-values; The multiple computing nodes are notified to obtain the gradients of the multiple sub-query matrices based on the gradients of the multiple queries, to obtain the gradients of the multiple sub-key matrices based on the gradients of the multiple sub-keys, and to obtain the gradients of the multiple sub-value matrices based on the gradients of the multiple sub-values. The multiple computing nodes are notified to update the multiple subquery matrices, the multiple subkey matrices, and the multiple subvalue matrices based on the gradients of the multiple subquery matrices, the multiple subkey matrices, and the multiple subvalue matrices, so as to obtain the trained model.

17. The cloud management platform according to claim 16, characterized in that, The completion time of the plurality of computing nodes obtaining the gradients of the plurality of subquery matrices is earlier than the start time of the plurality of computing nodes obtaining the gradients of the plurality of subkey matrices. The completion time of the plurality of computing nodes obtaining the gradients of the plurality of subkey matrices is earlier than the start time of the plurality of computing nodes obtaining the gradients of the plurality of subvalue matrices. The completion time of the plurality of computing nodes decomposing the gradients of the plurality of keys is earlier than the start time of the plurality of computing nodes decomposing the gradients of the plurality of values. The completion time of the plurality of computing nodes decomposing the gradients of the plurality of values ​​is earlier than the completion time of the plurality of computing nodes obtaining the gradients of the plurality of subvalue matrices.

18. The cloud management platform according to claim 16, characterized in that, The completion time of the plurality of computing nodes obtaining the gradients of the plurality of subquery matrices is earlier than the start time of the plurality of computing nodes obtaining the gradients of the plurality of subvalue matrices. The completion time of the plurality of computing nodes obtaining the gradients of the plurality of subvalue matrices is earlier than the start time of the plurality of computing nodes obtaining the gradients of the plurality of subkey matrices. The completion time of the plurality of computing nodes decomposing the gradients of the plurality of values ​​is earlier than the start time of the plurality of computing nodes decomposing the gradients of the plurality of keys. The completion time of the plurality of computing nodes decomposing the gradients of the plurality of keys is earlier than the completion time of the plurality of computing nodes obtaining the gradients of the plurality of subkey matrices.

19. A computing device cluster, characterized in that, The computing device cluster includes at least one computing device, each computing device including a processor and memory: The memory is used to store instructions; The processor is configured to, according to the instructions, cause the computing device cluster to perform the method of any one of claims 1 to 9.

20. A computer storage medium, characterized in that, The computer storage medium stores one or more instructions that, when executed by one or more computers, cause the one or more computers to perform the method of any one of claims 1 to 9.

21. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1 to 9.

Images