Model parameter processing method, model training method, computing device and computing cluster

Abstract

The embodiment of the invention discloses a model parameter processing method, a model training method, computing equipment and a computing cluster, and relates to the technical field of artificial intelligence. The model parameter processing method is applied to a target node. The method comprises the following steps: acquiring gradient data from each computing node which shares a local memory based on a computing fast link (CXL) protocol; the gradient data is stored in a local memory of the computing node; the gradient data is obtained by training a locally deployed target model by the computational node; the target model is a distributed training model. Determining gradient fusion data according to the gradient data of each computational node; the gradient fusion data is used for updating model parameters of the target model by each computing node; and storing the gradient fusion data in the local memory of each computing node. Therefore, the model training efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model parameter processing method, a model training method, a computing device, and a computing cluster. Background Art

[0002] Collective communication is an important communication mode in the distributed training of large-scale deep learning models. It defines a series of standardized information exchange interfaces and provides an efficient communication mechanism for parallel training and computation between multiple distributed training nodes. However, during distributed training, frequent cross-node data transmission is often required between multiple training nodes to process model parameters, which results in high communication costs and low training efficiency. Summary of the Invention

[0003] The embodiments of the present application provide a model parameter processing method, a model training method, a computing device, and a computing cluster, which can improve the efficiency of model training.

[0004] In a first aspect, embodiments of the present application provide a model parameter processing method, applied to a target node. The method comprises: obtaining gradient data from each computing node that shares local memory based on the Compute Express Link (CXL) protocol; storing the gradient data in the local memory of the computing node; the gradient data is obtained by the computing node training a locally deployed target model; the target model is a distributed training model; determining gradient fusion data based on the gradient data of each computing node; using the gradient fusion data on each computing node to update the model parameters of the target model; and storing the gradient fusion data in the local memory of each computing node.

[0005] In this embodiment, the target node shares local memory with each compute node based on the CXL protocol. This allows the target node to access the local memory of each compute node as if it were its own, reading and writing data from the compute node's local memory to process model parameters. This frees the data transmission process from network constraints, avoids frequent data exchange, improves data transmission efficiency between nodes, and enhances model parameter processing efficiency, thereby enhancing the efficiency of distributed training.

[0006] In one possible implementation, the method further includes: when the first computing node is in a faulty state, obtaining training information corresponding to the first computing node; wherein the first computing node is any computing node among multiple computing nodes; the training information includes current model parameters of the target model on the first computing node; and storing the training information corresponding to the first computing node in the local memory of a backup computing node that shares local memory based on the CXL protocol, so that the backup computing node takes over the first computing node based on the training information to continue training the target model.

[0007] In this embodiment, if the first computing node is in a faulty state, the training task cannot be continued, and the training task on the first computing node needs to be migrated to the backup computing node, which performs "breakpoint resumption training", that is, continues to train the target model based on the current model parameters of the target model. By monitoring the fault status of each computing node in real time by the target node, the faulty computing node (that is, the first computing node in a faulty state) can be discovered in time, and the training tasks on the faulty computing node can be migrated in time. In addition, the target node transfers the training information from the first computing node to the local memory of the backup computing node based on the CXL protocol, thereby improving the efficiency of training task migration.

[0008] In a possible implementation, the training information further includes the current training round number.

[0009] Since the current training round number clearly defines the training progress of the target model before the interruption, it can accurately continue training from the round where it was last stopped (that is, the current training round number) when resuming training at a breakpoint, avoiding repeated training of the completed part and improving training efficiency.

[0010] In one possible implementation, after acquiring gradient data from each compute node sharing local memory based on the CXL protocol, the method further includes storing the gradient data in the memory of a processor on the target node; and the processor is configured to calculate gradient fusion data based on the gradient data from each compute node. This reduces data transmission and improves communication performance.

[0011] In one possible implementation, after determining the gradient fusion data based on the gradient data of each computing node, the method further includes storing the gradient fusion data in shared local memory. This allows the gradient fusion data to be not only available to each computing node in real time but also to be fully recorded, facilitating review of training processes and data tracing.

[0012] In a second aspect, an embodiment of the present application provides a model training method, which is applied to a target computing node. The method includes: sending gradient data to a target node that shares local memory based on the Compute Quick Link (CXL) protocol; wherein the target node is used to calculate gradient fusion data; the gradient data is stored in the local memory of the computing node; the gradient data is obtained by the target computing node training a locally deployed target model; the target model is a model distributedly trained by multiple computing nodes; the target model is a distributedly trained model. Receive the gradient fusion data sent by the target node; wherein the gradient fusion data is determined by the target node based on the gradient data corresponding to each computing node; the gradient fusion data is used by each computing node to update the model parameters of the target model; based on the gradient fusion data, the model parameters of the locally deployed target model are updated.

[0013] In this way, for the target computing node, the CXL protocol can better improve the data exchange speed between the target node and the target node, reduce the communication overhead in the distributed training process, and thus better improve the efficiency of model training.

[0014] In one possible implementation, the method further includes: in the event of a fault, sending locally stored training information to a backup computing node that shares local memory based on the CXL protocol; the training information includes current model parameters of the target model.

[0015] In this way, the target computing node can send the training information to the backup computing node in a timely and direct manner, thereby improving the efficiency with which the backup computing node obtains its training information, thereby improving the training efficiency.

[0016] In a possible implementation, the method further includes: in the event of a fault, sending locally stored training information to the target node; the training information includes current model parameters of the target model.

[0017] In this way, the target computing node can send the training information to the target node in a timely and efficient manner, ensuring that the training information will not be lost due to failures, so that the target node can obtain the training information corresponding to the target computing node, which is more convenient for the target node to uniformly manage its corresponding training information and allocate it to the corresponding backup computing node, so that the backup computing node can take over the target computing node based on the training information to continue training the target model, ensuring training integrity and training continuity.

[0018] In a third aspect, an embodiment of the present application provides a computing device; the computing device can serve as a target node. The computing device includes: a memory and a processor; the memory and the processor are coupled; the memory is used to store program instructions; the processor is used to execute program instructions so that the computing device performs the following: obtaining gradient data from each computing node that shares local memory based on the Compute Quick Link (CXL) protocol; the gradient data is stored in the local memory of the computing node; the gradient data is obtained by the computing node training a locally deployed target model; the target model is a distributed training model. Based on the gradient data of each computing node, gradient fusion data is determined; the gradient fusion data is used by each computing node to update the model parameters of the target model. The gradient fusion data is stored in the local memory of each computing node.

[0019] In one possible implementation, the computing device further performs: when the first computing node is in a faulty state, obtaining training information corresponding to the first computing node; wherein the first computing node is any computing node among the multiple computing nodes; the training information includes the current model parameters of the target model on the first computing node; and storing the training information corresponding to the first computing node in the local memory of a backup computing node that shares local memory based on the CXL protocol, so that the backup computing node takes over the first computing node based on the training information to continue training the target model.

[0020] In one possible implementation, after the computing device obtains gradient data from each computing node sharing a local memory based on the CXL protocol, the computing device further performs: storing the gradient data in the memory of a processor of a target node; and the processor is configured to calculate gradient fusion data based on the gradient data of each computing node.

[0021] In a possible implementation, after the computing device determines the gradient fusion data according to the gradient data of each computing node, the computing device further performs: storing the gradient fusion data in a shared local memory.

[0022] In a fourth aspect, an embodiment of the present application provides a computing device; the computing device can serve as a target computing node.

[0023] The computing device includes: a memory and a processor; the memory and the processor are coupled; the memory is used to store program instructions; the processor is used to execute the program instructions, so that the computing device performs the following operations: sending gradient data to a target node that shares local memory based on the Compute Express Link (CXL) protocol; wherein the target node is used to calculate gradient fusion data; the gradient data is stored in the local memory of the computing node; the gradient data is obtained by the target computing node training a locally deployed target model; the target model is a distributed training model; receiving the gradient fusion data sent by the target node; wherein the gradient fusion data is determined by the target node based on the gradient data corresponding to each computing node; the gradient fusion data is used by each computing node to update the model parameters of the target model; and updating the model parameters of the locally deployed target model based on the gradient fusion data.

[0024] In one possible implementation, the computing device further performs: in the event of a failure, sending locally stored training information to a backup computing node that shares local memory based on the CXL protocol; the training information includes current model parameters of the target model.

[0025] In a possible implementation, the computing device further executes: in the case of a fault state, sending locally stored training information to the target node; the training information includes current model parameters of the target model.

[0026] In a fifth aspect, an embodiment of the present application provides a computing cluster; the computing cluster includes a target node and multiple computing nodes; wherein the target node is the computing device involved in the third aspect above; the target computing node is the computing device involved in the fourth aspect above.

[0027] In a possible implementation, the target node is a node other than the multiple computing nodes in the computing cluster.

[0028] In a sixth aspect, a storage medium is provided, on which computer program instructions are stored; when the computer program instructions are executed by a computing device, the computing device implements the method as described above.

[0029] In a seventh aspect, a computer program product is provided, which includes a computer program. When at least one processor executes the computer program, the at least one processor executes the above method.

[0030] Among them, the technical effects brought about by any implementation method in the third to seventh aspects can refer to the technical effects brought about by different implementation methods in the first and / or second aspects, and will not be repeated here.

[0031] Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A schematic diagram of the structure of a computing cluster provided in an embodiment of the present application;

[0033] Figure 2 A schematic diagram of the structure of another computing cluster provided in an embodiment of the present application;

[0034] Figure 3 A schematic diagram of the structure of another computing cluster provided in an embodiment of the present application;

[0035] Figure 4 A schematic diagram of the structure of another computing cluster provided in an embodiment of the present application;

[0036] Figure 5 A schematic diagram of another computing cluster structure provided in an embodiment of the present application

[0037] Figure 6 A schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0038] Figure 7 A flow chart of a model parameter processing method provided in an embodiment of the present application;

[0039] Figure 8A visual flowchart of a model parameter processing method provided in an embodiment of the present application;

[0040] Figure 9 A flowchart of a model training method provided in an embodiment of the present application;

[0041] Figure 10 A flow chart of another model parameter processing method provided in an embodiment of the present application;

[0042] Figure 11 A schematic diagram of the structure of a model parameter processing device provided in an embodiment of the present application;

[0043] Figure 12 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0044] Figure 13 A schematic diagram of the structure of another computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0046] In the description of this application, unless otherwise specified, " / " indicates that the objects associated before and after are in an "or" relationship, for example, A / B can represent A or B; "and / or" in this application is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural.

[0047] Furthermore, in the description of this application, unless otherwise specified, "plurality" means two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0048] In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. Those skilled in the art will understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit differences. At the same time, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or design schemes. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way for easy understanding.

[0049] In the distributed training of large-scale deep learning models, collective communication is a crucial communication mode. It defines a series of standardized information exchange interfaces, providing an efficient communication mechanism for parallel training and computation between multiple distributed training nodes (also known as computing nodes, such as training servers). However, during distributed training, frequent cross-node data transmission is often required between multiple training nodes to process model parameters, resulting in high communication costs and low training efficiency.

[0050] In light of this, embodiments of the present application provide a model parameter processing method in which a target node obtains gradient data from each compute node that shares local memory using the Compute Express Link (CXL) protocol. After calculating gradient fusion data based on the gradient data, the target node directly writes the gradient fusion data into the local memory of each compute node. In this way, the target node shares local memory with each compute node via the CXL protocol, allowing it to access the local memory of each compute node as if it were accessing its own local memory. This improves the data transfer rate between nodes, thereby increasing the efficiency of model parameter processing and model training.

[0051] The local memory of the computing node includes, but is not limited to, dynamic random-access memory (DRAM), non-volatile memory (NVM), and double data rate synchronous dynamic random access memory (DDRSDRAM). The embodiments of the present application do not limit the type of local memory.

[0052] Gradient data refers to the gradient parameters (i.e., adjustment values) for the parameters to be trained in the model, obtained after the computing nodes perform a round of training on the models deployed by each node (such as the target model). Gradient fusion data is obtained after the gradient data obtained by the training of each computing node is fused and calculated, and is used to update the parameters to be trained in the model. During the actual training process, the parameters to be trained of the model on each computing node are updated based on the gradient fusion data. The aforementioned fusion calculation is, for example, summation, averaging, or weighted average, and the embodiments of the present application do not limit the specific operations of the fusion calculation.

[0053] In some embodiments, the method provided in the embodiments of the present application is applied to the target node. The method includes: first, obtaining gradient data from each computing node that shares local memory based on the CXL protocol; the gradient data is stored in the local memory of the computing node, and the gradient data is obtained by the computing node training the locally deployed target model, and the target model is a distributed training model. Secondly, based on the gradient data of each computing node, the gradient fusion data is determined, and the gradient fusion data is used by each computing node to update the model parameters of the target model, that is, the parameters to be trained. Finally, the gradient fusion data is stored in the local memory of each computing node.

[0054] This allows the target node to access the local memory of each compute node as if it were its own, sharing local memory with the compute nodes based on the CXL protocol. This allows the target node to read and write data from the compute node's local memory to process model parameters. This frees the data transmission process from network constraints, avoids frequent data exchange, improves data transmission efficiency between nodes, and enhances model parameter processing efficiency, thereby enhancing the efficiency of distributed training.

[0055] The following is an exemplary introduction to the system architecture of the embodiment of the present application.

[0056] like Figure 1 As shown, an embodiment of the present application provides a computing cluster. The computing cluster includes multiple computing nodes, each of which has a target model locally deployed, and the target model is a distributed training model. The target node and the multiple computing nodes share local memory based on the CXL protocol.

[0057] The target node is used to obtain gradient data from each compute node that shares local memory using the CXL protocol. This gradient data is stored in the compute node's local memory and is obtained by the compute node training the locally deployed target model. The target node also determines gradient fusion data based on the gradient data from each compute node. This fusion data is used by each compute node to update the model parameters of the target model. Furthermore, the target node stores the fusion gradient data in the local memory of each compute node.

[0058] In some embodiments, the target node may be a node other than the plurality of computing nodes in the computing cluster. That is, the computing cluster includes the plurality of computing nodes and the target node.

[0059] Multiple compute nodes in a computing cluster are used for distributed training of the target model, while target nodes are not used for target model training. This allows the target nodes to focus on the fusion and distribution of gradient data, while multiple compute nodes focus on training the target model. This effectively avoids task conflicts and resource contention, improves the processing efficiency of model parameters, and enhances the overall operational efficiency of the computing cluster, thereby improving the efficiency of target model training. Furthermore, using nodes other than multiple compute nodes as target nodes ensures the computation of gradient fusion data even if some of the multiple compute nodes fail, minimizing the impact of these failed compute nodes on training tasks and improving the fault tolerance of the computing cluster.

[0060] Combine Figure 1 As shown, the computing cluster includes multiple computing nodes (computing nodes 1-3) and a target node. Local memory 1 represents the local memory of computing node 1, local memory 2 represents the local memory of computing node 2, and local memory 3 represents the local memory of computing node 3. The connection between the target node and local memory 1-3 indicates that the target node is connected to local memory 1-3 based on the CXL protocol to realize the sharing of local memory between the target node and each computing node. Gradient data is used to represent the adjustment value of each parameter to be trained in the target model. a, b, and c in the local memory can be understood as different values ​​in the gradient data, corresponding to different parameters to be trained in the model. Based on this, the gradient data obtained by the target node from computing node 1 includes a1, b1, and c1, the gradient data obtained from computing node 2 includes a2, b2, and c2, and the gradient data obtained from computing node 3 includes a3, b3, and c3. Specifically, the target node is the gradient data directly obtained from the local memory of each computing node.

[0061] In one implementation, the target node includes a CXL controller. The CXL protocol is a high-performance interconnect standard that allows a node to access the memory of other CXL-supported nodes as if they were accessing its own local memory. The CXL controller is the hardware implementation unit of the CXL protocol and can be a CXL memory expander. The target node exchanges data with each compute node through its CXL controller, enabling the target node to share local memory with each compute node. For example, the target node can directly read data from the local memory of each compute node into its own memory, or directly write data to the local memory of each compute node.

[0062] In one implementation, the local memory of a compute node includes a CXL controller and a memory controlled by the CXL controller. The memory can provide memory resources for storing gradient data, gradient fusion data, and the like. The CXL controller of the compute node is communicatively connected to the CXL controller of the target node via the CXL protocol, configured to receive data read requests from the target node and provide the target node with data stored in the memory (e.g., gradient data). Furthermore, the CXL controller of the compute node is configured to receive data write requests from the target node and provide storage space for the target node to write data (e.g., gradient fusion data) to the memory.

[0063] In this way, by storing gradient data, gradient fusion data, etc. in the local memory of each computing node, not only can the target node obtain and distribute data in a more targeted manner, improving data processing efficiency and thus improving the processing efficiency of model parameters, but also through the collaboration of the CXL protocol and the CXL controller, efficient sharing and utilization of memory resources can be achieved, avoiding task conflicts and resource contention, and ensuring the stability of the computing cluster operation and the training efficiency of the target model.

[0064] In some embodiments, the data stored in the local memory (such as gradient data, gradient fusion data, etc.) is organized as a key-value cache (KV cache). KV cache is a data organization format for storing data.

[0065] In some embodiments, the target node includes a graphics processing unit (GPU). The GPU is configured to calculate gradient fusion data based on the gradient data of each computation node. The processor's memory may be a cache in the GPU for temporarily storing frequently accessed data or computation results.

[0066] Specifically, the GPU in the target node acquires gradient data from each compute node, determines the gradient fusion data based on the gradient data from each compute node, and stores the gradient fusion data in the local memory of each compute node. Because GPUs have powerful parallel computing capabilities, they can improve the processing efficiency of model parameters, thereby improving the efficiency of distributed training.

[0067] In some embodiments, the target node is further configured to store the gradient data in the memory of the target node's processor. This allows the GPU to perform calculations directly based on the data in the processor's memory, reducing data transmission and further improving the processing efficiency of model parameters.

[0068] In some embodiments, the target node is configured to obtain gradient data from each computing node, including: the target node obtains gradient data from each computing node once each time the computing node trains a target model, thereby ensuring timely acquisition of gradient data.

[0069] In some embodiments, the target node is also configured to store the fused gradient data in shared local memory. Because the target node shares local memory with each compute node, storing the fused gradient data in shared local memory not only makes the fused gradient data available to each compute node in real time but also preserves a complete record, facilitating review of training processes and data tracing.

[0070] In some embodiments, the target computing node is configured to send gradient data to a target node sharing local memory based on the CXL protocol. The target computing device is further configured to receive gradient fusion data sent by the target node. Furthermore, the target computing device is further configured to update model parameters of a locally deployed target model based on the gradient fusion data.

[0071] In this way, after training the target model, each compute node directly shares the gradient data obtained with the target node via the CXL protocol, eliminating the need to exchange gradient data with other compute nodes. The node then directly receives the gradient fusion data sent by the target node to process the model parameters. This eliminates the need for network transmission, freeing the data transmission process from network constraints and avoiding frequent data exchange. This improves data communication efficiency and performance between nodes, enhances model parameter processing efficiency, and thus enhances the efficiency of distributed training.

[0072] Exemplarily, the target computing node is any computing node among multiple computing nodes of the computing cluster.

[0073] In some embodiments, the target computing node is further used to train a locally deployed target model to obtain gradient data. The target computing node locally deploys a target model to be trained and holds a training dataset for training the target model. The training dataset includes multiple training data, and each training data may include sample data and label data corresponding to the sample data.

[0074] Illustratively, the training data in the training data sets held by different computing nodes may be the same or different.

[0075] Exemplarily, the training datasets held by multiple computing nodes collectively constitute a complete training dataset. It will be appreciated that the complete training dataset is divided into multiple small batches of training datasets, which are then distributed to different computing nodes. These small batches of training datasets, i.e., the training datasets held by each computing node, can be stored in a storage unit other than local memory within each computing node, allowing each computing node to input them into its locally deployed target model for model training.

[0076] In some embodiments, the computing cluster further includes a scheduler for scheduling resources in the computing cluster.

[0077] In one implementation, the scheduler is further configured to deploy the target model on each computing node.

[0078] In one implementation, the scheduler is further configured to divide the complete training data set into multiple small batches of training data sets, and distribute the small batches of training data sets to different computing nodes.

[0079] In this way, multiple computing nodes can perform distributed training on the target model.

[0080] In some embodiments, the local memory of the computing node also stores training information, which is used to represent information generated by the computing node during the process of training the locally deployed target model.

[0081] In one implementation, the training information includes current model parameters of the target model on the computing node (which may be referred to as current model parameters).

[0082] Exemplarily, the training information stored in the local memory of the computing node is described as follows: Assume that the current model parameters of the target model on the computing node are the model parameters obtained in the i-th round of training, and the computing node performs the i+1th round of training based on the target model. If the i+1th round of training is not completed (such as training failure, training interruption, etc.), the current model parameters stored in the local memory of the computing node are the model parameters obtained in the i-th round of training. If the i+1th round of training is completed, the current model parameters stored in the local memory of the computing node are the model parameters obtained in the i+1th round of training. i is an integer greater than or equal to 0.

[0083] In another implementation, the training information includes the current training round number. For example, if the computing node has performed the i-th round of training on the target model, the current training round number is the i-th round.

[0084] It can be understood that the i-th round of training is based on the model parameters of the target model obtained in the i-1th round. The i-th round of training corresponds to the model parameters obtained in the i-th round of training, or the model parameters obtained in the i-1th round of training. In other words, when the target model is trained in the i-th round, the current training round is the i-th round. If the training is complete, the current model parameters corresponding to the i-th round are the model parameters obtained in the i-th round of training. If the training is not complete, the current model parameters corresponding to the i-th round are the model parameters obtained in the i-1th round of training.

[0085] This can be further understood as follows: within the same round of model training, the current training round is round i. If training is complete, the corresponding gradient data is the gradient data obtained in round i, and the current model parameters are obtained based on the gradient data obtained in round i. If training is incomplete, the corresponding gradient data is the gradient data obtained in round i-1, and the current model parameters are obtained based on the gradient data obtained in round i-1. This is because in cases of training failure or interruption, although the number of training rounds increases, the gradient data is not updated, resulting in the current model parameters not being updated.

[0086] Illustratively, the current training round number may be used as an index of the gradient data, to indicate that the gradient data is obtained through training through the training round number.

[0087] Specifically, when the training information includes the current training round number, the target node is also used to determine the gradient data corresponding to the maximum training round number in each computing node, and used as the gradient data for determining the gradient fusion data.

[0088] The maximum number of training rounds is the maximum number of training rounds currently stored in the local memory of each compute node. This ensures that the gradient data used to calculate the gradient fusion data is obtained from the most recent training round and is from the same training round. This enables asynchronous data storage and updates, as well as alignment of the training rounds corresponding to the gradient data, resulting in more accurate gradient fusion data determined from the gradient data.

[0089] In another implementation, the training information includes the current model parameters and the current number of training rounds of the target model on the computing node. In this way, the training information can more comprehensively reflect the current training status of the target model.

[0090] In some embodiments, the target node is further configured to obtain training information corresponding to the first computing node if the first computing node is in a faulty state. The target node is further configured to store the training information corresponding to the first computing node in the local memory of a backup computing node that shares local memory based on the CXL protocol, so that the backup computing node can take over from the first computing node and continue training the target model based on the training information.

[0091] The first computing node is any computing node among the plurality of computing nodes. The training information here includes the current model parameters of the target model on the first computing node.

[0092] In this way, the target node and the backup computing node share local memory based on the CXL protocol. If the first computing node is in a faulty state, the training task cannot be continued, and the training task on the first computing node needs to be migrated to the backup computing node, which performs "breakpoint resume training", that is, continues to train the target model based on the current model parameters. By real-time monitoring of the fault status of each computing node by the target node, the faulty computing node (that is, the first computing node in a faulty state) can be discovered in time, and the training tasks on the faulty computing node can be migrated in time.

[0093] Exemplarily, the computing cluster further includes a standby computing node. Specifically, the standby computing node is a node in the computing cluster that can be scheduled by a scheduler in the cluster and is not assigned a training task for the target model.

[0094] For example, the scheduler can determine a backup computing node in the computing cluster before the first computing node is in a faulty state. In this way, if any computing node fails, the backup computing node can promptly take over its training task, ensuring the feasibility of the training.

[0095] For example, the scheduler may determine a backup computing node when the first computing node is in a faulty state, so that the backup computing node does not need to perform unnecessary calculations, thereby releasing resources on the backup computing node.

[0096] In some embodiments, the standby computing node is deployed with a target model and holds a training dataset for training the target model.

[0097] The deployment time of deploying the target model to the spare computing nodes is relatively flexible.

[0098] For example, the target model can be deployed for the backup computing node before the first computing node is in a faulty state. In this way, when a fault occurs, the backup computing node can be quickly put into use and directly take over the first computing node to continue training, shortening the fault response time and improving the model parameter processing efficiency, thereby improving the training efficiency and ensuring the timely recovery and continuation of the training task.

[0099] Exemplarily, the target model can also be deployed for the backup computing node when the first computing node is in a faulty state. In this way, when it is determined that the first computing node has failed, the resources of the backup computing node are occupied to avoid unnecessary preparation work by the backup computing node when no failure occurs, thereby saving computing resources.

[0100] Exemplarily, the target model may be deployed on a spare computing node by the scheduler.

[0101] In one implementation, the training data set held by the standby computing node is the same as the training data set held by the first computing node.

[0102] Specifically, the scheduler can allocate the training dataset used by the first computing node to train the target model to the backup computing node. This allows the backup computing node to continue training the target model based on the training dataset and training information. In other words, the training task of the first computing node is migrated to the backup computing node, which can seamlessly take over the training task. This ensures that the backup computing node is aligned with the first computing node in many aspects when taking over training from the first computing node, avoiding model performance deviations or training interruptions caused by data inconsistencies.

[0103] In one implementation, the training information corresponding to the first computing node also includes the current training round number. In this way, the standby computing node not only continues training the target model based on the current model parameters, but also continues training the target model at the round number where the first computing node interrupted training.

[0104] In an embodiment of the present application, the training information may include the current model parameters of the target model on the first computing node. In this way, the standby computing node can continue to train the target model based on the training data set and current model parameters used by the first computing node to train the target model. In addition, the training information may also include the current number of training rounds. Since the current number of training rounds clarifies the training progress of the target model that has been completed before the interruption, it allows the training to be continued accurately from the last stopped round (i.e., the current number of training rounds) when resuming training at a breakpoint, thereby avoiding repeated training of the completed part and improving training efficiency.

[0105] In some embodiments, as Figure 2As shown, the target node is also connected to the local memory of the backup compute node via the CXL protocol. This allows the target node and the backup compute node to share local memory, allowing training information to be written directly to the local memory of the backup compute node. In addition, when the backup compute node takes over the distributed training of the target model from the first compute node, the backup compute node can be considered as a node among multiple compute nodes. In this case, since the target node and the backup compute node share local memory, data can be read from or written to the local memory of the backup compute node.

[0106] Combine Figure 2 As shown in the figure, A1, A2, and A3 are the training information of computing nodes 1-3 respectively. Taking computing node 1 as an example, A1 includes the model parameters (i.e., current model parameters) obtained by computing node 1 in the i-th round of training the target model. It may also include the i-th round of current training for computing node 1. When the target node obtains data from computing node 1, it first obtains a1, b1, and c1. Assume that computing node 1 is a faulty computing node. In this case, the target node obtains the training information A1 corresponding to computing node 1 from computing node 1 and stores A1 in the local memory of the backup computing node to achieve breakpoint resumption training.

[0107] Exemplarily, the local memory of the backup computing node includes a CXL controller and a memory controlled by the CXL controller. The CXL controller of the backup computing node communicates with the CXL controller of the target node via the CXL protocol to enable the target node and the backup computing node to share local memory.

[0108] Exemplarily, the number of backup computing nodes corresponds to the number of first computing nodes in a failed state. It is understood that if the number of first computing nodes in a failed state is one, then only one backup computing node is required to take over from the failed computing node and continue training the target model. If the number of first computing nodes in a failed state is three, then three backup computing nodes are required to take over from the failed computing node and continue training the target model.

[0109] In this way, if a computing node fails during a training task, the training task is migrated to a spare computing node, ensuring the continuity of the training task, improving the efficiency of model parameter processing, and improving the accuracy and efficiency of distributed training.

[0110] In some embodiments, the target computing node is further configured to store the corresponding training information in its local memory after each round of training of the target model. For example, after each round of training of the target model, the target computing node stores the training information corresponding to the round of training in the local memory.

[0111] In other embodiments, the target computing node is further configured to store the corresponding training information in its local memory after a preset number of training rounds. For example, the preset number of training rounds can be set to 2, 3, or 4 rounds. Based on this, the target computing node can store the training information corresponding to the next training round in its local memory after every two training rounds.

[0112] In other embodiments, the target node is also used to obtain training information from each computing node, and when the first computing node is in a faulty state, store the training information corresponding to the first computing node in the memory of the backup computing node, so that the backup computing node can take over the first computing node based on the training information and continue to train the target model.

[0113] In this way, not only can real-time detection of each computing node be achieved, but also faulty computing nodes can be discovered in time and training tasks on the faulty computing nodes can be migrated in time. The training information of each computing node can also be backed up to the memory of the target node, so that when the first computing node is unable to send the corresponding training information to the target node (such as when the first computing node is powered off or the system is abnormal), the most recent training information can be traced back and stored in the backup computing node, so that the backup computing node can resume the training process based on the most recent training information, ensuring the continuity and stability of the target model training and reducing the time and resource loss caused by training interruptions.

[0114] Exemplarily, the target node is configured to obtain training information from each computing node, including: the target node is configured to obtain training information corresponding to each computing node once each time the computing node trains the target model. This ensures that the most recent training information is the most up-to-date training information, which facilitates real-time monitoring of whether each computing node is in a fault state.

[0115] Exemplarily, the target node is configured to obtain training information from each computing node, including: the target node is configured to obtain the training information stored in the local memory of each computing node once every predetermined number of training rounds. This reduces the memory usage of the target node and frees up memory resources.

[0116] In some embodiments, if the training information includes the current model parameters of the target model, the backup computing node is used to update the parameters to be trained of the target model to the current model parameters when the training information is obtained, and then continue to train the target model based on the training data set, so that the backup computing node takes over the first computing node to continue training the target model.

[0117] In other embodiments, if the training information also includes the current number of training rounds, the backup computing node is further configured to set the number of training rounds to the current number of training rounds, and then continue training the target model based on the training dataset, thereby achieving more comprehensive alignment of the backup computing node with the training status of the first computing node. In this way, because the current number of training rounds clearly defines the training progress of the target model before the interruption, when resuming training from a breakpoint, training can be accurately resumed from the last stopped round, avoiding repeated training of the completed portion and improving training efficiency.

[0118] In one embodiment, after obtaining the training information, the target node can store the training information in the memory of the processor, thereby reducing data transmission and improving communication performance; or, the training information can be stored in the local memory of the target node, thereby reducing the memory usage of the processor and releasing more memory resources of the processor.

[0119] In some embodiments, the target computing node is further configured to send locally stored training information to the target node in the event of a failure. This allows the target computing node to promptly send training information to the target node, ensuring that the training information is not lost due to a failure, and allowing the target node to obtain the training information corresponding to the target computing node. Based on this, especially in the event of failures at multiple computing nodes, the target node can centrally manage its corresponding training information and assign it to the corresponding backup computing node, allowing the backup computing node to take over from the target computing node based on the training information and continue training the target model, ensuring training integrity and continuity.

[0120] Combine Figure 2 As shown, the target computing node can be computing node 1, computing node 2 or computing node 3. In the case of a fault state, its locally stored training information is shared with the target node through the CXL protocol, so that the target node can obtain its training information and uniformly manage and distribute its training information.

[0121] In other embodiments, the target computing node is further configured to send the training information corresponding to the target computing node to the target node after each round of training of the target model. This allows the target node to obtain the training information corresponding to each round of training of the target computing node, implement a backup of the training information, and monitor the failure of the target computing node in real time.

[0122] In some other embodiments, the target computing node is further configured to send the training information corresponding to the target computing node to the target node at a preset number of training rounds per interval, so as to free up more memory resources of the target node.

[0123] In other embodiments, the target compute node is further configured to, in the event of a failure, send locally stored training information to a backup compute node that shares local memory using the CXL protocol. In this manner, the target compute node and the backup compute node share local memory using the CXL protocol. This allows the target compute node to send training information to the backup compute node in a timely and direct manner, improving the backup compute node's ability to access its training information and thereby enhancing training efficiency.

[0124] like Figure 3 As shown, the target compute node is also connected to the backup compute node via the CXL protocol. Taking compute node 1 as the target compute node, for example, compute node 1 and the backup compute node are connected via the CXL protocol to enable them to share local memory. Therefore, if compute node 1 fails, training information A1 in local memory 1 of compute node 1 is directly sent to the backup compute node, improving the efficiency of the backup compute node taking over the training tasks of the first compute node, thereby enhancing training efficiency.

[0125] In other embodiments, the target computing node also shares local memory with other computing nodes based on the CXL protocol, where the other computing nodes are computing nodes other than the target computing node. The target computing node is further configured to obtain training information from the other computing nodes and, if the other computing nodes are in a faulty state, send the locally stored training information of the other computing nodes in the faulty state to a backup computing node or the target node.

[0126] In this way, in some specific scenarios, such as power outages or system failures in other computing nodes, the training information can be sent to the target computing node first, and the target computing node can determine whether the other computing nodes are in a faulty state. In the case of a faulty state, the most recent training information of the other computing nodes can be sent to the backup computing node or the target node to enable the other computing nodes to continue training the target model. At the same time, the resource usage of the target node can be reduced so that the target node can focus on the calculation of model parameters.

[0127] like Figure 4 As shown, using compute node 2 as the target compute node as an example, compute node 2 communicates with other compute nodes (compute nodes 1 and 3) and the backup compute node via the CXL protocol, enabling compute node 1 to share local memory with the other compute nodes and the backup compute node. Based on this, the target compute node (compute node 2) obtains the training information corresponding to compute nodes 1 and 3. If compute node 1 fails, the target compute node (compute node 2) directly sends the corresponding training information A1 of compute node 1 to the backup compute node or the target node, improving training efficiency.

[0128] In this way, the training information of other computing nodes is saved through the target computing node. When a computing node (such as computing node 1) fails, the training task of the failed node can be migrated to the backup computing node. Since the target computing node communicates with other computing nodes through the CXL protocol, the communication transmission overhead is reduced.

[0129] In other embodiments, Figure 5 As shown, each compute node shares local memory with the backup compute node and other compute nodes via the CXL protocol. This allows the target compute node, regardless of whether it is compute node 1, compute node 2, or compute node 3, to obtain training information from other compute nodes and send training information from the failed node to the target node or backup compute node. This provides greater flexibility and further ensures the continuity of training progress and tasks.

[0130] In the embodiments of the present application, the target computing node and / or target node is a computing device, which may specifically be a network device. The network device may include a server, etc. The server may be a single physical or logical server, or may be two or more physical or logical servers sharing different responsibilities and cooperating to implement various server functions.

[0131] Illustratively, the server may be a blade server, a high-density server, a rack server, a tower server, or the like.

[0132] The hardware of the computing device includes a processor, a basic input / output system (BIOS) chip, an out-of-band controller, and memory, while the software mainly includes the BIOS, an out-of-band management module, and an operating system (OS). Figure 6 As shown, an embodiment of the present application provides a structural schematic diagram of a computing device.

[0133] The processor may include a central processing unit (CPU), which includes one or more CPU cores. All CPU data processing operations are performed by the CPU cores. The more CPU cores a CPU includes, the faster the data processing speed. The processor may also include a GPU. In the embodiments of the present application, the processor in the target node and / or the target computing node refers to the CPU.

[0134] The BIOS chip is a chip installed on the motherboard that initializes and detects various hardware components during the computer's startup process. The BIOS chip includes a flash memory area.

[0135] The out-of-band management module is located in the out-of-band controller, and the operating system is located in the processor.

[0136] An out-of-band management module can be a management unit for non-business modules. For example, an out-of-band management module can remotely maintain and manage a computing device through a dedicated data channel. This out-of-band management module is completely independent of the computing device's operating system and can communicate with the BIOS and operating system through the computing device's out-of-band management interface.

[0137] Exemplarily, the out-of-band management module may include a management unit for the computing device's operating status, a management system in a management chip, a baseboard management controller (BMC), a system management module (SMM), etc. It should be noted that the embodiments of the present application do not limit the specific form of the out-of-band management module, and the above description is merely an example.

[0138] The operating system (OS) is a computer program that manages and controls the hardware and software resources of a computing device. Any other software must be supported by the OS to run. After a computing device is powered on, the BIOS first performs a series of operations, including self-tests and initialization, and then boots the OS, allowing the user to use the computing device normally.

[0139] BIOS is a set of programs embedded in the BIOS chip on the motherboard of a computing device. The main function of BIOS is to provide the lowest-level and most direct hardware settings and control for the computing device.

[0140] Memory, also known as internal memory or main memory, is installed in the memory slots on the motherboard of a computing device. It should be noted that the system architecture and application scenarios described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of system architecture and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0141] For ease of understanding, the model parameter processing method provided in the embodiment of the present application is exemplarily introduced below in combination with the above-mentioned computing cluster and the accompanying drawings.

[0142] like Figure 7 As shown, the embodiment of the present application provides a flow chart of a model parameter processing method, which is applied to the above-mentioned target node and executed by the target node. The method includes:

[0143] S701 , obtaining gradient data from each computing node sharing local memory based on the CXL protocol.

[0144] Gradient data is obtained by computing nodes when they train the locally deployed target model. Gradient data represents the adjusted values ​​of each parameter to be trained in the target model.

[0145] For example, gradient data is retrieved from the local memory of each compute node using the CXL protocol. CXL is a high-performance interconnect standard that allows nodes to access the memory of other CXL-supported nodes as if they were their own local memory. This allows the target node to share local memory with various computing devices, enabling data to be read and written to the local memory of each compute node.

[0146] In the embodiment of the present application, by obtaining data from the local memory of each computing node, not only can the target node obtain and distribute data in a more targeted manner, thereby improving data processing efficiency and thus improving the processing efficiency of model parameters, but also, through the collaboration of the CXL protocol and the CXL controller, efficient sharing and utilization of memory resources can be achieved, task conflicts and resource contention can be avoided, and the stability of the computing cluster operation and the efficiency of target model training can be guaranteed.

[0147] In some embodiments, obtaining gradient data from each computing node includes obtaining gradient data from each computing node once each time the computing node trains a target model, thereby ensuring timely acquisition of gradient data.

[0148] In some embodiments, the method further includes storing the gradient data in a memory of a processor of the target node. By storing the gradient data in the processor's memory, the GPU can directly calculate the gradient data, thereby reducing data transmission, further reducing communication overhead, and further improving the processing efficiency of model parameters.

[0149] In some embodiments, the method further includes: determining gradient data corresponding to the maximum number of training rounds in each computing node, and using the gradient data as the gradient data for determining the gradient fusion data.

[0150] The maximum number of training rounds is the maximum value of the current number of training rounds stored in the local memory of each computing node.

[0151] In this embodiment of the present application, the gradient data used to calculate the gradient fusion data is guaranteed to be obtained from the most recent training run and is obtained from the same training round. This allows for asynchronous data storage and updating, as well as alignment of the training rounds corresponding to the gradient data, resulting in more accurate gradient fusion data determined from the gradient data.

[0152] Exemplarily, the gradient data in the memory of the processor is stored in the form of a gradient data set.

[0153] S702: Determine gradient fusion data based on the gradient data of each computing node.

[0154] The gradient fusion data is used by each computing node to update the model parameters of the target model.

[0155] In some embodiments, determining the gradient fusion data according to the gradient data of each computing node includes: performing a reduction calculation according to the gradient data of each computing node to obtain the gradient fusion data.

[0156] In one implementation, the reduced calculation includes: summation, average value, floating point, maximum, minimum, etc. The embodiment of the present application does not limit the specific operation of the fusion calculation.

[0157] Exemplarily, performing reduction calculations based on the gradient data of each computing node to obtain gradient fusion data includes: summing the gradient data of each computing node to obtain gradient fusion data.

[0158] Exemplarily, performing reduction calculations based on the gradient data of each computing node to obtain the gradient fusion data includes: averaging the gradient data of each computing node to obtain the gradient fusion data.

[0159] In some embodiments, the method further includes storing the gradient fusion data in a shared local memory. The shared memory can be the local memory of the target node. Since the target node shares the local memory with each computing node, storing the gradient fusion data in the shared memory not only makes the gradient fusion data available to each computing node in real time but also retains a complete record, facilitating review of the training process and data tracing.

[0160] S703: Store the gradient fusion data in the local memory of each computing node.

[0161] It can be understood that storing the gradient fusion data in the local memory of each computing node is based on the CXL protocol to directly write the gradient fusion data into the local memory of each computing node.

[0162] In an embodiment of the present application, the target node shares local memory with each computing node based on the CXL protocol, and can directly read data in the local memory of each computing node or write data to the local memory of each computing node without the need for network transmission, so that the data transmission process is not restricted by network transmission, and at the same time avoids frequent data exchange, thereby improving the data communication efficiency and communication performance between the target node and each computing node, improving the processing efficiency of model parameters, and thus improving the efficiency of distributed training.

[0163] like Figure 8As shown, an embodiment of the present application provides a visual flow diagram of a model parameter processing method. The target node (specifically, the GPU in the target node) obtains gradient data from the local memory (such as local memory 1-3) of each computing node (such as computing nodes 1-3). The gradient data set can be (a1, b1, c1; a2, b2, c2; a3, b3, c3). According to the gradient data of each computing node, the gradient fusion data (such as a1+a2+a3; b1+b2+b3; c1+c2+c3) is determined, and the gradient fusion data is stored in the local memory 1-3 of each computing node. After the target node calculates the gradient fusion data, the gradient fusion data can be stored in the processor memory, or it may be stored in the local memory 4 of the target node.

[0164] In this way, based on the CXL protocol, the target node can directly read and write data in the local memory of the computing node without transmitting it through the network, so that the data transmission process is not restricted by network transmission. At the same time, frequent data exchange is avoided, which improves the data communication efficiency and communication performance between nodes, improves the efficiency of processing model parameters, and thus improves the efficiency of distributed training.

[0165] In some embodiments, the method further includes: when the first faulty node is in a faulty state, obtaining training information corresponding to the first computing node, and storing the training information corresponding to the first computing node in a local memory of a backup computing node that shares local memory based on the CXL protocol, so that the backup computing node takes over from the first computing node to continue training the target model based on the training information.

[0166] Training information is used to represent the information generated by the computing node during the process of training the locally deployed target model.

[0167] The first computing node is any computing node among the multiple computing nodes, and the training information includes current model parameters of the target model on the first computing node.

[0168] In this way, if the first computing node is in a faulty state, the training task cannot be continued, and the training task on the first computing node needs to be migrated to the backup computing node, which performs "breakpoint resumption training", that is, continuing to train the target model based on the current model parameters. By real-time monitoring of the fault status of each computing node by the target node, the faulty computing node can be discovered in time and the training tasks on the faulty computing node can be migrated in time. In addition, the target node transfers the training information from the first computing node to the local memory of the backup computing node based on the CXL protocol, thereby improving the efficiency of training task migration.

[0169] In one implementation, the training information corresponding to the first computing node includes the current model parameters of the target model on the first computing node. In this way, the standby computing node can continue training the target model based on the training dataset and current model parameters used by the first computing node to train the target model.

[0170] In another implementation, the training information corresponding to the first computing node also includes the current training round number. This allows the target model to accurately resume training from the last stopped round (i.e., the current training round number) when training is resumed, avoiding repetition of already completed training steps and improving training efficiency.

[0171] In other embodiments, the method further includes obtaining training information from each computing node. If the first computing node fails, storing the training information corresponding to the first computing node in a memory of a backup computing node, so that the backup computing node can take over from the first computing node and continue training the target model based on the training information.

[0172] In this way, not only can real-time detection of each computing node be achieved, faulty computing nodes can be discovered in time, and training tasks on the faulty computing nodes can be migrated in time, but the training information of each computing node can also be backed up to the memory of the target node, so that when the first computing node is unable to send the corresponding training information to the target node (such as when the first computing node is powered off or the system is abnormal, etc.), the most recent training information can be traced back and stored in the local memory of the backup computing node, so that the backup computing node can resume the training process based on the most recent training information, thereby ensuring the continuity and stability of the target model training and reducing the time and resource loss caused by training interruptions.

[0173] Exemplarily, obtaining training information from each computing node includes obtaining training information corresponding to each computing node each time the computing node trains the target model. This ensures that the most recent training information is the most up-to-date training information, which facilitates real-time monitoring of whether each computing node is in a fault state.

[0174] Exemplarily, obtaining training information from each computing node includes obtaining training information from the local memory of each computing node once every predetermined number of training rounds, thereby reducing memory usage of the target node and freeing up memory resources.

[0175] In one implementation, if the training information includes the current model parameters of the target model, the standby computing node updates the parameters to be trained of the target model to the current model parameters upon obtaining the training information, and then continues to train the target model based on the training data set, so that the standby computing node takes over from the first computing node to continue training the target model.

[0176] In another implementation, if the training information also includes the current training round number, the backup computing node sets the training round number to the current training round number and then continues training the target model based on the training dataset, achieving more comprehensive alignment of the backup computing node with the training status of the first computing node. In this way, because the current training round number clearly defines the training progress of the target model before the interruption, when resuming training from a breakpoint, training can be accurately resumed from the last stopped round, avoiding repeated training of the completed part and improving training efficiency.

[0177] In some embodiments, the method further includes storing the training information in a memory of a processor thereof or in a local memory of the target node.

[0178] In some embodiments, the method further includes determining whether the computing node is in a faulty state. In this manner, the target node monitors the fault status of each node in real time. In the event of a computing node failure, the training data corresponding to the failed computing node is stored in the local memory of a backup computing node, allowing the backup computing node to promptly take over from the failed computing node and continue training the target model.

[0179] In one implementation, determining whether a computing node is in a faulty state includes determining an update status of gradient data in a local memory of each computing node. If the gradient data in the local memory of a first computing node has not been updated for a predetermined period of time, the first computing node is determined to be in a faulty state.

[0180] For example, the preset duration can be set according to the training data set. If the training data set is small, the preset duration can be set in the range of a few seconds to a few minutes, such as 5 seconds to 5 minutes. If the training data set is large, the preset duration can be set in the range of tens of minutes, such as 30 minutes.

[0181] In another implementation, determining whether the computing node is in a fault state includes: if the number of training rounds in the local memory of the first computing node is not updated within a preset time period, determining that the first computing node is in a fault state.

[0182] In another implementation, determining whether a computing node is in a faulty state includes determining that the first computing node is in a faulty state if, within a preset time period, a data status of gradient data obtained by the first computing node during training of a target model is abnormal in format (e.g., data corruption). The data status indicates whether the data (including the data format) is normal.

[0183] In the embodiment of the present application, it is possible to determine whether a computing node is in a fault state in a more diverse and flexible manner through a variety of methods.

[0184] like Figure 9As shown, an embodiment of the present application provides a flow chart of a model training method, which is applied to the above-mentioned target computing node. The method is executed by the target computing node. The method includes:

[0185] S901 , sending gradient data to a target node that shares local memory based on the CXL protocol.

[0186] The target node is used to calculate the gradient fusion data based on the gradient data of each computing node.

[0187] Gradient data is stored in the local memory of the compute node. The gradient data is obtained by training the locally deployed target model on the target compute node. The target model is a distributed training model.

[0188] In some embodiments, the method further includes: training a locally deployed target model to obtain gradient data.

[0189] In one implementation, training a locally deployed target model to obtain gradient data includes: obtaining prediction data corresponding to sample data based on training data input to the target model, where each training data includes the sample data and label data corresponding to the sample data. A preset loss function is used to construct a prediction loss based on the difference between the prediction data corresponding to the sample data and the label data corresponding to the sample data. Partial derivatives of the prediction loss with respect to each model parameter of the target model are calculated, and the partial derivatives of each model parameter are used as the gradient data for each model parameter.

[0190] Exemplarily, the preset loss function may be any type of loss function, including but not limited to: a cross entropy loss function, a mean square error loss function, etc.

[0191] In this way, by training the target model, gradient data is obtained so that the subsequent target node can determine the gradient fusion data based on the gradient data, thereby enabling the target computing node to implement the next round of training for the target model.

[0192] S902: Receive gradient fusion data sent by the target node.

[0193] The gradient fusion data is determined by the target node based on the gradient data corresponding to each computing node. The gradient fusion data is used by each computing node to update the model parameters of the target model.

[0194] S903: Update the model parameters of the locally deployed target model according to the gradient fusion data.

[0195] In some embodiments, updating the model parameters of the locally deployed target model based on the gradient fusion data includes: calculating: target model parameters = current model parameters - learning rate * gradient fusion data to obtain the target model parameters, and replacing the current model parameters with the target model parameters.

[0196] The learning rate (lr) is used to indicate the magnitude of the change in the model parameters of the target model. For example, the learning rate can be manually adjusted by the algorithm personnel during the training process.

[0197] In this way, the target model parameters are calculated by fusion of the current model parameters, learning rate and gradient data, which can smooth the parameter update process, further control the parameter fluctuation range, further improve the stability of training, accelerate the convergence process and improve training efficiency.

[0198] In some embodiments, the method further includes: in the event of a failure, sending the locally stored training information to a backup computing node that shares a local memory based on a CXL protocol.

[0199] In an embodiment of the present application, the target computing node can send the training information to the backup computing node in a timely and direct manner, thereby improving the efficiency of the backup computing node in obtaining its training information, thereby improving the training efficiency.

[0200] In some other embodiments, the method further includes: in the event of a failure, sending the locally stored training information to the target node.

[0201] In an embodiment of the present application, the target computing node can send training information to the target node in a timely manner, ensuring that the training information will not be lost due to a fault, so that the target node can obtain the training information corresponding to the target computing node, which is more convenient for the target node to uniformly manage its corresponding training information and assign it to the corresponding backup computing node, so that the backup computing node can take over the target computing node based on the training information to continue training the target model, ensuring training integrity and training continuity.

[0202] In other embodiments, the method further includes: sending the training information corresponding to the target computing node to the target node during each round of training of the target model. In this way, the target node can obtain the training information corresponding to each round of training of the target computing node, realize the backup of the training information, and monitor the failure of the target computing node in real time.

[0203] In other embodiments, the method further includes: sending training information corresponding to the target computing node to the target node at intervals of a preset number of training rounds, thereby freeing up more memory resources at the target node and monitoring the failure of the target computing node through the target node.

[0204] like Figure 10As shown, the embodiment of the present application provides a flow diagram of another model parameter processing method, which is applied to a computing cluster and is jointly executed by the target node and each computing node (such as computing node 1 and computing node 2). The target node and each computing node share local memory based on the CXL protocol. The method includes:

[0205] S1001: Each computing node trains the locally deployed target model to obtain gradient data.

[0206] S1002: Each computing node sends gradient data to the target node.

[0207] S1003: The target node obtains gradient data from each computing node.

[0208] S1004: The target node determines gradient fusion data based on the gradient data of each computing node.

[0209] S1005: The target node stores the gradient fusion data in the local memory of each computing node.

[0210] S1006: Each computing node receives the gradient fusion data sent by the target node.

[0211] S1007: Each computing node updates the model parameters of the locally deployed target model based on the gradient fusion data.

[0212] In the embodiments of the present application, the CXL protocol enables efficient data exchange between each computing node and the target node, optimizing the communication process and performance during model parameter processing. Each computing node generates gradient data during model training, which the target node obtains, fuses, and distributes, allowing the computing node to update the model parameters based on the gradient fusion data. This reduces network communication overhead, avoids repeated data transmission, effectively alleviates the memory wall and communication wall problems in distributed training, improves the processing efficiency of model parameters, improves the overall operating efficiency of the computing cluster, and enhances the training efficiency of the target model.

[0213] like Figure 11 As shown, an embodiment of the present application provides a model parameter processing device 200, which is applied to a target node. The model parameter processing device 200 includes: an acquisition module 21, a determination module 22 and a storage module 23. Among them, the acquisition module 21 is used to obtain gradient data from each computing node that shares a local memory based on the CXL protocol. The gradient data is stored in the local memory of the computing node. The gradient data is obtained by the computing node training the locally deployed target model; the target model is a distributed training model. The determination module 22 is used to determine the gradient fusion data based on the gradient data of each computing node. The gradient fusion data is used by each computing node to update the model parameters of the target model. The storage module 23 is used to store the gradient fusion data in the local memory of each computing node.

[0214] like Figure 12 As shown, an embodiment of the present application provides a model training device 300, which is applied to a target computing node. The model training device 300 includes: a sending module 31, a receiving module 32, and an updating module 33. Among them, the sending module 31 is used to send gradient data to the target node that shares the local memory based on the CXL protocol. Among them, the target node is used to calculate gradient fusion data. The gradient data is stored in the local memory of the computing node. The gradient data is obtained by the target computing node training the locally deployed target model. The target model is a distributed training model. The receiving module 32 is used to receive the gradient fusion data sent by the target node. Among them, the gradient fusion data is determined by the target node based on the gradient data corresponding to each computing node. The gradient fusion data is used by each computing node to update the model parameters of the target model. The updating module 33 is used to update the model parameters of the locally deployed target model based on the gradient fusion data.

[0215] like Figure 13 As shown, an embodiment of the present application provides another computing device 500. The computing device 500 includes a processor 510 and a memory 520 for storing processor-executable instructions. When the processor 510 is configured to execute the instructions, the computing device 500 implements the model parameter processing method or model training method as described above.

[0216] Figure 13 The computing device 500 shown is merely an example and should not limit the functionality and scope of use of the embodiments of the present application.

[0217] Computing device 500 is implemented as a general-purpose computing device. Components of computing device 500 may include, but are not limited to, one or more processors 510, memory 520, a communication bus 540 connecting various system components (including memory 520 and processor 510), and a communication interface 530.

[0218] Communication bus 540 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnection (PCI) bus.

[0219] The computing device 500 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computing device, including volatile and non-volatile media, removable and non-removable media.

[0220] The memory 520 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. The computing device may further include other removable / non-removable, volatile / non-volatile computer system storage media. Figure 13 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a Compact Disc Read Only Memory (hereinafter referred to as: CD-ROM), a Digital Video Disc Read Only Memory (hereinafter referred to as: DVD-ROM), or other optical media) may be provided. In these cases, each drive can be connected to the communication bus 540 via one or more data medium interfaces. The memory 520 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present application.

[0221] A program / utility having a set (at least one) of program modules may be stored in memory 520. Such program modules include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. The program modules generally perform the functions and / or methods of the embodiments described herein.

[0222] The computing device 500 may also communicate with one or more external devices (e.g., keyboard, pointing device, display, etc.), one or more devices that enable a user to interact with the computing device, and / or any device that enables the computing device to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed through the communication interface 530. Furthermore, the computing device 500 may also communicate with the network adapter ( Figure 13 The network adapter can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via the communication bus 540. Figure 13 Not shown, other hardware and / or software modules may be used in conjunction with the computing device 500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, disk arrays (Redundant Arrays of Independent Drives; hereinafter referred to as: RAID) systems, tape drives, and data backup storage systems.

[0223] The processor 510 executes various functional applications and data processing by running the programs stored in the memory 520, such as implementing the above-mentioned method provided in the embodiment of the present application.

[0224] It is understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is merely illustrative and does not constitute a structural limitation on the computing device 500. In other embodiments of the present application, the computing device 500 may also adopt a different interface connection method from the above embodiments, or a combination of multiple interface connection methods.

[0225] It is understandable that, in order to realize the above functions, the above-mentioned computing devices and the like include hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily appreciate that, in combination with the various exemplary units and algorithm steps described in the embodiments disclosed herein, the embodiments of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiments of the present application.

[0226] The embodiment of the present application can divide the functional modules of the above-mentioned computing device etc. according to the above-mentioned method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0227] The present application also provides a storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a computing device, the computing device implements the above-mentioned method.

[0228] An embodiment of the present application also provides a computer program product, which includes a computer program. When at least one processor executes the computer program, the at least one processor executes the above method provided in the embodiment of the present application.

[0229] The computing device, storage medium or computer program product provided in the embodiments of the present application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here.

[0230] Through the description of the above embodiments, those skilled in the art will clearly understand that for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0231] The functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0232] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as flash memory, mobile hard disk, read-only memory, random access memory, magnetic disk or optical disk.

[0233] The above is only a specific embodiment of the present application, but the scope of protection of this application is not limited to this. Any changes or substitutions within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A model parameter processing method, characterized in that: Applied to a target node; the method comprises: Obtaining gradient data from each computing node that shares local memory based on the Compute Express Link (CXL) protocol; the gradient data is stored in the local memory of the computing node; the gradient data is obtained by the computing node training a locally deployed target model; the target model is a distributed training model; Determining gradient fusion data based on the gradient data of each computing node; the gradient fusion data is used by each computing node to update the model parameters of the target model; The gradient fusion data is stored in the local memory of each computing node.

2. The model parameter processing method according to claim 1, characterized in that: The method further comprises: When the first computing node is in a fault state, obtaining training information corresponding to the first computing node; The first computing node is any computing node among the multiple computing nodes; the training information includes the current model parameters of the target model on the first computing node; The training information corresponding to the first computing node is stored in the local memory of the backup computing node that shares the local memory based on the CXL protocol, so that the backup computing node takes over the training of the target model from the first computing node based on the training information.

3. The model parameter processing method according to claim 2, characterized in that: The training information also includes the current training round number.

4. The model parameter processing method according to any one of claims 1 to 3, characterized in that: After obtaining the gradient data from each computing node sharing the local memory based on the CXL protocol, the method further includes: The gradient data is stored in a memory of a processor of the target node; the processor is configured to calculate the gradient fusion data based on the gradient data of each computing node.

5. The model parameter processing method according to any one of claims 1 to 4, characterized in that: After determining the gradient fusion data according to the gradient data of each computing node, the method further includes: The gradient fusion data is stored in a shared local memory.

6. A model training method, characterized in that: Applied to a target computing node; the method comprises: Sending gradient data to a target node that shares local memory based on a computational fast link (CXL) protocol; wherein the target node is used to calculate gradient fusion data; The gradient data is stored in the local memory of the computing node; the gradient data is obtained by the target computing node training the locally deployed target model; the target model is a distributed training model; receiving the gradient fusion data sent by the target node; wherein the gradient fusion data is determined by the target node according to the gradient data corresponding to each of the computing nodes; and the gradient fusion data is used by each computing node to update the model parameters of the target model; The model parameters of the locally deployed target model are updated according to the gradient fusion data.

7. The model training method according to claim 6, characterized in that The method further comprises: In the event of a failure, sending locally stored training information to the backup computing node sharing local memory based on the CXL protocol; the training information includes current model parameters of the target model; or, In the case of a fault state, locally stored training information is sent to the target node; the training information includes current model parameters of the target model.

8. A computing device, characterized in that include: memory and processor, The memory is used to store program instructions; The processor is configured to execute the program instructions so as to enable the computing device to perform the method according to any one of claims 1 to 5 or 6 to 7.

9. A computing cluster, characterized in that: include: The computing cluster includes a target node and multiple computing nodes; The target node and the plurality of computing nodes share local memory based on a Compute Express Link (CXL) protocol; The target node is a computing device according to claim 8, and the computing device is used to execute the method according to any one of claims 1 to 5; The computing node is the computing device according to claim 8, and the computing device is used to execute the method according to any one of claims 6-7.

10. The computing cluster according to claim 9, characterized in that: The target node is a node in the computing cluster other than the multiple computing nodes.

Images