A model parallel training method and related equipment supporting heterogeneous clusters

Abstract

The present invention discloses a model parallel training method and related equipment supporting heterogeneous clusters. The method comprises: each cluster writes code according to its own framework, and then trains a part of data on its own framework, then first performs model splicing within the cluster, and merges the model parameters of multiple cards that are parallel-sliced into a complete model. The models obtained after splicing of different clusters are consistent, and then the spliced model parameters are sliced and transmitted to a parameter server. The parameter server fuses the model parameters, and then the parameter server sends the fused model to each cluster. Finally, each cluster trains the received model after model parallel slicing according to its own model parallel training strategy. The present invention realizes user privacy protection, data security, large model parallel training and large model transmission, can integrate resources of different computing centers, and meets the needs of heterogeneous frameworks and heterogeneous clusters to jointly train large models.

Description

Technical Field

[0001] The present invention relates to the field of communication technologies, and in particular to a method, system, terminal, and computer-readable storage medium that support parallel model training in heterogeneous clusters. Background Art

[0002] Federated learning and privacy-preserving computing enable the sharing of data value and knowledge while ensuring privacy and security. This allows data to be available but invisible, alleviating the concerns of partner organizations regarding data security. Although federated learning has been widely adopted, existing technical frameworks are not well-suited for large-scale model scenarios, limiting the use cases of federated learning training. Furthermore, large-scale model technology has become mainstream. Large models and big data have further improved model accuracy, addressing many of the pain points of smaller models. Therefore, large-scale model training using federated learning has a broad range of applications.

[0003] For example, two intelligent computing centers need to collaborate on training a model. Intelligent computing center A uses an NPU cluster and the Mindspore deep learning framework, while intelligent computing center B uses a GPU cluster and the Pytorch deep learning framework. Both intelligent computing centers A and B use model parallelism and data parallelism for training, but they use different model segmentation strategies.

[0004] Pre-trained large models have become a hot topic in recent years. Their performance is astonishing. A single pre-trained model can achieve impressive results across multiple tasks, even multimodal tasks, surpassing many carefully designed smaller models. Because large pre-trained models require enormous amounts of data, while the data a single institution can provide is limited and other institutions may also have some data, fully leveraging data from multiple institutions is a pressing need. However, this requires ensuring data security, which is where federated learning comes in. Currently, some other federated training frameworks are primarily designed for single-GPU training and do not support multi-GPU model parallel training. However, model parallelism is essential for training large models.

[0005] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention

[0006] The main purpose of the present invention is to provide a method, system, terminal and computer-readable storage medium that support model parallel training in heterogeneous clusters, aiming to solve the problem that the federated training framework in the existing technology is basically designed for single-card training and does not support multi-card model parallel training.

[0007] To achieve the above object, the present invention provides a model parallel training method supporting heterogeneous clusters, the model parallel training method supporting heterogeneous clusters comprising the following steps:

[0008] Initialize the numpy model based on the parameter server and send it to the GPU cluster and NPU cluster respectively. The GPU cluster and NPU cluster load the initialized numpy model into the actual training model according to their respective training frameworks. The actual training model is then split and trained in parallel.

[0009] The GPU cluster and NPU cluster each train a portion of data with a custom preset size locally;

[0010] The GPU cluster and NPU cluster perform model concatenation based on their respective training frameworks and model parallelization strategies. They merge the models distributed to different devices into a complete model and convert the complete model into a unified NumPy format, ensuring that the NumPy models generated by the GPU and NPU clusters are consistent.

[0011] After slicing the numpy model in a unified way, it is transferred to the parameter server, which then performs splicing processing.

[0012] After receiving the unified format numpy models from the GPU cluster and NPU cluster, the parameter server fuses the numpy models by parameter averaging to obtain the fused numpy model.

[0013] The parameter server distributes the fused model shards to each training cluster. Each training cluster splits the complete NumPy model according to the training framework and model parallel strategy used. The split models are assigned to different devices for training.

[0014] Optionally, the model parallel training method supporting heterogeneous clusters, wherein the GPU cluster and the NPU cluster load the initialized numpy model into the actual training model according to their respective training frameworks, and then perform model parallel segmentation and post-training on the actual training model, specifically includes:

[0015] If the training framework used by the NPU cluster is based on the Mindspore training framework, the NPU cluster will use a horizontal splitting strategy to split the actual training model horizontally;

[0016] If the training framework used by the GPU cluster is a Pytorch-based training framework, the model actually trained by the GPU cluster is vertically split using the vertical splitting strategy.

[0017] Optionally, in the method for supporting parallel model training in heterogeneous clusters, the horizontal segmentation includes:

[0018] When splitting horizontally, if the weight matrix is Wnxn , then a horizontal 4-way split is performed, and the matrix after splitting is W bxn , W bxn , W bxn , W bxn , where b = n / 4;

[0019] The longitudinal cutting includes:

[0020] When splitting vertically, if the weight matrix is W nxn , then a 4-way split is performed vertically, and the matrix after splitting is W nxb , W nxb , W nxb , W nxb , where b = n / 4.

[0021] Optionally, the method for supporting model parallel training in heterogeneous clusters, wherein the complete model is converted into a unified numpy format, is specifically:

[0022] Each training framework provides an interface to convert the complete model into a unified numpy format through the interface.

[0023] Optionally, the method for supporting model parallel training in heterogeneous clusters, wherein the parameter server receives the numpy models in a unified format from the GPU cluster and the NPU cluster, and then fuses the numpy models by parameter averaging to obtain a fused numpy model, specifically comprising:

[0024] If the model parameter format of the GPU cluster is [W1, W2, W3, ..., Wn], and the model parameter format of the NPU cluster is [K1, K2, K3, ..., Kn], then the fused model parameters are: [M1, M2, M3, ..., Mn]; where Wn, Kn, and Mn are weight arrays;

[0025] Among them, the parameter average fusion is: Mi = (Wi + Ki) / 2.

[0026] Optionally, the model parallel training method supporting heterogeneous clusters, wherein the parameter server distributes the fused model shards to each training cluster, each training cluster splits the complete numpy model according to the training framework and model parallel strategy used, and the split models are distributed to different devices and trained, and then further includes:

[0027] After the models assigned to different devices are trained, it is determined whether the models have converged. When the models reach any preset convergence condition, the model training is stopped.

[0028] Optionally, in the method for supporting parallel model training in heterogeneous clusters, the convergence condition includes:

[0029] The loss value is less than a pre-set threshold;

[0030] The change in weight between two iterations is less than the preset value;

[0031] Set the maximum number of iterations. When the number of iterations exceeds the maximum number of iterations,

[0032] Optionally, the model parallel training method supporting heterogeneous clusters, wherein the numpy model is essentially an array.

[0033] Optionally, the model parallel training method supporting heterogeneous clusters is described, wherein the code for in-cluster training is expanded to cloud-to-cloud collaborative training.

[0034] Optionally, the model parallel training method supports heterogeneous clusters, wherein the GPU cluster and the NPU cluster adopt different deep learning frameworks.

[0035] Optionally, the method for supporting model parallel training in heterogeneous clusters supports model parallel training on multiple cards.

[0036] In addition, to achieve the above-mentioned object, the present invention further provides a model parallel training system supporting heterogeneous clusters, wherein the model parallel training system supporting heterogeneous clusters includes: a parameter server, a GPU cluster and an NPU cluster, and the parameter server is respectively communicated with the GPU cluster and the NPU cluster;

[0037] The parameter server initializes the NumPy model and sends it to the GPU cluster and NPU cluster respectively. The GPU cluster and NPU cluster load the initialized NumPy model into the actual training model based on their respective training frameworks. The actual training model is then split and trained in parallel.

[0038] The GPU cluster and NPU cluster each train a portion of data with a custom preset size locally;

[0039] The GPU cluster and NPU cluster perform model concatenation based on their respective training frameworks and model parallelization strategies. They merge the models distributed to different devices into a complete model and convert the complete model into a unified NumPy format, ensuring that the NumPy models generated by the GPU and NPU clusters are consistent.

[0040] After slicing the numpy model in a unified way, it is transferred to the parameter server, which then performs splicing processing.

[0041] After receiving the unified format numpy models from the GPU cluster and NPU cluster, the parameter server fuses the numpy models by parameter averaging to obtain the fused numpy model.

[0042] The parameter server distributes the fused model shards to each training cluster. Each training cluster splits the complete NumPy model according to the training framework and model parallel strategy used. The split models are assigned to different devices for training.

[0043] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a model parallel training program supporting heterogeneous clusters, and when the model parallel training program supporting heterogeneous clusters is executed by the processor, the steps of the model parallel training method supporting heterogeneous clusters are implemented as described above.

[0044] In the present invention, each cluster writes code according to its own framework, and then trains a part of the data on its own framework. Then, the model is spliced within the cluster first, and the model parameters of the model that are divided into multiple cards in parallel are merged into a complete model. The models obtained after splicing of different clusters are consistent. Then the spliced model parameters are fragmented and transmitted to the parameter server. The parameter server fuses the model parameters, and then the parameter server sends the fused model to each cluster. Finally, each cluster trains the received model after model parallel segmentation according to its own model parallel training strategy. The present invention realizes user privacy protection, data security, large model parallel training and large model transmission, can integrate resources from different computing centers, meet the needs of heterogeneous frameworks and heterogeneous clusters to jointly train large models, and further optimizes the user interface, so that users only need to modify very small codes to expand the code trained in the cluster into cloud-to-cloud collaborative training. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a flowchart of a preferred embodiment of the present invention's method for supporting parallel model training in heterogeneous clusters;

[0046] Figure 2 This is a schematic diagram of a heterogeneous cluster and heterogeneous framework parallel training architecture in a preferred embodiment of the present invention supporting a model parallel training method under a heterogeneous cluster;

[0047] Figure 3 It is a schematic diagram of the principle of a preferred embodiment of the present invention supporting a model parallel training system under heterogeneous clusters. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the present invention more clear and distinct, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0049] This paper proposes a model parallel training method that supports heterogeneous clusters, such as Figure 1 and Figure 3 As shown, the model parallel training method supporting heterogeneous clusters includes the following steps:

[0050] Step S10: Initialize the numpy model based on the parameter server, and send the initialized numpy model to the GPU cluster and NPU cluster respectively. The GPU cluster and NPU cluster load the initialized numpy model into the actual training model according to their respective training frameworks, and then perform parallel splitting and training on the actual training model.

[0051] Specifically, the parameter server initializes a numpy model (an array representation format, essentially an array) and then distributes it to the GPU cluster and NPU cluster. Since the numpy model cannot be trained directly, each cluster needs to load the numpy model into the actual training model according to the training framework used, and then perform model parallel segmentation before training.

[0052] For example, if the training framework used by the NPU cluster is based on the Mindspore training framework, the NPU cluster uses a horizontal splitting strategy to split the actual training model horizontally; if the training framework used by the GPU cluster is based on the Pytorch training framework, the GPU cluster uses a vertical splitting strategy to split the actual training model vertically.

[0053] Different frameworks may have different model parallel splitting strategies. For example, for the same weight matrix, Mindspore uses a horizontal splitting strategy, while the Pytorch-based training framework uses a vertical splitting strategy. Therefore, the matrix needs to be split horizontally in the NPU cluster, while the GPU cluster uses vertical splitting.

[0054] Horizontal segmentation strategy display:

[0055] Let the weight matrix be W nxn , then a horizontal 4-way split is performed, and the matrix after splitting is W bxn , W bxn , W bxn , W bxn , where b = n / 4.

[0056] Vertical segmentation strategy display:

[0057] Let the weight matrix be W nxn , then a 4-way split is performed vertically, and the matrix after splitting is W nxb , W nxb , W nxb , W nxb , where b = n / 4.

[0058] In step S20, the GPU cluster and the NPU cluster respectively train a portion of data of a custom preset size locally.

[0059] Specifically, each cluster trains on a portion of the data locally. For example, the GPU cluster and NPU cluster each train on a portion of data of a custom preset size. Because collaborative training requires multiple model fusions during the entire training process, each fusion only trains on a portion of the data, and the training data size can be customized.

[0060] In step S30, the GPU cluster and the NPU cluster perform model splicing according to their respective training frameworks and model parallel strategies, merge the models divided into different devices into a complete model, and convert the complete model into a unified numpy format so that the numpy models obtained by the GPU cluster and the NPU cluster are consistent.

[0061] Specifically, the parameter server needs to integrate the data uploaded by each cluster, so the model format uploaded by each cluster must be consistent. Because different frameworks use different model splitting strategies, and the model formats of Pytorch and Mindspore are inconsistent, the following two steps are required: (1) Each cluster performs model splicing based on its own training framework and model parallelization strategy. The existing parallelization strategies are more commonly used in horizontal splitting and vertical splitting. A small number of frameworks also use block splitting (such as the Optmus framework using the SUMMA algorithm), that is, the models split to different devices are merged into a complete model. (2) Then the complete model is converted to a unified numpy format. The conversion method is relatively simple. Each framework provides an interface for converting to numpy (model weight w, convert to numpy format: w.asnumpy()), so that the numpy model obtained by each cluster is consistent.

[0062] Step S40: Slice the numpy model according to a unified slicing method and transmit it to the parameter server, which performs splicing processing.

[0063] Specifically, some communication frameworks have a maximum data transmission limit. For example, GRPC can only transmit 2G at a time, while large models often reach 10G, 100G, or even 1T. Therefore, a model needs to be sliced and transmitted to the parameter server in a unified way, and then spliced. The slicing process is as follows:

[0064] Assume that the format of the numpy model is [W1, W2, W3, …, Wn], where Wn is the weight array; set the slice size to size_limit = 1M.

[0065] Suppose the sliced model has slices = []

[0066] For I form 1to n:

[0067] If size(slices)>size_limit:

[0068] Use GPRC communication to transmit slices: grpc.send(slices)

[0069] slices=[]

[0070] else:

[0071] Insert WI into slices: slices.append(WI)

[0072] grpc.send(slices)

[0073] Finish.

[0074] Step S50: After receiving the numpy models in a unified format from the GPU cluster and the NPU cluster, the parameter server fuses the numpy models by parameter averaging to obtain a fused numpy model.

[0075] Specifically, after the parameter server receives the numpy models in a unified format from different clusters, it needs to fuse the models. The most common fusion method is to perform parameter averaging to obtain a fused numpy model. Parameter averaging fusion algorithm:

[0076] Assume there are only two parties involved (for example, a GPU cluster and an NPU cluster), the model parameters of the GPU cluster are in the format of [W1, W2, W3, ..., Wn], and the model parameters of the NPU cluster are in the format of [K1, K2, K3, ..., Kn]. Then the fused model parameters are: [M1, M2, M3, ..., Mn]; where Wn, Kn, and Mn are weight arrays.

[0077] Among them, the parameter average fusion is: Mi = (Wi + Ki) / 2.

[0078] Step S60: The parameter server distributes the fused model shards to each training cluster. Each training cluster splits the complete numpy model according to the training framework and model parallel strategy used. The split models are assigned to different devices and trained.

[0079] Specifically, the fused model is split into shards and distributed to various training clusters. Each cluster then splits the complete NumPy model based on the training framework and model parallelization strategy used, assigning the split models to different devices for training. After training, the models assigned to different devices are judged to determine whether they have converged. When the model meets any of the preset convergence conditions, model training is terminated.

[0080] Among them, the convergence conditions include: the loss value is less than a preset threshold; the change in weight between two iterations is less than a preset value; setting a maximum number of iterations, when the number of iterations exceeds the maximum number of iterations.

[0081] For example, the present invention can integrate the segmented models in intelligent computing centers A and B into a complete model, and then convert the models into numpy array format so that different centers have model files in the same format. Since the model file is too large to be transmitted directly, the model in numpy format is sliced, and then the GRPC communication library is used to transmit the model slices of each worker end to the server end for aggregation, and then the aggregated model is sent to each worker for the next step of training. Experiments have shown that this method can support 13B models, and theoretically can support larger models.

[0082] Possible scenarios for training a large model through federated learning in multiple clusters include: (1) Cluster resources are limited, and multiple homogeneous clusters need to collaborate in training; Cluster I has prepared data and algorithms and wants to train a pre-trained large model, but the cluster's internal resources are insufficient, and it hopes to use the computing power of Cluster II. In this case, it can copy part of the data to Cluster II, and then train in the two clusters separately, and then fuse the parameters of the two clusters through the parameter server, and then send the fused parameters to the two clusters. (2) Data does not leave the local area, and heterogeneous clusters collaborate in training. Due to user data privacy issues, cluster data cannot be transmitted externally. In this case, each cluster needs to implement a set of algorithm codes based on the framework it uses, and also implement a set of model splicing codes based on the model parallel scheme of each framework. (3) For example, multiple hospitals want to collaborate in training a large image recognition model for medical image recognition, but the user's medical data is very private. In this case, each center needs to train its own model in different frameworks and then aggregate the models of different frameworks. The framework of the present invention can very friendly support this scenario and does not require unnecessary model migration.

[0083] This invention realizes user privacy protection, data security, large-model parallel training, and large-model transmission. It can integrate resources from different computing centers, meet the needs of heterogeneous frameworks and heterogeneous clusters to jointly train large models, and further optimize the user interface, allowing users to expand the code trained in the cluster into cloud-to-cloud collaborative training by only modifying minimal code.

[0084] The present invention mainly solves the pain point problem that the existing collaborative training framework does not support large model training. The existing framework basically only supports single-card training or data parallel training, and cannot handle the situation of splitting a model into multiple cards, let alone the situation of adopting different model splitting strategies under heterogeneous frameworks. The solution adopted by the present invention is: each cluster writes code according to its own framework, and then trains a part of the data on its own framework. Then, the model is spliced within the cluster first, and the model parameters of the model that is split into multiple cards in parallel are merged into a complete model. The models obtained after splicing of different clusters are consistent; then the spliced model parameters are transmitted in pieces (because the model is too large to be transmitted in one go; the sharding method adopted by different clusters is consistent) to the server end, and the server fuses the model parameters; then the server sends the fused model to each cluster; finally, each cluster will perform model parallel splitting on the received model according to its own model parallel training strategy, and continue the next round of training.

[0085] The highlights of the present invention are: (1) support for heterogeneous clusters, allowing NPU clusters and GPU clusters to jointly train a large model; (2) support for different training frameworks and different model parallel segmentation strategies; (3) unified model transmission format, avoiding the model migration step; (4) adoption of model sharding transmission, breaking through communication limitations, and capable of transmitting larger models.

[0086] Further, if Figure 2 and Figure 3 As shown, based on the above-mentioned model parallel training method supporting heterogeneous clusters, the present invention also provides a model parallel training system supporting heterogeneous clusters, wherein the model parallel training system supporting heterogeneous clusters includes:

[0087] Parameter Server (i.e. Figure 2 and Figure 3 Parameter server in ), GPU cluster (i.e. Figure 2 and Figure 3 GPU cloud) and NPU cluster (i.e. Figure 2 and Figure 3 The parameter server communicates with the GPU cluster and NPU cluster respectively.

[0088] The parameter server initializes the numpy model and sends the initialized numpy model to the GPU cluster and NPU cluster respectively. The GPU cluster and NPU cluster load the initialized numpy model into the real training model according to their respective training frameworks, and then perform model parallel segmentation on the real training model for training. The GPU cluster and NPU cluster respectively train part of the data of custom preset size locally. The GPU cluster and NPU cluster perform model splicing according to their respective training frameworks and model parallel strategies, merge the models segmented to different devices into a complete model, and convert the complete model into a unified numpy format. formula, so that the numpy models obtained by the GPU cluster and the NPU cluster are consistent; the numpy model is sliced according to a unified segmentation method and transmitted to the parameter server, which performs splicing processing; after the parameter server receives the numpy models in a unified format from the GPU cluster and the NPU cluster, it fuses the numpy models by parameter averaging to obtain a fused numpy model; the parameter server distributes the fused model slices to each training cluster, and each training cluster segments the complete numpy model according to the training framework and model parallel strategy used. The segmented models are then assigned to different devices for training.

[0089] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a model parallel training program supporting heterogeneous clusters, and when the model parallel training program supporting heterogeneous clusters is executed by a processor, the steps of the model parallel training method supporting heterogeneous clusters are implemented as described above.

[0090] In summary, the present invention provides a model parallel training method and related equipment that supports heterogeneous clusters. The method includes: each cluster writes code according to its own framework, and then trains a part of the data on its own framework, and then first performs model splicing within the cluster, and merges the model parameters of multiple cards that are parallelly split into a complete model. The models obtained after splicing of different clusters are consistent, and then the spliced model parameters are fragmented and transmitted to the parameter server. The parameter server fuses the model parameters, and then the parameter server sends the fused model to each cluster. Finally, each cluster will train the received model after model parallel splitting according to its own model parallel training strategy; the present invention realizes user privacy protection, data security, large model parallel training and large model transmission, can integrate resources from different computing centers, meet the needs of heterogeneous frameworks and heterogeneous clusters to jointly train large models, and further optimizes the user interface, so that users only need to modify very small codes to expand the code trained in the cluster into cloud-to-cloud collaborative training.

[0091] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or terminal comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or terminal comprising the element.

[0092] Of course, those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware (such as a processor, controller, etc.) through a computer program. The program can be stored in a computer-readable storage medium that can be read by a computer. When the program is executed, it can include the processes in the above-described method embodiments. The computer-readable storage medium can be a memory, a magnetic disk, an optical disk, etc.

[0093] It should be understood that the application of the present invention is not limited to the above examples. For those skilled in the art, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A model parallel training method supporting heterogeneous clusters, characterized in that: The method for supporting parallel model training in heterogeneous clusters includes: Initialize the numpy model based on the parameter server and send it to the GPU cluster and NPU cluster respectively. The GPU cluster and NPU cluster load the initialized numpy model into the actual training model according to their respective training frameworks. The actual training model is then split and trained in parallel. The GPU cluster and NPU cluster each train a portion of data with a custom preset size locally; The GPU cluster and NPU cluster perform model concatenation based on their respective training frameworks and model parallelization strategies. They merge the models distributed to different devices into a complete model and convert the complete model into a unified NumPy format, ensuring that the NumPy models generated by the GPU and NPU clusters are consistent. After slicing the numpy model in a unified way, it is transferred to the parameter server, which then performs splicing processing. After receiving the unified format numpy models from the GPU cluster and NPU cluster, the parameter server fuses the numpy models by parameter averaging to obtain the fused numpy model. The parameter server distributes the fused model shards to each training cluster. Each training cluster splits the complete NumPy model according to the training framework and model parallel strategy used. The split models are assigned to different devices for training.

2. The model parallel training method supporting heterogeneous clusters according to claim 1, characterized in that: The GPU cluster and NPU cluster load the initialized numpy model into the actual training model according to their respective training frameworks, and then perform parallel splitting and post-training on the actual training model, specifically including: If the training framework used by the NPU cluster is based on the Mindspore training framework, the NPU cluster will use a horizontal splitting strategy to split the actual training model horizontally; If the training framework used by the GPU cluster is a Pytorch-based training framework, the model actually trained by the GPU cluster is vertically split using the vertical splitting strategy.

3. The model parallel training method supporting heterogeneous clusters according to claim 2, characterized in that: The horizontal cutting includes: When splitting horizontally, if the weight matrix is W nxn , then a horizontal 4-way split is performed, and the matrix after splitting is W bxn , W bxn , W bxn , W bxn , where b=n / 4; The longitudinal cutting includes: When splitting vertically, if the weight matrix is W nxn , then a 4-way split is performed vertically, and the matrix after splitting is W nxb , W nxb , W nxb , W nxb , where b=n / 4.

4. The model parallel training method supporting heterogeneous clusters according to claim 1, characterized in that: The complete model is converted into a unified numpy format, specifically: Each training framework provides an interface to convert the complete model into a unified numpy format through the interface.

5. The model parallel training method supporting heterogeneous clusters according to claim 1, characterized in that: After receiving the numpy models in a unified format from the GPU cluster and the NPU cluster, the parameter server fuses the numpy models by parameter averaging to obtain a fused numpy model, specifically including: If the model parameter format of the GPU cluster is [W1, W2, W3, ..., Wn], and the model parameter format of the NPU cluster is [K1, K2, K3, ..., Kn], then the fused model parameters are: [M1, M2, M3, ..., Mn]; where Wn, Kn, and Mn are weight arrays; Among them, the parameter average fusion is: Mi = (Wi+Ki) / 2.

6. The model parallel training method supporting heterogeneous clusters according to claim 1, characterized in that: The parameter server distributes the fused model shards to each training cluster. Each training cluster splits the complete NumPy model according to the training framework and model parallel strategy used. The split models are assigned to different devices and trained. The following steps are also included: After the models assigned to different devices are trained, it is determined whether the models have converged. When the models reach any preset convergence condition, the model training is stopped.

7. The model parallel training method supporting heterogeneous clusters according to claim 6, characterized in that: The convergence conditions include: The loss value is less than a pre-set threshold; The change in weight between two iterations is less than the preset value; Set the maximum number of iterations. When the number of iterations exceeds the maximum number of iterations, 8. The model parallel training method supporting heterogeneous clusters according to claim 1, characterized in that: The numpy model is essentially an array.

9. The model parallel training method supporting heterogeneous clusters according to claim 1, characterized in that: Expand the code for cluster training to cloud-to-cloud collaborative training.

10. The model parallel training method supporting heterogeneous clusters according to claim 1, characterized in that: The GPU cluster and the NPU cluster use different deep learning frameworks.

11. The model parallel training method supporting heterogeneous clusters according to claim 10, characterized in that: The model parallel training method supporting heterogeneous clusters supports multi-GPU parallel model training.

12. A model parallel training system supporting heterogeneous clusters, characterized by: The model parallel training system supporting heterogeneous clusters includes: a parameter server, a GPU cluster and an NPU cluster, and the parameter server is communicated with the GPU cluster and the NPU cluster respectively; The parameter server initializes the NumPy model and sends it to the GPU cluster and NPU cluster respectively. The GPU cluster and NPU cluster load the initialized NumPy model into the actual training model based on their respective training frameworks. The actual training model is then split and trained in parallel. The GPU cluster and NPU cluster each train a portion of data with a custom preset size locally; The GPU cluster and NPU cluster perform model concatenation based on their respective training frameworks and model parallelization strategies. They merge the models distributed to different devices into a complete model and convert the complete model into a unified NumPy format, ensuring that the NumPy models generated by the GPU and NPU clusters are consistent. After slicing the numpy model in a unified way, it is transferred to the parameter server, which then performs splicing. After receiving the unified format numpy models from the GPU cluster and NPU cluster, the parameter server fuses the numpy models by parameter averaging to obtain the fused numpy model. The parameter server distributes the fused model shards to each training cluster. Each training cluster splits the complete NumPy model according to the training framework and model parallel strategy used. The split models are assigned to different devices for training.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a model parallel training program supporting heterogeneous clusters. When the model parallel training program supporting heterogeneous clusters is executed by a processor, the steps of the model parallel training method supporting heterogeneous clusters as described in any one of claims 1-11 are implemented.

Images