A cluster-based model training method, device, equipment and storage medium

Abstract

The present invention relates to the field of model training technology, and in particular to a cluster-based model training method, device, equipment and storage medium. The present invention places the model to be trained on each cluster and uses the local data on each cluster to train the model. The present invention places the model to be trained on each cluster, so the present invention does not need to migrate the data on each cluster to the outside of each cluster, and can use the data of each cluster to train the model, thereby protecting the data privacy on each cluster to prevent data leakage.

Description

Technical Field

[0001] The present invention relates to the field of model training technology, and in particular to a cluster-based model training method, device, equipment and storage medium. Background Art

[0002] In recent years, the global development of artificial intelligence (various intelligent computing models) has been unstoppable, and has created huge social benefits in key areas such as smart healthcare, smart cities, and Internet finance. Before using the model to process related tasks in the fields of smart healthcare, smart cities, and Internet finance, the model needs to be trained, and model training requires a large amount of data. Clusters, as carriers of data, have the data required for training models, but the limitations of the data corresponding to a single cluster cannot meet the data requirements for model training. Therefore, multiple clusters are required to provide the model with more and different data to meet the data requirements for model training. The existing technology is to migrate the data on each cluster to aggregate the data on each cluster, and then train the model with the aggregated data. The existing technology has caused the leakage of cluster data because the data on each cluster is moved out of the cluster.

[0003] In summary, the existing technology of using clusters to train models will lead to data leakage.

[0004] Therefore, the prior art still needs to be improved and enhanced. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a cluster-based model training method, device, equipment and storage medium, which solves the problem that the existing technology uses clusters to train models, which may lead to data leakage.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a cluster-based model training method, which includes:

[0008] Send the model to each cluster, where the cluster is used to provide the data required for training the model;

[0009] Obtaining local parameters corresponding to the model when training on each of the clusters based on local data, where the local data is data on each of the clusters;

[0010] Based on each of the local parameters, the target parameters corresponding to the model are obtained to complete the training of the model. The trained model is used to complete the target task.

[0011] In one implementation, obtaining the target parameters corresponding to the model based on each of the local parameters, completing the training of the model, and using the trained model to complete the target task includes:

[0012] Obtaining each local gradient corresponding to the model located on each of the clusters during training;

[0013] According to each of the local gradients and each of the local parameters, the target parameters corresponding to the model are obtained to complete the training of the model.

[0014] In one implementation, obtaining the target parameter corresponding to the model based on each of the local gradients and each of the local parameters to complete the training of the model includes:

[0015] Aggregating each of the local gradients to obtain an aggregated gradient that matches the model;

[0016] Fusing the local parameters to obtain fusion parameters corresponding to the model;

[0017] According to the aggregation gradient and the fusion parameter, the target parameter corresponding to the model is obtained.

[0018] In one implementation, obtaining the target parameter corresponding to the model according to the aggregated gradient and the fusion parameter includes:

[0019] Sending the aggregated gradient and the fusion parameter to each of the clusters;

[0020] Obtaining the local target parameters corresponding to the model on each of the clusters when iterative training is performed according to the aggregated gradient, the fusion parameter and the local data until convergence;

[0021] Fusing the local target parameters to obtain target fusion parameters corresponding to the model;

[0022] According to the target fusion parameters, the target parameters corresponding to the model are obtained.

[0023] In one implementation, obtaining the target parameter corresponding to the model based on the target fusion parameter includes:

[0024] Obtaining the local target gradient corresponding to when the model on each of the clusters is iteratively trained according to the aggregated gradient, the fusion parameter and the local data until convergence;

[0025] Aggregating each of the local target gradients to obtain a target aggregate gradient that matches the model;

[0026] According to the target aggregation gradient, an optimizer matching the target fusion parameter is obtained;

[0027] According to the optimizer, the target fusion parameters are updated to obtain the target parameters corresponding to the model.

[0028] In one implementation, the model is trained based on local data to obtain local parameters corresponding to the model on each cluster, and the local data is located on each cluster;

[0029] Sending each of the local parameters to a global server, where the global server is used to fuse each of the local parameters;

[0030] According to the aggregation parameter and the local data, each local target parameter corresponding to the model on each cluster is obtained, wherein the aggregation parameter is a parameter obtained after the global server fuses each local parameter;

[0031] Each of the local target parameters is sent to the global server, and the global server is used to fuse the local target parameters to complete the training of the model. The trained model is used to complete the target task.

[0032] In one implementation, the training of the model based on the local data to obtain local parameters corresponding to the model on each cluster, wherein the local data is located on each cluster, includes:

[0033] According to each of the clusters, a local server and each working node included in each of the clusters are obtained, and the working node is used to load the local data onto the model;

[0034] Training the model according to the local data on each of the working nodes to obtain sub-part parameters corresponding to the model;

[0035] The parameters of each sub-part are integrated through the local server to obtain the local parameters corresponding to the model on each cluster.

[0036] In one implementation, the fusing of the sub-part parameters by the local server to obtain local parameters corresponding to the model on each cluster includes:

[0037] Calculate the sub-part gradient corresponding to the model through each of the working nodes;

[0038] The local server fuses the parameters of each sub-part according to the sub-part gradient to obtain the local parameters corresponding to the model on each cluster.

[0039] In one implementation, the method further includes:

[0040] Each of the clusters receives the model sent by the global server, and the global server initializes the model before sending the model.

[0041] In one implementation, each of the clusters trains the model synchronously, and each of the clusters is a heterogeneous cluster, and the heterogeneous clusters include an NPU cluster and a GPU cluster.

[0042] In a second aspect, an embodiment of the present invention further provides a device for a cluster-based model training method, wherein the device includes the following components:

[0043] A model sending module is used to send the model to each cluster, and the cluster is used to provide the data required for training the model;

[0044] A parameter receiving module, used for obtaining each local parameter corresponding to the model when training the model on each cluster according to the local data, wherein the local data is the data on each cluster;

[0045] The target parameter generation module is used to obtain the target parameters corresponding to the model based on each of the local parameters, complete the training of the model, and the trained model is used to complete the target task.

[0046] In a third aspect, an embodiment of the present invention further provides a device for a cluster-based model training method, wherein the device includes the following components:

[0047] A model training module, used to train the model based on local data to obtain local parameters corresponding to the model on each cluster, wherein the local data is located on each cluster;

[0048] A parameter sending module, used for sending each of the local parameters to a global server, and the global server is used for fusing each of the local parameters;

[0049] A local target parameter generation module, used to obtain local target parameters corresponding to the models on each of the clusters according to the aggregation parameters and the local data, wherein the aggregation parameters are the parameters after the global server fuses the local parameters;

[0050] A sending module is used to send each of the local target parameters to the global server, and the global server is used to fuse the local target parameters to complete the training of the model. The trained model is used to complete the target task.

[0051] In a fourth aspect, an embodiment of the present invention further provides a terminal device, wherein the terminal device includes a memory, a processor, and a cluster-based model training program stored in the memory and executable on the processor, and when the processor executes the cluster-based model training program, the steps of the above-mentioned cluster-based model training method are implemented.

[0052] In a fifth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a cluster-based model training program is stored. When the cluster-based model training program is executed by a processor, the steps of the above-mentioned cluster-based model training method are implemented.

[0053] Beneficial effect: The present invention places the model to be trained on each cluster and uses the local data on each cluster to train the model. The present invention places the model to be trained on each cluster. Therefore, the present invention does not need to migrate the data on each cluster to the outside of each cluster and can use the data of each cluster to train the model, thereby protecting the data privacy on each cluster to prevent data leakage.

[0054] In addition, the present invention places the model on each cluster for parallel training, which can improve the training speed of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is the overall flow chart of the present invention;

[0056] Figure 2 is a flow chart of an embodiment of the present invention;

[0057] Figure 3 This is a block diagram of the internal structure principle of the terminal device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The following is a clear and complete description of the technical solution of the present invention in combination with the embodiments and the accompanying drawings. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] Research has found that in recent years, the global development of artificial intelligence (various intelligent computing models) has been unstoppable, and has created huge social benefits in key areas such as smart healthcare, smart cities, and Internet finance. Before using the model to process related tasks in the fields of smart healthcare, smart cities, and Internet finance, the model needs to be trained, and model training requires a large amount of data. Clusters, as carriers of data, have the data required for training models, but the limitations of the data corresponding to a single cluster cannot meet the data requirements for model training. Therefore, multiple clusters are required to provide the model with more and different data to meet the data requirements for model training. The existing technology is to migrate the data on each cluster to aggregate the data on each cluster, and then train the model with the aggregated data. The existing technology has caused the leakage of cluster data because the data on each cluster has been moved out of the cluster.

[0060] In order to solve the above technical problems, the present invention provides a cluster-based model training method, device, equipment and storage medium, which solves the problem that the existing technology uses clusters to train models, which may lead to data leakage. In specific implementation, the model is trained on each cluster, and the target parameters corresponding to the model are obtained based on the local parameters obtained by training the model on each cluster to complete the training of the model. The present invention does not need to migrate the data on each cluster to the outside of each cluster, and can use the data of each cluster to train the model, thereby protecting the data privacy on each cluster to prevent data leakage.

[0061] For example, the model is a speech recognition model. Before using the speech recognition model to recognize speech, it is necessary to first use data to train the speech recognition model until the parameters on the speech recognition model are trained to meet the requirements. There are three clusters A, B, and C. The data on cluster A comes from the speech of people in a, the data on cluster B comes from the speech of people in b, and the data on cluster C comes from the speech of people in c. Since the data sources of the three clusters are different, the data formed by the three clusters are relatively rich and can meet the data required for training the speech recognition model. In this embodiment, the speech recognition model is synchronized to these three clusters. The three clusters use their own data to train the speech recognition model respectively. When each cluster uses its own data to recognize the speech recognition model, it can obtain the parameters of the speech recognition model. For example, the parameters obtained by cluster A are a1, the parameters obtained by cluster B are b1, and the parameters obtained by cluster C are c1. The average of a1, b1, and c1 is calculated, and the average is used as the model parameter when the training model is terminated.

[0062] Exemplary Methods

[0063] The cluster-based model training method of this embodiment can be applied to a terminal device, which can be a terminal product with computing functions, such as a computer. Figure 1 As shown in , the cluster-based model training method specifically includes the following steps:

[0064] S100, sending the model to each cluster, where the cluster is used to provide the data required for training the model.

[0065] In this embodiment, the master working node (which can be a server with computing functions) builds the model to be trained (builds the model calculation graph), and then the master working node initializes the parameters of the model to be trained. The global server synchronizes the initialized model to be trained to each cluster, and each cluster receives the initialized model to be trained sent by the global server.

[0066] The cluster of this embodiment includes a simulated GPU cloud and a simulated NPU cloud, and both the simulated GPU cloud and the simulated NPU cloud include local working nodes and local servers. The local server is used to receive the model to be trained after initialization sent by the global server, and the local working node is used to load the local data on the simulated GPU cloud or the simulated NPU cloud onto the model for model training. The simulated GPU cloud includes 2 GPU servers and a local parameter server; the simulated NPU cloud includes two NPU servers and a local parameter server. The global parameter server is connected to the simulated GPU cloud / simulated NPU cloud via a wide area network.

[0067] S200, obtaining local parameters corresponding to the model when training the model on each of the clusters based on local data, where the local data is data on each of the clusters.

[0068] In this embodiment, each cluster first trains the model based on local data to obtain local parameters corresponding to the model on each cluster, and then each cluster sends the local parameters to the global server through its own local server.

[0069] In this embodiment, the cluster first trains the model based on the local data to obtain local parameters corresponding to the model on each cluster, including the following steps:

[0070] S201, according to each of the clusters, obtain the local servers and the working nodes included in each of the clusters, and the working nodes are used to load the local data onto the model.

[0071] In this embodiment, each cluster has its own local data, which needs to be loaded into the model through the working node for model training.

[0072] S202, training the model according to the local data on each of the working nodes to obtain sub-component parameters corresponding to the model.

[0073] After all servers synchronize the models to be trained to each cluster, each cluster will perform synchronous training on the models on each working node. The synchronous training of the models by each cluster belongs to inter-center synchronous training, and the synchronous training of the models by each working node within each cluster belongs to intra-center synchronous training. That is, this implementation divides the intra-center / extra-center network into two layers. The first layer realizes local aggregation within the center, and the second layer realizes global aggregation between centers, which can improve the training speed of the model.

[0074] S203, fusing the parameters of each sub-part through the local server to obtain local parameters corresponding to the model on each cluster.

[0075] In each cluster, since the model is trained on each working node, the sub-part parameters corresponding to each working node will also be obtained. These sub-part parameters need to be fused to obtain the local parameters that can represent the entire cluster to train the model. This embodiment fuses the sub-part parameters, specifically including: calculating the sub-part gradient corresponding to the model through each working node; the local server fuses the sub-part parameters according to the sub-part gradient to obtain the local parameters corresponding to the model on each cluster. Among them, the sub-part gradient is the sub-part gradient obtained by deriving the model on each working node. After obtaining the sub-part gradient, a suitable optimizer is selected. The sub-part gradient can characterize the change trend of the local parameter. For example, according to the sub-part gradient, it can be judged whether the sub-part parameter has a trend of increasing or decreasing during the training process. According to the change trend of the sub-part parameter, a suitable optimizer is selected to fuse the sub-part parameters to obtain the local parameters corresponding to each cluster.

[0076] S300, obtaining the target parameters corresponding to the model based on each of the local parameters, completing the training of the model, and the trained model is used to complete the target task.

[0077] In this implementation, the target parameters corresponding to the model are obtained according to each of the local parameters, which includes four parts: in the first part, each local server sends each local parameter and the local gradient composed of each sub-part gradient to the global server, the global server fuses each local parameter to obtain a fused parameter (for example, averaging each local parameter), and the global server aggregates each local gradient (for example, averaging each local gradient) to obtain an aggregated gradient; in the second part, the global server sends the fused parameter and the aggregated gradient to the local servers on each cluster, and each cluster continues to train the model to be trained based on the fused parameter and the aggregated gradient; in the third part, iterative training is performed, that is, the first part and the second part are repeated until the output result of the model to be trained converges; in the fourth part, the global server receives the local target parameters and local target gradients sent by each local server when the model output result converges, and fuses the local target parameters to obtain the target fused parameter, aggregates the local target gradient to obtain the target aggregated gradient, and then the global server obtains the optimizer according to the target aggregated gradient, and updates the target fused parameter through the optimizer to obtain the target parameter.

[0078] The first part includes: the global server aggregates each of the local gradients to obtain an aggregated gradient that matches the model; the global server fuses each of the local parameters to obtain a fusion parameter corresponding to the model.

[0079] The second part includes: the global server sends the aggregated gradient and the fusion parameter to each of the clusters; the global server obtains the local target parameters corresponding to the model on each of the clusters and performs iterative training according to the aggregated gradient, the fusion parameter and the local data until convergence.

[0080] For example, Figure 2 For example, the general process of training the model of the present invention is explained: the global server synchronizes the model to be trained to each working node on each cluster, and each working node trains the model. The training will change the parameters of each model, and the parameters are sent to the global server through the local server. The global server fuses the parameters sent by each local server, and the global server sends the fused parameters to each working node on each cluster. Several working nodes continue to train the model, and the above steps are repeated until the results of the model output converge, and the training of the model is completed.

[0081] Figure 2 The simulated GPU cloud and simulated NPU cloud in the example are clusters, which are described in detail below. Figure 2 The process of training the model:

[0082] (1) The master work node is responsible for building the model calculation graph (i.e., building the relationship between the various variables in the model), initializing the model parameters, and delivering them to the global parameter server to initialize the global model. At the same time, the master work node is also responsible for setting the synchronization mode (each cluster synchronizes the training model), intermediate data compression algorithm, network communication optimization algorithm, etc.

[0083] (ii) The local servers in each cluster obtain the latest model data from the global server and synchronize it to all working nodes in each cluster.

[0084] (3) The working nodes in each cluster are responsible for loading data and calculating the gradient of the local model.

[0085] (iv) Upload the gradients of the working nodes in each cluster to the local parameter server for parameter aggregation.

[0086] (V) The global server is responsible for globally aggregating the local aggregated gradients in all clusters and updating the parameters using the optimization algorithm.

[0087] (6) The global server sends the aggregated parameters to the local servers of each cluster.

[0088] (VII) Update the local model with the parameters of the local server

[0089] (8) Iterate steps (3) to (7) until the model converges.

[0090] In summary, the present invention places the model to be trained on each cluster and uses the local data on each cluster to train the model. The present invention places the model to be trained on each cluster, so the present invention does not need to migrate the data on each cluster to the outside of each cluster, and can use the data of each cluster to train the model, thereby protecting the data privacy on each cluster to prevent data leakage. The present invention places the model on each cluster for parallel training, which can increase the training speed of the model.

[0091] In addition, the present invention has the following advantages: 1) The present invention only focuses on whether the model structure and optimizer calculation are consistent when training the model, and does not need to change the framework usage habits (the framework is the hardware structure where the cluster is located), nor does it need to change the parallel training method (the model is synchronously trained in parallel on each cluster);

[0092] 2) To solve the problem of insufficient resources in one cluster to train a large model, resources in other clusters can be used for training; 3) It is convenient for participants to join. Only a few lines of communication code need to be inserted in the corresponding places to realize collaborative training of large models with other participants; 4) When the model to be trained is a speech recognition model, for special needs such as multi-language translation, each institution has data in different languages ​​and uses different frameworks. This method can easily realize collaborative training among multiple institutions.

[0093] Exemplary Devices

[0094] This embodiment also provides a device for a cluster-based model training method, the device comprising the following components:

[0095] A model sending module is used to send the model to each cluster, and the cluster is used to provide the data required for training the model;

[0096] A parameter receiving module, used for obtaining each local parameter corresponding to the model when training the model on each cluster according to the local data, wherein the local data is the data on each cluster;

[0097] The target parameter generation module is used to obtain the target parameters corresponding to the model based on each of the local parameters, complete the training of the model, and the trained model is used to complete the target task.

[0098] Alternatively, the device comprises the following components:

[0099] A model training module, used to train the model based on local data to obtain local parameters corresponding to the model on each cluster, wherein the local data is located on each cluster;

[0100] A parameter sending module, used for sending each of the local parameters to a global server, and the global server is used for fusing each of the local parameters;

[0101] A local target parameter generation module, used to obtain local target parameters corresponding to the models on each of the clusters according to the aggregation parameters and the local data, wherein the aggregation parameters are the parameters after the global server fuses the local parameters;

[0102] A sending module is used to send each of the local target parameters to the global server, and the global server is used to fuse the local target parameters to complete the training of the model. The trained model is used to complete the target task.

[0103] Based on the above embodiments, the present invention further provides a terminal device, whose principle block diagram can be shown as follows: Figure 3 As shown. The terminal device includes a processor, a memory, a network interface, a display screen, and a temperature sensor connected through a system bus. Among them, the processor of the terminal device is used to provide computing and control capabilities. The memory of the terminal device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a cluster-based model training method is implemented. The display screen of the terminal device can be a liquid crystal display screen or an electronic ink display screen, and the temperature sensor of the terminal device is pre-set inside the terminal device to detect the operating temperature of the internal device.

[0104] Those skilled in the art will understand that Figure 3 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal device to which the scheme of the present invention is applied. The specific terminal device may include more or less components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0105] In one embodiment, a terminal device is provided, the terminal device comprising a memory, a processor, and a cluster-based model training program stored in the memory and executable on the processor, and when the processor executes the cluster-based model training program, the following operation instructions are implemented:

[0106] Send the model to each cluster, where the cluster is used to provide the data required for training the model;

[0107] Obtaining local parameters corresponding to the model when training on each of the clusters based on local data, where the local data is data on each of the clusters;

[0108] Based on each of the local parameters, the target parameters corresponding to the model are obtained to complete the training of the model. The trained model is used to complete the target task.

[0109] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0110] In summary, the present invention discloses a cluster-based model training method, device, equipment and storage medium, the method comprising: sending the model to each cluster, the cluster is used to provide the model with the data required for training; obtaining each local parameter corresponding to the model when training based on local data on each cluster, the local data is the data on each cluster; according to each local parameter, the target parameter corresponding to the model is obtained to complete the training of the model, and the model after training is used to complete the target task. The present invention does not need to migrate the data on each cluster to the outside of each cluster, and can use the data of each cluster to train the model, thereby protecting the data privacy on each cluster to prevent data leakage.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cluster-based model training method, characterized in that: include: Send the model to each cluster, the model is a speech recognition model, the cluster is used to provide the model with the data required for training, the cluster includes a local working node and a local server, the local server is used to receive the model sent by the global server, the local working node is used to load local data onto the model for model training, the local server sends local parameters to the global server, and each working node in each cluster performs synchronous training on the model; Obtaining local parameters corresponding to the model when it is trained on each of the clusters based on local data, where the local data is data on each of the clusters; According to each of the local parameters, the target parameters corresponding to the model are obtained to complete the training of the model, and the trained model is used to complete the target task; The model is trained based on the local data to obtain local parameters corresponding to the model on each cluster, wherein the local data is located on each cluster, including: According to each of the clusters, a local server and each working node included in each of the clusters are obtained, and the working node is used to load the local data onto the model; Training the model according to the local data on each of the working nodes to obtain sub-part parameters corresponding to the model; Calculate the sub-part gradient corresponding to the model through each of the working nodes; The local server fuses the parameters of each sub-part according to the sub-part gradient to obtain the local parameters corresponding to the model on each cluster.

2. The cluster-based model training method according to claim 1, characterized in that: The target parameters corresponding to the model are obtained according to each of the local parameters, and the training of the model is completed. The trained model is used to complete the target task, including: Obtaining each local gradient corresponding to the model located on each of the clusters during training; According to each of the local gradients and each of the local parameters, the target parameters corresponding to the model are obtained to complete the training of the model.

3. The cluster-based model training method according to claim 2, characterized in that: The step of obtaining target parameters corresponding to the model based on the local gradients and the local parameters to complete the training of the model includes: Aggregating each of the local gradients to obtain an aggregated gradient that matches the model; Fusing the local parameters to obtain fusion parameters corresponding to the model; According to the aggregated gradient and the fusion parameter, the target parameter corresponding to the model is obtained.

4. The cluster-based model training method according to claim 3, characterized in that: The step of obtaining target parameters corresponding to the model according to the aggregated gradient and the fusion parameter includes: Sending the aggregated gradient and the fusion parameter to each of the clusters; Obtaining the local target parameters corresponding to the model on each of the clusters when iterative training is performed according to the aggregated gradient, the fusion parameter and the local data until convergence; Fusing the local target parameters to obtain target fusion parameters corresponding to the model; According to the target fusion parameters, the target parameters corresponding to the model are obtained.

5. The cluster-based model training method according to claim 4, characterized in that: The step of obtaining target parameters corresponding to the model based on the target fusion parameters includes: Obtaining the local target gradient corresponding to when the model on each of the clusters is iteratively trained according to the aggregated gradient, the fusion parameter and the local data until convergence; Aggregating each of the local target gradients to obtain a target aggregate gradient that matches the model; According to the target aggregation gradient, an optimizer matching the target fusion parameter is obtained; According to the optimizer, the target fusion parameters are updated to obtain the target parameters corresponding to the model.

6. A cluster-based model training method, characterized in that: include: The model is trained according to the local data to obtain local parameters corresponding to the model on each cluster, wherein the model is a speech recognition model, the local data is located on each cluster, the cluster includes a local working node and a local server, the local server is used to receive the model sent by the global server, the local working node is used to load the local data onto the model for model training, the local server sends the local parameters to the global server, and each working node in each cluster synchronously trains the model; Sending each of the local parameters to a global server, where the global server is used to fuse each of the local parameters; According to the aggregation parameter and the local data, each local target parameter corresponding to the model on each cluster is obtained, wherein the aggregation parameter is a parameter obtained after the global server fuses each local parameter; Sending each of the local target parameters to the global server, the global server is used to fuse each of the local target parameters to complete the training of the model, and the trained model is used to complete the target task; The model is trained according to the local data to obtain local parameters corresponding to the model on each cluster, wherein the local data is located on each cluster, including: According to each of the clusters, a local server and each working node included in each of the clusters are obtained, and the working node is used to load the local data onto the model; Training the model according to the local data on each of the working nodes to obtain sub-part parameters corresponding to the model; Calculate the sub-part gradient corresponding to the model through each of the working nodes; The local server fuses the parameters of each sub-part according to the sub-part gradient to obtain the local parameters corresponding to the model on each cluster.

7. The cluster-based model training method according to claim 6, characterized in that: Also includes: Each of the clusters receives the model sent by the global server, and the global server initializes the model before sending the model.

8. The cluster-based model training method according to any one of claims 1 to 7, characterized in that: Each of the clusters trains the model synchronously, and each of the clusters is a heterogeneous cluster, and the heterogeneous clusters include an NPU cluster and a GPU cluster.

9. A device for a cluster-based model training method, characterized in that: The device comprises the following components: A model sending module is used to send the model to each cluster. The model is a speech recognition model. The cluster is used to provide the model with the data required for training. The cluster includes a local working node and a local server. The local server is used to receive the model sent by the global server. The local working node is used to load local data onto the model for model training. The local server sends local parameters to the global server. Each working node in each cluster performs synchronous training on the model. A parameter receiving module, used for obtaining each local parameter corresponding to the model when training the model on each cluster according to the local data, wherein the local data is the data on each cluster; A target parameter generation module, used to obtain the target parameters corresponding to the model according to each of the local parameters, complete the training of the model, and the trained model is used to complete the target task; The model is trained according to the local data to obtain local parameters corresponding to the model on each cluster, wherein the local data is located on each cluster, including: According to each of the clusters, a local server and each working node included in each of the clusters are obtained, and the working node is used to load the local data onto the model; Training the model according to the local data on each of the working nodes to obtain sub-part parameters corresponding to the model; Calculate the sub-part gradient corresponding to the model through each of the working nodes; The local server fuses the parameters of each sub-part according to the sub-part gradient to obtain the local parameters corresponding to the model on each cluster.

10. A device for a cluster-based model training method, characterized in that: The device comprises the following components: A model training module is used to train the model based on local data to obtain local parameters corresponding to the model on each cluster. The model is a speech recognition model. The local data is located on each cluster. The cluster includes a local working node and a local server. The local server is used to receive the model sent by the global server. The local working node is used to load the local data onto the model for model training. The local server sends the local parameters to the global server. Each working node in each cluster synchronously trains the model. A parameter sending module, used for sending each of the local parameters to a global server, and the global server is used for fusing each of the local parameters; A local target parameter generation module, used to obtain local target parameters corresponding to the models on each of the clusters according to the aggregation parameters and the local data, wherein the aggregation parameters are the parameters after the global server fuses the local parameters; A sending module, used for sending each of the local target parameters to the global server, the global server is used for fusing each of the local target parameters to complete the training of the model, and the trained model is used to complete the target task; The model is trained according to the local data to obtain local parameters corresponding to the model on each cluster, wherein the local data is located on each cluster, including: According to each of the clusters, a local server and each working node included in each of the clusters are obtained, and the working node is used to load the local data onto the model; Training the model according to the local data on each of the working nodes to obtain sub-part parameters corresponding to the model; Calculate the sub-part gradient corresponding to the model through each of the working nodes; The local server fuses the parameters of each sub-part according to the sub-part gradient to obtain the local parameters corresponding to the model on each cluster.

11. A terminal device, characterized in that: The terminal device includes a memory, a processor, and a cluster-based model training program stored in the memory and executable on the processor. When the processor executes the cluster-based model training program, the steps of the cluster-based model training method as described in any one of claims 1 to 8 are implemented.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a cluster-based model training program, and when the cluster-based model training program is executed by the processor, the steps of the cluster-based model training method as described in any one of claims 1-8 are implemented.

Images