A model distributed training method and system based on computing power network

Abstract

The present invention discloses a distributed model training method based on a computing power network, comprising: obtaining model information and model files uploaded by users, obtaining data set information, determining whether the model name in the obtained model information exists in a pre-established dictionary, and if not, creating a main process; utilizing the main process and loading the corresponding model and data set from a pre-established model repository and data set repository, respectively, based on the model name and data set name in a distributed call service request; performing distributed training on the model using the loaded data set to obtain a distributed trained model; and saving the distributed trained model to a distributed training result repository via the main process. This invention can address the technical issues of existing distributed systems, such as the limited dissemination range of user-built models, and the waste of computing resources caused by distributed training of mismatched models and data sets.

Description

Technical Field

[0001] The present invention belongs to the field of high-performance computing technology, and more specifically, relates to a model distributed training method and system based on a computing power network. Background Art

[0002] In recent years, the scope of machine learning research has continued to expand, with its applications in fields such as natural language processing, image recognition, and speech recognition becoming increasingly widespread, directly impacting our daily lives. However, as model size and training data volumes increase, single-machine model training can no longer meet real-time requirements. To address this issue, researchers have begun to incorporate distributed systems into machine learning, leading to the emergence of distributed training.

[0003] Currently, the mainstream distributed training systems are pytorch and TensorFlow.

[0004] PyTorch uses a dynamic computational graph model and implements distributed training through the torch.distributed module. This module provides a variety of distributed training strategies, encapsulates the model through the DistributedDataParallel module, implements data parallelism, and updates model parameters through gradient synchronization.

[0005] TensorFlow uses a static computational graph model and supports distributed training through the tf.distribute module. This module can be used for distributed training in different environments, allowing users to select the appropriate configuration based on their specific hardware and training requirements.

[0006] However, the two distributed training systems mentioned above have some problems that cannot be ignored:

[0007] First, the models built by users of the two distributed training systems mentioned above are not easily usable by other users, making these models difficult to be widely used.

[0008] Second, in the two distributed training systems mentioned above, mismatched models and datasets may cause distributed training to begin until errors occur during the training process, resulting in a waste of computing resources.

[0009] Third, the two distributed training systems mentioned above require a high level of professional expertise from users. For users without a basic knowledge of distributed training, additional learning costs are required, and the degree of automation is low.

[0010] Fourth, the above two distributed training systems are not convenient enough for managing models and distributed training result files. Summary of the Invention

[0011] In response to the above-mentioned defects or improvement needs of the prior art, the present invention provides a model distributed training method and system based on a computing power network, which aims to solve the technical problem that the model built by users of the existing distributed system has a small dissemination range; the technical problem that distributed training of mismatched models and data sets causes waste of computing resources; the technical problem that the degree of automation is low, resulting in additional learning costs for researchers; and the technical problem that the management of models and distributed training result files cannot be effectively realized.

[0012] To achieve the above objectives, according to one aspect of the present invention, a distributed model training method based on a computing power network is provided, comprising the following steps:

[0013] (1) Obtain model information and model files uploaded by users;

[0014] (2) Obtaining information about the dataset;

[0015] (3) Determine whether the model name in the model information obtained in step (1) exists in a pre-established dictionary. If so, obtain a list of dataset names corresponding to the model name from the dictionary and proceed to step (4). Otherwise, proceed to step (5).

[0016] (4) Determine whether the dataset name in the dataset information obtained in step (2) exists in the dataset name list. If so, generate a distributed training service request based on the model name and the dataset name corresponding to the dataset name list, and then proceed to step (5). Otherwise, proceed to step (8).

[0017] (5) Create a main process, use the main process, and load the corresponding model and dataset from the pre-established model warehouse and dataset warehouse respectively according to the model name and dataset name in the distributed call service request;

[0018] (6) Using the dataset loaded in step (5), the model loaded in step (5) is subjected to distributed training to obtain a distributed trained model.

[0019] (7) The distributed training model obtained in step (6) is saved to the distributed training result warehouse through the main process, and it is determined whether the distributed training result can be obtained. If so, the process ends, otherwise it goes to step (8).

[0020] (8) Re-acquire the data set information and return to step (3).

[0021] Preferably, the model includes a neural network model built by the user, or a deep neural network model;

[0022] Deep neural network models include convolutional neural networks (CNN), recurrent neural networks (RNN), and so on.

[0023] Model information includes model name, model type, and model purpose.

[0024] Preferably, the information of the dataset includes the dataset name, dataset introduction, dataset type, the size of the training set and test set in the dataset, and the classification category of the dataset.

[0025] In the dictionary, the model name is used as the key, and the dataset name list consisting of all dataset names supported by the model name is used as the corresponding value.

[0026] Preferably, in step (4), the process of generating a distributed training service request based on the model name and the dataset name corresponding to the dataset name list is to use the requests command in Python to package the model name and the dataset name into a distributed training service request.

[0027] Specifically, step (5) is as follows: first, parse the distributed call service request to obtain the model name and dataset name therein; then, use the importlib library in the python library to load the model file corresponding to the model name stored in the model warehouse; finally, use the datasets module in the torchvision library to load the dataset corresponding to the dataset name stored in the dataset warehouse; if the dataset is not stored in the dataset warehouse, use the datasets module to download the dataset corresponding to the dataset name from the dataset source provided by the open source deep learning framework pytorch to the dataset warehouse, and then load the dataset from the dataset warehouse.

[0028] Preferably, step (6) includes the following sub-steps:

[0029] (6-1) Create n worker processes through the main process, randomly number the created n worker processes, divide the data set loaded in step (5) to obtain multiple sub-data sets, and save the model loaded in step (5) globally and locally in turn to obtain a global model corresponding to the main thread and n local models corresponding to the n worker threads, where the value range of n is 1 to 8.

[0030] (6-2) All sub-datasets, 1 global model and n local models obtained in step (6-1) are assigned to corresponding work processes.

[0031] (6-3) Each working process trains the model loaded in step (5) according to its corresponding sub-dataset and local model to obtain the local parameters of the current cycle of the working process.

[0032] (6-4) Each worker process waits for all other worker processes to obtain their local parameters for the current cycle, and then sends its local parameters for the current cycle to the global model;

[0033] (6-5) After receiving the local parameters of the current cycle from all working processes, the global model aggregates all local parameters and uses the aggregation results to update the weights of the global model to obtain the updated global parameters of all working processes.

[0034] (6-6) Before each working process needs to train the local model, it uses the global parameters updated by all working processes obtained in step (6-5) to update the local parameters of the working process, and repeats the above steps (6-2) to step (6-5) until the preset number of iterative training times is reached, thereby obtaining a distributed trained model.

[0035] Preferably, the process of obtaining multiple sub-datasets in step (6-1) is as follows: first, obtain the size of the data set loaded in step (5); then, divide the size of the data set by the total number of preset working processes to obtain the size of the data set that each working process should process; then, perform a first division on the data set loaded in step (5) according to the size, and the obtained small part is the first sub-training set (numbered 1), and the large part is the remaining data set; then, divide the remaining data set according to the size to obtain the second sub-training set (numbered 2), ..., and so on, and finally, obtain multiple divided sub-datasets;

[0036] Step (6-1) is to save the model loaded in step (5) as a global model, which is used to store global parameters, including the number of layers, weights, and biases of the neural network. All working processes jointly maintain the global parameters;

[0037] Step (6-1) saves a model loaded in step (5) as a local model for each working process to store local parameters, including the number of layers, weights, and biases of the neural network.

[0038] Preferably, step (6-2) is specifically as follows: first, the global model is assigned to the main process, and all sub-datasets are assigned to corresponding working processes according to their numbers, that is, the sub-dataset numbered 1 is assigned to the working process numbered 1,..., and so on. Finally, each local model is assigned to its corresponding working process.

[0039] Preferably, step (6-3) is specifically,

[0040] First, each worker process sets a random seed using its unique number;

[0041] Then, each worker process uses the DataLoader tool in PyTorch to load the first batch of data samples from the sub-dataset according to the pre-set batch size, and performs forward propagation on the loaded first batch of data samples through the local model to obtain the output of the local model;

[0042] Subsequently, each working process calculates the loss function value based on the output of the local model and the target output, and then backpropagates the loss function value to obtain the gradient of the local model corresponding to the working process. These gradients will be used to guide the update of local parameters.

[0043] Subsequently, the gradient of the local model corresponding to the working process is used to update the local parameters corresponding to the working process.

[0044] Finally, the above process is repeated for the remaining batches of data samples in the sub-dataset until all data samples in the sub-dataset are processed, thereby obtaining the local parameters of the current cycle of the work process.

[0045] Preferably, step (6-6) is specifically,

[0046] First, before a new training cycle begins, each worker process will obtain the value of the global parameter updated by all worker processes in step (6-5) for each parameter name to update the value of the local parameter of the corresponding parameter name, so that each worker process obtains the new local parameter;

[0047] Each worker process then repeats the training process from steps (6-2) to (6-5) above, performing a new training cycle on the updated local parameters and the corresponding sub-dataset. This continues until the pre-set number of iterations are completed. The global model of the master process is then used as the distributed trained model.

[0048] According to another aspect of the present invention, a model distributed training system based on a computing power network is provided, comprising:

[0049] The first module is used to obtain the model information and model files uploaded by the user;

[0050] The second module is used to obtain the information of the data set;

[0051] The third module is used to determine whether the model name in the model information obtained by the first module exists in a pre-established dictionary. If so, it obtains the dataset name list corresponding to the model name from the dictionary and transfers to the fourth module; otherwise, it transfers to the fifth module;

[0052] The fourth module is used to determine whether the dataset name in the dataset information obtained in the second module exists in the dataset name list. If so, a distributed training service request is generated based on the model name and the dataset name corresponding to the dataset name list, and then the process is transferred to the fifth module. Otherwise, the process is transferred to the eighth module.

[0053] The fifth module is used to create a main process, and use the main process to load the corresponding model and dataset from the pre-established model warehouse and dataset warehouse respectively according to the model name and dataset name in the distributed call service request;

[0054] The sixth module is used to perform distributed training on the model loaded by the fifth module using the data set loaded by the fifth module to obtain a distributed trained model.

[0055] The seventh module is used to save the distributed training model obtained by the sixth module to the distributed training result warehouse through the main process, and determine whether the distributed training result can be obtained. If so, the process ends, otherwise it goes to the eighth module.

[0056] The eighth module is used to retrieve the information of the data set and return to the third module.

[0057] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0058] (1) Due to the adoption of step (1), the present invention stores the models uploaded by users in the model warehouse, which facilitates the storage and management of the models. Therefore, the models can also be used by other users, which can enhance the communication and dissemination between the models built by users to a greater extent. Therefore, it can solve the technical problem that the models built by users themselves are difficult to be widely used.

[0059] (2) The present invention adopts steps (3) to (4), which uses a pre-established dictionary to check the model and data set used to generate the distributed service request, thereby reducing the distributed training that may be performed on mismatched models and data sets before the distributed training begins, thereby solving the technical problem of wasting computing resources;

[0060] (3) Due to the adoption of step (5), the present invention realizes dynamic loading of the model through the importlib library in the Python library, and automatically loads the model from the model warehouse for training during training, thereby solving the technical problem that the distributed training system requires a high level of professionalism from the user;

[0061] (4) Due to the adoption of step (7), the present invention separately stores the distributed training result files through the distributed training result warehouse, which facilitates the download and management of the distributed training result files, thereby solving the technical problem of the difficulty in managing the model and the distributed training result files;

[0062] (5) The present invention also has strong scalability. Its distributed training algorithm is designed and implemented separately, so more distributed training algorithms can be expanded for different distributed training tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a flow chart of the model distributed training method based on computing power network of the present invention. DETAILED DESCRIPTION

[0064] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0065] The present invention provides a model distributed training method based on a computing power network, the purpose of which is to realize the management of user models, so that users can implement simple and convenient distributed model training tasks in a computing power network in a more automated manner without the knowledge base of distributed training.

[0066] like Figure 1 As shown, the present invention provides a model distributed training method based on a computing power network, comprising the following steps:

[0067] (1) Obtain model information and model files uploaded by users;

[0068] After obtaining the model file and model information uploaded by the user, this step names the model file to be consistent with the model name in the model information. Then, the model file is stored in the model warehouse for subsequent model loading in distributed services. The subsequent model loading process will use the importlib library in the Python library to load the model file to complete the model initialization.

[0069] The models of the present invention include neural network models built by users themselves, or deep neural network models such as convolutional neural networks (CNN) and recurrent neural networks (RNN).

[0070] The model information in the present invention includes model name, model type, model purpose, etc.

[0071] The advantage of this step (1) is that the model uploaded by the user will be stored and managed by the platform, so it can also be used by other users, which can enhance the communication and dissemination between models to a greater extent.

[0072] (2) Obtaining information about the dataset;

[0073] Specifically, the dataset information includes the dataset name, dataset introduction, dataset type, the size of the training set and test set in the dataset, the dataset classification category, and other information.

[0074] (3) Determine whether the model name in the model information obtained in step (1) exists in a pre-established dictionary. If so, obtain a list of dataset names corresponding to the model name from the dictionary and proceed to step (4). Otherwise, proceed to step (5).

[0075] Specifically, the dictionary uses the model name (such as CNN, RNN, etc.) as the dictionary key, and the dataset name list consisting of all dataset names supported by the model name as the corresponding value.

[0076] (4) Determine whether the dataset name in the dataset information obtained in step (2) exists in the dataset name list. If so, generate a distributed training service request based on the model name and the dataset name corresponding to the dataset name list, and then proceed to step (5). Otherwise, proceed to step (8).

[0077] Specifically, in this step, the process of generating a distributed training service request based on the model name and the dataset name corresponding to the dataset name list is to use the requests command in Python to package the model name and the dataset name into a distributed training service request.

[0078] The advantage of the above steps (3) to (4) is that the model and dataset used to generate the distributed service request are checked using a pre-established dictionary, thereby reducing the distributed training requests of mismatched models and datasets and avoiding the waste of computing resources.

[0079] (5) Create a main process, use the main process, and load the corresponding model and dataset from the pre-established model warehouse and dataset warehouse respectively according to the model name and dataset name in the distributed call service request;

[0080] Specifically, this step first parses the distributed call service request to obtain the model name and dataset name, and then uses the importlib library in the python library to load the model file corresponding to the model name stored in the model warehouse; finally, uses the datasets module in the torchvision library to load the dataset corresponding to the dataset name stored in the dataset warehouse. If the dataset is not stored in the dataset warehouse, the datasets module is used to download the dataset corresponding to the dataset name from the dataset source provided by the open source deep learning framework pytorch to the dataset warehouse, and then load the dataset from the dataset warehouse.

[0081] The torchvision library is a sub-library in the open source deep learning framework pytorch, which provides commonly used datasets, pre-trained models, and image conversion and processing tools; the datasets module has built-in datasets such as MNIST, CIFAR10, ImageNet, CoCo, etc.

[0082] The importlib library is a module in Python that provides tools for dynamically loading modules. It is suitable for importing modules at runtime or operating the import process programmatically, so that the program can load modules on demand at runtime instead of statically importing all modules at the beginning of the program.

[0083] The advantage of this step (5) is that it realizes the dynamic loading of the model, automatically loading the model from the model warehouse for training during training, and completing the model distributed training service in a more automated form to solve the problem of high professional requirements for distributed training.

[0084] (6) Using the dataset loaded in step (5), the model loaded in step (5) is subjected to distributed training to obtain a distributed trained model.

[0085] Specifically, this step includes the following sub-steps:

[0086] (6-1) Create n worker processes through the main process, randomly number the created n worker processes, divide the data set loaded in step (5) to obtain multiple sub-data sets, and save the model loaded in step (5) globally and locally in turn to obtain a global model corresponding to the main thread and n local models corresponding to the n worker threads (where n ranges from 1 to 8, preferably 4).

[0087] The process of obtaining multiple sub-datasets in this step is as follows: first, obtain the size of the dataset loaded in step (5); then, divide the size of the dataset by the total number of pre-set working processes (which is equal to the total number of local models n) to obtain the size of the dataset that each working process should process; then, perform the first division of the dataset loaded in step (5) according to the size, and the small part obtained is the first sub-training set (numbered 1), and the large part is the remaining dataset; then, divide the remaining dataset according to the size to obtain the second sub-training set (numbered 2), ..., and so on, and finally, obtain multiple divided sub-datasets.

[0088] In addition, this step is to save the model loaded in step (5) as a global model for storing global parameters (including the number of layers, weights, biases, etc. of the neural network). All working processes jointly maintain the global parameters, and these global parameters are accessed and updated by all working processes during the training process to ensure the consistency and accuracy of the model. In addition, this step saves a model loaded in step (5) as a local model for each working process to store local parameters (including the number of layers, weights, biases, etc. of the neural network). The global parameters and local parameters are saved by the model loaded in step (5). Therefore, the global model and the local model contain the same parameter name, and the value of the local parameter of each working process will be randomly initialized in step (6-3). The value of the global parameter will change by averaging the values ​​of the local parameters of each working process.

[0089] (6-2) All sub-datasets, 1 global model and n local models obtained in step (6-1) are assigned to corresponding work processes.

[0090] Specifically, this step is as follows: first, the global model is assigned to the main process, all sub-datasets are assigned to the corresponding working processes according to their numbers (that is, the sub-dataset numbered 1 is assigned to the working process numbered 1, ..., and so on), and finally, each local model is assigned to its corresponding working process.

[0091] (6-3) Each working process trains the model loaded in step (5) according to its corresponding sub-dataset and local model to obtain the local parameters of the current cycle of the working process.

[0092] Specifically, each worker process sets a random seed using its unique number. This allows each worker process to randomly initialize local parameter values, avoiding synchronization issues with local models. This helps reduce random initialization differences between worker processes during distributed training.

[0093] Then, each worker process uses the DataLoader tool in PyTorch to load the first batch of data samples from the sub-dataset according to the preset batch size (i.e., Batch size) (if Batch size is set to 64, 64 data samples from the sub-dataset are loaded in each batch), and performs forward propagation on the loaded first batch of data samples through the local model (this process involves passing the input data through various layers of the local model, such as fully connected layers, convolutional layers, activation layers, etc.) to obtain the output of the local model (this output can have different forms, such as the housing price value in the housing price prediction task);

[0094] Subsequently, each worker process calculates the loss function value based on the output of the local model and the target output (such as the actual price of the house in the house price prediction task), and then backpropagates the loss function value to obtain the gradient of the local model corresponding to the worker process (these gradients will be used to guide the update of local parameters).

[0095] Subsequently, the gradient of the local model corresponding to the working process is used to update the local parameters corresponding to the working process.

[0096] Finally, the above process is repeated for the remaining batches of data samples in the sub-dataset until all data samples in the sub-dataset are processed, thereby obtaining the local parameters of the current cycle of the work process.

[0097] In addition, the training speed of each worker process may be different. Therefore, after the training cycle ends, the worker process that completes the local model training ahead of time will call the synchronization mechanism to put the worker process into a blocked state, waiting for all other worker processes to complete the local model training.

[0098] The advantage of this sub-step (6-3) is that it uses a random seed to randomly initialize the local parameters of each worker process, avoiding the synchronization problem of the local model and reducing the random differences in initialization between different worker processes in distributed training.

[0099] (6-4) Each worker process waits for all other worker processes to obtain their local parameters for the current cycle, and then sends its local parameters for the current cycle to the global model;

[0100] (6-5) After receiving the local parameters of the current cycle from all working processes, the global model aggregates all local parameters and uses the aggregation results to update the weights of the global model to obtain the updated global parameters of all working processes.

[0101] Specifically, after receiving the local parameters of all worker processes for the current cycle, the global model, because the global model and the local models have the same parameter names, will traverse all local models in sequence according to each parameter name in the local parameters of the current cycle to obtain the parameter value corresponding to that parameter name. It will then take the average of all parameter values ​​as the parameter value corresponding to that parameter name in the new global model and save it to the global model. After all parameter names in the local parameters of the current cycle are updated, the updated global parameters of all worker processes are obtained. This ensures that the global model can obtain the latest local parameter information, thereby ensuring the convergence and consistency of the global model as a whole.

[0102] (6-6) Before each work process needs to train the local model, it uses the global parameters updated by all work processes obtained in step (6-5) to update the local parameters of the work process, and repeats the above steps (6-2) to (6-5) until the preset number of iterative training times is reached (the preset number of iterative training times ranges from 3 to 10, preferably 5), thereby obtaining a distributed trained model.

[0103] Specifically, first, before the start of a new training cycle, since the global model and the local model have the same parameter name, each worker process will obtain the value of the global parameter updated by all worker processes in step (6-5) for each parameter name to update the value of the local parameter of the corresponding parameter name, so that each worker process obtains the new local parameter.

[0104] Each worker process then repeats the training process from steps (6-2) to (6-5) above, performing a new training cycle on the updated local parameters and the corresponding sub-dataset. This continues until the pre-set number of iterations are completed. The global model of the master process is then used as the distributed trained model.

[0105] (7) The distributed training model obtained in step (6) is saved to the distributed training result warehouse through the main process, and it is determined whether the distributed training result can be obtained. If so, the process ends, otherwise it goes to step (8).

[0106] This step stores the distributed training model obtained in step (6) into the distributed training result warehouse (the warehouse is used to store all the result files after distributed training) and determines whether the distributed training result can be obtained. If so, the process ends, otherwise it goes to step (8).

[0107] The advantage of this step (7) is that the distributed training results are saved separately, which facilitates the downloading and management of the distributed training result files.

[0108] (8) Re-acquire the data set information and return to step (3);

[0109] Through the above steps, a distributed training platform is finally realized, which can realize the acquisition of model information and model files, the acquisition of data sets, and the selection of appropriate models and data sets to call distributed training services. This allows users to implement distributed training tasks of models in a relatively automatic manner without any knowledge of distributed training.

[0110] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A distributed model training method based on a computing power network, characterized in that: The following steps are involved: (1) Obtain model information and model files uploaded by users; (2) Obtaining information about the dataset; (3) Determine whether the model name in the model information obtained in step (1) exists in a pre-established dictionary. If so, obtain a list of dataset names corresponding to the model name from the dictionary and proceed to step (4). Otherwise, proceed to step (5). (4) Determine whether the dataset name in the dataset information obtained in step (2) exists in the dataset name list. If so, generate a distributed training service request based on the model name and the dataset name corresponding to the dataset name list, and then proceed to step (5). Otherwise, proceed to step (8). (5) Create a main process, use the main process, and load the corresponding model and dataset from the pre-established model warehouse and dataset warehouse respectively according to the model name and dataset name in the distributed call service request; Step (5) is specifically as follows: first parse the distributed call service request to obtain the model name and dataset name therein, then use the importlib library in the python library to load the model file corresponding to the model name stored in the model warehouse; finally, use the datasets module in the torchvision library to load the dataset corresponding to the dataset name stored in the dataset warehouse. If the dataset is not stored in the dataset warehouse, use the datasets module to download the dataset corresponding to the dataset name from the dataset source provided by the open source deep learning framework pytorch to the dataset warehouse, and then load the dataset from the dataset warehouse; (6) Using the data set loaded in step (5) to perform distributed training on the model loaded in step (5) to obtain a distributed trained model; step (6) includes the following sub-steps: (6-1) creating n worker processes through the main process, randomly numbering the created n worker processes, dividing the data set loaded in step (5) to obtain multiple sub-data sets, and successively saving the model loaded in step (5) globally and locally to obtain a global model corresponding to the main thread and n local models corresponding to the n worker threads, where the value of n ranges from 1 to 8; (6-2) Allocate all sub-datasets, 1 global model, and n local models obtained in step (6-1) to corresponding work processes; (6-3) Each working process trains the model loaded in step (5) according to its corresponding sub-dataset and local model to obtain the local parameters of the current cycle of the working process; (6-4) Each worker process waits for all other worker processes to obtain their local parameters for the current cycle, and then sends its local parameters for the current cycle to the global model; (6-5) After receiving the local parameters of the current cycle from all work processes, the global model aggregates all local parameters and uses the aggregation results to update the weights of the global model to obtain the updated global parameters of all work processes; (6-6) Before each worker process needs to train the local model, it uses the global parameters updated by all worker processes obtained in step (6-5) to update the local parameters of the worker process, and repeats the above steps (6-2) to (6-5) until the preset number of iterative training times is reached, thereby obtaining a distributed trained model; (7) The distributed training model obtained in step (6) is saved to the distributed training result warehouse through the main process, and it is determined whether the distributed training result can be obtained. If so, the process ends, otherwise it goes to step (8); (8) Re-acquire the data set information and return to step (3).

2. The distributed model training method based on computing power network according to claim 1 is characterized in that: Models include user-built neural network models or deep neural network models; Deep neural network models include convolutional neural network (CNN) and recurrent neural network (RNN); Model information includes model name, model type, and model purpose.

3. The distributed model training method based on computing power network according to claim 1 or 2, characterized in that: The dataset information includes the dataset name, dataset introduction, dataset type, the size of the training set and test set in the dataset, and the dataset classification category; In the dictionary, the model name is used as the key, and the dataset name list consisting of all dataset names supported by the model name is used as the corresponding value.

4. The distributed model training method based on computing power network according to claim 3 is characterized in that: In step (4), the process of generating a distributed training service request based on the model name and the dataset name corresponding to the dataset name list is to use the requests command in Python to package the model name and the dataset name into a distributed training service request.

5. The distributed model training method based on computing power network according to claim 4 is characterized in that: The process of obtaining multiple sub-datasets in step (6-1) is as follows: first, obtain the size of the data set loaded in step (5); then, divide the size of the data set by the total number of preset working processes to obtain the size of the data set that each working process should process; then, perform a first division on the data set loaded in step (5) according to the size, and the obtained small part is the first sub-training set, and the large part is the remaining data set; then, divide the remaining data set according to the size to obtain the second sub-training set, ..., and so on, and finally, obtain multiple divided sub-datasets; Step (6-1) is to save the model loaded in step (5) as a global model, which is used to store global parameters, including the number of layers, weights, and biases of the neural network. All working processes jointly maintain the global parameters; Step (6-1) saves a model loaded in step (5) as a local model for each working process to store local parameters, including the number of layers, weights, and biases of the neural network.

6. The distributed model training method based on computing power network according to claim 5 is characterized in that: Step (6-2) is as follows: first, the global model is assigned to the main process, and all sub-datasets are assigned to the corresponding working processes according to their numbers, that is, the sub-dataset numbered 1 is assigned to the working process numbered 1, ..., and so on. Finally, each local model is assigned to its corresponding working process.

7. The distributed model training method based on computing power network according to claim 6 is characterized in that: Step (6-3) is specifically, First, each worker process sets a random seed using its unique number; Then, each worker process uses the DataLoader tool in PyTorch to load the first batch of data samples from the sub-dataset according to the pre-set batch size, and performs forward propagation on the loaded first batch of data samples through the local model to obtain the output of the local model; Subsequently, each worker process calculates the loss function value based on the obtained local model output and the target output, and then performs backpropagation on the loss function value to obtain the gradient of the local model corresponding to the worker process. These gradients will be used to guide the update of local parameters; Then, the gradient of the local model corresponding to the working process is used to update the local parameters corresponding to the working process; Finally, the above process is repeated for the remaining batches of data samples in the sub-dataset until all data samples in the sub-dataset are processed, thereby obtaining the local parameters of the current cycle of the work process.

8. The distributed model training method based on computing power network according to claim 7 is characterized in that: Step (6-6) is specifically, First, before a new training cycle begins, each worker process will obtain the value of the global parameter updated by all worker processes in step (6-5) for each parameter name to update the value of the local parameter of the corresponding parameter name, so that each worker process obtains the new local parameter; Then, each working process repeats the training process from step (6-2) to step (6-5) above, and performs a new cycle of training on the updated local parameters and the corresponding sub-dataset; Until the training of the preset number of iterations is completed, the global model of the main process will be used as the distributed trained model.

9. A model distributed training system based on computing power network, characterized in that: include: The first module is used to obtain the model information and model files uploaded by the user; The second module is used to obtain the information of the data set; The third module is used to determine whether the model name in the model information obtained by the first module exists in a pre-established dictionary. If so, it obtains the dataset name list corresponding to the model name from the dictionary and transfers to the fourth module; otherwise, it transfers to the fifth module; The fourth module is used to determine whether the dataset name in the dataset information obtained in the second module exists in the dataset name list. If so, a distributed training service request is generated based on the model name and the dataset name corresponding to the dataset name list, and then the process is transferred to the fifth module. Otherwise, the process is transferred to the eighth module. The fifth module is used to create a main process, use the main process, and load the corresponding model and dataset from the pre-established model warehouse and dataset warehouse respectively according to the model name and dataset name in the distributed call service request; the fifth module is specifically: first parse the distributed call service request to obtain the model name and dataset name therein, and then use the importlib library in the python library to load the model file corresponding to the model name stored in the model warehouse; finally, use the datasets module in the torchvision library to load the dataset corresponding to the dataset name stored in the dataset warehouse. If the dataset is not stored in the dataset warehouse, use the datasets module to download the dataset corresponding to the dataset name from the dataset source provided by the open source deep learning framework pytorch to the dataset warehouse, and then load the dataset from the dataset warehouse; The sixth module is used to perform distributed training on the model loaded in the fifth module using the dataset loaded in the fifth module to obtain a distributed trained model; the sixth module includes the following submodules: The first submodule is used to create n working processes through the main process, randomly number the n created working processes, divide the data set loaded by the fifth module to obtain multiple sub-data sets, and save the model loaded by the fifth module globally and locally in sequence to obtain a global model corresponding to the main thread and n local models corresponding to the n working threads, where the value range of n is 1 to 8; the process of obtaining multiple sub-data sets in the first submodule is specifically as follows: first, obtain the size of the data set loaded by the fifth module, then divide the size of the data set by the total number of preset working processes to obtain the size of the data set that each working process should process, then, divide the data set loaded by the fifth module for the first time according to the size, and the small part obtained is the first sub-training set, and the large part is the remaining data set, then, divide the remaining data set according to the size to obtain the second sub-training set, ..., and so on, finally, obtain multiple divided sub-data sets; The second submodule is used to assign all sub-datasets, 1 global model and n local models obtained by the first submodule to corresponding work processes; The third submodule is used for each working process to train the model loaded by the fifth module according to its corresponding sub-dataset and local model to obtain the local parameters of the current cycle of the working process; The fourth submodule is used for each worker process to wait until all other worker processes have obtained their local parameters for the current cycle, and then send their local parameters for the current cycle to the global model; The fifth submodule is used for the global model to aggregate all local parameters after receiving the local parameters of the current cycle from all working processes, and use the aggregation results to update the weights of the global model to obtain the updated global parameters of all working processes; The sixth submodule is used for each working process to update the local parameters of the working process using the global parameters updated by all working processes obtained in the fifth submodule before training the local model, and repeat the above second to fifth submodules until the preset number of iterative training times is reached, thereby obtaining a distributed trained model; The seventh module is used to save the distributed training model obtained by the sixth module to the distributed training result warehouse through the main process, and determine whether the distributed training result can be obtained. If so, the process ends, otherwise it goes to the eighth module; The eighth module is used to retrieve the information of the data set and return to the third module.

Images