Model distributed training method and system based on computing power network

Abstract

The invention discloses a model distributed training method based on a computing power network, and the method comprises the steps: obtaining model information and a model file uploaded by a user, obtaining the information of a data set, judging whether a model name in the obtained model information exists in a pre-established dictionary or not, if not, creating a host process, and if not, creating a database; respectively loading a corresponding model and a corresponding data set from a pre-established model warehouse and a pre-established data set warehouse according to the model name and the data set name in the distributed calling service request by utilizing the host process; and performing distributed training on the model by using the loaded data set to obtain a distributed trained model, and storing the distributed trained model in a distributed training result warehouse through the host process. The technical problem that the propagation range of a model built by a user of an existing distributed system is small can be solved, and the technical problem that computing resources are wasted due to the fact that distributed training is carried out on an unmatched model and a data set can be solved.

Description

Technical Field

[0001] The present invention belongs to the field of high performance computing technology, and more specifically, relates to a model distributed training method and system based on a computing power network. Background Art

[0002] In recent years, the research scope of machine learning has been continuously expanded, and its application in natural language processing, image recognition, speech recognition and other fields has become more and more extensive, directly affecting our daily lives. However, with the increase in model size and the amount of training data, single-machine model training can no longer meet the real-time requirements. In order to solve this problem, researchers began to introduce distributed systems into machine learning. Therefore, distributed training came into being.

[0003] The current mainstream distributed training systems are mainly pytorch and TensorFlow.

[0004] PyTorch adopts a dynamic computational graph mode and implements distributed training through the torch.distributed module. This module provides a variety of distributed training strategies, encapsulates the model through the DistributedDataParallel module, implements data parallelism, and updates model parameters through gradient synchronization.

[0005] TensorFlow uses a static computational graph mode and supports distributed training through the tf.distribute module. This module can be used for distributed training in different environments, allowing users to choose the appropriate configuration based on specific hardware and training requirements.

[0006] However, the above two distributed training systems have some problems that cannot be ignored:

[0007] First, the models built by users of the above two distributed training systems are not easily used by other users, which makes it difficult for these models to be widely used;

[0008] Second, in the above two distributed training systems, incompatible models and data sets may also start distributed training during the training process until errors occur during the training process, which will cause a waste of computing resources;

[0009] Third, the above two distributed training systems require a high level of professionalism from the users. For users who do not have a knowledge base of distributed training, additional learning costs are required, and the degree of automation is low;

[0010] Fourth, the above two distributed training systems are not convenient enough for the management of models and distributed training result files. Summary of the invention

[0011] In response to the above defects or improvement needs of the prior art, the present invention provides a model distributed training method and system based on a computing power network, which aims to solve the technical problem that the model built by users of the existing distributed system has a small propagation range; the technical problem that distributed training of mismatched models and data sets results in a waste of computing resources; the technical problem that the degree of automation is low, resulting in additional learning costs for researchers; and the technical problem that the management of models and distributed training result files cannot be effectively realized.

[0012] To achieve the above object, according to one aspect of the present invention, a model distributed training method based on a computing power network is provided, comprising the following steps:

[0013] (1) Obtain model information and model files uploaded by users;

[0014] (2) Obtaining information about the data set;

[0015] (3) Determine whether the model name in the model information obtained in step (1) exists in a pre-established dictionary. If so, obtain a list of data set names corresponding to the model name from the dictionary and proceed to step (4). Otherwise, proceed to step (5).

[0016] (4) Determine whether the dataset name in the dataset information obtained in step (2) exists in the dataset name list. If so, generate a distributed training service request according to the model name and the dataset name corresponding to the dataset name list, and then proceed to step (5); otherwise, proceed to step (8);

[0017] (5) Creating a main process, using the main process and loading the corresponding model and dataset from the pre-established model repository and dataset repository respectively according to the model name and dataset name in the distributed call service request;

[0018] (6) Use the data set loaded in step (5) to perform distributed training on the model loaded in step (5) to obtain a distributed trained model.

[0019] (7) The distributed training model obtained in step (6) is saved in the distributed training result warehouse through the main process, and it is determined whether the distributed training result can be obtained. If so, the process ends, otherwise it proceeds to step (8).

[0020] (8) Re-obtain the data set information and return to step (3).

[0021] Preferably, the model includes a neural network model built by the user, or a deep neural network model, etc.;

[0022] Deep neural network models include convolutional neural networks (CNN), recurrent neural networks (RNN), and so on.

[0023] Model information includes model name, model type, and model purpose.

[0024] Preferably, the information of the data set includes the name of the data set, a brief introduction to the data set, the type of the data set, the size of the training set and the test set in the data set, and the classification category of the data set.

[0025] In the dictionary, the model name is used as the dictionary key, and the dataset name list consisting of all dataset names supported by the model name is used as the corresponding value.

[0026] Preferably, in step (4), the process of generating a distributed training service request according to the model name and the dataset name corresponding to the dataset name list is to use the requests command in Python to package the model name and the dataset name into a distributed training service request.

[0027] Step (5) is as follows: first, parse the distributed call service request to obtain the model name and dataset name therein; then, use the importlib library in the python library to load the model file corresponding to the model name stored in the model warehouse; finally, use the datasets module in the torchvision library to load the dataset corresponding to the dataset name stored in the dataset warehouse; if the dataset is not stored in the dataset warehouse, use the datasets module to download the dataset corresponding to the dataset name from the dataset source provided by the open source deep learning framework pytorch to the dataset warehouse, and then load the dataset from the dataset warehouse.

[0028] Preferably, step (6) comprises the following sub-steps:

[0029] (6-1) Create n worker processes through the main process, randomly number the created n worker processes, divide the data set loaded in step (5) to obtain multiple sub-data sets, and save the model loaded in step (5) globally and locally in turn to obtain a global model corresponding to the main thread and n local models corresponding to the n worker threads, where the value range of n is 1 to 8.

[0030] (6-2) All sub-datasets, 1 global model and n local models obtained in step (6-1) are assigned to corresponding working processes.

[0031] (6-3) Each working process trains the model loaded in step (5) according to its corresponding sub-dataset and local model to obtain the local parameters of the current cycle of the working process.

[0032] (6-4) Each worker process waits for all other worker processes to obtain the local parameters of its current cycle, and then sends its local parameters of the current cycle to the global model;

[0033] (6-5) After receiving the local parameters of the current cycle from all working processes, the global model aggregates all local parameters and uses the aggregation results to update the weights of the global model to obtain the updated global parameters of all working processes.

[0034] (6-6) Before each working process needs to train the local model, it uses the global parameters updated by all working processes obtained in step (6-5) to update the local parameters of the working process, and repeats the above steps (6-2) to step (6-5) until the preset number of iterative training times is reached, thereby obtaining a distributed trained model.

[0035] Preferably, the process of obtaining multiple sub-datasets in step (6-1) is specifically as follows: first, obtain the size of the data set loaded in step (5), then divide the size of the data set by the total number of preset working processes to obtain the size of the data set that each working process should process, then divide the data set loaded in step (5) for the first time according to the size, and the small part obtained is the first sub-training set (numbered 1), and the large part is the remaining data set, then divide the remaining data set according to the size to obtain the second sub-training set (numbered 2), ..., and so on, and finally obtain multiple divided sub-datasets;

[0036] Step (6-1) is to save the model loaded in step (5) as a global model, which is used to store global parameters, including the number of layers, weights, and biases of the neural network. All working processes jointly maintain the global parameters;

[0037] Step (6-1) saves a model loaded in step (5) as a local model for each working process, which is used to store local parameters, including the number of layers, weights, and biases of the neural network.

[0038] Preferably, step (6-2) is specifically as follows: first, the global model is assigned to the main process, and all sub-datasets are assigned to corresponding working processes according to their numbers, that is, the sub-dataset numbered 1 is assigned to the working process numbered 1, ..., and so on. Finally, each local model is assigned to its corresponding working process.

[0039] Preferably, step (6-3) is specifically,

[0040] First, each worker process sets a random seed using its unique number;

[0041] Then, each worker process uses the DataLoader tool in pytorch to load the first batch of data samples from the sub-dataset according to the preset batch size, and forward propagates the first batch of loaded data samples through the local model to obtain the output of the local model;

[0042] Subsequently, each working process calculates the loss function value based on the output of the local model and the target output, and then back-propagates the loss function value to obtain the gradient of the local model corresponding to the working process. These gradients will be used to guide the update of local parameters.

[0043] Thereafter, the gradient of the local model corresponding to the working process is used to update the local parameters corresponding to the working process.

[0044] Finally, the above process is repeated for the remaining batches of data samples in the sub-dataset until all data samples in the sub-dataset have been processed, thereby obtaining the local parameters of the current cycle of the work process.

[0045] Preferably, step (6-6) is specifically,

[0046] First, before a new training cycle starts, each worker process will obtain the value of the global parameter updated by all worker processes in step (6-5) for each parameter name to update the value of the local parameter of the corresponding parameter name, so that each worker process obtains the new local parameter;

[0047] Then, each working process repeats the training process from step (6-2) to step (6-5) above, and performs a new cycle of training on the updated local parameters and the corresponding sub-dataset. Until the training of the preset number of iterative training is completed, the global model of the master process will be used as the distributed trained model.

[0048] According to another aspect of the present invention, a model distributed training system based on a computing power network is provided, comprising:

[0049] The first module is used to obtain the model information and model files uploaded by the user;

[0050] The second module is used to obtain the information of the data set;

[0051] The third module is used to determine whether the model name in the model information obtained by the first module exists in the pre-established dictionary. If yes, a list of data set names corresponding to the model name is obtained from the dictionary and transferred to the fourth module; otherwise, the list is transferred to the fifth module.

[0052] The fourth module is used to determine whether the data set name in the data set information obtained by the second module exists in the data set name list. If yes, a distributed training service request is generated according to the model name and the data set name corresponding to the data set name list, and then the fifth module is entered; otherwise, the eighth module is entered;

[0053] The fifth module is used to create a main process, and use the main process to load the corresponding model and dataset from the pre-established model warehouse and dataset warehouse respectively according to the model name and dataset name in the distributed call service request;

[0054] The sixth module is used to perform distributed training on the model loaded by the fifth module using the data set loaded by the fifth module to obtain a distributed trained model.

[0055] The seventh module is used to save the distributed training model obtained by the sixth module into the distributed training result warehouse through the main process, and determine whether the distributed training result can be obtained. If so, the process ends, otherwise it goes to the eighth module.

[0056] The eighth module is used to retrieve the information of the data set and return to the third module.

[0057] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0058] (1) Due to the adoption of step (1), the present invention stores the models uploaded by users in the model warehouse, which facilitates the storage and management of the models. Therefore, the models can also be used by other users, which can enhance the communication and dissemination between the models built by users themselves to a greater extent. Therefore, it can solve the technical problem that the models built by users themselves are difficult to be widely used.

[0059] (2) The present invention adopts steps (3) to (4), which uses a pre-established dictionary to check the model and data set used to generate the distributed service request, thereby reducing the distributed training that may be performed on the mismatched model and data set before the distributed training begins, thereby solving the technical problem of wasting computing resources;

[0060] (3) Due to the adoption of step (5), the present invention realizes dynamic loading of the model through the importlib library in the python library, and automatically loads the model from the model warehouse for training during training, thereby solving the technical problem that the distributed training system requires a high level of professionalism from the user;

[0061] (4) The present invention adopts step (7), which uses a distributed training result warehouse to separately save the distributed training result files, making it convenient to download and manage the distributed training result files, thereby solving the technical problem of difficult management of models and distributed training result files;

[0062] (5) The present invention also has strong scalability. Its distributed training algorithm is designed and implemented independently, so more distributed training algorithms can be expanded for different distributed training tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a flow chart of the model distributed training method based on computing power network of the present invention. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0065] The present invention provides a model distributed training method based on a computing power network, the purpose of which is to realize the management of user models, so that users can implement simple and convenient distributed model training tasks in a computing power network in a more automated manner without the knowledge base of distributed training.

[0066] like Figure 1 As shown, the present invention provides a model distributed training method based on a computing power network, comprising the following steps:

[0067] (1) Obtain model information and model files uploaded by users;

[0068] After obtaining the model file and model information uploaded by the user, this step names the model file to be consistent with the model name in the model information, and then stores the model file in the model warehouse for subsequent model loading in distributed services. The subsequent model loading process will use the importlib library in the python library to load the model file to complete the model initialization.

[0069] The model of the present invention includes a neural network model built by the user, or a deep neural network model such as a convolutional neural network (CNN) or a recurrent neural network (RNN).

[0070] The model information in the present invention includes model name, model type, model purpose, etc.

[0071] The advantage of this step (1) is that the model uploaded by the user will be stored and managed by the platform, so it can also be used by other users, which can enhance the communication and dissemination between models to a greater extent.

[0072] (2) Obtaining information about the data set;

[0073] Specifically, the information of the dataset includes the dataset name, dataset introduction, dataset type, the size of the training set and test set in the dataset, the classification category of the dataset, and other information.

[0074] (3) Determine whether the model name in the model information obtained in step (1) exists in a pre-established dictionary. If so, obtain a list of data set names corresponding to the model name from the dictionary and proceed to step (4). Otherwise, proceed to step (5).

[0075] Specifically, the dictionary uses the model name (such as CNN, RNN, etc.) as the dictionary key, and the dataset name list consisting of all dataset names supported by the model name as the corresponding value.

[0076] (4) Determine whether the dataset name in the dataset information obtained in step (2) exists in the dataset name list. If so, generate a distributed training service request according to the model name and the dataset name corresponding to the dataset name list, and then proceed to step (5); otherwise, proceed to step (8);

[0077] Specifically, in this step, the process of generating a distributed training service request according to the model name and the dataset name corresponding to the dataset name list is to use the requests command in Python to package the model name and the dataset name into a distributed training service request.

[0078] The advantages of the above steps (3) to (4) are that the model and data set used to generate the distributed service request are checked using a pre-established dictionary, thereby reducing the distributed training requests for mismatched models and data sets and avoiding the waste of computing resources.

[0079] (5) Creating a main process, using the main process and loading the corresponding model and dataset from the pre-established model repository and dataset repository respectively according to the model name and dataset name in the distributed call service request;

[0080] Specifically, this step first parses the distributed call service request to obtain the model name and dataset name, and then uses the importlib library in the python library to load the model file corresponding to the model name stored in the model warehouse; finally, uses the datasets module in the torchvision library to load the dataset corresponding to the dataset name stored in the dataset warehouse. If the dataset is not stored in the dataset warehouse, the datasets module is used to download the dataset corresponding to the dataset name from the dataset source provided by the open source deep learning framework pytorch to the dataset warehouse, and then load the dataset from the dataset warehouse.

[0081] The torchvision library is a sub-library in the open source deep learning framework pytorch, which provides commonly used datasets, pre-trained models, and image conversion and processing tools; the datasets module has built-in MNIST, CIFAR10, ImageNet, CoCo and other datasets.

[0082] The importlib library is a module in Python that provides tools for dynamically loading modules. It is suitable for importing modules at runtime or operating the import process programmatically, so that the program can load modules on demand at runtime instead of statically importing all modules at the beginning of the program.

[0083] The advantage of this step (5) is that it realizes dynamic loading of the model. When training is in progress, the model is automatically loaded from the model warehouse for training, and the model distributed training service is completed in a more automated form to solve the problem of high professional requirements for distributed training.

[0084] (6) Use the data set loaded in step (5) to perform distributed training on the model loaded in step (5) to obtain a distributed trained model.

[0085] Specifically, this step includes the following sub-steps:

[0086] (6-1) Create n working processes through the main process, randomly number the created n working processes, divide the data set loaded in step (5) to obtain multiple sub-data sets, and save the model loaded in step (5) globally and locally in turn to obtain a global model corresponding to the main thread and n local models corresponding to the n working threads (where n ranges from 1 to 8, preferably 4).

[0087] The process of obtaining multiple sub-datasets in this step is specifically as follows: first, obtain the size of the data set loaded in step (5), then divide the size of the data set by the total number of preset working processes (which is equal to the total number of local models n) to obtain the size of the data set that each working process should process, then divide the data set loaded in step (5) for the first time according to the size, and the small part obtained is the first sub-training set (numbered 1), and the large part is the remaining data set, then divide the remaining data set according to the size to obtain the second sub-training set (numbered 2), ..., and so on, and finally obtain multiple divided sub-datasets.

[0088] In addition, this step is to save the model loaded in step (5) as a global model for storing global parameters (including the number of layers, weights, biases, etc. of the neural network). All working processes jointly maintain the global parameters, which are accessed and updated by all working processes during the training process to ensure the consistency and accuracy of the model. In addition, this step saves a model loaded in step (5) as a local model for each working process for storing local parameters (including the number of layers, weights, biases, etc. of the neural network). The global parameters and local parameters are saved by the model loaded in step (5). Therefore, the global model and the local model contain the same parameter name, and the value of the local parameter of each working process will be randomly initialized in step (6-3). The value of the global parameter will change by averaging the values ​​of the local parameters of each working process.

[0089] (6-2) All sub-datasets, 1 global model and n local models obtained in step (6-1) are assigned to corresponding working processes.

[0090] Specifically, this step is as follows: first, the global model is assigned to the main process, all sub-datasets are assigned to the corresponding working processes according to their numbers (that is, the sub-dataset numbered 1 is assigned to the working process numbered 1, ..., and so on), and finally, each local model is assigned to its corresponding working process.

[0091] (6-3) Each working process trains the model loaded in step (5) according to its corresponding sub-dataset and local model to obtain the local parameters of the current cycle of the working process.

[0092] Specifically, in this step, first, each worker process uses its unique number to set a random seed; the purpose is to allow each worker process to randomly initialize the value of local parameters to avoid synchronization issues of local models. This helps to reduce the randomness differences in initialization between different worker processes in distributed training;

[0093] Then, each worker process uses the DataLoader tool in pytorch to load the first batch of data samples from the sub-dataset according to the preset batch size (i.e., Batch size) (if Batch size is set to 64, 64 data samples in the sub-dataset are loaded in each batch), and forward propagates the first batch of data samples through the local model (this process involves passing the input data through various layers of the local model, such as fully connected layers, convolutional layers, activation layers, etc.) to obtain the output of the local model (the output can be in different forms, such as the house price value in the house price prediction task);

[0094] Subsequently, each working process calculates the loss function value based on the output of the local model and the target output (such as the actual price of the house in the house price prediction task), and then backpropagates the loss function value to obtain the gradient of the local model corresponding to the working process (these gradients will be used to guide the update of local parameters).

[0095] Thereafter, the gradient of the local model corresponding to the working process is used to update the local parameters corresponding to the working process.

[0096] Finally, the above process is repeated for the remaining batches of data samples in the sub-dataset until all data samples in the sub-dataset have been processed, thereby obtaining the local parameters of the current cycle of the work process.

[0097] In addition, the training speed of each worker process may be different. Therefore, after the training cycle is over, the worker process that completes the local model training in advance will call the synchronization mechanism to make the worker process enter a blocked state, waiting for all other worker processes to complete the local model training.

[0098] The advantage of this sub-step (6-3) is that it uses a random seed to randomly initialize the local parameters of each working process, avoiding the synchronization problem of the local model and reducing the random differences in initialization between different working processes in distributed training.

[0099] (6-4) Each worker process waits for all other worker processes to obtain the local parameters of its current cycle, and then sends its local parameters of the current cycle to the global model;

[0100] (6-5) After receiving the local parameters of the current cycle from all working processes, the global model aggregates all local parameters and uses the aggregation results to update the weights of the global model to obtain the updated global parameters of all working processes.

[0101] Specifically, after receiving the local parameters of all working processes in the current cycle, the global model will traverse all local models in turn according to each parameter name in the local parameters of the current cycle, because the global model and the local model have the same parameter name, to obtain the parameter value corresponding to the parameter name, and take the average of all parameter values ​​as the parameter value corresponding to the parameter name in the new global model, and save it to the global model. After all parameter names in the local parameters of the current cycle are updated, the updated global parameters of all working processes are obtained. This is to ensure that the global model can obtain the latest local parameter information, thereby ensuring the overall convergence and consistency of the global model.

[0102] (6-6) Before each work process needs to train the local model, it uses the global parameters updated by all work processes obtained in step (6-5) to update the local parameters of the work process, and repeats the above steps (6-2) to (6-5) until the preset number of iterative training times is reached (the preset number of iterative training times ranges from 3 to 10, preferably 5), thereby obtaining a distributed trained model.

[0103] Specifically, first, before the start of a new training cycle, since the global model and the local model have the same parameter name, each work process will obtain the value of the global parameter updated by all work processes in step (6-5) for each parameter name to update the value of the local parameter of the corresponding parameter name, so that each work process obtains the new local parameter.

[0104] Then, each working process repeats the training process from step (6-2) to step (6-5) above, and performs a new cycle of training on the updated local parameters and the corresponding sub-dataset. Until the training of the preset number of iterative training is completed, the global model of the master process will be used as the distributed trained model.

[0105] (7) The distributed training model obtained in step (6) is saved in the distributed training result warehouse through the main process, and it is determined whether the distributed training result can be obtained. If so, the process ends, otherwise it proceeds to step (8).

[0106] This step stores the distributed training model obtained in step (6) into the distributed training result warehouse (the warehouse is used to store all the result files after distributed training), and determines whether the distributed training result can be obtained. If so, the process ends, otherwise it goes to step (8).

[0107] The advantage of this step (7) is that the distributed training results are saved separately, which facilitates the downloading and management of the distributed training result files.

[0108] (8) Re-acquire the information of the data set and return to step (3);

[0109] Through the above steps, a distributed training platform is finally realized, which can realize the acquisition of model information and model files, the acquisition of data sets, and the selection of appropriate models and data sets to call distributed training services, so that users can realize the distributed training tasks of models more automatically without the knowledge of distributed training.

[0110] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A distributed model training method based on a computing power network, characterized in that: The following steps are involved: (1) Obtain model information and model files uploaded by users; (2) Obtaining information about the data set; (3) Determine whether the model name in the model information obtained in step (1) exists in a pre-established dictionary. If so, obtain a list of data set names corresponding to the model name from the dictionary and proceed to step (4). Otherwise, proceed to step (5). (4) Determine whether the dataset name in the dataset information obtained in step (2) exists in the dataset name list. If so, generate a distributed training service request according to the model name and the dataset name corresponding to the dataset name list, and then proceed to step (5); otherwise, proceed to step (8); (5) Creating a main process, using the main process and loading the corresponding model and dataset from the pre-established model repository and dataset repository respectively according to the model name and dataset name in the distributed call service request; (6) Use the data set loaded in step (5) to perform distributed training on the model loaded in step (5) to obtain a distributed trained model. (7) The distributed training model obtained in step (6) is saved in the distributed training result warehouse through the main process, and it is determined whether the distributed training result can be obtained. If so, the process ends, otherwise it proceeds to step (8). (8) Re-obtain the data set information and return to step (3).

2. The distributed model training method based on computing power network according to claim 1 is characterized in that: Models include neural network models built by users themselves, or deep neural network models, etc. Deep neural network models include convolutional neural networks (CNN), recurrent neural networks (RNN), and so on. Model information includes model name, model type, and model purpose.

3. The model distributed training method based on computing power network according to claim 1 or 2, characterized in that: The information of the dataset includes the dataset name, dataset introduction, dataset type, the size of the training set and test set in the dataset, and the dataset classification category. In the dictionary, the model name is used as the dictionary key, and the dataset name list consisting of all dataset names supported by the model name is used as the corresponding value.

4. The distributed model training method based on computing power network according to any one of claims 1 to 3, characterized in that: In step (4), the process of generating a distributed training service request based on the model name and the dataset name corresponding to the dataset name list is to use the requests command in Python to package the model name and the dataset name into a distributed training service request. Step (5) is as follows: first, parse the distributed call service request to obtain the model name and dataset name therein; then, use the importlib library in the python library to load the model file corresponding to the model name stored in the model warehouse; finally, use the datasets module in the torchvision library to load the dataset corresponding to the dataset name stored in the dataset warehouse; if the dataset is not stored in the dataset warehouse, use the datasets module to download the dataset corresponding to the dataset name from the dataset source provided by the open source deep learning framework pytorch to the dataset warehouse, and then load the dataset from the dataset warehouse.

5. The distributed model training method based on computing power network according to claim 4 is characterized in that: Step (6) includes the following sub-steps: (6-1) Create n worker processes through the main process, randomly number the created n worker processes, divide the data set loaded in step (5) to obtain multiple sub-data sets, and save the model loaded in step (5) globally and locally in turn to obtain a global model corresponding to the main thread and n local models corresponding to the n worker threads, where the value range of n is 1 to 8. (6-2) All sub-datasets, 1 global model and n local models obtained in step (6-1) are assigned to corresponding working processes. (6-3) Each working process trains the model loaded in step (5) according to its corresponding sub-dataset and local model to obtain the local parameters of the current cycle of the working process. (6-4) Each worker process waits for all other worker processes to obtain the local parameters of its current cycle, and then sends its local parameters of the current cycle to the global model; (6-5) After receiving the local parameters of the current cycle from all working processes, the global model aggregates all local parameters and uses the aggregation results to update the weights of the global model to obtain the updated global parameters of all working processes. (6-6) Before each working process needs to train the local model, it uses the global parameters updated by all working processes obtained in step (6-5) to update the local parameters of the working process, and repeats the above steps (6-2) to step (6-5) until the preset number of iterative training times is reached, thereby obtaining a distributed trained model.

6. The distributed model training method based on computing power network according to claim 5 is characterized in that: The process of obtaining multiple sub-datasets in step (6-1) is specifically as follows: first, obtain the size of the data set loaded in step (5), then divide the size of the data set by the total number of preset working processes to obtain the size of the data set that each working process should process, then divide the data set loaded in step (5) for the first time according to the size, and the small part obtained is the first sub-training set (numbered 1), and the large part is the remaining data set, then divide the remaining data set according to the size to obtain the second sub-training set (numbered 2), ..., and so on, and finally obtain multiple divided sub-datasets; Step (6-1) is to save the model loaded in step (5) as a global model, which is used to store global parameters, including the number of layers, weights, and biases of the neural network. All working processes jointly maintain the global parameters; Step (6-1) saves a model loaded in step (5) as a local model for each working process, which is used to store local parameters, including the number of layers, weights, and biases of the neural network.

7. The distributed model training method based on computing power network according to claim 6 is characterized in that: Step (6-2) is specifically as follows: first, the global model is assigned to the main process, and all sub-datasets are assigned to corresponding working processes according to their numbers, that is, the sub-dataset numbered 1 is assigned to the working process numbered 1, ..., and so on. Finally, each local model is assigned to its corresponding working process.

8. The distributed model training method based on computing power network according to claim 7 is characterized in that: Step (6-3) is specifically: First, each worker process sets a random seed using its unique number; Then, each worker process uses the DataLoader tool in pytorch to load the first batch of data samples from the sub-dataset according to the preset batch size, and forward propagates the first batch of loaded data samples through the local model to obtain the output of the local model; Subsequently, each working process calculates the loss function value based on the output of the local model and the target output, and then back-propagates the loss function value to obtain the gradient of the local model corresponding to the working process. These gradients will be used to guide the update of local parameters. Thereafter, the gradient of the local model corresponding to the working process is used to update the local parameters corresponding to the working process. Finally, the above process is repeated for the remaining batches of data samples in the sub-dataset until all data samples in the sub-dataset have been processed, thereby obtaining the local parameters of the current cycle of the work process.

9. The distributed model training method based on computing power network according to claim 8, characterized in that: Step (6-6) is specifically, First, before a new training cycle starts, each worker process will obtain the value of the global parameter updated by all worker processes in step (6-5) for each parameter name to update the value of the local parameter of the corresponding parameter name, so that each worker process obtains the new local parameter; Then, each working process repeats the training process from step (6-2) to step (6-5) above, and performs a new cycle of training on the updated local parameters and the corresponding sub-dataset. Until the training of the preset number of iterative training is completed, the global model of the master process will be used as the distributed trained model.

10. A model distributed training system based on a computing power network, characterized in that: include: The first module is used to obtain the model information and model files uploaded by the user; The second module is used to obtain the information of the data set; The third module is used to determine whether the model name in the model information obtained by the first module exists in the pre-established dictionary. If yes, a list of data set names corresponding to the model name is obtained from the dictionary and transferred to the fourth module; otherwise, the list is transferred to the fifth module. The fourth module is used to determine whether the data set name in the data set information obtained by the second module exists in the data set name list. If yes, a distributed training service request is generated according to the model name and the data set name corresponding to the data set name list, and then the fifth module is entered; otherwise, the eighth module is entered; The fifth module is used to create a main process, and use the main process to load the corresponding model and dataset from the pre-established model warehouse and dataset warehouse respectively according to the model name and dataset name in the distributed call service request; The sixth module is used to perform distributed training on the model loaded by the fifth module using the data set loaded by the fifth module to obtain a distributed trained model. The seventh module is used to save the distributed training model obtained by the sixth module into the distributed training result warehouse through the main process, and determine whether the distributed training result can be obtained. If so, the process ends, otherwise it goes to the eighth module. The eighth module is used to retrieve the information of the data set and return to the third module.

Images