Model training method and device

Abstract

The invention discloses a model training method and device, and relates to the technical field of cloud computing. The model training method comprises the steps that in the process that a first working node and at least one second working node cooperatively train a model, a check point file stored by the first working node is only part of data used by a complete model, and the check point file is stored by the at least one second working node; the check point files respectively stored by the first working node and the at least one second working node are integrated to obtain complete data utilized by the model. And if the first working node loses data in the process of training the model, the first working node obtains part or all of the check point file which is stored in the second working node and contains the B sub-data. And furthermore, the first working node only stores partial data in the data utilized by the complete model, so that the stored data volume is reduced, the time consumed by storage is reduced, and the efficiency of training the model is improved.

Claims

1. A model training method, characterized in that: The method is applied to a first working node, where the first working node is deployed on a cloud computing platform. The cloud computing platform also has at least one second working node deployed thereon. The first working node and the at least one second working node each use different training data to collaboratively train a model or a portion of a network layer in the model. The method includes: The first working node receives a storage request; the storage request carries: a first field used for splitting data used by the model or a portion of the network layer in the model, where the data used by the model or the portion of the network layer in the model includes parameters and / or training data of the model or the portion of the network layer in the model; The first working node splits the data required for the model or a portion of the network layer in the model on the first working node into N sub-data according to the first field in the storage request, where N is greater than or equal to 2; The first working node stores A sub-data among the N sub-data as a checkpoint file; 1≤A≤N, and the checkpoint file stored by the at least one second working node includes B sub-data among the N sub-data, where B is the difference between N and A; If the first working node loses data during the process of training the model, the first working node obtains part or all of the B sub-data.

2. The method according to claim 1, characterized in that The value of the first field is N, and the first working node and N-1 second working nodes each use different training data to collaboratively train the model or part of the network layer in the model.

3. The method according to claim 1 or 2, characterized in that The cloud computing platform further includes a cloud management platform, the cloud management platform being used to manage the first working node and the second working node. The cloud management platform provides a storage request configuration interface, the storage request configuration interface being used to obtain configuration information for model training storage data input by a user, the configuration information being used to determine whether to split the data required for use by the model on the first working node into N sub-data. The first field in the storage request is determined based on the configuration information. The first working node receiving the storage request includes: The first working node receives the storage request from the cloud management platform.

4. The method according to any one of claims 1 to 3, characterized in that If the data used by the model includes the training data, the checkpoint file in the first working node includes: first sub-data and compressed second sub-data, the first sub-data being the A sub-data, the second sub-data being training data used by at least one of the second working nodes in a process of training the model, and the B sub-data including the second sub-data; If the first working node loses data during the training of the model, the first working node obtains part or all of the B sub-data, including: If the first working node loses data during the training of the model, the first working node obtains part of the B sub-data.

5. The method according to any one of claims 1 to 4, characterized in that If the data used by the model includes the parameters, the checkpoint file in the first working node includes: the A sub-data, 1≤A <N; If the first working node loses data during the training of the model, the first working node obtains part or all of the B sub-data, including: If the first working node loses data during the training of the model, the first working node obtains the B sub-data.

6. The method according to any one of claims 1 to 5, characterized in that The first working node stores A sub-data among the N sub-data as checkpoint files, including: The first working node stores the checkpoint file containing the A sub-data locally on the first working node; If the first working node loses data during the training of the model, the first working node obtains part or all of the B sub-data, including: If the first working node loses data during the training of the model, the first working node obtains the B sub-data from the at least one second working node.

7. The method according to any one of claims 1 to 6, characterized in that The cloud computing platform further includes a storage node, and the method further includes: The first working node sends the checkpoint file containing the A sub-data to the storage node; If the first working node loses data during the training of the model, the first working node obtains part or all of the B sub-data, including: If the first working node loses data during the training of the model, the first working node obtains B sub-data from the at least one second working node and obtains the A sub-data from the storage node.

8. The method according to any one of claims 1 to 7, characterized in that The storage request also carries: a status field instructing the first working node to store a checkpoint file, the status field is used to indicate a target network layer in the model, and the first working node is used to train the target network layer.

9. The method according to any one of claims 1 to 8, characterized in that The storage request further carries: a second field for indicating a storage frequency, wherein the storage frequency includes a time frequency or a training round frequency; The first working node stores A sub-data among the N sub-data as checkpoint files, including: If the first working node meets the time frequency or the training round frequency during the training of the model or part of the network layer in the model, A sub-data of the N sub-data are stored as checkpoint files.

10. The method according to any one of claims 1 to 9, characterized in that The data loss is used to indicate one or more of the following: failure of the first working node, interruption of the first working node.

11. A model training device, characterized in that: The device is applied to a first working node, the first working node is deployed on a cloud computing platform, and the cloud computing platform is further deployed with at least one second working node. The first working node and the at least one second working node each use different training data to collaboratively train a model or a portion of a network layer in the model. The device includes: A receiving module, configured to receive a storage request; the storage request carries: a first field used for splitting data utilized by the model or a portion of the network layer in the model, wherein the data utilized by the model or the portion of the network layer in the model includes parameters and / or training data of the model or the portion of the network layer in the model; A splitting module, configured to split data required by the model or some network layers in the model into N sub-data according to the first field in the storage request, where N≥2; A storage module, configured to store A sub-data among the N sub-data as checkpoint files; 1≤A≤N, and the checkpoint files stored by at least one second working node include B sub-data among the N sub-data, and B is the difference between N and A; An obtaining module, configured to obtain some or all of the B sub-data if data is lost during the training of the model.

12. The device according to claim 11, characterized in that The value of the first field is N, and the first working node and N-1 second working nodes respectively use different training data to co-train the model or some network layers in the model.

13. The device according to claim 11 or 12, characterized in that The cloud computing platform further includes a cloud management platform, and the cloud management platform is configured to manage the first working node and the second working nodes. The cloud management platform provides a storage request configuration interface, and the storage request configuration interface is configured to obtain configuration information input by a user for storing data for model training. The configuration information is used to determine to split data required by the model on the first working node into N sub-data, and the first field in the storage request is determined according to the configuration information. The receiving module is specifically configured to receive the storage request from the cloud management platform.

14. The device according to any one of claims 11 to 13, characterized in that If the data used by the model includes the training data, the checkpoint file in the first working node includes: a first sub-data and a compressed second sub-data. The first sub-data is the A sub-data, and the second sub-data is training data used by at least one second working node during the training of the model. The B sub-data includes the second sub-data. The obtaining module is specifically configured to obtain some of the B sub-data if data is lost during the training of the model.

15. The device according to any one of claims 11 to 14, characterized in that If the data used by the model includes the parameters, the checkpoint file in the first working node includes: the A sub-data, 1≤A<N. The obtaining module is specifically configured to obtain the B sub-data if data is lost during the training of the model.

16. The device according to any one of claims 11 to 15, characterized in that The storage module is specifically configured to store the checkpoint file including the A sub-data locally on the first working node. The obtaining module is specifically configured to obtain some or all of the B sub-data from at least one second working node if data is lost during the training of the model.

17. The device according to any one of claims 11 to 16, characterized in that The cloud computing platform further includes a storage node, and the device further includes: a sending module; The sending module is configured to send the checkpoint file including the A sub-data to the storage node; The obtaining module is specifically configured to obtain B sub-data from at least one second working node and obtain the A sub-data from the storage node if data is lost during the training of the model.

18. A computer cluster, characterized in that: It includes at least one computing device, and each computing device includes a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in a memory of the at least one computing device, so that the computer cluster executes the method according to any one of claims 1 to 10.

19. A computer-readable storage medium, characterized in that The storage medium stores a computer program or instruction, and when the computer program or instruction is executed by a processing device, the method according to any one of claims 1 to 10 is implemented.

20. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed on a processing device, the method according to any one of claims 1 to 10 is implemented.

Images