Data recovery method and system for distributed deep learning and readable storage medium

Abstract

The application provides a data recovery method and system for distributed deep learning and a readable storage medium, the method comprising the following steps: S1, collecting parameter information of a neural network model for training, and obtaining global information of data distribution to be recovered by using an erasure code; S2, using artificial intelligence to calculate a priority of recovery of training data of each category, and sorting strips where the data to be recovered is located; and S3, combining the calculated data recovery priority and the global information of the data distribution to be recovered, and constructing a parallel recovery scheme. The application establishes a weighted priority for the data to be recovered by using artificial intelligence, and combines the global information of the data block distribution obtained by using the erasure code to construct a parallel recovery scheme, so that the ability of obtaining the global information of the data by using the erasure code and the ability of recovering the partially lost data by using the artificial intelligence are combined, resource consumption is greatly reduced under acceptable training accuracy, and data reconstruction of the distributed deep learning in the cloud storage system is accelerated.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed deep learning, and particularly relates to a data recovery method and system for distributed deep learning and a readable storage medium. BACKGROUND

[0002] Distributed deep learning is a typical machine learning method running in a distributed environment such as a cloud computing system. The corresponding training, verification and test data sets are generally very large. Since the disk failure rate of a cloud storage system is very high, a key problem of distributed deep learning is how to effectively tolerate disk failures in the training program. These failures will cause a large amount of data loss, reducing the accuracy of training and slowing down the training process.

[0003] Currently, there are two ways for data recovery and reconstruction. One is an EC (error correction code) based recovery method, which is characterized by random recovery and has no awareness of data. However, the entire lost data needs to be reconstructed, which consumes a large amount of computing, network and I / O resources. Although a parallel recovery method can be applied in this environment, the upper machine learning application is easily affected due to long data recovery waiting time and reduced computing resources.

[0004] The other is an AI (artificial intelligence) based recovery method. A generative adversarial network is often used to approximately recover lost image data. One of the shortcomings of this recovery scheme is that the generative model must be trained before the training set is lost to generate new data with the same statistics as the training set. Another shortcoming is the high cost of resource consumption, which requires a long recovery time. SUMMARY

[0005] The present application aims to provide a data recovery method and system for distributed deep learning, which can reduce resource consumption and speed up data reconstruction of distributed deep learning in a cloud storage system under acceptable training accuracy.

[0006] To achieve the above-mentioned purpose, the present application provides a data recovery method for distributed deep learning, comprising the following steps:

[0007] S1, collecting parameter information of a neural network model for training, and obtaining global information of data distribution to be recovered by using an error correction code;

[0008] S2, using artificial intelligence to calculate the priority of recovery of each type of training data, and sorting the bands where the data to be recovered is located;

[0009] S3, combining the calculated data recovery priority and the global information of the data distribution to be recovered to construct a parallel recovery scheme.

[0010] Alternatively, methods for using artificial intelligence to calculate the priority of recovering training data of different categories include:

[0011] in, α represents the weight update frequency, which is the number of times the weight parameters corresponding to different classes of training data are updated within a unit iteration period during model training. α and β are two hyperparameters. Indicates stripe The Local imbalance of p-class training data at nodes Indicates stripe The Imbalance of p-class training data across all nodes. This indicates the frequency of weight updates for training data of category p.

[0012] Optionally, the weight update frequency for each category of training data is derived from the parameter information.

[0013] Optionally, the weight update frequency is proportional to the number of samples in the training data for each category.

[0014] Optionally, the local imbalance and the global imbalance are derived from the global information obtained by the erasure coding.

[0015] Optionally, the recovery scheme includes strip selection and parallel strip management.

[0016] Optionally, when performing stripe selection and parallel stripe management, the remaining stripes that can be recovered in parallel with the current stripe are explored according to the priority of stripe recovery, and then the data that can be recovered in parallel with the current stripe is selected.

[0017] Based on the same inventive concept, this application also proposes a distributed deep learning data recovery system, comprising:

[0018] The parameter collection module is configured to collect parameter information for training the neural network model and use erasure coding to obtain global information on the distribution of the data to be recovered.

[0019] The priority sorting module is configured to use artificial intelligence to calculate the priority of recovering training data of each category and sort the strips in which the data to be recovered is located.

[0020] The scheme construction module is configured to combine the calculated data recovery priority with global information on the distribution of the data to be recovered to construct a parallel recovery scheme.

[0021] Among them, methods that utilize artificial intelligence to calculate the priority of recovering training data of different categories include:

[0022] wherein, is the weight update frequency, i.e., the number of times of updating the weight parameter of the p-th class training data in a unit iteration period in the model training process, and a and β are two hyperparameters, denotes the local imbalance of the p-th class training data on the node of the strip denoted by denotes the local imbalance of the p-th class training data on the node of the strip denoted by denotes the imbalance of the p-th class training data on all nodes of the strip denoted by denotes the weight update frequency of the p-th class training data.

[0023] Based on the same inventive concept, the application further provides a readable storage medium having a computer program stored thereon, wherein the computer program can realize the distributed deep learning data recovery method when executed by a processor.

[0024] In the distributed deep learning data recovery method, system and readable storage medium provided by the application, the artificial intelligence is used to establish a weighted priority for the data to be recovered, and the global information of the data blocks obtained by the erasure code is used to construct a parallel recovery scheme, so that the ability of the erasure code to obtain global information is combined with the ability of the artificial intelligence to recover the partially lost data, thereby greatly reducing the cost and accelerating the data reconstruction of the distributed deep learning in the cloud storage system under the acceptable training accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0025] Those skilled in the art will understand that the provided drawings are for better understanding of the application, and do not constitute any limitation on the scope of the application. Among them:

[0026] Figure 1 a flowchart of the distributed deep learning data recovery method provided by an embodiment of the application;

[0027] Figure 2 a schematic diagram of the distributed deep learning data recovery system provided by an embodiment of the application.

[0028] In the drawings:

[0029] 100 - parameter collection module; 200 - priority sorting module; 300 - scheme construction module. DETAILED DESCRIPTION

[0030] In order to make the objects, advantages and features of the present application more clearly, the following further describes the present application in conjunction with the drawings and specific embodiments. It should be noted that the drawings are very simplified and all use non-precise proportions, only for the purpose of facilitating and clearly assisting the description of the embodiments of the present application. In order to make the objects, features and advantages of the present application more obvious and easy to understand, please refer to the drawings. It should be understood that the structures, proportions, sizes, etc. shown in the drawings attached to the present specification are only used to understand and read by those skilled in the art, and are not used to limit the conditions of the implementation of the present application. Any modification of structure, change of proportion relationship or adjustment of size, as long as it is the same or similar to the effect and purpose that can be achieved by the present application, should still fall within the scope of the technical content disclosed by the present application.

[0031] As used in the present application, the singular forms "a", "an" and "the" include plural referents unless the content clearly dictates otherwise. As used in the present application, the term "or" is generally used in the sense of "and / or" unless the content clearly dictates otherwise. As used in the present application, the term "several" is generally used in the sense of "at least one" unless the content clearly dictates otherwise. As used in the present application, the term "at least two" is generally used in the sense of "two or more" unless the content clearly dictates otherwise. In addition, the terms "first", "second", "third" are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second", "third" can be explicitly or implicitly included one or at least two features.

[0032] In the description of the present application, unless otherwise explicitly specified and limited, the terms "installation", "connection", "connection", "fixation" should be understood broadly, for example, it can be fixed connection, or detachable connection, or integral; it can be mechanical connection, or electrical connection; it can be direct connection, or indirect connection through intermediate medium; it can be the internal communication of two elements or the interaction relationship between two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0033] Please refer to Figure 1 , Figure 1 The flow chart of the data recovery method of distributed deep learning provided by an embodiment of the present application. The embodiment provides a data recovery method of distributed deep learning, which comprises the following steps:

[0034] S1. Collect parameter information for training the neural network model and use erasure coding to obtain global information on the distribution of the data to be recovered;

[0035] S2. Use artificial intelligence to calculate the priority of recovering training data of each category and sort the strips in which the data to be recovered is located;

[0036] S3. Combine the calculated data recovery priority with the global information on the distribution of the data to be recovered to construct a parallel recovery scheme.

[0037] This invention uses artificial intelligence to establish a weighted priority for the data to be recovered, and combines it with the global information of data block distribution obtained by erasure coding to construct a parallel recovery scheme. It combines the ability of erasure coding to obtain global data information with the ability of artificial intelligence to recover partially lost data, which significantly reduces resource consumption while maintaining acceptable training accuracy and accelerates the data reconstruction of distributed deep learning in cloud storage systems.

[0038] First, step S1 is executed to collect parameter information for training the neural network model and to obtain global information about the distribution of the data to be recovered using erasure coding. The parameter information includes the weight update frequency of each category of training data, erasure coding parameters (such as the number of data blocks, the number of parity blocks, and the coding matrix), and data storage information (distribution of different stripes, distribution of individual stripe blocks), etc.

[0039] Then, step S2 is executed, using artificial intelligence to calculate the priority of recovering training data of each category, and sorting the strips containing the data to be recovered. Specifically, the method for using artificial intelligence to calculate the priority of recovering training data of each category includes:

[0040] in, α represents the weight update frequency, which is the number of times the weight parameters corresponding to different classes of training data are updated within a unit iteration period during model training. α and β are two hyperparameters. Indicates stripe The Local imbalance of p-class training data at nodes Indicates stripe The Imbalance of p-class training data across all nodes. This indicates the frequency of weight updates for training data of category p.

[0041] The local and global imbalances of the training data for each category can be calculated using erasure coding to obtain global information about the distribution of the data to be restored. Specifically, the imbalance of the training data by category refers to the significant difference in the number of samples (i.e., training data) of different categories within the training dataset processed in this application.

[0042] greater indicates that more p-class training data is missing at the node greater indicates that the data of the p-class training data missing accounts for a larger percentage of the cluster at the node greater greater

[0043] For a training model with Q neurons in the hidden layer, the weight of the p-class training data needs to be monitored. In order to monitor the weight update of each class of training data in the neural network model training process, the weight difference ΔW = W (t + 1) - W(t) from the tth round to the t+1th round of training iteration needs to be calculated. In order to calculate the weight update frequency from the tth round to the t+1th round, the global model of the tth round needs to be downloaded first, then the sample data of each class in the auxiliary data is sent into the global model, and finally the weight difference is used to determine the update rate.

[0044] In this embodiment, the weight update frequency is proportional to the number of samples of each class of training data.

[0045] After calculating the current training weight update frequency of each class of training data and the local imbalance and global imbalance, the weighted priority of the strip recovery of the to-be-recovered data can be calculated.

[0046] Finally, S3 is executed to combine the calculated data recovery priority and the global information of the to-be-recovered data distribution to construct a parallel recovery scheme. The recovery scheme includes strip selection and parallel strip management. When performing strip selection and parallel strip management, the strip that can be recovered in parallel with the current strip is explored from the remaining strips according to the priority order of the strip recovery, and then the data that can be recovered in parallel with the current strip is selected.

[0047] Please refer to Figure 2 , based on the same inventive concept, the embodiment of the present application also provides a data recovery system for distributed deep learning, comprising:

[0048] A parameter collection module 100 configured to collect parameter information of a neural network model for training, and obtain global information of a to-be-recovered data distribution using an erasure code;

[0049] A priority sorting module 200 configured to calculate the priority of each class of training data recovery using artificial intelligence, and sort the strips where the to-be-recovered data is located;

[0050] A scheme construction module 300 configured to combine the calculated data recovery priority and the global information of the to-be-recovered data distribution to construct a parallel recovery scheme.

[0051] Based on the same inventive concept, the application further provides a readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, can implement the distributed deep learning data recovery method as described above in the above feature description.

[0052] The readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The readable storage medium can be, for example but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or punched tape, a holographic storage medium, or any suitable combination of the foregoing. A computer program product, such as the computer program 1120, can be downloaded to the respective computing / processing device from a readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network, and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer program product from the network and forwards the computer program product for storage in a readable storage medium in the respective computing / processing device. The computer program 1120 for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer program 1120 can execute entirely on the user's computing / processing device, partly on the user's computing / processing device, as a stand-alone software package, partly on the user's computing / processing device and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computing / processing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0053] The computer program can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A typical computer program can be stored in any tangible medium of storage, including volatile or non-volatile storage. Any appropriate tangible medium of storage can be used, including volatile or non-volatile storage. The computer program can be deployed for use by one or many types of computer system or other computing systems including general purpose computers of one of many types, special purpose computer systems that can be peripheral devices, computer system on a chip, and / or other computer systems or processing systems.

[0054] The computer program can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the computer program implemented process implements the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0055] In summary, the embodiments of the present application provide a data recovery method and system for distributed deep learning and a readable storage medium. The artificial intelligence establishes a weighted priority for the data to be recovered, and combines the global information of the data blocks obtained by the erasure code to construct a parallel recovery scheme. The ability of the erasure code to obtain global information is combined with the ability of the artificial intelligence to recover partially lost data, which greatly reduces the overhead under acceptable training accuracy and accelerates the data reconstruction of distributed deep learning in a cloud storage system.

[0056] The above description is merely that of the preferred embodiments of the application, and is not intended in any way to limit the scope of the application. Any modifications, variations or improvements made to the application in light of the above teachings by persons skilled in the art without departing from the spirit and scope of the application should be covered by the application. Obviously, those skilled in the art can make various modifications and variations to the application without departing from the spirit and scope of the application. Thus, if these modifications and variations belong to the scope of the application and its equivalent technologies, the application is intended to include them.

Claims

1. A distributed deep learning-based data recovery method, characterized in that, Includes the following steps: S1. Collect parameter information for training the neural network model and use erasure coding to obtain global information on the distribution of the data to be recovered; S2. Use artificial intelligence to calculate the priority of recovering training data of each category and sort the strips in which the data to be recovered is located; S3. Combine the calculated data recovery priority with the global information on the distribution of the data to be recovered to construct a parallel recovery scheme; Among them, methods that utilize artificial intelligence to calculate the priority of recovering training data of different categories include: ; in, α represents the weight update frequency, which is the number of times the weight parameters corresponding to different classes of training data are updated within a unit iteration period during model training. α and β are two hyperparameters. Indicates stripe The Local imbalance of p-class training data at nodes Indicates stripe The Imbalance of p-class training data across all nodes. This indicates the frequency of weight updates for training data of category p.

2. The data recovery method for distributed deep learning according to claim 1, characterized in that, The weight update frequency for each category of training data is derived from the aforementioned parameter information.

3. The data recovery method for distributed deep learning according to claim 2, characterized in that, The weight update frequency is proportional to the number of samples in each category of training data.

4. The data recovery method for distributed deep learning according to claim 1, characterized in that, The local imbalance and the global imbalance are derived from the global information obtained by the erasure coding.

5. The data recovery method for distributed deep learning according to claim 1, characterized in that, The recovery scheme includes strip selection and parallel strip management.

6. The data recovery method for distributed deep learning according to claim 5, characterized in that, When performing stripe selection and parallel stripe management, the remaining stripes are explored in order of priority of stripe recovery to find stripes that can be recovered in parallel with the current stripe, and then the data that can be recovered in parallel with the current stripe is selected.

7. A distributed deep learning data recovery system, characterized in that, include: The parameter collection module is configured to collect parameter information for training the neural network model and use erasure coding to obtain global information on the distribution of the data to be recovered. The priority sorting module is configured to use artificial intelligence to calculate the priority of recovering training data of each category and sort the strips in which the data to be recovered is located. The scheme construction module is configured to combine the calculated data recovery priority with global information on the distribution of the data to be recovered to construct a parallel recovery scheme. Among them, methods that utilize artificial intelligence to calculate the priority of recovering training data of different categories include: ; in, α represents the weight update frequency, which is the number of times the weight parameters corresponding to different classes of training data are updated within a unit iteration period during model training. α and β are two hyperparameters. Indicates stripe The Local imbalance of p-class training data at nodes Indicates stripe The Imbalance of p-class training data across all nodes. This indicates the frequency of weight updates for training data of category p.

8. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it can implement the distributed deep learning data recovery method according to any one of claims 1-6.

Images