A processing method for accelerating training

Abstract

The application relates to the technical field of electric digital data processing, in particular to a processing method for accelerating training. The method is applied to a distributed training architecture which comprises a plurality of training nodes and a distributed memory file system, the system comprising a control node and a plurality of storage nodes, the control node being used for realizing management on the storage nodes and storage directories, and each training node realizing read-write operation on the storage directories through the control node; the method comprises the following steps: in response to architecture initialization, the control node acquires initial training stage data from external storage and stores the initial training stage data in the storage directories; in response to the architecture starting an initial stage training task, a target training node sends a request for acquiring the initial training stage data to the control node; and the control node returns the corresponding storage directories to the training node, so that the training node acquires the initial training stage data for training. The application can pre-store external data in the storage directories during initialization and training, and realizes memory-level read-write speed.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing technology, and in particular to a processing method for accelerating training. Background Technology

[0002] In existing technologies, when using distributed framework systems for model training, the data required for training is usually retrieved from external storage after the training task has started. The type of external storage can be a disk, etc., and the external storage is usually provided by the user. This method of retrieving data from external storage has the drawback of long retrieval time, which means that the distributed training framework is in the stage of retrieving data from external storage for a long time after the training task has started. This results in the existing distributed training framework system having a long model training time and low training efficiency. Summary of the Invention

[0003] The purpose of this invention is to provide a method for accelerating training, thereby reducing model training time and improving training efficiency.

[0004] According to the present invention, a method for accelerating training is provided. This method is applied to a distributed training architecture, which includes several training nodes, each including a graphics processing unit (GPU), with different GPUs executing different training tasks. The distributed training architecture also includes a distributed memory file system, comprising a control node and several storage nodes, each storage node consisting of the memory of a server. The distributed memory file system further includes several storage directories. The control node manages the storage nodes and storage directories, and each training node can perform read and write operations on the storage directories through the control node. The method includes the following steps:

[0005] S100, in response to the initialization of the distributed training architecture, the control node obtains the initial training stage data from the external storage and stores the initial training stage data in the storage directory; the initial training stage data is the data required by the distributed training architecture to perform the initial stage training task, and the initial stage includes several stages.

[0006] S200, in response to the start of the initial training phase of the distributed training architecture, the target training node sends a request to the control node to obtain the initial training phase data; the target training node is a training node included in the distributed training architecture.

[0007] S300, after receiving the request from the target training node to obtain the initial training stage data, the control node returns the storage directory containing the initial training stage data to the target training node.

[0008] S400, after the target training node receives the storage directory containing the initial training stage data returned by the control node, it retrieves the initial training stage data from the storage directory containing the initial training stage data.

[0009] S500, the target training node is trained based on the acquired initial training phase data.

[0010] Compared with the prior art, the present invention has at least the following beneficial effects:

[0011] This invention provides a method for accelerating training. This method is applied to a distributed training framework, which includes several training nodes, each including a graphics processing unit (GPU). Different GPUs execute different training tasks. Furthermore, the distributed training framework includes a distributed memory file system, comprising several storage nodes, each consisting of the memory of a server. The distributed memory file system also includes several storage directories. A control node is used to manage the storage nodes and directories. Each training node can perform read and write operations on the storage directories through the control node. Specifically, in response to the initialization of the distributed training architecture, the control node pre-acquires the data required for the initial training phase (i.e., initial training phase data) from external storage. Therefore, when the distributed training architecture starts the initial training phase, it is no longer necessary to retrieve the initial training phase data from external storage. Instead, the initial training phase data is retrieved from the storage directory containing the initial training phase data in the distributed memory file system. Compared to retrieving the initial training phase data from external storage, the process of retrieving the initial external storage data from the storage directory is faster, operating at memory-level read / write speeds. This results in a shorter initial training phase and improved training efficiency.

[0012] Furthermore, during the training process, the present invention can read data from external storage into a distributed memory file system in advance according to the needs of the task for continuous use by the training task. This enables continuous pre-reading of external data and continuously provides memory-level read and write speeds for the training task, resulting in shorter training time in subsequent stages and improved training efficiency in those stages. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 A flowchart of a method for accelerating training provided in an embodiment of the present invention. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] According to this embodiment, a method for accelerating training is provided. This method is applied to a distributed training architecture, which includes several training nodes, each including a graphics processing unit (GPU). Different GPUs are used to execute different training tasks. The distributed training architecture also includes a distributed memory file system, which includes a control node and several storage nodes, each consisting of the memory of a server. The distributed memory file system also includes several storage directories. The control node manages the storage nodes and storage directories, and each training node can perform read and write operations on the storage directories through the control node.

[0017] In this embodiment, each storage node is composed of the memory of the server included in the training node or the memory of other servers, wherein the other servers are servers other than the server included in the training node.

[0018] Optionally, the distributed memory file system includes storage directories divided into a first type of storage directory and a second type of storage directory. The first type of storage directory is used to store data obtained from external storage, and the second type of storage directory is used to store training data generated by training nodes during training. The ratio of storage space included in the first storage directory to storage space included in the second storage directory can be preset or adjusted as needed.

[0019] like Figure 1 As shown, the method includes the following steps:

[0020] S100, in response to the initialization of the distributed training architecture, the control node obtains the initial training stage data from the external storage and stores the initial training stage data in the storage directory; the initial training stage data is the data required by the distributed training architecture to perform the initial stage training task, and the initial stage includes several stages.

[0021] In this embodiment, the distributed training architecture is initialized before the initial training phase of the distributed training architecture starts. This embodiment reads data from external storage into the storage directory of the distributed memory file system during the initialization phase of the distributed training architecture, without occupying time after the initial training phase starts. Furthermore, since the initial training phase data is already stored in the storage directory of the distributed memory file system, training nodes that need to obtain the initial training phase data can directly read it from the storage directory after the initial training phase starts. Compared to reading the initial training phase data from external storage, reading the initial training phase data from the storage directory is a memory-level read / write operation, which is much faster.

[0022] In this embodiment, the initial stage includes several stages, where several means greater than or equal to 1. For example, the initial stage only includes the first stage, which is the first training stage entered after the distributed training architecture starts the training task; or, for another example, the initial stage includes the first, second and third stages, where the second stage is the next training stage after the first training stage, and the third stage is the next training stage after the second training stage.

[0023] S200, in response to the start of the initial training phase of the distributed training architecture, the target training node sends a request to the control node to obtain the initial training phase data; the target training node is a training node included in the distributed training architecture.

[0024] S300, after receiving the request from the target training node to obtain the initial training stage data, the control node returns the storage directory containing the initial training stage data to the target training node.

[0025] S400, after the target training node receives the storage directory containing the initial training stage data returned by the control node, it retrieves the initial training stage data from the storage directory containing the initial training stage data.

[0026] S500, the target training node is trained based on the acquired initial training phase data.

[0027] This embodiment provides a method for accelerating training. This method is applied to a distributed training framework, which includes several training nodes, each including a graphics processing unit (GPU). Different GPUs execute different training tasks. Furthermore, the distributed training framework of this embodiment also includes a distributed memory file system, which includes several storage nodes, each consisting of the memory of a server. The distributed memory file system also includes several storage directories. A control node is used to implement read and write operations on the storage nodes and storage directories. Each training node can perform read and write operations on the several storage directories through the control node. Specifically, in response to distributed training... During architecture initialization, the control node pre-acquires the data required for the initial training phase of the distributed training architecture (i.e., initial training phase data) from external storage. Therefore, when the distributed training architecture starts the initial training phase, it is no longer necessary to retrieve the initial training phase data from external storage. Instead, the initial training phase data is retrieved from the storage directory containing the initial training phase data in the distributed memory file system. Compared to retrieving the initial training phase data from external storage, the process of retrieving the initial external storage data from the storage directory in this embodiment takes less time, which is at the memory level read / write speed. This makes the training process in the initial phase shorter and improves the training efficiency in the initial phase.

[0028] In one specific implementation, the distributed training architecture includes a first training node and a second training node, wherein the second training node is a downstream training node of the first training node, and the method further includes:

[0029] S600, the first training node sends a request to the control node to store the first training data.

[0030] In this embodiment, when multiple training nodes in the distributed training framework jointly execute a training task, a training node may be an upstream or downstream training task of other training nodes. For example, when a training node (worker1) is an upstream training task of another training node (worker2), it means that the training data of worker1 is the basic data for worker2 to execute the training task, and worker2 depends on the training data of worker1 when executing the training task. When a training node (worker1) is a downstream training task of another training node (worker3), it means that the training data of worker3 is the basic data for worker1 to execute the training task, and worker1 depends on the training data of worker3 when executing the training task.

[0031] S610, after receiving the request from the first training node to store the first training data, the control node returns the first storage directory to the first training node; the first storage directory is the storage directory included in the distributed memory file system.

[0032] S620, after receiving the first storage directory returned by the control node, the first training node stores the first training data in the first storage directory.

[0033] S630, the control node sends preset information including the first storage directory to the second training node.

[0034] In this embodiment, the second training node is a downstream training node of the first training node. When the second training node performs the training task, it relies on the first training data. By sending preset information including the first storage directory to the second training node through the control node, the second training node can know that the first training node has stored the first training data in the first storage directory.

[0035] S640, after receiving the preset information including the first storage directory sent by the control node, the second training node retrieves the first training data from the first storage directory and performs training based on the first training data.

[0036] In one specific implementation, after S640, the method further includes:

[0037] S650, if the second training node has write permission to the first storage directory, and the first training data is only used by the second training node to execute the training task of the current stage, then the second training node deletes the first training data stored in the first storage directory after successfully reading the first training data; if the second training node does not have write permission to the first storage directory, and the first training data is only used by the second training node to execute the training task of the current stage, then proceed to S660.

[0038] In this embodiment, if the first training data stored in the first storage directory is only used by the second training node for the current stage of training, then after the second training node successfully reads the first training data, the first training data becomes useless data. In this embodiment, if the second training node has the permission to write to the first storage directory, the second training node performs the operation of deleting the first training data in the first storage directory.

[0039] S660, after successfully reading the first training data, the second training node sends a message to the first training node indicating that it has successfully read the first training data.

[0040] S670, after the first training node receives the message from the second training node that the first training data has been successfully read, it deletes the first training data stored in the first storage directory.

[0041] Based on S650-S670, when the first training data is only used by the second training node to perform the training task of the current stage, the deletion operation after the second training node successfully reads the first training data is realized, saving the space of the first storage directory.

[0042] In this embodiment, if the first training data is not only used by the second training node to perform the training task of the current stage, then proceed to S680.

[0043] S680, if the preset external storage conditions are met, the control node will store the first training data in external storage; the preset external storage conditions include one of the following conditions: the ratio of the storage space occupied by the data in all storage directories included in the distributed memory file system to the total storage space of all storage directories is greater than or equal to a preset ratio; the number of times the first training data is accessed in a preset historical time period is less than or equal to a preset number of accesses; the total number of times the first training data is accessed is less than or equal to a preset number of accesses.

[0044] In this embodiment, when the first training data is not only used by the second training node to perform the training task of the current stage, the first training data is stored in external storage according to preset external storage conditions, thus avoiding the loss of the first training data.

[0045] In one specific implementation, after S100, the method further includes the following steps:

[0046] S110, if the storage directory included in the distributed memory file system still has remaining storage space, then proceed to S120.

[0047] S120, determine whether the current progress of the initial stage training task is greater than or equal to a preset progress threshold. If it is greater than or equal to, proceed to S130.

[0048] In this embodiment, the preset progress threshold is an empirical value, for example, the preset progress threshold is 60%.

[0049] S130, the control node obtains the subsequent training stage data from the external storage and stores it in the storage directory; the subsequent training stage data is the data required for the training tasks of the training stage after the initial training stage is executed by the distributed training architecture.

[0050] Based on S110-S130, when there is still remaining storage space in the storage directory included in the distributed memory file system, and the current progress of the initial stage training task is greater than or equal to a preset progress threshold, this embodiment retrieves the data required for the training tasks of the subsequent training stages from external storage to the storage directory while the distributed training architecture is executing the initial stage training task (i.e., before executing the training tasks of subsequent stages). Compared with the method of retrieving the subsequent training stage data from external storage after the subsequent stage has started, this embodiment does not occupy the training time of the subsequent stage, and the first external data is retrieved from the storage directory in advance, which is a memory-level read and write operation, reducing the time required for the distributed training architecture to execute the training tasks of the subsequent stage and improving the training efficiency of the subsequent stage.

[0051] In a preferred embodiment, each storage directory in the distributed memory file system corresponds to a backup quantity, and the control node is also used to manage the backup quantity; after S630, the method further includes: the control node obtains the backup quantity corresponding to the first storage directory, and backs up the first training data on other storage nodes according to the backup quantity; the other storage nodes are storage nodes other than the target storage node among the storage nodes included in the distributed memory file system, and the target storage node is the storage node in the distributed memory file system that receives the first training data.

[0052] As an optional implementation, the number of backups corresponding to the first storage directory is an empirical value, for example, 2 or 3. If the number of backups corresponding to the first storage directory is 2, then the first training data is backed up on one of the storage nodes in the distributed memory file system, excluding the target storage node; if the number of backups corresponding to the first storage directory is 3, then the first training data is backed up on two storage nodes in the distributed memory file system, excluding the target storage node.

[0053] As a preferred embodiment, the process of obtaining the number of backups corresponding to the first storage directory includes:

[0054] S631, Obtain the importance of the training data to be stored in the first storage directory.

[0055] As an optional specific implementation, the importance of the training data to be stored in the first storage directory is an empirical value, and the importance of the training data to be stored in the first storage directory is a value between 0 and 1. For example, the importance of the training data to be stored in the first storage directory is 0.5 or 0.8.

[0056] As a preferred embodiment, the process of determining the importance of the training data to be stored in the first storage directory includes:

[0057] S6311, obtain the training progress of the training node corresponding to the training data to be stored in the first storage directory.

[0058] In this embodiment, the training progress of the training nodes corresponding to the training data to be stored in the first storage directory reflects the overall training progress of the training nodes corresponding to all the training data to be stored in the first storage directory. Optionally, the training progress of the training nodes corresponding to the training data to be stored in the first storage directory is the average of the training progress of the training nodes corresponding to all the training data to be stored in the first storage directory. For a certain training data to be stored in the first storage directory, if the training data is generated when the training task of its corresponding training node is executed to 10%, then the training progress of the training node corresponding to that training task is 10%; if the training data is generated when the training task of its corresponding training node is executed to 20%, then the training progress of the training node corresponding to that training task is 20%; if the training data is generated when the training task of its corresponding training node is executed to 63%, then the training progress of the training node corresponding to that training task is 63%; if the training data is generated when the training task of its corresponding training node is executed to 82%, then the training progress of the training node corresponding to that training task is 82%. The training node corresponding to the training data is the training node to which the graphics processor that generated the training data belongs.

[0059] S6312, obtain the number of downstream training nodes corresponding to the training nodes to be stored in the first storage directory.

[0060] In this embodiment, when multiple training nodes in the distributed training framework jointly execute a training task, a training node may be an upstream or downstream training task of other training nodes. For example, when a training node (worker1) is an upstream training task of another training node (worker2), it means that the training data of worker1 is the basic data for worker2 to execute the training task, and worker2 depends on the training data of worker1 when executing the training task. When a training node (worker1) is a downstream training task of another training node (worker3), it means that the training data of worker3 is the basic data for worker1 to execute the training task, and worker1 depends on the training data of worker3 when executing the training task.

[0061] In this embodiment, the number of downstream training nodes of the training nodes corresponding to the training data to be stored in the first storage directory reflects the overall number of downstream training nodes of the training nodes corresponding to all training data to be stored in the first storage directory. Optionally, the number of downstream training nodes of the training nodes corresponding to the training data to be stored in the first storage directory is an average value of the numbers of downstream training nodes of the training nodes corresponding to all training data to be stored in the first storage directory.

[0062] S6313, determining the importance of the training data to be stored in the first storage directory according to the training progress and the number of downstream training nodes.

[0063] In this embodiment, the training progress of the training nodes corresponding to the training data to be stored in the first storage directory is positively correlated with the importance of the training data to be stored in the first storage directory, and the number of downstream training nodes of the training nodes corresponding to the training data to be stored in the first storage directory is also positively correlated with the importance of the training data to be stored in the first storage directory.

[0064] As a preferred specific implementation, the importance of the training data to be stored in the first storage directory is d, d=w1×a+w2×b / c, where a is the training progress of the training nodes corresponding to the training data to be stored in the first storage directory, b is the number of downstream training nodes of the training nodes corresponding to the training data to be stored in the first storage directory, c is the number of training nodes included in the distributed training framework, w1 and w2 are respectively weights corresponding to the training progress and the number of downstream training nodes, 0<w1<1, 0<w2<1, and w1+w2=1. Optionally, both w1 and w2 are empirical values, for example, w1=w2=0.5.

[0065] S632, acquiring the historical number of failures of the training nodes included in the distributed training framework.

[0066] In this embodiment, the historical number of failures of the training nodes included in the distributed training framework is determined according to the number of failed training nodes when the distributed training framework executes historical training tasks. For example, the distributed training framework has executed multiple training tasks within a historical time period, and optionally, the average value or maximum value of the number of failed training nodes during the execution of the multiple training tasks within the historical time period is determined as the historical number of failures of the training nodes included in the distributed training framework.

[0067] S633, acquiring the storage space utilization rate of the storage nodes included in the distributed memory file system.

[0068] In this embodiment, the storage space utilization rate of the storage nodes included in the distributed memory file system is the ratio of the storage space occupied by the training data generated during the execution of the current training task by the distributed training framework to the total storage space of the storage nodes included in the distributed memory file system. A larger value of the storage space utilization rate of the storage nodes included in the distributed memory file system indicates that the total storage space of the storage nodes included in the distributed memory file system is smaller relative to the training task. Optionally, the storage space occupied by the training data generated during the execution of the training task by the distributed training framework is an empirical value.

[0069] S634, determining the number of backups corresponding to the first storage directory according to the importance degree, the number of historical faults and the storage space utilization rate.

[0070] In this embodiment, the importance degree of the training data to be stored in the first storage directory is positively correlated with the number of backups corresponding to the first storage directory, the number of historical faults of the training nodes included in the distributed training framework is positively correlated with the number of backups corresponding to the first storage directory, and the storage space utilization rate of the storage nodes included in the distributed memory file system is negatively correlated with the number of backups corresponding to the first storage directory.

[0071] As a preferred specific embodiment, the number of backups corresponding to the first storage directory is g, where g=int((w3×e+w4 / f)×d / k), int() represents rounding, k is a preset importance degree threshold, 0<k<1, w3 and w4 are respectively weights corresponding to the number of historical faults of the training nodes included in the distributed training framework and the storage space utilization rate, 0<w3<1, 0<w4<1, w3+w4=1, e is the number of historical faults of the training nodes included in the distributed training framework, and f is the storage space utilization rate of the storage nodes included in the distributed memory file system. Optionally, k, w3 and w4 are all empirical values, for example, w3=w4=0.5. Based on this preferred specific embodiment, even if some training nodes in the distributed training framework fail, the first training data can be obtained from the storage directories included in the distributed file system with a high probability, so that the training node that needs to obtain the first training data can obtain the first training data from the storage directories included in the distributed file system, implement memory-level reading and writing, and thereby improve training efficiency.

[0072] While some specific embodiments of the present invention have been described in detail by way of examples, those skilled in the art should understand that the foregoing examples are only for the purpose of illustration and not for limiting the scope of the present invention. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of the present invention. The scope of the present invention is defined by the appended claims.

Claims

1. A method for accelerating training, characterized in that, The accelerated training processing method is applied to a distributed training architecture, which includes several training nodes, each including a graphics processing unit (GPU), with different GPUs used to execute different training tasks. The distributed training architecture also includes a distributed memory file system, comprising a control node and several storage nodes, each storage node consisting of the memory of a server. The distributed memory file system further includes several storage directories, and the control node manages the storage nodes and directories. Each training node can perform read and write operations on the storage directories through the control node. The method includes the following steps: S100, in response to the initialization of the distributed training architecture, the control node obtains the initial training stage data from the external storage and stores the initial training stage data in the storage directory; the initial training stage data is the data required by the distributed training architecture to execute the initial stage training task, and the initial stage includes several stages; S200, in response to the start of the initial training phase of the distributed training architecture, the target training node sends a request to the control node to obtain the initial training phase data; the target training node is a training node included in the distributed training architecture; S300, after receiving the request from the target training node to obtain the initial training stage data, the control node returns the storage directory containing the initial training stage data to the target training node. S400, after the target training node receives the storage directory containing the initial training stage data returned by the control node, it retrieves the initial training stage data from the storage directory containing the initial training stage data. S500, the target training node is trained based on the acquired initial training phase data; The distributed training architecture includes a first training node and a second training node, wherein the second training node is a downstream training node of the first training node, and the method further includes: S600, the first training node sends a request to the control node to store the first training data; S610, after receiving the request from the first training node to store the first training data, the control node returns the first storage directory to the first training node; the first storage directory is the storage directory included in the distributed memory file system; S620, after receiving the first storage directory returned by the control node, the first training node stores the first training data in the first storage directory; S630, the control node sends preset information including the first storage directory to the second training node; S640, after receiving the preset information including the first storage directory sent by the control node, the second training node retrieves the first training data from the first storage directory and performs training based on the first training data. Following S100, the method further includes the following steps: S110, if the storage directory included in the distributed memory file system still has remaining storage space, then proceed to S120; S120, determine whether the current progress of the initial stage training task is greater than or equal to a preset progress threshold; if it is greater than or equal to, proceed to S130. S130, the control node obtains the subsequent training stage data from the external storage and stores it in the storage directory; the subsequent training stage data is the data required for the training tasks of the training stage after the initial training stage is executed by the distributed training architecture.

2. The accelerated training processing method according to claim 1, characterized in that, Following S640, the method further includes: S650, if the second training node has write permission to the first storage directory, and the first training data is only used by the second training node to execute the training task of the current stage, then the second training node deletes the first training data stored in the first storage directory after successfully reading the first training data; if the second training node does not have write permission to the first storage directory, and the first training data is only used by the second training node to execute the training task of the current stage, then proceed to S660. S660, after successfully reading the first training data, the second training node sends a message to the first training node indicating that it has successfully read the first training data. S670, after the first training node receives the message from the second training node that the first training data has been successfully read, it deletes the first training data stored in the first storage directory.

3. The accelerated training processing method according to claim 2, characterized in that, S650 further includes: if the first training data is not only used by the second training node to perform the training task of the current stage, then proceed to S680; S680, if the preset external storage conditions are met, the control node stores the first training data in external storage; the preset external storage conditions include one of the following: the ratio of the storage space occupied by the data in all storage directories included in the distributed memory file system to the total storage space of all storage directories is greater than or equal to a preset ratio; the number of times the first training data is accessed in a preset historical time period is less than or equal to a preset number of accesses; the total number of times the first training data is accessed is less than or equal to a preset number of accesses.

4. The accelerated training processing method according to claim 1, characterized in that, The distributed memory file system includes storage directories divided into a first type of storage directory and a second type of storage directory. The first type of storage directory is used to store data obtained from external storage, and the second type of storage directory is used to store training data generated by training nodes during the training process.

5. The accelerated training processing method according to claim 1, characterized in that, Each storage node consists of the memory of the servers included in the training node or the memory of other servers, which are servers other than those included in the training node.

Images