A computing method, device, storage medium, and equipment in parallel training
By deploying conventional and elastic layers between computing nodes and dynamically adjusting the load, the problem of computing delay in pipeline parallelism is solved, and the efficiency of large-scale model training is improved.
Patent Information
- Application Number
- CN202411058896.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-02
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-08-02
AI Technical Summary
Traditional distributed computing models have challenges in load balancing and resource utilization, especially in pipeline parallelism, where computing delay causes computing nodes to wait, affecting the efficiency of large-scale model training.
A parallel training method is adopted to deploy the model layer in multiple computing nodes, each node deploys a conventional layer and an elastic layer, and completes some computing tasks without relying on the post-node conditions, and dynamically monitors and adjusts the computing nodes. Load, reduce waiting time, and improve system fault tolerance.
Effectively balance the load between computing nodes, optimize the overall computing efficiency, reduce computing waiting time, and improve the efficiency of model training.
Smart Images

Figure CN118585342B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a computing method, apparatus, storage medium, and device in parallel training. Background Art
[0002] As the scale and complexity of deep learning models continue to increase, the computing resources of a single node cannot meet the needs of large-scale model training. Distributed computing technology can significantly improve the training efficiency of large-scale models by allocating computing tasks to multiple computing nodes and executing computing tasks in parallel.
[0003] However, traditional distributed computing models still face many challenges in load balancing and resource utilization, especially in pipeline parallelism, where the same batch of data needs to be pipelined through each computing node to obtain the final calculation result. During this calculation process, any delay in the calculation of any computing node will cause the subsequent computing nodes to wait. It can be seen that how to more effectively utilize computing resources and balance the computing load in multi-node pipeline parallelism, thereby improving the overall training efficiency of the model, is an urgent problem to be solved.
[0004] Therefore, the present invention provides a computing method, device, storage medium, and equipment in parallel training. Summary of the invention
[0005] This specification provides a computing method and device for parallel training to partially solve the above-mentioned problems existing in the prior art.
[0006] This manual adopts the following technical solutions:
[0007] The present specification provides a computing method in parallel training, which is applied to model parallel training, wherein all model layers of the model are used as regular layers, and the regular layers are deployed in a plurality of computing nodes, wherein at least one regular layer is deployed in each computing node, and the regular layers deployed in each computing node are different from each other, and an elastic layer is also deployed in the computing node, and for any computing node, the elastic layer of the computing node is at least one regular layer of a post-node of the computing node, and the method comprises:
[0008] For any computing node, the computing result of the computing node is obtained according to the current batch of training samples and the conventional layer deployed on the computing node;
[0009] If the post-node of the computing node has completed the computing of the training samples of the previous batch of the current batch, the computing result is used as the output result; otherwise, the output result of the computing node is obtained according to the computing result and the elastic layer of the computing node;
[0010] The output result is transmitted to the subsequent node.
[0011] Optionally, the calculation result of the computing node is obtained according to the current batch of training samples and the conventional layer deployed on the computing node, specifically including:
[0012] When the computing node has a predecessor node, the output result transmitted by the predecessor node is used as the input sample of the computing node;
[0013] A calculation result of the calculation node is obtained according to the input sample and the normal layer of the calculation node.
[0014] Optionally, obtaining a calculation result of the calculation node according to the input sample and a normal layer of the calculation node specifically includes:
[0015] If the input sample has not been calculated by the elastic layer of the predecessor node, the calculation result of the computing node is obtained according to the input sample and the conventional layer deployed on the computing node; otherwise, the calculation result of the computing node is obtained according to the input sample and the independent layer of the computing node, wherein the independent layer is a model layer in the conventional layer of the computing node that is different from the elastic layer of the predecessor node.
[0016] Optionally, transmitting the output result to the post-node specifically includes:
[0017] Transmitting the output result and the elastic layer usage identifier corresponding to the output result to the post-node;
[0018] Before obtaining the calculation result of the calculation node, the method further includes:
[0019] According to the elastic layer usage identifier corresponding to the input sample, it is determined whether the input sample has been calculated by the elastic layer of the preceding node.
[0020] Optionally, after transmitting the output result to the post-node, the method further includes:
[0021] Determine a batch of training samples corresponding to the output result as a completed batch of the computing node;
[0022] Transmitting the completed batch to a predecessor node of the computing node;
[0023] After obtaining the calculation result of the calculation node, the method further includes:
[0024] According to the completed batches of the subsequent nodes of the computing node and the current batch, it is determined whether the subsequent nodes of the computing node have completed the calculation of the input samples of the previous batch of the current batch.
[0025] Optionally, also include:
[0026] When updating the model layer parameters, for any computing node, the computing node updates the parameters of the conventional layer of the computing node according to the final calculation result of the model and the preset loss function.
[0027] Optionally, also include:
[0028] For any computing node, the computing node updates the parameters of the elastic layer of the computing node according to the updated parameters of the regular layer of the subsequent node of the computing node.
[0029] The present specification also provides a computing device in parallel training, the device is applied to model parallel training, all model layers of the model are used as regular layers, the regular layers are deployed in a number of computing nodes, each computing node is deployed with at least one regular layer, the regular layers deployed by each computing node are different from each other, the computing nodes are also deployed with elastic layers, for any computing node, the elastic layer of the computing node is at least one regular layer of the post-node of the computing node, the device comprises:
[0030] The computing module obtains the computing result of any computing node according to the current batch of training samples and the conventional layer deployed on the computing node;
[0031] The elastic module uses the calculation result as the output result if the post-node of the computing node has completed the calculation of the training samples of the previous batch of the current batch; otherwise, obtains the output result of the computing node according to the calculation result and the elastic layer of the computing node;
[0032] A transmission module transmits the output result to the post-node.
[0033] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the calculation method in the above-mentioned parallel training.
[0034] This specification provides a device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the calculation method in the above-mentioned parallel training when executing the program.
[0035] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:
[0036] It can be seen from the above method that this method can dynamically monitor and adjust the load of computing nodes, reduce the computing waiting time of the overall computing process, improve the system fault tolerance, balance the load between computing nodes, and thus optimize the overall computing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings:
[0038] Figure 1 A flowchart of a calculation method in parallel training in this specification;
[0039] Figure 2 A schematic diagram of the segmentation of an original model in this manual;
[0040] Figure 3 Schematic diagram of the relationship between the elastic layer and the conventional layer in this specification;
[0041] Figure 4 It is a schematic diagram of the relationship between the independent layer, the elastic layer and the conventional layer in this specification;
[0042] Figure 5 A schematic diagram of a computing device in parallel training provided in this specification;
[0043] Figure 6 The corresponding Figure 1 Schematic diagram of electronic equipment. DETAILED DESCRIPTION
[0044] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0045] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.
[0046] In pipeline parallel distributed computing, the complete model is usually divided and arranged in several computing nodes. When a batch of sample data is calculated, each computing node uses the model layer arranged in its own computing node to perform pipeline calculation on the sample data according to the position order of the model layer arranged in the original model of the node. For example, taking the original model divided into three parts as an example, the first part, the second part and the third part are combined to form the complete original model. The model output of the first part is the model input of the second part, and the model output of the second part is the model input of the third part. The first part is arranged in node A, the second part is arranged in node B, and the third part is arranged in node C. Therefore, when performing pipeline parallel computing, after node A completes the calculation for the first batch of data, it transmits the model output of node A to node B. Node B starts to calculate according to the data transmitted by node A, and node A starts to calculate the second batch of data at the same time. Therefore, each node performs pipeline calculation on the same batch of data, and each node can calculate different batches of data in parallel at the same time. However, in the above process, after node A completes the calculation for the first batch of data, if node B has not completed the calculation for the data transmitted by node A for the first batch, then obviously, node B cannot immediately start calculating the data transmitted by node A for the second batch, and the calculation progress of node C is also affected by the calculation delay of node B. Therefore, this specification provides a calculation method in parallel training, which can be applied to distributed model training with parallel pipeline arrangement.
[0047] Figure 1 This is a flow chart of a computing method in parallel training in this specification, in which all model layers of the model are used as regular layers, and the regular layers are deployed in several computing nodes. At least one regular layer is deployed in each computing node, and the regular layers deployed in each computing node are different from each other. An elastic layer is also deployed in the computing node. For any computing node, the elastic layer of the computing node is at least one regular layer of the post-node of the computing node.
[0048] Specifically, before deploying the model layer to the computing node, the original model can be segmented according to the existing model segmentation method to obtain model groups, and then each model group is assigned to a different computing node in subsequent steps. The model layer in each model group assigned to each computing node is a regular layer.
[0049] like Figure 2As shown, for an original model, the original model is composed of several model layers arranged in the order of the position of each model layer in the original model. The original model in this figure has 8 model layers. The model groups after the original model is segmented can be combined into the original model in sequence. A model group can contain several consecutive model layers, and the model layers contained in different model groups are completely different.
[0050] Among them, this specification does not limit the number of model layers in each model group after the original model is divided into model groups. In actual application scenarios, the number of model layers in each model group can be determined and adjusted by the user.
[0051] After the original model is split, each model group obtained by the split can be assigned to each computing node, that is, the number of model groups obtained by the split is equal to the number of computing nodes, and each model layer in the model group assigned to each computing node is the regular layer of the computing node; after each model group is arranged on each computing node, the number of the model group arranged on each computing node can be determined as the number of the computing node, wherein the number of the model group represents the position order of the model group in the original model, for example: model group No. 1 includes the first model layer of the original model and several consecutive model layers after the first model layer, and model group No. 2 includes several consecutive model layers after model group No. 1 in the original model; after each regular layer is deployed on each computing node, an elastic layer can be deployed for each computing node according to the regular layer of the subsequent node of each computing node.
[0052] In one or more embodiments of the present specification, the elastic layer of a computing node includes at least part of the regular layers of the nodes behind the computing node, and the elastic layer of the computing node is the first several layers of the regular layers of the nodes behind the computing node. Figure 3 As shown, the regular layers of computing node A are the 1st to 5th model layers of the original model, and the regular layers of computing node B, the post-node of computing node A, are the 6th to 10th model layers. For a batch of sample data, after the calculation of the 5th model layer in computing node A, the calculation of the 6th to 10th layers should continue in sequence. Then, the elastic layers of computing node A can be 6-n layers, where n<=10 and n>=6.
[0053] Therefore, when calculating a batch of data, computing node No. 1 first calculates the training sample, and then transmits the output result to the post-node of computing node No. 1 (i.e., computing node No. 2), and the remaining computing nodes calculate and transmit the received data in sequence according to the number of the computing node. Among them, the post-node of a computing node is only the computing node with a number just greater than that of the computing node, for example, the post-node of computing node No. 1 is computing node No. 2. The pre-node of a computing node is only the computing node with a number just less than that of the computing node, which is not elaborated in this specification.
[0054] It should be noted that a computing node refers to a physical device that performs executable computing tasks, including but not limited to a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), etc.
[0055] Therefore, the elastic layer of a computing node can use the computing power of the computing node itself to replace the subsequent nodes of the computing node to complete part of the computing tasks based on the calculation results of the conventional layer of the computing node without relying on the subsequent nodes.
[0056] In addition, methods for segmenting the model include but are not limited to equal layer segmentation, layer segmentation, block segmentation, etc.
[0057] Layer averaging: All model layers of a deep neural network are evenly distributed according to the number of computing nodes, so that each computing node is assigned to an equal number of consecutive model layers.
[0058] Layer splitting: Assign model layers to different nodes in sequence. For example, the first few model layers of a deep neural network are assigned to the first node, the middle few model layers are assigned to the second node, and the last few model layers are assigned to the third node.
[0059] Block-level sharding: Grouping the model layers of a model into blocks and then distributing these blocks to different nodes can better balance the computational load in some cases.
[0060] After each model layer of the original model is deployed on each computing node, the calculation method in the parallel training provided in this specification can be continued. The current model is used to represent the original model after segmentation, which specifically includes the following steps:
[0061] S100: For any computing node, a computing result of the computing node is obtained according to the current batch of training samples and the conventional layer deployed on the computing node.
[0062] Specifically, when the current batch of training samples is input into the current model, the computing node No. 1 that deploys the first model layer of the current model directly uses the conventional layer deployed by itself to process the training samples, obtains the calculation result of the computing node No. 1, and obtains the output result of the computing node No. 1 based on the calculation result, and transmits the output result to the post-node of the computing node No. 1 - computing node No. 2, so that computing node No. 2 and subsequent computing nodes continue to calculate. In other words, the computing node that deploys the first model layer of the current model directly uses the training samples for calculation, while other computing nodes use the training samples processed by the front nodes.
[0063] In one or more embodiments of the present specification, when the computing node has a predecessor node, the output result transmitted by the predecessor node is used as an input sample of the computing node, and the computing result of the computing node is obtained based on the input sample and the normal layer of the computing node.
[0064] Specifically, for any computing node other than computing node No. 1, the computing node has a predecessor node. When the computing node performs calculations, the output result transmitted by the predecessor node of the computing node is used as the input sample of the computing node, and the input sample is calculated using the regular layer deployed in the computing node to obtain the calculation result of the computing node.
[0065] S102: If the post-node of the computing node has completed computing the training samples of the previous batch of the current batch, the computing result is used as the output result; otherwise, the output result of the computing node is obtained according to the computing result and the elastic layer of the computing node.
[0066] For any computing node, after the computing node obtains the calculation results for the current batch of training samples, it can determine whether the subsequent node has completed the calculation of the previous batch of training samples for the current batch based on the calculation progress of the subsequent node sent by the subsequent node in history.
[0067] If the subsequent node of the computing node has completed the calculation of the previous batch of training samples, the computing node uses the calculation result of the computing node as the output result to transmit to the subsequent node. If the subsequent node of the computing node has not completed the calculation of the previous batch of training samples, the elastic layer of the computing node is used to further calculate the calculation result of the computing node to obtain the output result of the elastic layer of the computing node, which is transmitted to the subsequent node of the computing node as the output result of the computing node.
[0068] Among them, for the previous batch of the current batch, if the current batch training samples calculated by the computing node S100 are the training samples of the second batch, then the previous batch is the first batch; to determine whether the subsequent node of the computing node has completed the calculation of the previous batch of input samples, the computing node can request the subsequent node for the batch of training samples corresponding to the computing task being executed by the subsequent node, and determine it based on the batch replied by the subsequent node and the current batch corresponding to the training samples calculated by the computing node.
[0069] Therefore, for any batch of training samples, each computing node performs calculations in sequence according to the numbering order, and the final calculation results of the current model for the batch of training samples can be obtained.
[0070] According to the above calculation process, when the calculation of any computing node is slow, that is, the predecessor node of the computing node has completed the calculation task of the current batch of training samples, but the computing node has not completed the calculation task of the previous batch of training samples of the current batch, in the conventional calculation method, the computing node can only wait for the computing node to complete the calculation task of the previous batch before continuing the calculation task of the current batch, causing the overall calculation process to stagnate and wait. However, the method provided in this specification, under the above situation, can use the predecessor node of the computing node to replace the computing node to perform part or all of the current batch of calculation tasks of the computing node while the computing node executes the calculation task of the previous batch, so that the calculation progress of the current batch will not stagnate due to the delay of the computing node.
[0071] It should be noted that the method provided in this specification does not formulate a fixed load balancing strategy in advance, but dynamically adjusts the task amount of each computing node according to the completion status of the computing tasks of each computing node during the calculation process.
[0072] S104: Transmit the output result to the subsequent node.
[0073] For any computing node, after the computing node obtains the output result of the computing node, the output result can be transmitted to the subsequent node of the computing node. When the computing node is the computing node with the largest number, the output result of the computing node with the largest number is the final calculation result of the current model.
[0074] like Figure 1 The computing method in a parallel training shown can dynamically monitor and adjust the load of computing nodes, reduce the computing waiting time of the overall computing process, improve the system fault tolerance, balance the load between computing nodes, and thus optimize the overall computing efficiency.
[0075] In such Figure 1In the step S100 shown, if the input sample has not been calculated by the elastic layer of the predecessor node, the calculation result of the computing node is obtained according to the input sample and the conventional layer deployed on the computing node; otherwise, the calculation result of the computing node is obtained according to the input sample and the independent layer of the computing node, wherein the independent layer is a model layer in the conventional layer of the computing node that is different from the elastic layer of the predecessor node.
[0076] When the computing node is not computing node No. 1, after the computing node uses the output result transmitted by the predecessor node of the computing node as the input sample, it needs to determine whether the input sample has been calculated by the elastic layer of the predecessor node to determine whether all the regular layers in the computing node should be used to calculate the input sample, so as to avoid errors caused by repeated calculations using model layers. If the input sample has not been calculated by the elastic layer of the predecessor node, all the regular layers of the computing node can be used to calculate the input sample. If the input sample has been calculated by the elastic layer of the predecessor node, it is necessary to use the independent layer of the computing node - the model layer in the regular layer of the computing node that is different from the elastic layer of the predecessor node to calculate the input sample. Figure 4 As shown in the figure, the C computing node is the predecessor node of the D computing node, the regular layers of the C computing node are 1-3 layers, the elastic layers are 4-5 layers, the regular layers of the D computing node are 4-6 layers, and the 6th model layer is the independent layer of the D computing node. If the output result transmitted by the C computing node to the D computing node is calculated by the elastic layer of the C computing node, the output result of the C computing node is the input sample of the D computing node. When the D computing node uses the regular layer to calculate the input sample, it can use the independent layer for calculation.
[0077] Among them, for any computing node, it is determined whether the input sample of the computing node has been calculated by the elastic layer of the predecessor node of the computing node. When the predecessor node of the computing node calculates any batch of training samples using the elastic layer, the batch number of the batch of training samples and the elastic layer usage notification corresponding to the batch number are sent to the computing node. Therefore, when the computing node calculates the input sample, if the computing node does not store the elastic layer usage notification of the batch corresponding to the input sample, it can be determined that the training samples of the batch have been calculated by the elastic layer of the predecessor node of the computing node.
[0078] In addition, in Figure 1 In step S104, the output result and the elastic layer usage identifier corresponding to the output result are transmitted to the post-position node. That is, for any computing node, the computing node can transmit the output result of the computing node and the elastic layer usage identifier corresponding to the output result to the post-position node.
[0079] Thus, in Figure 1 Before step S100 shown, it is determined whether the input sample has been calculated by the elastic layer of the predecessor node according to the elastic layer usage identifier corresponding to the input sample. For example, the elastic layer usage identifier can be: 0-unused, 1-used, and other common representations. Therefore, when the computing node receives the output result of the predecessor node and the elastic layer usage identifier, it can determine whether the output result has been calculated by the elastic layer of the predecessor node according to the elastic layer usage identifier. It should be noted that the output result of the predecessor node here is the input sample of the computing node.
[0080] In addition, in Figure 1 In step S104, the batch of training samples corresponding to the output result is determined as the completed batch of the computing node, and the completed batch is transmitted to the predecessor node of the computing node. For any computing node, after the computing node completes the calculation of the current batch of training samples, the batch can be regarded as the completed batch, and the batch number of the completed batch can be sent to the predecessor node of the computing node. Figure 1 In S102 shown, it is determined whether the subsequent node of the computing node has completed the calculation of the input samples of the previous batch of the current batch according to the completed batch of the subsequent node of the computing node and the current batch. For any computing node, the subsequent node of the computing node will send the completed batch corresponding to the computing task of each batch to the computing node after completing the calculation task of the batch, so that after the computing node completes the calculation of the input samples of the current batch, it can determine whether the subsequent node has completed the calculation of the input samples of the previous batch of the current batch according to the completed batch of the subsequent node of the computing node stored in the computing node.
[0081] Therefore, for the same batch of training samples, using the method provided in this specification can ensure that no repeated model layers are used for calculation in pipeline parallel calculations, thereby avoiding possible calculation errors.
[0082] In addition, in Figure 1 After step S102 shown, when the model layer parameters are updated, for any computing node, the computing node updates the parameters of the regular layer of the computing node according to the final calculation result of the model and the preset loss function, and can also update the parameters of the elastic layer of the computing node according to the updated parameters of the regular layer of the subsequent node of the computing node.
[0083] When the highest-numbered computing node completes the computing task for the current batch of training samples, the final computing result of the current model for the current batch of training samples can be obtained. Each computing node can update the parameters of its own layer in descending order according to the final computing result and the preset loss function (this process can refer to the reverse transfer process in the model parameter update). For any computing node, after updating its own regular parameters, the computing node can transmit the model layer parameters of its regular layer with the same number of layers as the elastic layer of the predecessor node of the computing node to the predecessor node of the computing node, so that the predecessor node of the computing node updates the parameters of the elastic layer according to the parameters transmitted by the computing node.
[0084] In one or more embodiments of the present specification, for a model group, several computing nodes that deploy the same model group can be preset, that is, several computing nodes that deploy the same conventional layer and elastic layer can be preset, as a group of data parallel groups, the number of each computing node in a data parallel group is the same, for a group of data parallel groups, when calculating training samples, each computing node in the data parallel group obtains the calculation result of the data parallel group according to the training samples of the current batch and the conventional layer deployed on the group of data parallel groups, if the rear node of the data parallel group has completed the calculation of the training samples of the previous batch, the calculation result is used as the output result and transmitted to the rear node of the group of data parallel groups, otherwise, each computing node in the data parallel group obtains the output result of the computing node according to the calculation result and the elastic layer of the data parallel group, and transmits the output result to the rear node of the group of data parallel groups.
[0085] In one or more embodiments of the present specification, the method provided in the present specification is used for training a large language model, the large language model includes several model layers, the model layers are deployed in several computing nodes, the model layers deployed in each computing node include a regular layer and an elastic layer, the elastic layer is determined according to the regular layer of the post-node of the computing node, the method includes:
[0086] For any computing node, the computing result of the computing node is obtained according to the multimodal training samples of the current batch and the conventional layer deployed on the computing node;
[0087] If the post-node of the computing node has completed the calculation of the previous batch of multimodal training samples, the calculation result is used as the output result; otherwise, the output result of the computing node is obtained according to the calculation result and the elastic layer of the computing node;
[0088] The output result is transmitted to the subsequent node.
[0089] Among them, multimodal training samples can be obtained by encoding and aligning multimodal data. Multimodal data may include text data, audio data, image data, etc. Common encoding methods can be used to encode multimodal data. For example, text data is encoded using a Doc2Vec encoder, image data is encoded using an NFNet-F6 encoder, and audio data is encoded using a C-Former encoder. The vectors obtained after encoding different modalities are aligned to a unified vector length range to obtain multimodal training samples.
[0090] The above is a calculation method in parallel training provided in one or more embodiments of this specification. Based on the same idea, this specification also provides a corresponding calculation device in parallel training, such as Figure 5 shown.
[0091] Figure 5 A schematic diagram of a computing device in parallel training provided in this specification, wherein the device is applied to model parallel training, wherein all model layers of the model are used as regular layers, wherein the regular layers are deployed in a plurality of computing nodes, wherein at least one regular layer is deployed in each computing node, and the regular layers deployed in each computing node are different from each other, and an elastic layer is also deployed in the computing node, and for any computing node, the elastic layer of the computing node is at least one regular layer of a post-node of the computing node, wherein the device comprises:
[0092] The computing module 500 obtains the computing result of any computing node according to the current batch of training samples and the conventional layer deployed on the computing node;
[0093] The elastic module 502 uses the calculation result as the output result if the post-node of the computing node has completed the calculation of the training samples of the previous batch of the current batch; otherwise, obtains the output result of the computing node according to the calculation result and the elastic layer of the computing node;
[0094] The transmission module 504 transmits the output result to the subsequent node.
[0095] Optionally, the computing module 500 is specifically used to: when the computing node has a predecessor node, use the output result transmitted by the predecessor node as the input sample of the computing node, and obtain the computing result of the computing node according to the input sample and the normal layer of the computing node.
[0096] Optionally, the computing module 500 is specifically used to: if the input sample has not been calculated by the elastic layer of the predecessor node, obtain the computing result of the computing node according to the input sample and the conventional layer deployed on the computing node; otherwise, obtain the computing result of the computing node according to the input sample and the independent layer of the computing node, wherein the independent layer is a model layer in the conventional layer of the computing node that is different from the elastic layer of the predecessor node.
[0097] Optionally, the transmission module 504 is specifically configured to: transmit the output result and the elastic layer usage identifier corresponding to the output result to the post-node;
[0098] The elastic module 502 is further configured to determine whether the input sample has been subjected to elastic layer calculation of the preceding node according to an elastic layer usage identifier corresponding to the input sample.
[0099] Optionally, the transmission module 504 is further used to: determine a batch of training samples corresponding to the output result as a completed batch of the computing node, and transmit the completed batch to a preceding node of the computing node;
[0100] The elastic module 502 is further used to determine whether the subsequent node of the computing node has completed the calculation of the input samples of the previous batch of the current batch according to the completed batches of the subsequent node of the computing node and the current batch.
[0101] Optionally, the computing module 500 is further used to: when updating the model layer parameters, for any computing node, the computing node updates the parameters of the conventional layer of the computing node according to the final calculation result of the model and a preset loss function;
[0102] Optionally, the elastic module 502 is further configured to update the parameters of the elastic layer of the computing node according to the updated parameters of the regular layer of the subsequent node of the computing node.
[0103] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 Provides computational methods for parallel training.
[0104] This manual also provides Figure 6 The schematic structure diagram of the electronic device shown in FIG. Figure 6 As mentioned above, at the hardware level, the computing device in the parallel training includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and of course may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1The calculation method in the parallel training described above. Of course, in addition to the software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.
[0105] In the 1990s, it was very clear whether the improvement of a technology was hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with hardware entity modules. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.
[0106] The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, and the memory controller may also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that, in addition to implementing the controller in a purely computer-readable program code manner, the controller may be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller may be considered as a hardware component, and the devices for implementing various functions included therein may also be considered as structures within the hardware component. Or even, the devices for implementing various functions may be considered as both software modules for implementing the method and structures within the hardware component.
[0107] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
[0108] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.
[0109] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0110] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0111] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0112] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0113] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0114] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0115] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0116] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0117] It should be understood by those skilled in the art that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0118] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.
[0119] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0120] The above description is only an embodiment of this specification and is not intended to limit this specification. For those skilled in the art, this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification should be included in the scope of the claims of this application.
Claims
1. A computing method in parallel training, characterized in that: The method is applied to model parallel training, all model layers of the model are used as regular layers, the regular layers are deployed in a number of computing nodes, at least one regular layer is deployed in each computing node, the regular layers deployed in each computing node are different from each other, an elastic layer is also deployed in the computing node, for any computing node, the elastic layer of the computing node is at least one regular layer of a post-node of the computing node, the method includes: For any computing node, the computing result of the computing node is obtained according to the current batch of training samples and the conventional layer deployed on the computing node; If the post-node of the computing node has completed the computing of the training samples of the previous batch of the current batch, the computing result is used as the output result; otherwise, the output result of the computing node is obtained according to the computing result and the elastic layer of the computing node; The output result is transmitted to the subsequent node.
2. The method according to claim 1, characterized in that According to the current batch of training samples and the conventional layers deployed on the computing node, the computing results of the computing node are obtained, including: When the computing node has a predecessor node, the output result transmitted by the predecessor node is used as the input sample of the computing node; A calculation result of the calculation node is obtained according to the input sample and the normal layer of the calculation node.
3. The method according to claim 2, characterized in that According to the input sample and the conventional layer of the computing node, a computing result of the computing node is obtained, which specifically includes: If the input sample has not been calculated by the elastic layer of the predecessor node, the calculation result of the computing node is obtained according to the input sample and the conventional layer deployed on the computing node; otherwise, the calculation result of the computing node is obtained according to the input sample and the independent layer of the computing node, wherein the independent layer is a model layer in the conventional layer of the computing node that is different from the elastic layer of the predecessor node.
4. The method according to claim 3, characterized in that Transmitting the output result to the post-node specifically includes: Transmitting the output result and the elastic layer usage identifier corresponding to the output result to the post-node; Before obtaining the calculation result of the calculation node, the method further includes: According to the elastic layer usage identifier corresponding to the input sample, it is determined whether the input sample has been calculated by the elastic layer of the preceding node.
5. The method according to claim 1, characterized in that After transmitting the output result to the post-node, the method further comprises: Determine a batch of training samples corresponding to the output result as a completed batch of the computing node; Transmitting the completed batch to a predecessor node of the computing node; After obtaining the calculation result of the calculation node, the method further includes: According to the completed batches of the subsequent nodes of the computing node and the current batch, it is determined whether the subsequent nodes of the computing node have completed the calculation of the input samples of the previous batch of the current batch.
6. The method according to claim 1, characterized in that Also includes: When updating the model layer parameters, for any computing node, the computing node updates the parameters of the conventional layer of the computing node according to the final calculation result of the model and the preset loss function.
7. The method according to claim 6, characterized in that Also includes: For any computing node, the computing node updates the parameters of the elastic layer of the computing node according to the updated parameters of the regular layer of the subsequent node of the computing node.
8. A computing device for parallel training, characterized in that: The device is applied to model parallel training, and all model layers of the model are used as regular layers. The regular layers are deployed in a plurality of computing nodes, and at least one regular layer is deployed in each computing node. The regular layers deployed in each computing node are different from each other. An elastic layer is also deployed in the computing node. For any computing node, the elastic layer of the computing node is at least one regular layer of a post-node of the computing node. The device includes: The computing module obtains the computing result of any computing node according to the current batch of training samples and the conventional layer deployed on the computing node; The elastic module uses the calculation result as the output result if the post-node of the computing node has completed the calculation of the training samples of the previous batch of the current batch; otherwise, obtains the output result of the computing node according to the calculation result and the elastic layer of the computing node; A transmission module transmits the output result to the post-node.
9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 7 is implemented.
10. A device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method described in any one of claims 1 to 7 is implemented.
