Method and apparatus for parallel training of neural network models

Abstract

A method and apparatus for parallel training of a neural network model are provided. The method includes: generating a plurality of partial data sets by dividing training data for a neural network model; identifying a plurality of pipeline stages of the neural network model; determining a plurality of save policies, where each of the plurality of save policies corresponds to a value generated by a selected one of the plurality of pipeline stages using a selected one of the plurality of partial data sets; and training a neural network model by processing the values generated by the selected pipeline stage based on the selected partial data set according to a respective save policy for each of the plurality of save policies.

Description

[0001] This application claims priority based on and claims the benefit of Korean Patent Application No. 10-2023-0185702, filed with the Korean Intellectual Property Office on December 19, 2023, the content of which is incorporated herein by reference in its entirety. Technical Field

[0002] The following disclosure relates to methods and devices for parallel training of neural network models. Background Art

[0003] In recent years, the fields of machine learning and artificial intelligence have witnessed rapid development mainly driven by the development and deployment of deep neural network models. The models have shown remarkable performance in various tasks such as image recognition, natural language processing, and reinforcement learning. However, one of the main challenges associated with training deep neural networks is the required computational complexity and time, especially when dealing with large-scale datasets and complex architectures. Traditional sequential training methods often result in long training times and inefficient resource utilization, hindering the scalability and practical deployment of deep learning systems.

[0004] In some cases, parallel training techniques can utilize distributed computing resources such as multi-core CPUs, GPUs (graphics processing units), and dedicated accelerators (e.g., TPU - tensor processing units) to accelerate the training process. By distributing the workload simultaneously across multiple processing units, parallel training can significantly reduce the training time and enhance scalability, effectively ensuring the training of large models and the processing of large-scale datasets. However, the design and implementation of effective parallel training algorithms and architectures pose technical and computational challenges, including optimized communication overhead, load balancing, and synchronization between distributed computing nodes. Therefore, there is a need in the art for advanced systems and methods for parallel training of neural network models. Summary of the Invention

[0005] The Summary of the Invention is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. The Summary of the Invention is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to help determine the scope of the claimed subject matter.

[0006] This disclosure describes systems and methods for parallel training of neural network models. Embodiments of this disclosure are configured to perform adaptive activation recomputation in pipeline parallelism. In some cases, adaptive recomputation can achieve a reduction in training time by improving memory efficiency. According to an embodiment, a neural network model can be divided into partial models that can be trained using multiple pipeline stages.

[0007] In some cases, the methods and systems of the present disclosure include computing the activation size of each in a partial model during a forward propagation operation. According to an embodiment, one or more layer gradients may be computed based on the activations during a backpropagation process. In some cases, during parallel processing, the activations may be recomputed, selectively recomputed, or not recomputed for a data set at a pipeline stage among multiple pipeline stages. In some cases, the total learning time of a neural network may be reduced by adaptively applying an activation recomputation operation.

[0008] According to an embodiment, a method for parallel training includes: generating a plurality of partial data sets by partitioning training data for a neural network model; identifying a plurality of pipeline stages of the neural network model; determining a plurality of save policies, wherein each of the plurality of save policies corresponds to a value generated by a selected pipeline stage among the plurality of pipeline stages using a selected partial data set among the plurality of partial data sets; and training the neural network model by processing the value generated by the selected pipeline stage based on the selected partial data set according to the corresponding save policy for each of the plurality of save policies.

[0009] According to an embodiment, an electronic device includes: a processor; and a memory configured to store instructions, wherein when executed by the processor, the instructions cause the electronic device to: generate a plurality of partial data sets by partitioning training data for a neural network model; identify a plurality of pipeline stages of the neural network model; determine a plurality of save policies, wherein each of the plurality of save policies corresponds to a value generated by a selected pipeline stage among the plurality of pipeline stages using a selected partial data set among the plurality of partial data sets; and train the neural network model by processing the value generated by the selected pipeline stage based on the selected partial data set according to the corresponding save policy for each of the plurality of save policies.

[0010] According to an embodiment, a method for training a neural network model includes: computing a first value at a first pipeline stage of the neural network model based on a first partial data set; processing the first value according to a first save policy based on the first pipeline stage and the first partial data set; computing a second value at the first pipeline stage of the neural network model based on a second partial data set; processing the second value according to a second save policy based on the first pipeline stage and the second partial data set; computing a third value at a second pipeline stage of the neural network model based on the first partial data set; processing the third value according to a third save policy based on the second pipeline stage and the first partial data set; computing a fourth value at the second pipeline stage of the neural network model based on the second partial data set; and processing the fourth value according to a fourth save policy based on the second pipeline stage and the second partial data set.

[0011] Other features and aspects will be apparent from the following detailed description, the drawings, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 An example of a pipelined parallelization operation without recomputation according to an embodiment is shown.

[0013] Figure 2 A training process of a neural network model using gradients according to an embodiment is shown.

[0014] Figure 3 A graph showing an example of memory usage for each pipeline stage without recomputation according to an embodiment.

[0015] Figure 4 An example of an activation keeping policy according to an embodiment is shown.

[0016] Figure 5 An example of the structure of a neural network model according to an embodiment is shown.

[0017] Figure 6 An example of a pipelined parallelization operation including recomputation according to an embodiment is shown.

[0018] Figure 7 A flowchart showing an example of a process for selecting a saving policy using priorities according to an embodiment.

[0019] Figure 8 An example of applying the priority of a saving policy according to an embodiment is shown.

[0020] Figure 9 A flowchart showing an example of a parallel training method according to an embodiment.

[0021] Figure 10 An example of the configuration of an electronic device is shown.

[0022] Figure 11 A flowchart showing an example of a parallel training process according to an embodiment.

[0023] Figure 12 A flowchart showing an example of a neural network training process according to an embodiment.

[0024] Throughout the drawings and the detailed description, unless otherwise described or provided, the same reference numerals will be understood to refer to the same elements, features, and structures. The drawings may not be drawn to scale, and for clarity, illustration, and convenience, the relative dimensions, proportions, and depictions of the elements in the drawings may be exaggerated. DETAILED DESCRIPTION

[0025] The technical automation of recognition processing can be achieved by implementing a neural network model by a processor serving as a dedicated computing structure. The model can provide a computationally intuitive mapping between input patterns and output patterns based on extensive training. In some cases, the ability of a neural network model trained to generate such a mapping can be referred to as the training ability of the neural network. Additionally, such a dedicated and well-trained neural network based on training can include a generalization ability that provides the generation of accurate outputs for input patterns. In some cases, the output corresponding to the input pattern may not be part of the training data set. In some cases, methods such as model parallelization and / or data parallelization can be employed to achieve the rapid convergence of the neural network towards the desired result and to process training and inference operations. This disclosure describes systems and methods for parallel training of neural network models. Embodiments of this disclosure are configured to perform adaptive activation recomputation in the case of pipeline parallelism. In some cases, adaptive recomputation can achieve a reduction in training time by improving memory efficiency. According to an embodiment, a neural network model can be divided into partial models that can be trained using multiple pipeline stages. In one example embodiment, the training data can include multimedia data. For example, the multimedia data can include one or more of text data, voice data, and image data. The neural network model can perform corresponding inference operations based on the input multimedia data.

[0026] In some cases, the methods and systems of this disclosure include: calculating the activation size of each in a partial model during the forward propagation operation of neural network training processing. According to an embodiment, one or more layer gradients can be calculated based on the activation during the backpropagation operation of neural network training processing. In some cases, during parallelization processing, the activation can be recomputed, selectively recomputed, or not recomputed for each of the multiple pipeline stages. In some cases, the total learning time of the neural network can be reduced by adaptively applying the activation recomputation operation.

[0027] Conventionally used techniques for parallel training can calculate the memory capacity occupied by the size of the activation in the training method of large deep learning models. In some cases, conventional systems can only generate the activation size. However, such methods may not be able to calculate the activation size in different cases and the variation of the activation recomputation range for each pipeline stage. Additionally, such methods may not be able to calculate the activation recomputation range for each pipeline stage that achieves the optimization of memory usage efficiency and training time.

[0028] In contrast, embodiments of this disclosure include an adaptive activation recomputation method applicable to the training of large models based on multiple pipeline stages under pipeline parallelism. For example, an embodiment can be configured to calculate the activation recomputation range for each pipeline stage based on a full recomputation, selective recomputation, or no recomputation strategy.

[0029] According to an embodiment, the method disclosed herein may be configured to maximize memory usage and reduce training time by selecting a policy for each pipeline stage. In some cases, the training process for a neural network model may include a forward propagation operation and a backpropagation operation. According to an embodiment, the activations obtained during forward propagation may be used to compute gradients during backpropagation. In some cases, an activation checkpoint may indicate a training method in which the activations generated during forward propagation are not stored in memory until backpropagation to prevent memory shortage, and the activations may be recomputed during backpropagation.

[0030] According to an embodiment, the activation checkpoint may include a full recomputation policy, a selective recomputation policy, and a non-recomputation policy. In some cases, a policy may be selected for each data set of each pipeline stage. In some cases, a policy may be selected to maximize memory usage. Embodiments of the present invention may include priorities for policy selection. In some cases, priorities may be assigned in the order of the non-recomputation policy, the selective recomputation policy, and the full recomputation policy (i.e., what is referred to herein as the keeping policy). In the case where data sets compete, the closest data set may be selected. In some examples, a policy may be selected to minimize training time.

[0031] Embodiments of the present disclosure include a method for pipeline parallelism that may reduce the size of a neural network model. In some cases, for example, when the size of the model to be trained is large compared to the availability of memory (such as GPU memory), the model may be partitioned into multiple layer units. According to an embodiment, in the case of a pipeline parallelization method that alternates forward propagation processing and backpropagation processing, the forward computed values may be sequentially transmitted from the first stage to the last stage. In some cases, backpropagation may be started, and backpropagation may transmit the computation results of each unit from the last stage to the first stage.

[0032] According to an embodiment, since the activation results from the forward operation may be used to compute gradients during the back operation, the forward operation values may be stored during the training process until the back operation starts. In some cases, when the number of partial data sets is high, multiple activation results may be accumulated in the initial pipeline stage. Thus, the early pipeline stages may include a large amount of memory usage.

[0033] Embodiments of the present disclosure are configured to perform activation checkpointing. In some cases, activation checkpointing may refer to a method in which activation result values used for gradient calculation during backpropagation may not be stored until the start of backpropagation. In some cases, activation values may be recomputed during the backpropagation operation. In some examples, the scope of activation recomputation may be optimized (e.g., selectively optimized) to reduce the recomputation cost (e.g., the recomputation cost may affect the total training time).

[0034] Accordingly, embodiments of the present disclosure may be configured to calculate the activation sizes for each activation recomputation scope. For example, the activation sizes may be calculated based on model parallelization during the model construction phase (i.e., before actual training is performed). In some examples, the activation sizes may be calculated based on the application of model parallelization (e.g., tensors, pipelining, sequences, etc.). In some cases, the memory size availability of each partial dataset in each pipeline stage may be performed.

[0035] In some cases, the activation recomputation scope may be set based on priorities (such as no recomputation policy, then selective recomputation policy, and finally full recomputation policy). In some examples, in the case where the partial dataset is not the last in the dataset index and there is available memory space, since the total training time depends on the training of the last partial dataset in pipeline parallelization, the last partial dataset may be preferentially assigned the no recomputation policy.

[0036] Embodiments of the present disclosure provide a method for parallel training of a neural network. In some cases, the method includes generating a plurality of partial datasets by partitioning the training data of a neural network model. Additionally, the method identifies a plurality of pipeline stages of the model and determines a plurality of save policies. As described herein, a corresponding save policy among the plurality of save policies corresponds to a value generated using a selected partial dataset among the plurality of partial datasets by a selected pipeline stage among the plurality of pipeline stages. Finally, the neural network model may be trained by processing the value generated by the selected pipeline stage based on the selected partial dataset according to the corresponding save policy.

[0037] Additionally, according to an embodiment, a method of training a neural network includes calculating a set of values corresponding to pipeline stages of a neural network model based on partial datasets. In some cases, each of the calculated set of values may be further processed according to a save policy based on the corresponding pipeline stage and partial dataset.

[0038] Thus, embodiments of the present disclosure can achieve an increase in memory usage efficiency by applying activation recomputation ranges for each pipeline stage during pipeline parallelization processing. Additionally, by adaptively applying the activation recomputation order in the case of a large number of partial data sets, the embodiments are able to achieve a reduction in the total learning time, which helps reduce the associated training costs.

[0039] The following detailed structural or functional descriptions are provided only as examples, and various changes and modifications can be made to the examples. Here, the examples should not be construed as limiting the present disclosure, but should be understood to include all changes, equivalents, and alternatives within the spirit and technical scope of the disclosure.

[0040] Terms (such as first, second, etc.) may be used herein to describe components. Each of these terms does not define the essence, order, or sequence of the corresponding component, but is only used to distinguish the corresponding component from one or more other components. For example, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

[0041] It should be noted that if a component is described as "connected", "coupled", or "joined" to another component, although the first component may be directly connected, coupled, or joined to the second component, a third component may be "connected", "coupled", and "joined" between the first component and the second component.

[0042] Unless the context clearly indicates otherwise, the singular forms are also intended to include the plural forms. It will also be understood that the terms "comprises" and / or "comprising", when used herein, specify the presence of the stated features, wholes, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or groups thereof.

[0043] As used herein, "at least one of A and B", "at least one of A, B, or C", etc., each of "at least one of A and B", "at least one of A, B, or C", etc. may include: any one of the items listed together in the corresponding phrase, or all possible combinations thereof.

[0044] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. It will also be understood that terms (such as those defined in a common dictionary) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and will not be interpreted in an idealized or overly formal sense.

[0045] In the following, examples will be described in detail with reference to the accompanying drawings. When describing embodiments with reference to the accompanying drawings, the same reference numerals indicate the same elements, and repeated descriptions related thereto will be omitted.

[0046] Figure 1 An example of a pipelined parallelization operation without recalculation according to an embodiment is shown. As Figure 1 shown, the horizontal axis may represent different time periods, and the vertical axis may represent different pipeline stages. Referring to Figure 1 , the first pipeline stage 110, the second pipeline stage 120, the third pipeline stage 130, and the fourth pipeline stage 140 may be used to perform parallel training of a neural network model. The first pipeline stage 110 to the fourth pipeline stage 140 may represent different parts of the neural network model that can operate sequentially. For example, each of the first pipeline stage 110 to the fourth pipeline stage 140 may include different layers or blocks of the neural network model.

[0047] Machine learning parameters (also referred to as model parameters or weights) are variables that provide the behavior and characteristics of a machine learning model. Machine learning parameters can be learned or estimated from training data and are used to make predictions or perform tasks based on the learned patterns and relationships in the data. Machine learning parameters are typically adjusted during the training process to minimize a loss function or maximize a performance metric. The goal of the training process is to find the optimal values of the parameters that allow the machine learning model to make accurate predictions or perform well on a given task.

[0048] For example, during the training process, an algorithm adjusts the machine learning parameters according to optimization techniques such as gradient descent, stochastic gradient descent, or other optimization algorithms to minimize the error or loss between the predicted output and the actual target. Once the machine learning parameters are learned from the training data, the machine learning parameters are used to make predictions on new unseen data.

[0049] An artificial neural network (ANN) has many parameters including weights and biases associated with each neuron in the network, and many parameters control the degree of connection between neurons and affect the ability of the neural network to capture complex patterns in the data. An ANN is a hardware component or a software component that includes a plurality of connected nodes (i.e., artificial neurons) that loosely correspond to the neurons in the human brain. Each connection or edge transmits a signal from one node to another (similar to the physical synapses in the brain). When a node receives a signal, the node processes the signal and then sends the processed signal to other connected nodes.

[0050] In some cases, the signals between nodes include real numbers, and the output of each node is calculated as a function of the sum of the inputs to each node. In some examples, a node may use other mathematical algorithms (such as selecting the maximum value from the inputs as the output, or any other suitable algorithm for activating the node) to determine the output of the node. Each node and edge is associated with one or more node weights that determine how to process and send signals.

[0051] In an ANN, the hidden (or intermediate) layer includes hidden nodes and is located between the input layer and the output layer. The hidden layer performs a non-linear transformation of the inputs into the network. Each hidden layer is trained to produce a defined output that contributes to the combined output of the output layer of the ANN. A hidden representation is a machine-readable data representation of the inputs learned from the hidden layer of the ANN and produced by the output layer. As the ANN's understanding of the inputs improves as the ANN is trained, the hidden representation gradually differentiates from earlier iterations.

[0052] During the training process of the ANN, the node weights are adjusted to improve the accuracy of the results (i.e., by minimizing a loss that somehow corresponds to the difference between the current result and the target result). The weights of the edges increase or decrease the strength of the signals transmitted between nodes. In some cases, a node has a threshold below which the signal is not transmitted at all. In some examples, nodes are aggregated into layers. Different layers perform different transformations on the inputs to different layers. The initial layer is called the input layer, and the last layer is called the output layer. In some cases, the signals traverse a particular layer multiple times.

[0053] Referring again to Figure 1 , the first pipeline stage 110 to the fourth pipeline stage 140 may be executed by different processors. For example, the processor may include at least one of a graphics processing unit (GPU), a neural processing unit (NPU), and an accelerator. For example, the first pipeline stage 110 to the fourth pipeline stage 140 may be executed by a first GPU, a second GPU, a third GPU, and a fourth GPU. Although Figure 1 four first pipeline stages 110 to 140 are shown, embodiments are not limited thereto.

[0054] According to an embodiment, the neural network model and the training dataset may be divided based on the first pipeline stage 110 to the fourth pipeline stage 140. For example, the neural network model may be divided into partial models, and the training dataset may be divided into partial datasets. For example, the layers of the neural network model may be divided into partial models, and the weights of the neural network model may be divided into partial models. In some cases, the partial dataset may correspond to (or be referred to as) a micro-batch. Hereinafter, a representative example in which the partial model includes partial layers of the neural network model will be described. However, embodiments are not limited thereto.

[0055] Partial models can be trained in the first pipeline stage 110 to the fourth pipeline stage 140. For example, a neural network model can be divided into a first partial model, a second partial model, a third partial model, and a fourth partial model, where the first partial model to the fourth partial model can be trained in the first pipeline stage 110 to the fourth pipeline stage 140. For example, the first partial model to the fourth partial model can be trained by the first GPU to the fourth GPU in the first pipeline stage 110 to the fourth pipeline stage 140.

[0056] According to an embodiment, partial data sets can be trained in the first pipeline stage 110 to the fourth pipeline stage 140. For example, as Figure 1 shown, the training data set can be divided into a first partial data set to an eighth partial data set. The partial data sets can be trained sequentially and in parallel using a pipeline method. For example, a first corresponding data set of the first partial data set can be trained in the first pipeline stage 110. Then, based on the training operation result of the first corresponding data set of the first partial data set in the first pipeline stage 110, a second corresponding data set of the first partial data set can be trained in the second pipeline stage 120.

[0057] In some cases, the first partial data set can be represented in different states in the first pipeline stage 110 to the fourth pipeline stage 140. For example, the representation of the first partial data can depend on the training operation result. In some cases, the partial data set of each state can be referred to as a corresponding data set. For example, when training the i-th partial data set in the j-th pipeline stage, the i-th partial data set can be referred to as the j-th corresponding data set of the i-th partial data set. As used herein, i can be a natural number between 1 and I, and j can be a natural number between 1 and J. Referring to Figure 1 the example, I can be 8, and J can be 4. As Figure 1 shown, the numbers 1 to 8 can represent the index value i values of the partial data sets.

[0058] The training of the neural network model can include forward propagation and backward propagation. According to forward propagation, the training operation result can be an activation. According to an embodiment, when the forward propagation using each partial data set is completed, backward propagation can be performed using each partial data set. For example, referring to Figure 1 the training process of the partial data set identified as value 1 in

[0059] Gradients can be used to train a neural network model. Activations can be generated through forward propagation, and the activations can be used to calculate gradients in backpropagation. The activations generated during forward propagation can be saved in memory and used to calculate gradients. All or a part of the activations can be stored in memory. In some cases, since the calculation of the activations is not separately required when calculating the gradients, the training time can be reduced. In some cases, since the memory space is used by the activations before calculating the gradients, when the size of the activations generated during forward propagation is large, a memory shortage may occur.

[0060] In some cases, a strategy of recalculating the activations when calculating the gradients can be used, that is, instead of the strategy of saving all or a part of the activations in memory to calculate the gradients. In this case, when the activations are generated in forward propagation, the memory space occupied by the activations can be released. The released memory space can be used by other data. In some cases, when the activations are required to calculate the gradients during backpropagation, the activations can be newly generated by recalculation. Therefore, since the memory space occupied by the activations can be minimized, a memory shortage can be prevented. In some cases, when calculating the gradients, the recalculation of the activations can increase the training time.

[0061] According to an embodiment, a saving strategy for the activations generated by the forward propagation of a partial model can be separately selected for each corresponding dataset in a partial dataset processed in the first pipeline stage 110 to the fourth pipeline stage 140. In some cases, the activations can be used to calculate the gradients of the backpropagation of a partial model of the neural network model. For example, the first saving strategy in the saving strategies can be used to train the corresponding dataset of the first partial dataset in the partial dataset in the first pipeline stage 110, and the second saving strategy different from the first saving strategy in the saving strategies can be used to train the corresponding dataset of the second partial dataset in the partial dataset in the first pipeline stage 110. According to an embodiment, the saving strategy can be selected based on the memory usage status. In some cases, the saving strategy can be selected to maximize the memory usage within the available memory capacity.

[0062] In some cases, when the selection of the saving strategy is completed, a strategy for parallel training of the neural network model can be generated based on the selected saving strategy. For example, the strategy can include information about the first pipeline stage 110 to the fourth pipeline stage 140, a partial model of the neural network model, a partial dataset of the training dataset, the saving strategy, or a combination thereof. The neural network model can be trained based on the strategy.

[0063] Figure 2 Illustrates a training process of a neural network model using gradients according to an embodiment. Refer to Figure 2, the neural network model 210 may include an n-2 layer 211, an n-1 layer 212, and an n layer 213. The layers 211, 212, and 213 may each include weights, and the weights may be updated during the training process. Each of the weights may be referred to as a parameter. As used herein, n may represent the number of layers of the neural network model 210. Figure 2 An example where n = 3 is shown, but the embodiments are not limited thereto.

[0064] Forward propagation operations 221, 222, and 223, activation gradient calculation operations 231, 232, and 233, layer gradient calculation operations 241, 242, and 243, and model update operations 251, 252, and 253 may be performed to train the neural network model 210. The activation gradient calculation operations 231, 232, and 233 and the layer gradient calculation operations 241, 242, and 243 may correspond to backpropagation operations.

[0065] In some cases, the forward propagation operation 221 may generate a hidden activation H1 by executing the n-2 layer 211 based on the input data I. Additionally, the forward propagation operation 222 may generate a hidden activation H2 by executing the n-1 layer 212 based on the hidden activation H1. Additionally, the forward propagation operation 223 may generate an output activation O by executing the n layer 213 based on the hidden activation H2.

[0066] Activation gradient data AG1 may be determined based on the comparison result between the output activation O and the ground truth GT. In some cases, the activation gradient data AG1 may correspond to a loss. The activation gradient calculation operation 231 may use the activation gradient data AG1 and the n layer 213 to generate activation gradient data AG2. Additionally, the activation gradient calculation operation 232 may use the activation gradient data AG2 and the n-1 layer 212 to generate activation gradient data AG3. Additionally, the activation gradient calculation operation 233 may use the activation gradient data AG3 and the n-2 layer 211 to generate activation gradient data AG4.

[0067] The layer gradient calculation operation 241 may use the activation gradient data AG1 and the hidden activation H2 to generate layer gradient data LG1. Similarly, the layer gradient calculation operation 242 may use the activation gradient data AG2 and the hidden activation H1 to generate layer gradient data LG2. Additionally, the layer gradient calculation operation 243 may use the activation gradient data AG3 and the input data I to generate layer gradient data LG3. The gradient may be represented as ΔW, where W may represent a weight. The term "gradient" may indicate a layer gradient. For example, as described herein, a layer gradient may indicate the gradient of a loss function with respect to the parameters (weights and biases) of a particular layer within the network. In some cases, a layer gradient may represent the rate of change of a loss function with respect to the parameters of that layer during the process of training a neural network using backpropagation.

[0068] In the case of a backpropagation operation, the activations H1, H2, and O generated by the forward propagation operations 221, 222, and 223 can be used to calculate layer gradients. According to an embodiment, one or more of the activations H1, H2, and O can be stored in a memory based on a saving strategy.

[0069] The model update operation 251 can update the weights of the nth layer 213 based on the layer gradient data LG1. Additionally, the model update operation 252 can update the weights of the (n - 1)th layer 212 based on the layer gradient data LG2. Additionally, the model update operation 253 can update the weights of the (n - 2)th layer 211 based on the layer gradient data LG3. The model update operations 251, 252, and 253 can be expressed as W - (γ × ΔW). Here, γ can represent the learning rate. The trained nth layer 261, the trained (n - 1)th layer 262, and the trained (n - 2)th layer 263 can be generated based on the model update operations 251, 252, and 253.

[0070] Figure 3 An example of the memory usage for each pipeline stage without performing recomputation according to an embodiment is shown. Referring to Figure 3 , when advancing from an initial pipeline stage (e.g., Figure 1 the first pipeline stage 110 of Figure 1 ) to a final pipeline stage (e.g.,

[0071] Figure 4 the fourth pipeline stage 140 of Figure 4 ), the memory usage can gradually decrease. Thus, as the pipeline stage approaches the initial pipeline stage, the interval between the time point of forward propagation and the time point of backpropagation can increase. Additionally, the activations to be saved from the time point of completing forward propagation to the time point of backpropagation can increase. According to an embodiment, the saving strategy of activations can be individually selected for each corresponding dataset of each pipeline stage using the memory usage characteristics described herein.

[0072] A save policy to be used can be determined from multiple save policies based on the memory usage status. In one example, a save policy to be used can be determined from multiple save policies based on the memory usage status so that the memory usage is optimized. In Figure 4 the example, the first save policy 411 to the third save policy 413 can respectively correspond to the save policy to be used for dataset 1, the save policy to be used for dataset 2, and the save policy to be used for dataset 3. The first save policy 411 to the third save policy 413 can be selected based on the memory usage status and the available memory capacity. For example, the first save policy 411 to the third save policy 413 can be selected to maximize the memory usage within the available memory capacity. The first save policy 411 to the third save policy 413 can include at least one of a full recomputation policy, a partial recomputation policy, and a non-recomputation policy. In some cases, among the first save policy 411 to the third save policy 413, the full recomputation policy generates activations by performing recomputation during backpropagation without saving (e.g., storing) the activations. In some cases, the save policy can include a partial recomputation policy that saves (e.g., stores) a part of the activations and generates the remaining part of the activations by performing recomputation during backpropagation. In some cases, the save policy can include a non-recomputation policy that saves (e.g., stores) the activations and does not perform recomputation and partial recomputation during backpropagation.

[0073] Figure 5 An example showing the structure of a neural network model according to an embodiment is provided. Referring to Figure 5 , the neural network model 500 can include a first layer 510, a k-th layer 520, and an n-th layer 530, and the first layer 510 to the n-th layer 530 can include various operations, where k and n are positive integers. The operations can be grouped based on various criteria. For example, the k-th layer 520 can include a first operation group 521, a second operation group 522, and a third operation group 523, and the n-th layer 530 can include a fourth operation group 531 and a fifth operation group 532. In some cases, various amounts of computation of the first operation group 521 to the fifth operation group 532 and various capacities of the operation results of the first operation group 521 to the fifth operation group 532 can be used. In some cases, a partial recomputation policy can be defined based on the characteristics of the first operation group 521 to the fifth operation group 532.

[0074] For example, the fourth operation group 531 may also have a smaller amount of computation than the fifth operation group 532. In some cases, the operation result of the fourth operation group 531 may occupy a larger capacity than the operation result of the fifth operation group 532. Therefore, using the fourth operation group 531 as the partial recalculation target of the partial recalculation strategy may be effective because the memory space efficiency can be increased and the time delay caused by recalculation can be minimized. In some cases, the k-th layer 520 and the n-th layer 530 may include different internal structures (such as the first operation group 521 to the third operation group 523, and the fourth operation group 531 and the fifth operation group 532). In the case of the k-th layer 520 and the n-th layer 530, different operation groups can be used as the target of partial recalculation.

[0075] In some examples, the neural network model 500 may include layers with the same internal structure. For example, the layer can be a transformer layer. In some cases, one or more transformer layers may include a self-attention operation group and a multi-layer perceptron (MLP) operation group. The self-attention operation group may have a smaller amount of computation than the MLP operation group, and the operation result of the self-attention operation group may have (e.g., occupy) a larger capacity than the operation result of the MLP operation group. Therefore, in the case of one or more transformer layers, the self-attention operation group can be used as the partial recalculation target of the partial recalculation strategy.

[0076] For example, a layer in a pipeline stage (e.g., the pipeline stage described with reference to Figure 1 ) may include the same structure as the k-th layer 520, and the layers in other pipeline stages may have the same structure as the n-th layer 530. For example, the layer in the first pipeline stage may have the same structure as the n-th layer 530. In some cases, the operation group of the layer in the first pipeline stage can be selected as the target of partial recalculation. For example, the fourth operation group 531 can be selected as the target of partial recalculation. In some cases, the memory space storing the activation of the fourth operation group 531 can be released after the forward propagation of the layer in the first pipeline stage is executed. The activation of the fifth operation group 532 can be saved (e.g., stored) for gradient calculation during backpropagation. In some cases, when the backpropagation of the layer in the first pipeline stage is executed, partial recalculation of the fourth operation group 531 can be performed to calculate the gradient. The activation of the fifth operation group 532 can be loaded from the memory without performing a separate recalculation process. For example, the n-th layer 530 can be a transformation layer, the fourth operation group 531 can be a self-attention group, and the fifth operation group 532 can be an MLP operation group.

[0077] Figure 6 An example of pipeline parallelization operation including recalculation according to an embodiment is shown. Refer to Figure 6, the first training process 610 may represent an example of selecting a non-recalculation strategy for a first corresponding data set of a first partial data set of the first pipeline stage 611, a second corresponding data set of a first partial data set of the second pipeline stage 612, a third corresponding data set of a first partial data set of the third pipeline stage 613, a third corresponding data set of a second partial data set of the third pipeline stage 613, and a fourth corresponding data set of a first partial data set of the fourth pipeline stage 614. Additionally, a full recalculation strategy may be selected for a first corresponding data set of a second partial data set of the first pipeline stage 611, a first corresponding data set of a third partial data set of the first pipeline stage 611, a first corresponding data set of a fourth partial data set of the first pipeline stage 611, a second corresponding data set of a second partial data set of the second pipeline stage 612, and a second corresponding data set of a third partial data set of the second pipeline stage 612.

[0078] In the case of the forward calculation of the first training process 610, the activations of the corresponding data sets to which the non-recalculation strategy is applied may be saved (e.g., stored) in the memory. Additionally, the activations of the corresponding data sets to which the full recalculation strategy is applied may not be saved in the memory. In the case of the backward calculation of the first training process 610, the activations of the corresponding data sets to which the non-recalculation strategy is applied may be loaded from the memory for gradient calculation, and the activations of the corresponding data sets to which the full recalculation strategy is applied may be newly generated for gradient calculation.

[0079] In some cases, the second training process 620 may represent an example of selecting a partial recalculation strategy for a second corresponding data set of a second partial data set of the second pipeline stage 622 and a second corresponding data set of a third partial data set of the second pipeline stage 622.

[0080] In the case of the forward calculation of the second training process 620, the activations of the corresponding data sets to which the non-recalculation strategy is applied may be saved in the memory. In some cases, the activations of the corresponding data sets to which the full recalculation strategy is applied may not be saved in the memory. In some cases, the activations of the corresponding data sets to which the partial recalculation strategy is applied may be partially stored in the memory. In the case of the backward calculation of the second training process 620, the activations of the corresponding data sets to which the non-recalculation strategy is applied may be loaded from the memory for gradient calculation. In some cases, the activations of the corresponding data sets to which the full recalculation strategy is applied may be newly generated for gradient calculation. In some cases, the activations of the corresponding data sets to which the partial recalculation strategy is applied may be loaded from the memory, and other activations of the corresponding data sets to which the partial recalculation strategy is applied may be newly generated.

[0081] In some cases, a partial recomputation strategy may be selectively applied to the second training process 620. Some of the activations of the corresponding dataset to which the partial recomputation strategy is applied may be loaded from the memory without performing a separate recomputation process. Thus, the memory usage level of the second training process 620 may be high compared to the first training process 610. Additionally, when using the partial recomputation strategy (e.g., compared to the first training process 610), the training time of the second training process 620 may be reduced based on the reduced time of recomputation. For ease of description, Figure 6 the example involves a limited number of pipeline stages 611 to 614 and 621 to 624 and a limited number of partial datasets. However, in some embodiments, a higher number of pipeline stages and a higher number of partial datasets may be used (higher than the Figure 6 number shown), and thus, the training time may be significantly reduced.

[0082] Figure 7 A flowchart showing an example of a process of selecting a save strategy based on priority according to an embodiment is shown. Referring to Figure 7 , in operation 710, the electronic device may calculate the memory usage of the activations for each save strategy of the corresponding dataset in the partial dataset. The electronic device may calculate the memory usage of the activations for each save strategy of the corresponding dataset by simulating the training process of each partial model.

[0083] For example, the save strategy may include at least one of a full recomputation strategy, a partial recomputation strategy, and a non-recomputation strategy. When at least one of the full recomputation strategy, the partial recomputation strategy, and the non-recomputation strategy is applied to each corresponding dataset, the electronic device may calculate the capacity occupied in the memory until backpropagation is performed after generating the activations of each corresponding dataset. In some cases, when using full recomputation, the activations may not occupy a separate memory space until backpropagation. In the case of using non-recomputation, the activations may occupy the memory space until backpropagation. In the case of using partial recomputation, a part of the activations (e.g., the part generated by the operation group in which recomputation is not performed) may occupy the memory space until backpropagation is performed.

[0084] In operation 720, the electronic device may calculate the available memory capacity of each corresponding dataset for each pipeline stage. The memory usage level may increase as the training process uses the corresponding dataset. When the activations are saved in the memory according to each save strategy, the available memory capacity may be calculated.

[0085] In operation 730, the electronic device may select an activation save policy for each respective data set based on priority. In some cases, a non-recalculation policy may have a higher priority than a partial recalculation policy, and a partial recalculation policy may have a higher priority than a full recalculation policy. For example, each of the full recalculation policy, the partial recalculation policy, and the non-recalculation policy applies to a respective data set. In some examples, the non-recalculation policy may be selected based on considering the memory usage status.

[0086] In some cases, when there is competition between training tasks for respective data sets based on priority, the priority may be preferentially applied to the respective data set among the respective data sets that is closer to the final data set. For example, the competition scenario may include a case where there are multiple respective data sets to which a high-priority policy (e.g., non-recalculation policy) may apply. However, in some examples, the high-priority policy may only apply to some of the multiple respective data sets.

[0087] For example, the respective data set of the fourth partial data set of the first pipeline stage may correspond to the final data set, and the competition between the respective data set of the fourth partial data set and the respective data set of the third partial data set may occur according to the priority. For example, by considering the memory usage status, a case may occur where the non-recalculation policy applies to one of the respective data set in the fourth partial data set and the respective data set in the third partial data set. In some cases, the priority may be preferentially applied to the training task of the respective data set of the fourth partial data set. Accordingly, the fourth partial data set may be trained based on the non-recalculation policy.

[0088] Figure 8 An example of applying the priority of the save policy according to an embodiment is shown. Refer to Figure 8 , the first training process 810 may represent an example where the non-recalculation policy is selected for the respective data sets of the partial data sets for the pipeline stages 811, 812, 813, and 814. In some cases, the non-recalculation policy may represent a high memory usage level. For example, as Figure 8 shown, the first pipeline stage 811 of the first training process 810 may exceed the memory limit.

[0089] The second training process 820 may represent an example where a full recomputation strategy is selected for a first corresponding data set of a fourth partial data set for the first pipeline stage 821. During the forward computation of the second training process 820, the activations of the corresponding data sets to which the non-recomputation strategy is applied may be saved (e.g., stored) in the memory, and the activations of the corresponding data sets to which the full recomputation strategy is applied may not be saved (e.g., stored) in the memory. During the backward computation of the second training process 820, the activations of the corresponding data sets to which the non-recomputation strategy is applied may be loaded from the memory for gradient computation, and the activations of the corresponding data sets to which the full recomputation strategy is applied may be newly generated for gradient computation.

[0090] Unlike the first training process 810, the full recomputation strategy may be selectively applied to the second training process 820. In some cases, the activations of the corresponding data sets to which the full recomputation strategy is applied may be newly generated. Thus, the memory usage level of the second training process 820 may be lower than that of the first training process 810 and may not exceed the memory limit.

[0091] The third training process 830 may represent an example where a full recomputation strategy is selected for a first corresponding data set of a third partial data set for the first pipeline stage 831. When the first corresponding data set of the fourth partial data set and the first corresponding data set of the third partial data set compete for priority, the priority may be preferentially applied to the first corresponding data set of the fourth partial data set. Thus, the non-recomputation strategy may be applied to the first corresponding data set of the fourth partial data set, and the full recomputation strategy may be applied to the first corresponding data set of the third partial data set. Thus, compared with the second training process 820, the third training process 830 may show a reduced training time while (as in the second training process 820) the memory limit is not exceeded.

[0092] For ease of description, Figure 8 the examples involve a finite number of pipeline stages 811 to 814, 821 to 824, and 831 to 834 and a finite number of partial data sets. However, more pipeline stages and more partial data sets may be used, and as also Figure 8 described, the training time may be significantly reduced.

[0093] Figure 9 shows a flowchart of an example of a parallel training method according to an embodiment. Refer to Figure 9, in operation 910, the electronic device may determine a partial model of the neural network model and a partial dataset of the training dataset by partitioning the neural network model and the training dataset at a pipeline stage for parallel training of the neural network model. In operation 920, the electronic device may separately select a saving strategy for the activations generated by the forward propagation of the partial model for the corresponding datasets in the partial dataset processed at the pipeline stage, where the activations may be used to calculate the gradients for the backpropagation of the partial model. In operation 930, the electronic device may generate a strategy for parallel training of the neural network model based on the saving strategy.

[0094] The first saving strategy in the saving strategy may be used to train the corresponding datasets of the first partial dataset in the partial dataset at the first pipeline stage in the pipeline stage, and the second saving strategy different from the first saving strategy in the saving strategy may be used to train the corresponding datasets of the second partial dataset in the partial dataset at the first pipeline stage.

[0095] Operation 920 may include an operation of selecting a saving strategy based on the memory usage status.

[0096] The operation of selecting a saving strategy based on the memory usage status may include an operation of selecting a saving strategy to maximize the memory usage within the available memory capacity.

[0097] The saving strategy may include a full recomputation strategy that generates activations by performing recomputation during backpropagation without saving (e.g., storing) the activations. Additionally, the saving strategy may include a partial recomputation strategy that saves (e.g., stores) a part of the activations and generates the remaining part of the activations by performing recomputation during backpropagation. Additionally, the saving strategy may include a non-recomputation strategy that saves (e.g., stores) the activations and does not perform recomputation and partial recomputation during backpropagation.

[0098] In some cases, operation 920 may include an operation of selecting a saving strategy based on the priority of the saving strategy.

[0099] For example, the non-recomputation strategy may have a higher priority than the partial recomputation strategy, and the partial recomputation strategy may have a higher priority than the full recomputation strategy.

[0100] In some cases, in the case of competition between the training tasks of the corresponding datasets at the pipeline stage based on priority, the priority may be preferentially applied to the corresponding datasets closer to the final dataset among the corresponding datasets.

[0101] In some cases, operation 920 may include an operation of selecting or determining a saving strategy based on the order of multiple partial datasets.

[0102] In some cases, when a non-recalculation strategy applies to either the final dataset in the corresponding dataset or a dataset that is not the final dataset in the corresponding dataset, the non-recalculation strategy can be preferentially applied to the final dataset.

[0103] Figure 10 An example of the configuration of an electronic device is shown. Refer to Figure 10 , the electronic device 1000 may include a processor 1010, a memory 1020, a camera 1030, a storage device 1040, an input device 1050, an output device 1060, and a network interface 1070 that can communicate with each other via a communication bus 1080. For example, the electronic device 1000 may be implemented as at least a part of a computing device (such as a desktop computer or a server).

[0104] The processor 1010 may execute functions and instructions to be executed in the electronic device 1000. For example, the processor 1010 may process instructions stored in the memory 1020 or the storage device 1040. The processor 1010 may execute one or more operations described by Figures 1 to 9 . The memory 1020 may include a computer-readable storage medium or a computer-readable storage device. The memory 1020 may store instructions to be executed by the processor 1010 and may store relevant information when software and / or applications are executed by the electronic device 1000.

[0105] The camera 1030 may capture photos (images) and / or record videos. For example, the camera 1030 may capture a facial image including the user's face. In some cases, the camera 1030 may provide a three-dimensional (3D) image including depth information associated with an object.

[0106] The storage device 1040 may include a computer-readable storage medium or a computer-readable storage device. The storage device 1040 may store a large amount of information for a long time compared to the memory 1020. For example, the storage device 1040 may include a magnetic hard disk, an optical disk, a flash memory, a floppy disk, or other non-volatile memories known in the art.

[0107] The input device 1050 may receive input from a user via a keyboard and a mouse. In some cases, the input device 1050 may receive input from a user based on touch input, voice input, and image input. For example, the input device 1050 may include a keyboard, a mouse, a touch screen, a microphone, or any other device that detects input from a user and sends the detected input to the electronic device 1000.

[0108] The output device 1060 can provide the output of the electronic device 1000 to the user through visual, auditory, or tactile channels. The output device 1060 can include, for example, a display, a touch screen, a speaker, a vibration generator, or any other device that provides output to the user. The network interface 1070 can communicate with external devices through a wired or wireless network.

[0109] Figure 11 A flowchart showing an example of a parallel training method according to an embodiment.

[0110] Referring to Figure 11 , in operation 1110, the system can generate multiple partial data sets by partitioning the training data for the neural network model. In some cases, the neural network model can be partitioned into partial models, and the training data set can be partitioned into partial data sets. For example, the layers of the neural network model can be partitioned into partial models, and the weights of the neural network model can be partitioned into partial models. In some examples, the partial data sets can correspond to or be referred to as mini-batches.

[0111] In operation 1120, the system can identify multiple pipeline stages of the neural network model. In some cases, the pipeline stages can represent different parts of the neural network model that can operate sequentially. For example, each of the pipeline stages can include different layers or blocks of the neural network model. Referring to Figure 1 , Figure 6 and Figure 8 describe further details regarding the identification of the pipeline stages.

[0112] In operation 1130, the system can determine multiple save policies, where each of the multiple save policies corresponds to a value generated using a selected partial data set from the multiple partial data sets by a selected pipeline stage among the multiple pipeline stages.

[0113] According to an embodiment, a save policy for activations generated by the forward propagation operation of the partial model can be individually selected for the corresponding data sets of the partial data sets processed in the pipeline stage, where the activations can be used to calculate the gradients of the backpropagation operation (e.g., as described with reference to Figure 1 the backpropagation operation after the forward propagation operation) of the partial model of the neural network model.

[0114] In some cases, the save policy can be selected based on the memory usage status. The save policy can be selected to maximize the memory usage within the available memory capacity. For example, after the selection of the save policy is completed, a policy for parallel training of the neural network model can be generated based on the selected save policy. For example, the policy can include information about the pipeline stage, the partial model of the neural network model, the partial data set of the training data set, the save policy, or a combination thereof.

[0115] Then, the neural network model can be trained. Referring again to Figure 11 In operation 1140, the system performs training of the neural network model by processing values generated by a selected pipeline stage based on a selected partial data set according to a respective save policy of each of a plurality of save policies.

[0116] Figure 12 A flowchart illustrating an example of a method for training a neural network model according to an embodiment.

[0117] Embodiments of the present disclosure may include a neural network model and training data that can be partitioned based on pipeline stages. In some cases, the neural network model can be partitioned into partial models, and the training data set can be partitioned into partial data sets. The partial models can be trained at pipeline stages (as described with reference to Figure 1 ). Additionally, the partial data sets can be trained at pipeline stages.

[0118] According to an example, the training data set can be partitioned into a plurality of partial data sets. In some cases, the partial data sets can be trained sequentially and in parallel using a pipeline approach. For example, a first corresponding data set of a first partial data set can be trained in a first pipeline stage of a plurality of pipeline stages. Additionally or alternatively, a second corresponding data set of the first partial data set can be trained in a second pipeline stage based on a training operation result of the first corresponding data set of the first partial data set of the first pipeline stage.

[0119] In some cases, the training process of the neural network model can include a forward propagation operation and a backpropagation operation. The training operation result can be an activation based on the forward propagation. After the forward propagation is completed using each partial data set, the backpropagation can be performed using each partial data set. In some cases, the neural network model can be trained using gradients, and the gradients can be calculated using the activation in the backpropagation.

[0120] According to an embodiment of the present disclosure, a save policy for the activation generated by the forward propagation of the partial model can be individually selected for the corresponding data set of the partial data set processed at the pipeline stage, where the activation can be used to calculate the gradient of the backpropagation of the partial model of the neural network model.

[0121] Referring to Figure 12 In operation 1210, the system calculates a first value at a first pipeline stage of the neural network model based on a first partial data set.

[0122] In operation 1220, the system processes the first value according to a first save policy based on the first pipeline stage and the first partial data set.

[0123] In operation 1230, the system calculates a second value at the first pipeline stage of the neural network model based on a second partial data set.

[0124] In operation 1240, the system processes a second value according to a second save policy based on a first pipeline stage and a second partial data set.

[0125] In operation 1250, the system calculates a third value at a second pipeline stage of a neural network model based on a first partial data set.

[0126] In operation 1260, the system processes the third value according to a third save policy based on the second pipeline stage and the first partial data set.

[0127] In operation 1270, the system calculates a fourth value at a second pipeline stage of a neural network model based on a second partial data set.

[0128] In operation 1280, the system processes the fourth value according to a fourth save policy based on the second pipeline stage and the second partial data set.

[0129] As described herein, for example, the first save policy in the save policy can be used to train the corresponding data set of the first partial data set in the partial data set at the first pipeline stage, and the second save policy different from the first save policy in the save policy can be used to train the corresponding data set of the second partial data set in the partial data set at the first pipeline stage. Additionally or alternatively, multiple save policies can be implemented for data sets of the partial data set that can be different from each other. The neural network model can be trained based at least in part on the selection of the save policy.

[0130] The units described herein can be implemented using hardware components, software components, and / or combinations thereof. The processing device can be implemented using one or more general-purpose or special-purpose computers (such as, for example, a processor, a controller, and an arithmetic logic unit (ALU), a DSP, a microcomputer, an FPGA, a programmable logic unit (PLU), a microprocessor, or any other device capable of responding and executing instructions in a defined manner). The processing device can run an operating system (OS) and one or more software applications running on the OS. The processing device can also access, store, manipulate, process, and create data in response to the execution of software. For simplicity, the description of the processing device is used as a singular; however, those skilled in the art will understand that the processing device can include multiple processing elements and multiple types of processing elements. For example, the processing device can include multiple processors, or a single processor and a single controller. Additionally, different processing configurations (such as, parallel processors) are feasible.

[0131] Software may include a computer program, code segments, instructions, or some combination thereof, which independently or jointly direct or configure a processing device to operate as required. Software and data may be stored on any type of machine, component, physical or virtual device, or computer storage medium or device capable of providing instructions or data to, or being interpreted by, a processing device. Software may also be distributed over networked computer systems such that the software is stored and executed in a distributed manner. Software and data may be stored by one or more non-transitory computer-readable recording media.

[0132] The method according to the above example may be recorded in a non-transitory computer-readable medium, which includes program instructions for implementing the various operations of the above example. The medium may also include data files, data structures, etc., either alone or in combination with the program instructions. The program instructions recorded on the medium may be program instructions specially designed and constructed for the purposes of the example, or they may be of the type well-known and available to those skilled in the art of computer software. Examples of non-transitory computer-readable media include: magnetic media (such as hard disks, floppy disks, and magnetic tapes); optical media (such as CD-ROM disks, DVDs, and / or Blu-ray disks); magneto-optical media (such as optical disks); and hardware devices specifically configured to store and execute program instructions (such as read-only memory (ROM), random access memory (RAM), flash memory (e.g., USB flash drives, memory cards, memory sticks, etc.)). Examples of program instructions include both machine code such as that produced by a compiler and files containing high-level code that can be executed by a computer using an interpreter.

[0133] The above device may be configured to act as one or more software modules to perform the operations of the above example, and vice versa.

[0134] As described above, although the examples have been described with reference to a limited number of drawings, those skilled in the art may apply various technical modifications and variations based thereon. For example, suitable results may be achieved if the described techniques are performed in a different order and / or if the components in the described system, architecture, device, or circuit are combined in a different manner and / or replaced or supplemented by other components or their equivalents.

[0135] Accordingly, other embodiments are within the scope of the appended claims.

Claims

1. A method for parallel training, the method comprising: Generating a plurality of partial data sets by dividing the training data for the neural network model; Identify multiple pipeline stages of a neural network model; determining a preservation strategy to be used from a plurality of preservation strategies, wherein the preservation strategy to be used corresponds to a value generated by a selected pipeline stage of the plurality of pipeline stages using a selected partial data set of the plurality of partial data sets; as well as The neural network model is trained by processing the values ​​generated by the selected pipeline stage based on the selected partial data set according to the preservation strategy to be used.

2. The method according to claim 1, wherein: A first preservation strategy of the plurality of preservation strategies is used for a first value corresponding to a first partial data set of the plurality of partial data sets in a first pipeline stage of the plurality of pipeline stages, and A second preservation strategy of the plurality of preservation strategies that is different from the first preservation strategy is used for a second value corresponding to a second partial data set of the plurality of partial data sets in a second pipeline stage of the plurality of pipeline stages.

3. The method according to claim 1, wherein: The step of determining a preservation strategy to be used from the plurality of preservation strategies comprises: A preservation policy to be used is determined from the plurality of preservation policies based on the memory usage status.

4. The method according to claim 3, wherein: The preservation strategy to be used is determined based on the memory usage status and the available memory capacity to optimize the memory usage.

5. The method according to claim 1, wherein: The plurality of preservation strategies include one or more of the following: A full recomputation strategy generates the value by performing recomputation during backpropagation without storing the value, a partial recomputation strategy that stores a portion of the value and generates the remainder of the value by performing partial recomputation during backpropagation, and A non-recomputation strategy stores the value without performing recomputation during backpropagation.

6. The method according to claim 5, wherein: The step of determining the preservation policy to be used from the plurality of preservation policies includes determining the preservation policy to be used from the plurality of preservation policies based on priorities among the plurality of preservation policies.

7. The method according to claim 6, wherein: A non-recalculation policy has a higher priority than a partial-recalculation policy, and a partial-recalculation policy has a higher priority than a full-recalculation policy.

8. The method according to claim 5, wherein: The plurality of preservation strategies are determined based on an order of the plurality of partial data sets.

9. A non-transitory computer-readable storage medium storing instructions which, when executed by a processor, cause the processor to perform the method according to claim 1.

10. An electronic device comprising: processor; as well as a memory configured to store instructions, Wherein, when executed by a processor, the instructions cause the electronic device to: Generating a plurality of partial data sets by dividing the training data for the neural network model; Identify multiple pipeline stages of a neural network model; determining a preservation strategy to be used from a plurality of preservation strategies, wherein the preservation strategy to be used corresponds to a value generated by a selected pipeline stage of the plurality of pipeline stages using a selected partial data set of the plurality of partial data sets; and The neural network model is trained by processing the values ​​generated by the selected pipeline stage based on the selected partial data set according to the preservation strategy to be used.

11. The electronic device according to claim 10, wherein: A first preservation strategy of the plurality of preservation strategies is used for a first value corresponding to a first partial data set of the plurality of partial data sets in a first pipeline stage of the plurality of pipeline stages, and A second preservation strategy of the plurality of preservation strategies that is different from the first preservation strategy is used for a second value corresponding to a second partial data set of the plurality of partial data sets in a second pipeline stage of the plurality of pipeline stages.

12. The electronic device according to claim 10, wherein: The instructions, in response to being executed by a processor, cause the processor to determine a preservation strategy to be used from the plurality of preservation strategies based on a memory usage status.

13. The electronic device according to claim 12, wherein: The instructions, in response to being executed by a processor, cause the processor to determine a preservation strategy to be used to optimize memory usage based on a memory usage status and an available memory capacity.

14. The electronic device according to claim 10, wherein: The plurality of preservation strategies include one or more of the following: A full recomputation strategy generates the value by performing recomputation during backpropagation without storing the value, a partial recomputation strategy that stores a portion of the value and generates the remainder of the value by performing partial recomputation during backpropagation, and A non-recomputation strategy stores the value without performing recomputation during backpropagation.

15. The electronic device according to claim 14, wherein: The instructions, in response to being executed by a processor, cause the processor to determine a preservation strategy to use from the plurality of preservation strategies based on a priority among the plurality of preservation strategies.

16. The electronic device according to claim 15, wherein: A non-recalculation policy has a higher priority than a partial-recalculation policy, and a partial-recalculation policy has a higher priority than a full-recalculation policy.

17. The electronic device according to claim 15, wherein: The plurality of preservation strategies are determined based on an order of the plurality of partial data sets.

18. A method for training a neural network model, the method comprising: Calculating a first value at a first pipeline stage of the neural network model based on the first portion of the data set; processing the first value according to a first preservation strategy based on the first pipeline stage and the first portion of the data set; Calculating a second value at a first pipeline stage of the neural network model based on the second portion of the data set; processing the second value according to a second preservation strategy based on the first pipeline stage and the second portion of the data set; calculating a third value at a second pipeline stage of the neural network model based on the first portion of the data set; processing the third value according to a third preservation strategy based on the second pipeline stage and the first portion of the data set; calculating a fourth value at a second pipeline stage of the neural network model based on the second portion of the data set; as well as The fourth value is processed according to a fourth preservation strategy based on the second pipeline stage and the second partial data set.

19. The method according to claim 18, wherein: The third value and the second value are calculated simultaneously.

20. The method according to claim 18, wherein: The first preservation strategy, the second preservation strategy, the third preservation strategy, and the fourth preservation strategy are selected from a group of preservation strategy types including one or more of the following: a full recalculation strategy to generate one or more of the first value, the second value, the third value, and the fourth value by performing recalculation during backpropagation without storing the one or more values, a partial recomputation strategy that stores a portion of the one or more values ​​and generates a remainder of the one or more values ​​by performing partial recomputation during backpropagation, and A non-recomputation strategy stores the one or more values ​​without performing recomputation during backpropagation.

Images