Data processing method, device and medium during training of deep learning model

Abstract

Provided are a data processing method, device, and medium for training a deep learning model. The data processing method may include: determining a target activation data set from candidate activation data sets to be unloaded based on at least one of a storage state of a first storage device associated with a first processor carrying the deep learning model and a communication state associated with the first processor; unloading the target activation data set from the first storage device to a second storage device associated with a second processor during a forward propagation phase of the training process; and loading each target activation data in the target activation data set from the second storage device to the first storage device before performing a backpropagation calculation using the target activation data during a backpropagation phase of the training process.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a data processing method, device, processor, and computer storage medium in the training process of a deep learning model. Background Art

[0002] In recent years, deep learning technology has achieved significant breakthroughs and is widely used in image recognition, natural language processing, and various intelligent applications. However, training efficient deep learning models requires extensive computing resources. Traditional central processing units (CPUs) struggle to meet the growing demand for deep learning model training due to their low computational efficiency. Therefore, to improve training efficiency, high-performance computing accelerators such as graphics processing units (GPUs), general-purpose graphics processing units (GPGPUs), or tensor processing units (TPUs) are often required.

[0003] On the other hand, as the scaling law in deep learning has been gradually validated, researchers are pushing for larger models. Larger models have demonstrated superior performance and stronger generalization across multiple tasks. However, current computing accelerators, whether GPUs, GPGPUs, or TPUs, have limited video memory capacity. As model size continues to grow, insufficient video memory becomes increasingly prominent, leading to increased memory pressure during training. This phenomenon not only affects model training efficiency but also restricts the development and application of larger models.

[0004] Therefore, it is necessary to propose effective solutions to the memory pressure faced by deep learning models during training. Summary of the Invention

[0005] According to one aspect of the present application, a data processing method for a training process of a deep learning model is provided, which may include: determining a target activation data set from a candidate activation data set to be unloaded based on a storage state of a first storage device associated with a first processor carrying the deep learning model and at least one of a communication state associated with the first processor; unloading the target activation data set from the first storage device to a second storage device associated with a second processor in a forward propagation phase of the training process; and loading the target activation data from the second storage device to the first storage device in a backward propagation phase of the training process for each target activation data in the target activation data set before performing a backward propagation calculation using the target activation data.

[0006] According to another aspect of the present application, a data processing method for a training process of a deep learning model is provided, which may include: in response to the available storage space of a first storage device associated with a first processor carrying the deep learning model being greater than or equal to a first storage threshold, only unloading the target activation data set to be unloaded to a second storage device associated with a second processor; in response to the available storage space of the first storage device being less than the first storage threshold and greater than or equal to a second storage threshold, unloading i) the optimizer state of the optimizer used for the training process or the weight value set associated with the target activation data set, and ii) the target activation data set to the second storage device; and in response to the available storage space of the first storage device being less than the second storage threshold, unloading at least a portion of the optimizer state used for the training process, the weight value set associated with the target activation data set, and the target activation data set to the second storage device.

[0007] According to another aspect of the present application, a data processing device for use in a training process of a deep learning model is provided, which may include: a determination module for determining a target activation data set from a candidate activation data set to be unloaded based on a storage state of a first storage device associated with a first processor carrying the deep learning model and at least one of a communication state associated with the first processor; an unloading module for unloading the target activation data set from the first storage device to a second storage device associated with a second processor in a forward propagation phase of the training process; and a loading module for loading the target activation data from the second storage device to the first storage device before performing backpropagation calculations on each target activation data in the target activation data set in a backward propagation phase of the training process.

[0008] According to yet another aspect of the present application, a processor is provided, which is configured to execute a computer program or instruction to implement the method described above.

[0009] According to another aspect of the present application, a computer-readable storage medium is provided, on which a computer program or instruction is stored. When the computer program or instruction is executed by a processor, the processor executes the method as described above.

[0010] Therefore, according to the data processing scheme for the training process of the deep learning model of the present application, a relatively reasonable target activation data set that can be unloaded can be determined from the candidate activation data based on at least one of the storage state and the communication state associated with the first processor according to the actual application scenario. Therefore, the activation data that needs to be unloaded can be automatically determined in accordance with the current scenario, thereby being applicable to dynamic changes in storage resources. In addition, since the target activation data is determined by considering the communication state, the activation data with a long loading time can be unloaded as little as possible, so that the waiting time can be reduced as much as possible, thereby maximizing the efficiency of training based on the unloading scheme. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments of the present application or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings of the embodiments of the present application.

[0012] Figure 1 A schematic diagram showing an application scenario of training a deep learning model according to an embodiment of the present application is shown.

[0013] Figure 2 A schematic diagram showing the principle of an offloading technical solution for solving the problem of storage resource pressure is shown.

[0014] Figure 3 A flow chart of a data processing method for a deep learning model training process according to an embodiment of the present application is shown.

[0015] Figure 4A A schematic diagram of uninstalling activation data according to an embodiment of the present application is shown.

[0016] Figure 4B A schematic diagram of the trial operation process according to an embodiment of the present application is shown.

[0017] Figure 5 A flow chart of a data processing method for a deep learning model training process according to an embodiment of the present application is shown.

[0018] Figure 6 A schematic diagram of the process of unloading the optimizer state according to an embodiment of the present application is shown.

[0019] Figure 7 A flow chart of a data processing method for a deep learning model training process according to an embodiment of the present application is shown.

[0020] Figure 8A schematic diagram illustrating a process of unloading activation data and its associated weight values ​​according to an embodiment of the present application is shown.

[0021] Figure 9 A flow chart of another data processing method for the training process of a deep learning model according to an embodiment of the present application is shown.

[0022] Figure 10 A structural block diagram of a data processing device used in a training process of a deep learning model according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0023] The following will be combined with the accompanying drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the embodiments described are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0024] Before describing the present application in detail, various terms that may be used in the present application are explained as follows.

[0025] 1. Activation data or activation value (Activation): The data transferred between each two adjacent network layers for a training sample during the forward propagation phase of the deep learning training process. This data has been processed by the activation layer (i.e., nonlinear mapping layer) at the previous neural network layer and can be equivalent to the output data of the previous network layer in the adjacent network layer.

[0026] 2. Optimizer: An algorithm used in deep learning to adjust neural network parameters to improve model prediction accuracy. It can change the weights of neurons in the deep learning model network to make the network better fit the data. Examples of optimizers include SGD (stochastic gradient descent) or Adam (adaptive momentum estimation).

[0027] 3. Optimizer State: During the training process, the optimizer maintains a set of variables, such as variables that store historical information about weight gradients. These variables record updates to model parameters (i.e., the weights of each network layer) during training, helping the optimizer adjust model parameters more effectively.

[0028] 4. Offload: The process of transferring some data from an accelerator (such as a GPU, GPGPU, or TPU) to a central processing unit (CPU) during training.

[0029] 5. Load: The process of loading certain data from the CPU to the accelerator;

[0030] 6. D2H: Transfers data from the device side (Device) (for example, GPU, GPGPU or TPU) to the host side (Host) (for example, CPU).

[0031] 7. H2D: Transfers data from the host (e.g., CPU) to the device (e.g., GPU, GPGPU, or TPU).

[0032] 8. Graphics card: A hardware device used to handle graphics rendering tasks, located on the device side, such as a GPU, GPGPU, or TPU. It processes images and videos independently of the CPU, which can significantly improve processing performance.

[0033] 9. Video memory: The memory in the graphics card used to store data required for graphics operations, specifically used for operations such as GPU, GPGPU or TPU.

[0034] Figure 1 A schematic diagram showing an application scenario of training a deep learning model according to an embodiment of the present application is shown.

[0035] like Figure 1 As shown, application scenario 100 may include a host side 110 and a device side 120, wherein the host side includes a second processor such as a central processing unit (CPU), and the device side 120 may include a first processor such as a graphics processing unit (GPU). This application uses an example in which the host side includes a CPU and the device side 120 includes a GPU, but it should be understood that this application is not limited thereto. For example, the device side 120 may also include a general-purpose graphics processing unit (GPGPU) or a tensor processing unit (TPU). In addition, both the first processor and the second processor have associated storage devices (e.g., memory) to store various data during processing.

[0036] The host 110 is typically considered the main device or "host," whose primary responsibility is to control task execution, manage program logic flow, and handle relatively low-intensity computational tasks. The device 120 is considered an auxiliary device or "device," specifically designed to perform highly parallel computational tasks. For example, when a deep learning model is deployed on the device 120, the device 120 needs to perform a large number of matrix and vector operations, such as the forward propagation, backpropagation, and parameter update processes in deep learning.

[0037] In deep learning, to efficiently utilize computing resources and increase training speed, the entire training dataset is typically not fed into the model all at once. Instead, the training dataset is split into multiple small batches. For each batch, the model performs a forward pass to calculate the loss, a backward pass, and parameter updates; this constitutes a time step. Of course, the entire training dataset can also be fed into the model all at once for training. Therefore, the training process for a deep learning model typically involves the following steps: First, in the forward pass, the GPU performs forward pass to calculate the loss based on the current model parameters and each training example as input data. In the backward pass, the GPU calculates gradients, and in the optimization phase, the optimizer updates the model parameters. This training process requires a large number of matrix operations, where GPUs play a significant role, significantly improving computational efficiency. Meanwhile, the CPU can continue to handle other tasks during this time, such as monitoring training progress, saving model checkpoints, or performing data augmentation.

[0038] As mentioned earlier, a solution is needed to address the pressure on storage resources (such as GPU memory) faced by deep learning models during training.

[0039] To address the pressure on storage resources, a variety of common methods have emerged, such as data parallelism, model parallelism, pipeline parallelism, recomputing, and offloading technologies. Offloading technology is a common solution to the problem of storage resource pressure. Its principle is as follows: Figure 2 shown.

[0040] As mentioned earlier, the training process of a deep learning model can be roughly divided into three stages: the forward propagation stage, the backpropagation stage, and the parameter update stage performed by the optimizer. The forward propagation stage calculates the loss value (loss), the backpropagation stage calculates the gradient of the loss value (loss) relative to each weight (model parameter), and finally the optimizer updates the parameters according to an optimization algorithm such as gradient descent. Figure 2 The scheme shown in the figure offloads some intermediate activation data needed for backpropagation to the CPU during the forward propagation phase, and then loads it to the GPU for backpropagation when it is needed. After backpropagation, the activation data can be released immediately, thereby minimizing video memory consumption.

[0041] For example, Figure 2In this example, the deep learning model consists of multiple neural network layers (illustrated as five in the figure, but this number may vary in practice). During the forward propagation phase, a single input (e.g., a training example from the batch corresponding to this forward propagation phase) is fed into the first layer. After computations are performed by these five layers, the output data (i.e., predicted data) is generated. The data transferred between adjacent layers is the activation data. The GPU predetermines the activation data between the second and third layers, and between the third and fourth layers, as the data to be offloaded to the CPU. After the second layer outputs the activation data, it is not only passed to the third layer, but also to the CPU, offloading this activation data to the GPU (a D2H process). This activation data is then used in the backpropagation phase for the third layer's computations. Therefore, before backpropagation computations are performed on the third layer, the previously offloaded activation data must be retrieved from the CPU and reloaded to the GPU (a H2D process). The activation data output by the third layer during the forward propagation phase follows a similar process. In addition, for the activation data output by the 1st and 4th layers, since these activation data are not selected to be offloaded to the CPU, the GPU will store these activation data and can be released after the back propagation phase is completed. For example, the activation data output by the 4th layer can be used to calculate the 5th layer in the back propagation phase and then the activation data output by the 4th layer can be released, or it can be released after the back propagation phase is completed. In the forward propagation phase, multiple training samples are involved as the input of the deep learning model, so each training sample will correspond to multiple activation data transferred between adjacent network layers of the model, and correspondingly, these activation data are used to perform back propagation calculations for each training sample in the back propagation phase.

[0042] but, Figure 2 The offloading technique shown in may not be flexible enough to adapt to various training situations and may have low training efficiency, as follows:

[0043] 1. It is difficult to determine which parts to offload to meet the GPU memory requirements, and it requires engineering experience and multiple attempts to gradually adjust. For example, Figure 2 In the example, the activation data output by the second layer and the third layer as the target activation data for offloading are predetermined, need to be determined in advance and are fixed once determined.

[0044] 2. The backpropagation calculation process during the backpropagation phase is dependent on the process of reloading data previously offloaded to the CPU. For example, before the backpropagation calculation for layer 3 in the backpropagation phase, the activation data that has been offloaded to the CPU (the activation data passed between layers 2 and 3 during the forward propagation phase) needs to be reloaded to the GPU. This may sometimes require a certain waiting time, making it difficult to maximize training efficiency based on the offloading solution.

[0045] 3. During training, dynamic or abnormal situations often occur, affecting the entire training process when all training samples are utilized. For example, during training, available storage resources may change dynamically due to video memory fragmentation or server resource preemption. This makes the current offloading strategy unstable and unable to reduce the pressure on the already changing video memory.

[0046] Therefore, in response to the above problems, an embodiment of the present application provides a data processing solution for the training process of a deep learning model, which adaptively selects which activation data will be unloaded by introducing an unloading scheduler, and optionally considers the weight values ​​and optimizer status of the main sources of storage resource consumption during the unloading training process in addition to unloading activation data. Therefore, various levels of unloading strategies can be implemented according to actual needs.

[0047] The following combination Figure 3-Figure 10 The following describes in detail the data processing scheme used in the training process of the deep learning model according to the embodiment of the present application.

[0048] Figure 3 A flow chart of a data processing method for a deep learning model training process according to an embodiment of the present application is shown. The data processing method can be executed by a first processor equipped with the deep learning model, for example, Figure 1 The GPU shown here can be used to execute the Figure 1 The deep learning model and / or various data generated during the training process can be stored in a first storage device associated with the first processor.

[0049] like Figure 3 As shown, in step S310, based on the storage status of the first storage device associated with the first processor carrying the deep learning model and at least one of the communication status associated with the first processor, a target activation data set is determined from the candidate activation data sets to be unloaded.

[0050] For example, the candidate activation data set to be offloaded includes all activation data to be transferred between each two adjacent network layers in the deep learning model in the forward propagation phase. Optionally, when determining the candidate activation data set, it may be equivalent to determining the indication information of these activation data (for example, a deep learning module, such as a function or network layer of a specific task, which can be used to specify each candidate activation data). For example, the candidate activation data set to be offloaded may be as follows: Figure 2 The following figure shows the activation data Act 1 to Act 4 passed between various network layers during the forward propagation phase after training begins. Alternatively, it can be a subset of activation data selected from all activation data according to predefined rules. These candidate activation data are pre-labeled and registered with the offload scheduler object so that the offload scheduler can know which candidate activation data exist.

[0051] In addition, all activation data passed between every two network layers of the deep learning model are to be used in the backpropagation phase of the deep learning model. The target activation data set is the set of activation data to be unloaded from the first storage device associated with the first processor to the second storage device associated with the second processor during the forward propagation phase, and reloaded back to the first storage device during the backpropagation phase.

[0052] For example, the storage state associated with the first processor may include the available storage space of the first storage device associated with the first processor (e.g., the video memory of the GPU). The available storage space may be storage space that can be used for deep learning model training and may change dynamically, for example due to server resource preemption. The communication state associated with the first processor may include the communication bandwidth and communication volume of the data communication link used to perform unloading of activation data and reloading of activation data, which may affect the unloading speed and reloading speed of activation data, thereby potentially affecting the training speed and training efficiency. Optionally, the available storage space of the first storage device may be actively detected each time the deep learning model needs to be trained, and optionally, the current available storage space may be detected in real time during the entire model training period, or the available storage space may be passively detected (e.g., upon receiving an alarm signal indicating insufficient storage space), and whether part of the activation data needs to be unloaded and the unloading strategy update as will be described later may be determined based on the detected available storage space.

[0053] Therefore, when the available storage space of the first storage device (for example, the video memory of the GPU) is not large enough, it may be necessary to select some candidate activation data (i.e., target activation data sets) for unloading and reloading. For example, when the available storage space is greater than or equal to a preset threshold, the target activation data set can be determined as an empty set, that is, no activation data can be unloaded, which can maximize the use of the first storage device. If the available storage space is less than or equal to the preset threshold, it is determined that some candidate activation data need to be selected for unloading and reloading. The preset threshold can be determined based on experience. For example, since the communication state associated with the first processor is associated with the expected time required to reload each activation data, the expected time required to reload each candidate activation data can be further considered to select the target activation data set to be unloaded.

[0054] Alternatively, when determining the target activation data set, the target activation data set may be selected by only considering the expected time required to reload each candidate activation data, for example, selecting activation data with less time required to be unloaded regardless of the size of the available storage space.

[0055] The specific steps of determining the target activation data set will be described in detail later.

[0056] Therefore, since the storage status of the first storage device (such as available storage space) and the communication status associated with the first processor may be different in different application scenarios or at different times, the most suitable target activation data set can be adaptively selected according to actual training, so that the activation data to be unloaded does not need to be fixed, so as to improve training efficiency and quality.

[0057] In step S320 , during the forward propagation phase of the training process, the target activation data set is unloaded from the first storage device to a second storage device associated with a second processor.

[0058] As mentioned above, in the forward propagation stage, the activation data output by the previous layer can be used as the input data of the next layer, and if the activation data is included in the target activation data set (i.e., determined as activation data to be unloaded), the activation data will also be unloaded to the second storage device associated with the second processor. At the same time, if the activation data is not included in the target activation data set (i.e., determined as activation data that does not need to be unloaded), it means that the activation data is not suitable for unloading, so it can be stored in the first storage device associated with the first processor for subsequent use in the backward propagation stage.

[0059] In step S330, in the back propagation phase of the training process, for each target activation data in the target activation data set, the target activation data is reloaded from the second storage device to the first storage device before the target activation data is used for back propagation calculation.

[0060] As previously described, since each target activation data has been unloaded to the second storage device, when the first processor needs to use the target activation data for backpropagation calculation, it is necessary to reload the target activation data from the second storage device associated with the second processor to the first storage device associated with the first processor before using the backpropagation calculation for use by the first processor. After the backpropagation phase is completed, all candidate activation data can be released.

[0061] Optionally, to improve training efficiency, it is desirable to minimize training time. As previously mentioned, the backpropagation calculation for each network layer requires corresponding activation data. If this activation data is target activation data that has already been offloaded to the second storage device and is included in the target activation dataset, then this target activation data should be reloaded to the first storage device before the backpropagation calculation using this target activation data. Therefore, it is possible to consider loading this activation data in parallel during the backpropagation calculation. That is, during the backpropagation phase, while the backpropagation calculation for the nth network layer of the deep learning model (for a certain training sample) is being performed, the target activation data (the activation data output by the nth network layer during the forward propagation phase) required for the backpropagation calculation for the n-1th network layer (for the same training sample) and offloaded to the second storage device is reloaded to the first storage device in parallel. Here, n is an integer greater than 2 and less than or equal to the total number of network layers of the deep learning model, and the network layer number n increases gradually in the direction of the forward propagation phase. This can shorten the time spent waiting for activation data to be reloaded, thereby improving training speed. In addition, the start time of the backpropagation calculation for the nth network layer can be aligned with the start time of reloading the corresponding target activation data, so that the reloading of the target activation data will not be performed too early, thereby reducing the storage pressure of the first storage device.

[0062] For example, Figure 4A A schematic diagram of uninstalling activation data according to an embodiment of the present application is shown.

[0063] like Figure 4A As shown, in the forward propagation stage, activation data is transferred between every two network layers. Figure 4AIn the example, 5 network layers and 4 activation data (Act 1-Act 4) for a training sample are shown. These 4 activation data are registered as objects of the offload scheduler, so the offload scheduler can determine the target activation data to be offloaded as described above, such as Figure 4A The activation data Act 2 and the activation data Act 4 shown in the figure are determined as the target activation data to be unloaded. In the back propagation stage, for the same training sample, the back propagation calculation of the 5th network layer requires the activation data Act 4, so the unloaded activation data Act 4 needs to be reloaded before the back propagation calculation starts. Similarly, the back propagation calculation of the 3rd network layer requires the activation data Act 2, so the unloaded activation data Act 2 needs to be reloaded before the back propagation calculation starts. For the activation data Act 1 and the activation data Act 3, since these two activation data have not been unloaded, they are still stored in the first storage device and can be directly used by the first processor for back propagation calculation. The unloading process of the activation data for other training samples is similar.

[0064] Although not in Figure 4A As shown in FIG, the process of reloading the activation data Act 2 can be performed in parallel with the back propagation calculation process for the 4th network layer, and the process of reloading the activation data Act 4 needs to be performed before the back propagation calculation for the 5th network layer.

[0065] Optionally, when determining the target activation data set, the target activation data set can be determined from the candidate activation data set to be unloaded based on the expected loading time corresponding to each candidate activation data in the candidate activation data set to be unloaded, wherein the communication status associated with the first processor mentioned above is associated with the expected loading time corresponding to each candidate activation data, because the communication status affects the loading speed of data to the first storage device.

[0066] For example, a trial run can be used to determine the expected loading time of each candidate activation data in the current application scenario, wherein during the trial run, the unloading scheduler will attempt to unload all candidate activation data to the second storage device associated with the second processor, and load each candidate activation data to the first storage device associated with the first processor when needed in the backpropagation phase, thereby ensuring that the training process can operate normally.

[0067] For example, the trial run can be performed by the following operations: in the forward propagation phase of the trial run training process, each candidate activation data in the candidate activation data set to be unloaded is unloaded from the first storage device to the second storage device; and in the reverse propagation phase of the trial run training process, for each candidate activation data in the candidate activation data set to be unloaded, before performing reverse propagation calculation using the candidate activation data, the candidate activation data is loaded from the second storage device to the first storage device. Similarly, the reverse propagation calculation and the activation data loading can be performed in parallel. For example, in the reverse propagation phase of the trial run training process, during the reverse propagation calculation for the nth network layer, the candidate activation data required for the reverse propagation calculation for the n-1th network layer is loaded into the first storage device in parallel, where n is an integer greater than 2, and the number n of the network layer gradually increases in the direction of the forward propagation phase.

[0068] In addition, considering that the storage state of the first storage device and the communication state associated with the first processor may change with the training process, a trial run operation can be performed regularly or periodically to more accurately determine the expected loading time corresponding to each candidate activation data, thereby more accurately determining the target activation data set.

[0069] Through trial operation, the expected loading time corresponding to each candidate activation data can be obtained. Therefore, when using it to determine the target activation data set, a subset of candidate activation data can be determined from the candidate activation data set to be unloaded based on the expected loading time corresponding to each candidate activation data, wherein the expected loading time corresponding to each candidate activation data in the candidate activation data subset is greater than the expected computational time of the back propagation calculation performed in parallel with the loading of the candidate activation data. Then, for each candidate activation data in the candidate activation data subset, the time difference between the expected loading time corresponding to the candidate activation data and the expected computational time of the back propagation calculation performed in parallel with the loading of the candidate activation data is determined as the expected idle duration corresponding to the candidate activation data; and the target activation data set is determined based on the expected idle duration corresponding to each candidate activation data in the candidate activation data subset.

[0070] Optionally, each candidate activation data in the candidate activation data subset whose corresponding expected idle time is greater than a predetermined time threshold can be excluded from the candidate activation data set to be unloaded to obtain a target activation data set. For example, if the expected idle time corresponding to the candidate activation data is long, it means that the loading time of the candidate activation data is long, which will cause an excessively long waiting time after the parallel execution of the back propagation calculation is completed. Therefore, such candidate activation data should be unloaded as much as possible; conversely, if the expected idle time corresponding to the candidate activation data is short or does not exist (that is, the expected loading time corresponding to the candidate activation data is less than or equal to the expected calculation time of the parallel execution of the back propagation calculation), it means that the loading time of the candidate activation data is short, which will not cause an excessively long waiting time after the parallel execution of the back propagation calculation is completed. Therefore, such candidate activation data should be unloaded as much as possible.

[0071] In addition, the target activation data set can also be determined in combination with the available storage space of the first storage device. For example, the amount of activation data to be uninstalled can be determined based on the size of the available storage space of the first storage device. For example, if the available storage space of the first storage device is large enough, no candidate activation data may be uninstalled, that is, the target activation data set is an empty set; if the available storage space of the first storage device is limited and the storage resource pressure is high, a portion of the candidate activation data may be uninstalled. For example, candidate activation data that is expected to take a long time to uninstall may be preferentially excluded and candidate activation data may be uninstalled as little as possible so that the available storage space can be used to the utmost, or all candidate activation data that is expected to take a time to uninstall is less than or equal to a predetermined threshold may be uninstalled. Alternatively, all candidate activation data that is expected to take a time to uninstall is less than or equal to a predetermined threshold may be uninstalled regardless of the available storage space of the first storage device.

[0072] Figure 4B A schematic diagram of the trial run process is shown. Figure 4B The present invention mainly focuses on how the candidate activation data that has been unloaded to the second storage device associated with the second processor during the back-propagation phase during the trial run is reloaded to the first storage device associated with the first processor and the associated timing. Figure 4B Still Figure 2 as well as Figure 4A The deep learning model shown includes 5 neural network layers for illustration.

[0073] like Figure 4B As shown, in the back-propagation phase, prefetching can be performed asynchronously to ensure that when a candidate activation data is needed, the candidate activation data has been loaded into the first storage device associated with the first processor. At the same time, this prefetching operation should not be performed too early, otherwise it will bring video memory pressure. For example, Figure 4BAs shown, at the beginning of the backpropagation calculation period of each network layer, a reload (prefetch) of candidate activation data is started simultaneously and in parallel.

[0074] For example, in Figure 4B In the forward pass, layer 5 must first be backpropagated. Prior to this, the candidate activation data (Act 4) from layer 4, which had already been offloaded to the second storage device, must be reloaded to the first storage device (i.e., H2D). This process can be performed immediately after starting the backward pass, so that the candidate activation data Act 4 is not pre-fetched too early.

[0075] In addition, when performing backpropagation calculations for the 5th layer, the operation of reloading the candidate activation data (Act 3) output by the 3rd layer of the forward propagation stage that has been unloaded to the second storage device to the first storage device (i.e., H2D) can be carried out synchronously, because the backpropagation calculations for the 4th layer require the use of the candidate activation data Act 3.

[0076] In some cases, due to the large number of candidate activation data and / or the influence of communication status, the loading process of the candidate activation data (Act n) output by the nth network layer in the forward propagation phase may take longer than the backpropagation calculation of the n+2th network layer in parallel. In this case, the backpropagation calculation of the n+1th network layer needs to wait for the loading process of the candidate activation data (Act n) to be completed before it can start again, because the backpropagation calculation of the n+1th network layer needs to use the candidate activation data (Act n). Figure 3 In the forward propagation phase, the execution of the backpropagation calculation for the second network layer cannot be executed until the loading process of the candidate activation data (Act 1) output by the first network layer is completed. This results in idle time in the backpropagation phase. Therefore, the offload scheduler can determine the candidate activation data to be offloaded as the target activation data set based on the idle time of the backpropagation calculation for each network layer in the backpropagation phase (which may have a duration of 0 or greater than 0) and the available storage space of the optional first storage device.

[0077] For example, the offload scheduler can calculate the performance overhead (corresponding to the idle time during the backpropagation phase) incurred by reloading each candidate activation data after unloading it. If the performance overhead of a particular candidate activation data is significant, when it is determined that some candidate activation data needs to be unloaded, the offload of candidate activation data with high performance overhead (corresponding to longer idle time) can be minimized. Furthermore, as previously described, the need to offload candidate activation data can be determined based on the available storage space of the first storage device, thereby appropriately disabling the offloading of candidate activation data with high performance overhead.

[0078] Optionally, if the idle time of the back propagation calculation for certain network layers in the back propagation stage is less than a predetermined threshold or there is no idle time, the candidate activation data to be unloaded during the back propagation calculation of these network layers can be determined as the target activation data set, regardless of the size of the available storage space of the first storage device.

[0079] Therefore, according to the reference Figures 3 to 4B The described data processing method for the training process of a deep learning model can determine a relatively reasonable set of target activation data that can be offloaded from candidate activation data based on at least one of the storage state and communication state associated with the first processor according to the actual application scenario. Therefore, the activation data that needs to be offloaded can be automatically determined in an adaptive manner to the current scenario, thereby being applicable to dynamic changes in storage resources. In addition, because the target activation data is determined by considering the communication state, activation data with a long loading time can be offloaded as little as possible, thereby minimizing waiting time, thereby maximizing training efficiency based on the offloading scheme.

[0080] Figure 5 A flow chart of a data processing method for a deep learning model training process according to an embodiment of the present application is shown. Figure 5 The data processing method may be Figure 3 Supplement or replace certain steps of the data processing method.

[0081] As mentioned earlier, the training process of the deep learning model also includes an optimization phase after the backpropagation phase in which the optimizer is used to update the parameters. The optimizer will generate an optimizer state in the optimization phase, which needs to be used in the next optimization phase of the optimizer.

[0082] like Figure 5 As shown, in step S510, after the current back propagation phase ends, at least a portion of the optimizer state of the optimizer used in the optimization phase is unloaded from the first storage device to the second storage device.

[0083] For example, the optimizer can be considered an algorithmic program module running on the first processor or another processor, and the optimizer state therein is only used during the optimization phase. Therefore, after completing the optimization phase for a time step (e.g., corresponding to a batch), the offload scheduler can offload the current optimizer state to the second storage device (corresponding to the D2H process). The optimizer state is then stored on the second storage device during the next forward propagation phase.

[0084] In step S520 , during the next back-propagation phase, the at least a portion of the optimizer state is loaded from the second storage device to the first storage device during an idle period other than a period for loading target activation data in the target activation data set.

[0085] Since the optimizer state unloaded to the second storage device (CPU) is to be used in the next optimization stage to help better adjust the model parameters, the optimizer state needs to be reloaded to the first storage device for use by the first processor (GPU) before the start of the next optimization stage, and the next optimization stage is after the next back propagation stage ends. In addition, during the next back propagation stage, the corresponding saved activation data of the forward propagation stage can be gradually released after being used for back propagation calculations, and the storage resource pressure will be reduced. Therefore, during the next back propagation stage, the optimizer state can be gradually reloaded to the first storage device asynchronously with the back propagation calculations, so that it can be used by the first processor for the next round of optimization update of the model parameters.

[0086] For example, Figure 6 An example process of an optimizer state offloading process is shown.

[0087] The optimizer state may include states corresponding to all weight parameters of the model (e.g., w1_state, w2_state, etc.), which may be registered as objects of the offload scheduler. The offload scheduler may write at least a portion of the optimizer state to the second storage device immediately after completing an optimization phase (i.e., a D2H process), retain the optimizer state in the second storage device during the next forward propagation phase, and reload the optimizer state to the first storage device (i.e., an H2D process) during the next backward propagation phase for use in the next optimization phase after the next backward propagation phase.

[0088] Optionally, the offload scheduler can determine the specific loading time for the state corresponding to each weight parameter. Since the expected computation time of the backpropagation calculation at some network layers is longer than the expected loading time of the target activation data in parallel, there will be a certain time interval between the loading of two target activation data. This time interval can be used to load the state corresponding to all weight parameters.

[0089] Since the calculations in the back-propagation phase do not depend on the optimizer state, the optimizer state loading process does not need to consider the activation data required by each network layer in the back-propagation phase, that is, it does not need to consider the back-propagation calculations. It only needs to ensure that the optimizer state loading process does not occupy the communication resources (e.g., timing and bandwidth) required for loading the activation data of each target. Figure 4B ,Although Figure 4B The trial run in the example assumes that each candidate activation data is unloaded, but assumes that candidate activation data Act 3 and candidate activation data Act 2 are target activation data and are thus unloaded to the second storage device. Since the time taken for the back propagation calculations performed in parallel with the loading of these two target activation data is greater than the time taken for each of the two target activation data to be loaded, the idle period between the loading processes of activation data Act 3 and activation data Act 2, the idle period between the loading processes of activation data Act 2 and activation data Act 1, and the idle period after the loading of activation data Act 1 is completed can be used to load the optimizer state.

[0090] In addition, combined Figure 5 and Figure 6 The scheme described in the offloading optimizer state to save the memory space of the first processor (eg GPU) can be used in conjunction with the reference Figures 3 to 4B The dynamic unloading activation data scheme is combined to form unloading strategies of different granularities and levels. Figure 5 The method shown may also include: in response to the available storage space of the first storage device associated with the first processor being less than a first storage threshold, determining that in addition to unloading the target activation data set to the second storage device associated with the second processor, at least a portion of the optimizer state of the optimizer used for the training process is also unloaded from the first storage device to the second storage device. Optionally, the method may also include: in response to the available storage space of the first storage device associated with the first processor being greater than or equal to the first storage threshold, determining that only the target activation data set is unloaded to the second storage device. The various data after unloading can be loaded back to the first storage device from the second storage device at an appropriate time as described above.

[0091] pass Figure 5 and Figure 6The data processing method shown in the training process of the deep learning model further considers the offloading of the optimizer state, so that it can form a variety of granularity offloading strategies together with the activation data, so that the appropriate offloading strategy can be selected according to actual needs (for example, the available storage space of the current video memory), thereby reducing the pressure on the storage device while improving the training effect and efficiency.

[0092] In addition to the activation data and optimizer state output by each network layer, the weight values ​​of the weight parameters of each network layer are also a significant factor contributing to the storage pressure on the first storage device. Therefore, when the storage resources of the first storage device are extremely limited and the deep learning model is relatively large and thus has a large number of weight parameters, it is also possible to consider offloading the weight values ​​of the weight parameters of each network layer.

[0093] In addition, since the current weight values ​​of the weight parameters of each network layer are used in both the forward propagation stage and the backward propagation stage, after the forward propagation stage is completed, similar to the activation data, the target activation data and corresponding weight values ​​of certain network layers required for the backward propagation stage can be unloaded from the first storage device to the second storage device at the same time, and when to be used in the backward propagation stage, the target activation data and corresponding weight values ​​for each of these network layers will be reloaded to the first storage device at the same time for use by the first processor (such as a GPU) for backward propagation calculations.

[0094] For example, Figure 7 A flow chart of a data processing method for a deep learning model training process according to an embodiment of the present application is shown. Figure 7 The data processing method may be Figure 3 and Figure 5 Supplement or replace certain steps of the data processing method.

[0095] like Figure 7 As shown, in step 710, in the forward propagation phase, each target activation data and its associated weight value in the target activation data set are unloaded from the first storage device to the second storage device.

[0096] The target activation data set here can also be determined based on the storage state of the first storage device and the communication state associated with the first processor, as described above in step S310. The specific determination process can be referred to the above description. In other words, each target activation data in the target activation data set, along with the weight value calculated by the target activation data in the forward propagation calculation, can be offloaded to the second storage device to further reduce the storage pressure of the first storage device.

[0097] In step S720, in the back propagation stage, for each target activation data in the target activation data set, before using the target activation data and its associated weight value to perform back propagation calculation, the target activation data and its associated weight value are loaded from the second storage device to the first storage device.

[0098] In this way, the first processor can normally utilize the target activation data and its associated weight values ​​to perform back propagation calculations.

[0099] Optionally, in step S730, in the back propagation stage, for each target activation data in the target activation data set, after the target activation data and its associated weight value are used for back propagation calculation, the target activation data in the first storage device is released but the weight value associated with the target activation data is not released, or the weight value associated with the target activation data is unloaded to the second storage device again.

[0100] Since the current weight values ​​associated with each network layer need to be updated by the subsequent optimization stage after the corresponding back propagation calculation is completed and then used in the next forward propagation stage, after each target activation data and its associated weight value are used for the back propagation calculation, the weight value can be saved at the first storage device without releasing it, which can avoid the reloading process and thus improve performance. In this case, the unreleased weight value can be used in subsequent stages. After the updated weight value is used in the next forward propagation stage, the updated weight value is unloaded to the second storage device together with the new activation data, and then reloaded to the first storage device when it is to be used in the next back propagation stage. In addition, since the activation data output by each network layer is different, and the activation data output by the same network layer for different training samples is also different, it can be released directly.

[0101] Alternatively, after the target activation data and its associated weight values ​​are used for back propagation calculation, the target activation data can be released and the associated weight values ​​can be unloaded to the second storage device again and reloaded to the first storage device before being used in the subsequent stage. This approach can save storage space in the first storage device, for example, Figure 5-Figure 6 The described case of unloading the optimizer state may enable the optimizer state to be better loaded into the first storage device.

[0102] Figure 8 An example process for offloading activation data and its associated weight values ​​is shown.

[0103] Figure 8 and Figure 4BSimilar to the above, the activation data Act 1-Act 4 for a training sample are replaced with a combination of activation data and weight values ​​(represented by tensor data tensor). Similarly, these four tensor data are registered as objects of the offload scheduler, so the offload scheduler can determine the target activation data and its associated weight value combination to be offloaded (called target tensor data) as described above, such as Figure 4B . In the back propagation phase, the back propagation calculation of the fifth network layer requires tensor data 4, so the unloaded tensor data 4 needs to be reloaded before the back propagation calculation begins. Similarly, the back propagation calculation of the third network layer requires tensor data 2, so the unloaded tensor data 2 needs to be reloaded before the back propagation calculation begins. As for tensor data 1 and tensor data 3, since these two tensor data have not been unloaded, they are still stored in the first storage device and can be directly used by the first processor for back propagation calculation.

[0104] In addition, since this offloading solution can also be combined with the optimizer state offloading mentioned above, Figure 8 The figure also schematically illustrates the process of unloading the optimizer state using the unloading scheduler.

[0105] pass Figure 7 and Figure 8 The data processing method shown in the training process of the deep learning model further considers the unloading of weight values ​​associated with the target activation data, so that multiple granularity unloading strategies can be formed together with the activation data and the optimizer state, so that the appropriate unloading strategy can be selected according to the actual situation (for example, the available storage space of the current video memory), thereby reducing the pressure on the storage device while improving the training effect and efficiency.

[0106] Therefore, in summary, three different levels of uninstallation strategies can be configured:

[0107] Uninstallation strategy 1: Only the activation data is uninstalled, and the activation data can be a part of all activation data.

[0108] Unloading strategy 2: Unloading the optimizer state, one of the weight values ​​associated with the activation data, and the activation data simultaneously.

[0109] Unloading strategy 3: Unload the activation data, optimizer status, and weight values ​​associated with the activation data at the same time.

[0110] For example, the offload policy may be determined according to available storage space at a first storage device associated with the first processor.

[0111] For example, in response to the available storage space of the first storage device being greater than or equal to a first storage threshold, it is determined that only the target activation data set (determined in the manner described above) is unloaded to the second storage device; in response to the available storage space of the first storage device being less than the first storage threshold and greater than or equal to the second storage threshold, it is determined that i) the optimizer state of the optimizer used for the training process or the weight value set associated with the target activation data set, and ii) the target activation data set are unloaded to the second storage device; and in response to the available storage space of the first storage device being less than the second storage threshold, it is determined that the optimizer state of the optimizer used for the training process, the weight value set associated with the target activation data set, and the target activation data set are unloaded to the second storage device.

[0112] Alternatively, in response to the available storage space of the first storage device being greater than or equal to the first storage threshold, the available storage space of the first storage device being greater than or equal to the first storage threshold for a predetermined period of time. A similar explanation also applies to the comparison of the available storage space of the first storage device with other storage thresholds.

[0113] Therefore, with this approach, even when storage resources are dynamically changing, such as when cloud servers or others preempt resources, and the system reports an OOM (out of memory) error, the program doesn't crash immediately. Instead, it updates the offload policy to schedule training, allowing training to continue. If a high level of remaining storage resources is detected for an extended period, the offload scheduler will also update the offload policy in real time to allocate more storage resources, improving training efficiency.

[0114] On the other hand, an embodiment of the present application also provides a data processing method for the training process of a deep learning model.

[0115] Figure 9 FIG2 shows a flow chart of another data processing method for training a deep learning model according to an embodiment of the present application. Similarly, the data processing method can be executed by a first processor equipped with the deep learning model, for example, Figure 1 The GPU shown here can be used to execute the Figure 1 The deep learning model and / or various data generated during the training process can be stored in a first storage device associated with the first processor.

[0116] like Figure 9As shown, in step S910, in response to the available storage space of the first storage device associated with the first processor carrying the deep learning model being greater than or equal to a first storage threshold, only the target activation data set to be unloaded is unloaded to the second storage device associated with the second processor.

[0117] For example, the target activation data set to be offloaded may include a portion of all activation data to be transferred between every two adjacent network layers in the deep learning model in the forward propagation phase. The portion of activation data may be a fixed activation data set or a fixed activation data set as previously described. Figure 3 The activation data set determined in the manner described in FIG. 4 may be an activation data set dynamically determined based on at least one of a storage state of the first storage device and a communication state associated with the first processor.

[0118] In step S920, in response to the available storage space of the first storage device being less than a first storage threshold and greater than or equal to a second storage threshold, i) at least a portion of the optimizer state of the optimizer used for the training process or the set of weight values ​​associated with the target activation data set, and ii) the target activation data set are unloaded to the second storage device.

[0119] In step S930, in response to the available storage space of the first storage device being less than a second storage threshold, the optimizer state used for the training process, the weight value set associated with the target activation data set, and the target activation data set are unloaded to the second storage device.

[0120] In other words, the currently applicable unloading strategy can be determined from the unloading strategies at different levels according to the different situations of available storage space, and the unloading operation from the first storage device to the second storage device can be performed accordingly, and the system can be reloaded from the second storage device to the first storage device at an appropriate time. Therefore, in the process of dynamic changes in storage resources, such as cloud servers or when others seize resources, when the system reports OOM (out of memory), the program will not crash directly, but will re-update the unloading strategy to schedule so that training can continue. When it is found that there is a lot of remaining storage space for a long time, the unloading scheduler will also update the unloading strategy in real time to occupy more storage space and improve training efficiency.

[0121] According to another aspect of the present application, a data processing device for use in the training process of a deep learning model is also provided.

[0122] Figure 10 The structure block diagram of the data processing device in the training process of the deep learning model according to the embodiment of the present application is shown. The device 1000 may include Figure 1The portion of the device side shown may include, for example, a first processor and an associated first storage device (such as a memory), or may include Figure 1 Other processors and associated storage devices on the device side are shown.

[0123] like Figure 10 As shown, the apparatus 1000 may include a determining module 1010 , an unloading module 1020 , and a loading module 1030 .

[0124] The determination module 1010 can be configured to determine a target activation data set from the candidate activation data sets to be unloaded based on at least one of a storage state of a first storage device associated with a first processor carrying a deep learning model and a communication state associated with the first processor.

[0125] Optionally, the set of candidate activation data to be offloaded includes all activation data to be transferred between every two adjacent network layers in the deep learning model in the forward propagation phase, or a portion of all activation data.

[0126] Optionally, when determining the target activation data set, determination module 1010 may determine the target activation data set from the candidate activation data set to be unloaded based on the expected loading time corresponding to each candidate activation data set, wherein the communication state associated with the first processor is associated with the expected loading time corresponding to each candidate activation data set. Alternatively, the expected loading time corresponding to each candidate activation data set may be determined through a trial run. The specific operation of the trial run has been described above and will not be repeated here.

[0127] The unloading module 1020 may be configured to unload the target activation data set from the first storage device to the second storage device associated with the second processor during a forward propagation phase of the training process.

[0128] Optionally, the second processor may be Figure 1 The processor on the host side shown is, for example, a central processing unit (CPU), and the second storage device may be a storage device at the CPU, for example, a memory.

[0129] The loading module 1030 can be configured to load the target activation data from the second storage device to the first storage device before performing backpropagation calculation using the target activation data for each target activation data in the target activation data set during the backpropagation phase of the training process.

[0130] Optionally, the loading module 1030 can load the target activation data required for the back propagation calculation for the n-1th network layer and unloaded to the second storage device into the first storage device in parallel during the back propagation calculation for the nth network layer of the deep learning model in the back propagation phase, where n is an integer greater than 2.

[0131] Optionally, the training process includes an optimization phase utilizing the optimizer after the back-propagation phase, and the unloading module 1020 may be configured to unload the optimizer state of the optimizer used for the optimization phase from the first storage device to the second storage device after the back-propagation phase ends, and the loading module 1030 may load the optimizer state from the second storage device to the first storage device during an idle period other than a period for loading the target activation data in the target activation data set during the next back-propagation phase.

[0132] Optionally, the unloading module 1020 may be further configured to unload each of the target activation data sets and their associated weight values ​​from the first storage device to the second storage device during the forward propagation phase of the training process, and the loading module 1030 may be further configured to load the target activation data and their associated weight values ​​from the second storage device to the first storage device during the reverse propagation phase of the training process for each target activation data in the target activation data set before performing reverse propagation calculations using the target activation data and their associated weight values. Optionally, during the reverse propagation phase of the training process, for each target activation data in the target activation data set, after the target activation data and its associated weight values ​​are used for reverse propagation calculations, the target activation data in the first storage device may be released, but the weight value associated with the target activation data may not be released, or the weight value associated with the target activation data may be unloaded to the second storage device again.

[0133] Optionally, the determination module 1010 can be configured to automatically determine an appropriate unloading strategy according to different actual situations, for example, in response to the available storage space of the first storage device being greater than or equal to a first storage threshold, determining to unload only the target activation data set to the second storage device; in response to the available storage space of the first storage device being less than the first storage threshold and greater than or equal to a second storage threshold, determining to unload i) at least a portion of the optimizer state of the optimizer used for the training process or the weight value set associated with the target activation data set, and ii) the target activation data set to the second storage device; and in response to the available storage space of the first storage device being less than the second storage threshold, determining to unload the optimizer state of the optimizer used for the training process, the weight value set associated with the target activation data set, and the target activation data set to the second storage device.

[0134] Therefore, according to the reference Figure 10 The described data processing device for training a deep learning model can determine a reasonable set of target activation data to be offloaded from candidate activation data based on at least one of the storage state and communication state associated with the first processor, depending on the actual application scenario. This automatically determines the activation data to be offloaded, adapting to the current scenario and adapting to dynamic changes in storage resources. Furthermore, because the target activation data is determined based on the communication state, activation data with long loading times can be minimized from being offloaded, minimizing latency and maximizing training efficiency based on the offloading scheme. Furthermore, different levels of offloading strategies can be selected based on available storage space. Therefore, when storage resources dynamically change, such as when a cloud server or another server preempts resources, and the system reports an out-of-memory (OOM) error, the program does not crash immediately. Instead, the offloading strategy is updated to schedule the training process, allowing training to continue. If a high level of remaining storage space is detected for an extended period, the offloading strategy is updated in real time to occupy more storage space, improving training efficiency.

[0135] In addition, in other embodiments, the data processing device 1000 can determine the currently applicable unloading strategy from different levels of unloading strategies according to different situations of available storage space, and perform an unloading operation from the first storage device to the second storage device accordingly, and reload from the second storage device to the first storage device at an appropriate time.

[0136] For example, the determination module 1010 may be configured to, in response to the available storage space of a first storage device associated with the first processor carrying the deep learning model being greater than or equal to a first storage threshold, determine to offload only the target activation data set to be offloaded to a second storage device associated with the second processor, and the unloading module 1020 may be configured to perform the offloading operation accordingly, and then the loading module 1030 may load the offloaded data back to the first storage device at an appropriate time (e.g., in parallel with the backpropagation calculation). Alternatively, the determination module 1010 may be configured to, in response to the available storage space of the first storage device being less than the first storage threshold and greater than or equal to a second storage threshold, determine to offload i) at least a portion of the optimizer state of the optimizer used for the training process or the set of weight values ​​associated with the target activation data set, and ii) the target activation data set to the second storage device, and the unloading module 1020 may be configured to perform the offloading operation accordingly, and then the loading module 1030 may load the offloaded data back to the first storage device at an appropriate time. Alternatively, the determination module 1010 can be configured to determine that the optimizer state used for the training process, the set of weight values ​​associated with the target activation data set, and the target activation data set will be unloaded to the second storage device in response to the available storage space of the first storage device being less than a second storage threshold, and the unloading module 1020 can be configured to perform the unloading operation accordingly, and then the loading module 1030 can load the unloaded data back to the first storage device at an appropriate time.

[0137] More details on the operations in each module are in the previous reference Figure 3 The contents described are basically the same, so the detailed description of these operations is omitted here.

[0138] In addition, although Figure 10 The above modules are shown in an exemplary manner, but it should be understood that the device 1000 can be divided into more or fewer modules according to different functions, or each module can be divided into further sub-modules, or two or more modules can be combined into one module. In some example embodiments, each module or further divided sub-module can be implemented using electronic hardware (for example, the first processor or other processors on the device side, such as DSP, ASIC, FPGA or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, etc.), computer software (for example, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), erasable programmable ROM (EPROM), etc.), or a combination of the two.

[0139] According to another aspect of the present application, a processor is provided for executing the above-mentioned Figure 3-Figure 9Optionally, the processor may include Figure 1 The first processor at the device end shown may include Figure 1 The processor may have an associated memory, so that the computer program or instruction on the memory can be executed to implement the aforementioned reference Figure 3-Figure 9 The data processing method shown. As an example, other processors that perform the above method may include integrated circuit chips that have signal processing capabilities. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, to implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The processor may also include or be combined with a memory, and the memory may include a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and may also store a computer-executable program or instructions. When the computer-executable program or instructions are executed, the processor may perform the various operations described above. The internal memory may also store a computer-executable program. When the computer-executable program is executed, the processor may perform the various operations described above. The non-volatile storage medium may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. It should be noted that memory of the methods described herein is intended to comprise, but not be limited to, these and any other suitable types of memory.

[0140] According to another aspect of the present application, a computer readable storage medium is provided, on which a computer program or instruction is stored. When the computer program or instruction is executed by a processor, the processor executes the above-mentioned Figure 3-Figure 9 The computer-readable storage medium may be the non-volatile storage medium as described above.

[0141] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, and the module, program segment, or part of the code contains at least one executable instruction for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0142] In general, various example embodiments of the present invention may be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When aspects of embodiments of the present invention are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented, as non-limiting examples, in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or a controller or other computing device, or some combination thereof.

[0143] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It should also be understood that terms such as those defined in common dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology and should not be interpreted in an idealized or highly formal sense, unless expressly defined as such herein.

[0144] The above is an illustration of the present invention and should not be considered as limiting thereof. Although several exemplary embodiments of the present invention have been described, it will be readily understood by those skilled in the art that many modifications may be made to the exemplary embodiments without departing from the novel teachings and advantages of the present invention. Therefore, all such modifications are intended to be included within the scope of the present invention as defined by the claims. It should be understood that the above is an illustration of the present invention and should not be considered as being limited to the specific embodiments disclosed, and modifications to the disclosed embodiments and other embodiments are intended to be included within the scope of the appended claims. The present invention is defined by the claims and their equivalents.

Claims

1. A data processing method for training a deep learning model, comprising: Determining a target activation data set from candidate activation data sets to be offloaded based on at least a communication state associated with a first processor hosting the deep learning model; During a forward propagation phase of the training process, unloading the target activation data set from a first storage device associated with the first processor to a second storage device associated with a second processor; as well as In a back-propagation phase of the training process, for each target activation data in the target activation data set, before performing back-propagation calculation using the target activation data, the target activation data is loaded from the second storage device to the first storage device, The step of determining the target activation data set from the candidate activation data set to be uninstalled includes: performing a trial run operation on the deep learning model to determine an expected loading time corresponding to each candidate activation data, wherein the trial run operation is performed regularly or periodically, and wherein the communication state associated with the first processor is associated with the expected loading time corresponding to each candidate activation data; and The target activation data set is determined from the candidate activation data sets to be unloaded based on the expected loading time corresponding to each candidate activation data set.

2. The data processing method according to claim 1, wherein: The candidate activation data set to be offloaded includes activation data to be transferred between every two adjacent network layers in the deep learning model in the forward propagation stage.

3. The data processing method according to claim 1, further comprising: In the back-propagation phase, during the back-propagation calculation for the n-th network layer of the deep learning model, the target activation data required for the back-propagation calculation for the n-1-th network layer and unloaded to the second storage device are loaded into the first storage device in parallel, where n is an integer greater than 2.

4. The data processing method according to claim 1, wherein: The trial operation includes the following operations: In a forward propagation phase of a trial run training process of the trial run operation, unloading each candidate activation data in the set of candidate activation data to be unloaded from the first storage device to the second storage device; In a back-propagation phase of the trial run training process of the trial run operation, for each candidate activation data in the set of candidate activation data to be unloaded, before performing back-propagation calculation using the candidate activation data, loading the candidate activation data from the second storage device to the first storage device; In which, during the back propagation stage of the trial run training process of the trial run operation, during the back propagation calculation for the nth network layer of the deep learning model, the candidate activation data required for the back propagation calculation for the n-1th network layer is loaded into the first storage device in parallel, where n is an integer greater than 2.

5. The data processing method according to claim 4, wherein: Based on the expected loading time corresponding to each candidate activation data, a target activation data set is determined from the candidate activation data set to be unloaded, including: Determining a subset of candidate activation data from the set of candidate activation data to be unloaded based on an expected loading time corresponding to each candidate activation data, wherein the expected loading time corresponding to each candidate activation data in the subset of candidate activation data is greater than an expected computation time of a backpropagation computation performed in parallel with the loading of the candidate activation data; For each candidate activation data in the candidate activation data subset, determining a time difference between an expected loading time corresponding to the candidate activation data and an expected computation time of a backpropagation calculation performed in parallel with the loading of the candidate activation data as the expected idle duration corresponding to the candidate activation data; and The target activation data set is determined based on the expected idle duration corresponding to each candidate activation data in the candidate activation data subset.

6. The data processing method according to claim 5, wherein: Determining the target activation data set based on the expected idle duration corresponding to each candidate activation data in the candidate activation data subset includes: excluding, from the candidate activation data set to be uninstalled, each candidate activation data in the candidate activation data subset whose corresponding expected idle duration is greater than a predetermined duration threshold, to obtain the target activation data set; or When the sum of the expected idle durations corresponding to the activation data in the candidate activation data subset is less than or equal to a second predetermined duration threshold, the candidate activation data set to be uninstalled is used as the target activation data set.

7. The data processing method according to claim 1, wherein the training process includes an optimization phase using an optimizer after the back-propagation phase, The method further comprises: After the back-propagation phase ends, unloading at least a portion of the optimizer state of the optimizer used in the optimization phase from the first storage device to the second storage device; as well as During a next back-propagation phase, the at least a portion of the optimizer state is loaded from the second storage device to the first storage device during an idle period other than a period for loading target activation data in the target activation data set.

8. The data processing method according to claim 1 or 7, further comprising: During the forward propagation phase of the training process, unloading each target activation data and its associated weight value in the target activation data set from a first storage device to a second storage device; as well as In the back-propagation phase of the training process, for each target activation data in the target activation data set, the target activation data and its associated weight value are loaded from the second storage device to the first storage device before back-propagation calculation is performed using the target activation data and its associated weight value.

9. The data processing method according to claim 8, further comprising: In the back-propagation phase of the training process, for each target activation data in the target activation data set, after the target activation data and its associated weight value are used for back-propagation calculation, the target activation data in the first storage device is released but the weight value associated with the target activation data is not released, or the weight value associated with the target activation data is unloaded to the second storage device again.

10. The data processing method according to claim 1, further comprising: After determining the target activation data set, In response to an available storage space of the first storage device being greater than or equal to a first storage threshold, determining to offload the target activation data set only to the second storage device; In response to the available storage space of the first storage device being less than the first storage threshold and greater than or equal to a second storage threshold, determining to offload i) at least a portion of optimizer states of an optimizer used in the training process or a set of weight values ​​associated with the target activation data set, and ii) the target activation data set to the second storage device; as well as In response to the available storage space of the first storage device being less than a second storage threshold, determining to unload at least a portion of the optimizer state of the optimizer used for the training process, the set of weight values ​​associated with the target activation data set, and the target activation data set to the second storage device.

11. The data processing method according to claim 1 , further comprising: In response to the available storage space of the first storage device being greater than or equal to a preset threshold, determining that the target activation data set is an empty set; as well as In response to the available storage space of the first storage device being less than a preset threshold, the target activation data set is determined to be at least a part of the candidate activation data set.

12. The data processing method according to claim 1, wherein: The first processor includes a graphics processing unit, a general-purpose graphics processing unit, or a tensor processing unit; and The second processor includes a central processing unit.

13. A data processing method for training a deep learning model, comprising: Determining a target activation data set from candidate activation data sets to be offloaded based on at least a communication state associated with a first processor hosting the deep learning model; In response to an available storage space of a first storage device associated with the first processor being greater than or equal to a first storage threshold, only offloading the target activation data set to be offloaded to a second storage device associated with a second processor; In response to the available storage space of the first storage device being less than the first storage threshold and greater than or equal to a second storage threshold, offloading i) an optimizer state of an optimizer used in the training process or a set of weight values ​​associated with the target activation data set, and ii) the target activation data set to the second storage device; as well as In response to the available storage space of the first storage device being less than a second storage threshold, unloading the optimizer state used for the training process, the set of weight values ​​associated with the target activation data set, and the target activation data set to the second storage device, The step of determining the target activation data set from the candidate activation data set to be uninstalled includes: performing a trial run operation on the deep learning model to determine an expected loading time corresponding to each candidate activation data, wherein the trial run operation is performed regularly or periodically, and wherein the communication state associated with the first processor is associated with the expected loading time corresponding to each candidate activation data; and The target activation data set is determined from the candidate activation data sets to be unloaded based on the expected loading time corresponding to each candidate activation data set.

14. A data processing device for use in a deep learning model training process, comprising: a determination module, configured to determine a target activation data set from candidate activation data sets to be offloaded based at least on a communication state associated with a first processor carrying the deep learning model; an unloading module, configured to unload the target activation data set from a first storage device associated with the first processor to a second storage device associated with a second processor during a forward propagation phase of the training process; as well as a loading module configured to, in a back-propagation phase of the training process, for each target activation data in the target activation data set, load the target activation data from the second storage device to the first storage device before performing back-propagation calculation using the target activation data; The uninstallation module determines the target activation data set from the candidate activation data set to be uninstalled, and is configured to: performing a trial run operation on the deep learning model to determine an expected loading time corresponding to each candidate activation data, wherein the trial run operation is performed regularly or periodically, and wherein the communication state associated with the first processor is associated with the expected loading time corresponding to each candidate activation data; and The target activation data set is determined from the candidate activation data sets to be unloaded based on the expected loading time corresponding to each candidate activation data set.

15. A processor configured to execute computer programs or instructions to implement the method according to any one of claims 1 to 13.

16. A computer-readable storage medium having a computer program or instructions stored thereon, wherein when the computer program or instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 13.

Images