Data processing method and device in training process of deep learning model, and medium

During the training of the deep learning model, the activation data is dynamically unloaded according to the processor's storage state and communication state, and the problem of insufficient video memory in deep learning model training is solved, and the training efficiency and adaptability are improved.

CN118981344AActive Publication Date: 2024-11-19北京壁仞科技开发有限公司 +1

Patent Information

Application Number
CN202411471636.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-22
Publication Date
2024-11-19
Estimated Expiration
2044-10-22

AI Technical Summary

Technical Problem

The video memory pressure problem caused by insufficient video memory during deep learning model training affects the efficiency of model training and limits the development and application of larger-scale models.

Method used

During the training of the deep learning model, the appropriate activation data set is dynamically determined and unloaded based on the processor-associated storage and communication states, thereby unloading and loading of data between the forward propagation and backpropagation stages. , optimize the use of storage resources.

Benefits of technology

Improve training efficiency, adapt to dynamic changes in storage resources, reduce waiting time, and minimize storage pressure.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118981344A_ABST
    Figure CN118981344A_ABST
Patent Text Reader

Abstract

The invention provides a data processing method and device in a training process of a deep learning model and a medium. The data processing method can comprise the steps of determining a target activation data set from candidate activation data sets to be unloaded based on at least one of a storage state of a first storage device associated with a first processor carrying the deep learning model and a communication state associated with the first processor; offloading the target activation data set from the first storage device to a second storage device associated with a second processor in a forward propagation phase of the training process; and in a back propagation phase of the training process, for each piece of target activation data in the set of target activation data, loading the target activation data from the second storage device to the first storage device before back propagation calculation is performed by using the target activation data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a data processing method, device, processor, and computer storage medium in a training process of a deep learning model. Background Art

[0002] In recent years, deep learning technology has made significant breakthroughs and is widely used in image recognition, natural language processing, and various intelligent applications. However, training efficient deep learning models must rely on a large amount of computing resources. Traditional central processing units (CPUs) are unable to meet the growing demand for training deep learning models due to their low computing efficiency. Therefore, in order to improve training efficiency, high-performance computing accelerators such as graphics processing units (GPUs), general-purpose graphics processing units (GPGPUs), or tensor processing units (TPUs) are usually required.

[0003] On the other hand, as the Scaling Law in the field of deep learning is gradually verified, researchers continue to push for the expansion of model scale. Large-scale models have shown superior performance and stronger generalization capabilities in multiple tasks. However, the current computing accelerators, whether GPU, GPGPU or TPU, have certain limitations in their video memory capacity. As the scale of the model continues to expand, the problem of insufficient video memory becomes more prominent, resulting in increased video memory pressure during training. This phenomenon not only affects the training efficiency of the model, but also limits the development and application of larger-scale models.

[0004] Therefore, it is necessary to propose an effective solution to the memory pressure faced by deep learning models during the training process. Summary of the invention

[0005] According to one aspect of the present application, a data processing method for a training process of a deep learning model is provided, the method may include: determining a target activation data set from a candidate activation data set to be unloaded based on a storage state of a first storage device associated with a first processor carrying the deep learning model and at least one of a communication state associated with the first processor; in a forward propagation phase of the training process, unloading the target activation data set from the first storage device to a second storage device associated with a second processor; and in a backward propagation phase of the training process, for each target activation data in the target activation data set, loading the target activation data from the second storage device to the first storage device before using the target activation data for backward propagation calculations.

[0006] According to another aspect of the present application, a method for processing data in a training process of a deep learning model is provided, which may include: in response to the available storage space of a first storage device associated with a first processor carrying the deep learning model being greater than or equal to a first storage threshold, only unloading a target activation data set to be unloaded to a second storage device associated with a second processor; in response to the available storage space of the first storage device being less than the first storage threshold and greater than or equal to a second storage threshold, unloading i) an optimizer state of an optimizer used for the training process or a weight value set associated with the target activation data set, and ii) the target activation data set to the second storage device; and in response to the available storage space of the first storage device being less than the second storage threshold, unloading at least a portion of the optimizer state used for the training process, the weight value set associated with the target activation data set, and the target activation data set to the second storage device.

[0007] According to another aspect of the present application, a data processing device for use in a training process of a deep learning model is provided, the device may include: a determination module for determining a target activation data set from a candidate activation data set to be unloaded based on a storage state of a first storage device associated with a first processor carrying the deep learning model and at least one of a communication state associated with the first processor; an unloading module for unloading the target activation data set from the first storage device to a second storage device associated with a second processor in a forward propagation phase of the training process; and a loading module for loading, in a reverse propagation phase of the training process, for each target activation data in the target activation data set from the second storage device to the first storage device before using the target activation data for reverse propagation calculations.

[0008] According to yet another aspect of the present application, a processor is provided, which is configured to execute a computer program or instruction to implement the method as described above.

[0009] According to another aspect of the present application, a computer-readable storage medium is provided, on which a computer program or instruction is stored. When the computer program or instruction is executed by a processor, the processor executes the method as described above.

[0010] Therefore, according to the data processing scheme used in the training process of the deep learning model of the present application, a relatively reasonable target activation data set that can be unloaded can be determined from the candidate activation data based on at least one of the storage state and the communication state associated with the first processor according to the actual application scenario, so that the activation data that needs to be unloaded can be automatically determined in accordance with the current scenario, so that it can be suitable for dynamic changes in storage resources. In addition, since the target activation data is determined by considering the communication state, the activation data with a long loading time can be unloaded as little as possible, so the waiting time can be reduced as much as possible, so that the efficiency of training can be maximized on the basis of the unloading scheme. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments of the present application or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings of the embodiments of the present application.

[0012] Figure 1 A schematic diagram is shown of an application scenario of training a deep learning model according to an embodiment of the present application.

[0013] Figure 2 A schematic diagram showing the principle of an offloading technical solution for solving the problem of storage resource pressure is shown.

[0014] Figure 3 A flow chart of a data processing method in a training process of a deep learning model according to an embodiment of the present application is shown.

[0015] Figure 4A A schematic diagram of uninstalling activation data according to an embodiment of the present application is shown.

[0016] Figure 4B A schematic diagram of the trial operation process according to an embodiment of the present application is shown.

[0017] Figure 5 A flow chart of a data processing method in a training process of a deep learning model according to an embodiment of the present application is shown.

[0018] Figure 6 A process schematic diagram of an optimizer state unloading process according to an embodiment of the present application is shown.

[0019] Figure 7 A flow chart of a data processing method in a training process of a deep learning model according to an embodiment of the present application is shown.

[0020] Figure 8A process schematic diagram of an unloading process of activation data and its associated weight values ​​according to an embodiment of the present application is shown.

[0021] Fig. 9 A flow chart of another data processing method used in the training process of a deep learning model according to an embodiment of the present application is shown.

[0022] Fig.10 A structural block diagram of a data processing device used in a training process of a deep learning model according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only embodiments of a part of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0024] Before describing the present application in detail, various terms that may be used in the present application are explained as follows.

[0025] 1. Activation data or activation value (Activation): The data transmitted between every two adjacent network layers for a training sample in the forward propagation phase of the deep learning training process. This data has been processed by the activation layer (i.e., nonlinear mapping layer) at the previous neural network layer and can be equivalent to the output data of the previous network layer in the adjacent network layer.

[0026] 2. Optimizer: An algorithm used in deep learning to adjust neural network parameters to improve the prediction accuracy of the model. It can change the weights of neurons in the network of the deep learning model so that the network can better fit the data. Examples of optimizers include SGD (stochastic gradient descent) or Adam (adaptive momentum estimation).

[0027] 3. Optimizer State: During the training process, the optimizer maintains a set of variables, such as variables that store historical information about weight gradients. These variables record the update information of model parameters (i.e., the weights of each network layer) during the training process, helping the optimizer to adjust model parameters more effectively.

[0028] 4. Offload: The process of transferring some data from an accelerator (such as a GPU, GPGPU, or TPU) to a central processing unit (CPU) during training. 5. Load: The process of loading certain data from the CPU to the accelerator; 6. D2H: Transfer data from the device side (Device) (for example, GPU, GPGPU or TPU) to the host side (Host) (for example, CPU).

[0029] 7. H2D: Transfers data from the host (e.g., CPU) to the device (e.g., GPU, GPGPU, or TPU).

[0030] 8. Graphics card: A hardware device used to process graphics rendering tasks, located on the device side, such as a GPU, GPGPU, or TPU. It processes images and videos independently of the CPU, which can greatly improve processing performance.

[0031] 9. Video memory: The memory in the graphics card used to store data required for graphics operations, specifically used for operations such as GPU, GPGPU or TPU.

[0032] Figure 1 A schematic diagram is shown of an application scenario of training a deep learning model according to an embodiment of the present application.

[0033] like Figure 1 As shown, the application scenario 100 may include a host end 110 and a device end 120, wherein the host end includes a second processor such as a central processing unit (CPU), and the device end 120 may include a first processor such as a graphics processing unit (GPU). In the present application, an example is given in which the host end includes a CPU and the device end 120 includes a GPU, but it should be understood that the present application is not limited thereto, for example, the device end 120 may also include a general purpose graphics processing unit (GPGPU) or a tensor processing unit (TPU). In addition, both the first processor and the second processor have an associated storage device (e.g., memory), so that various data during processing can be stored.

[0034] The host side 110 is usually regarded as the main device or "host", whose main responsibility is to control the execution of tasks, manage the logical flow of the program, and handle relatively low-intensity computing tasks. The device side 120 is regarded as an auxiliary device or "device", which is specially used to perform high-parallel computing tasks, such as when the deep learning model is deployed on the device side 120, so the device side 120 needs to perform a large number of matrix operations and vector calculations, such as forward propagation, back propagation and parameter update process in deep learning.

[0035] In deep learning, in order to efficiently utilize computing resources and increase training speed, the entire training data set is usually not input into the model for training at one time, but the training data set is split into multiple small batches. Each time a batch is processed, the model performs a forward propagation to calculate the loss, back propagation, and update the parameters, which is a time step. Of course, the entire training data set can also be input into the model for training at one time. Therefore, a training process of a deep learning model usually includes the following steps: First, in the forward propagation stage, the GPU performs forward propagation to calculate the loss value based on the current model parameters and each training sample as input data. In the back propagation stage, the GPU calculates the gradient and uses the optimizer to update the model parameters in the optimization stage. This training process requires a lot of matrix operations, so the GPU plays its advantages here and greatly improves computing efficiency. At the same time, the CPU can continue to handle other tasks during this period, such as monitoring training progress, saving model checkpoints, or performing data enhancement.

[0036] As mentioned earlier, there is a need for a solution to the pressure on storage resources (e.g., GPU memory) faced by deep learning models during training.

[0037] To solve the pressure of storage resources, there are many common methods such as data parallelism, model parallelism, pipeline parallelism, recomputation, offloading and other technologies. Among them, offloading technology is a common solution to the problem of storage resource pressure. Its principle is as follows: Figure 2 shown.

[0038] As mentioned earlier, a training process of a deep learning model can be roughly divided into three stages, including the forward propagation stage, the back propagation stage, and the parameter update stage performed by the optimizer. The forward propagation stage calculates the loss value (loss), and the back propagation stage calculates the gradient of the loss value (loss) relative to each weight (model parameter). Finally, the optimizer updates the parameters according to an optimization algorithm such as the gradient descent method. Figure 2 The scheme shown first offloads some intermediate activation data needed for back propagation to the CPU in the forward propagation stage, and then loads them to the GPU for back propagation when these activation data are needed for back propagation. After back propagation, these activation data can be released immediately, thereby minimizing the video memory consumption.

[0039] For example, Figure 2In the figure, the deep learning model includes multiple neural network layers (5 layers are shown as examples in the figure, but it may not be 5 layers in the actual application process). In the forward propagation stage, an input data (for example, a training sample in the batch corresponding to the forward propagation stage) is used as the input of the first layer, and the output data (i.e., prediction data) is obtained after the calculation of these 5 network layers. The data transmitted between adjacent network layers are various activation data. The GPU predetermines the selection of activation data between the second layer and the third layer and the activation data between the third layer and the fourth layer as the data to be unloaded to the CPU. After the second layer outputs the activation data, in addition to passing the activation data to the third layer, the activation data is also passed to the CPU, that is, the activation data is unloaded to the GPU (i.e., the D2H process), and the activation data is needed when calculating the third layer in the reverse propagation stage. Therefore, before calculating the third layer in the reverse propagation stage, it is necessary to obtain the previously unloaded activation data from the CPU, that is, reload the activation data to the GPU (i.e., the H2D process). The activation data output by the third layer in the forward propagation stage is also a similar transmission process. In addition, for the activation data output by the 1st and 4th layers, since these activation data are not selected to be unloaded to the CPU, the GPU will store these activation data and can be released after the back propagation phase is completed. For example, the activation data output by the 4th layer can be used in the back propagation phase to calculate the 5th layer and then release the activation data output by the 4th layer, or release it after the back propagation phase is completed. In the forward propagation phase, multiple training samples are involved as inputs of the deep learning model, so each training sample will correspond to multiple activation data transmitted between adjacent network layers of the model, and correspondingly, these activation data are used for back propagation calculations for each training sample in the back propagation phase.

[0040] but, Figure 2 The offloading techniques shown in may not be flexible enough to adapt to various training situations and have low training efficiency, as follows: 1. It is difficult to determine which parts to offload to meet the GPU memory requirements, and it requires engineering experience and multiple attempts to gradually adjust. Figure 2 In the present invention, the activation data output by the second layer and the third layer is predetermined as the target activation data for offloading, needs to be determined in advance and is fixed once determined.

[0041] 2. There is a dependency between the back-propagation calculation process in the back-propagation phase and the process of reloading the data previously unloaded to the CPU. For example, before the back-propagation calculation for the third layer in the back-propagation phase, the activation data that has been unloaded to the CPU (the activation data transferred between the second and third layers in the forward propagation phase) needs to be reloaded to the GPU. Therefore, it may take a certain amount of waiting time at some point, and it is difficult to maximize the training efficiency based on the unloading solution.

[0042] 3. During the training process, there are often some dynamic or abnormal situations that affect the entire training process when all training samples are used. For example, during the training process, due to the fragmentation of video memory or the dynamic change of available storage resources due to server resource preemption, the current unloading strategy is unstable and cannot reduce the pressure of the video memory that has changed.

[0043] Therefore, in response to the above problems, an embodiment of the present application provides a data processing solution for the training process of a deep learning model, which adaptively selects which activation data will be unloaded by introducing an unloading scheduler, and optionally considers the weight values ​​and optimizer states of the main sources of storage resource consumption during the unloading training process in addition to unloading activation data. Therefore, various levels of unloading strategies can be implemented according to actual needs.

[0044] The following combination Figure 3-Figure 10 The following is a detailed introduction to the data processing scheme used in the training process of the deep learning model according to the embodiment of the present application.

[0045] Figure 3 A flow chart of a data processing method for a deep learning model training process according to an embodiment of the present application is shown. The data processing method can be executed by a first processor equipped with the deep learning model, for example, Figure 1 The GPU shown here can be used to execute the Figure 1 The deep learning model and / or various data generated during the training process can be stored in a first storage device associated with the first processor.

[0046] like Figure 3 As shown, in step S310, based on the storage status of the first storage device associated with the first processor carrying the deep learning model and at least one of the communication status associated with the first processor, a target activation data set is determined from the candidate activation data sets to be uninstalled.

[0047] For example, the candidate activation data set to be uninstalled includes all activation data to be transferred between every two adjacent network layers in the deep learning model in the forward propagation phase. Optionally, when determining the candidate activation data set, it may be equivalent to determining the indication information of these activation data (for example, a deep learning module, such as a function or network layer of a specific task, which can be used to specify each candidate activation data). For example, the candidate activation data set to be uninstalled may be as follows: Figure 2 The activation data Act 1 to Act 4 are transmitted between the network layers in the forward propagation phase after the training starts. Alternatively, it can be part of the activation data selected from all the activation data according to predefined rules. These candidate activation data are marked in advance and registered with the object of the offload scheduler so that the offload scheduler can know which candidate activation data exist.

[0048] In addition, all activation data passed between every two network layers of the deep learning model are to be used in the back-propagation phase of the deep learning model. The target activation data set is a set of activation data to be unloaded from the first storage device associated with the first processor to the second storage device associated with the second processor in the forward propagation phase, and reloaded back to the first storage device in the back-propagation phase.

[0049] For example, the storage state associated with the first processor may include the available storage space of the first storage device (e.g., the video memory of the GPU) associated with the first processor. The available storage space may be a storage space that can be used for deep learning model training, and may be dynamically changing, for example, due to server resource preemption, etc. The communication state associated with the first processor may include the communication bandwidth and communication volume of the data communication link used to perform unloading of activation data and reloading of activation data, etc., which may affect the unloading speed and reloading speed of activation data, thereby possibly affecting the training speed and training efficiency. Optionally, the available storage space of the first storage device can be actively detected each time the deep learning model needs to be trained, and optionally, the current available storage space can also be detected in real time during the entire model training period, or the available storage space can be passively detected (e.g., an alarm signal of insufficient storage space is received), and it is determined whether part of the activation data needs to be unloaded and the unloading strategy update as will be described later based on the detected available storage space.

[0050] Therefore, when the available storage space of the first storage device (for example, the video memory of the GPU) is not large enough, it may be necessary to select some candidate activation data (i.e., the target activation data set) for unloading and reloading. For example, when the available storage space is greater than or equal to a preset threshold, the target activation data set can be determined as an empty set, that is, no activation data can be unloaded, which can maximize the use of the first storage device. If the available storage space is less than or equal to the preset threshold, it is determined that some candidate activation data need to be selected for unloading and reloading. The preset threshold can be determined based on experience. For example, since the communication state associated with the first processor is associated with the expected time required to reload each activation data, the expected time required to reload each candidate activation data can be further considered to select the target activation data set to be unloaded.

[0051] Alternatively, when determining the target activation data set, the target activation data set may be selected by only considering the expected time consumption of reloading each candidate activation data, for example, selecting activation data with less time consumption for unloading regardless of the size of the available storage space.

[0052] The specific steps of determining the target activation data set will be described in detail later.

[0053] Therefore, since the storage status of the first storage device (such as available storage space) and the communication status associated with the first processor may be different in different application scenarios or at different times, the most suitable target activation data set can be adaptively selected according to actual training, thereby eliminating the need to fix the activation data to be unloaded, thereby improving training efficiency and quality.

[0054] In step S320, during the forward propagation phase of the training process, the target activation data set is unloaded from the first storage device to a second storage device associated with a second processor.

[0055] As mentioned above, in the forward propagation stage, the activation data output by the previous layer can be used as the input data of the next layer, and if the activation data is included in the target activation data set (i.e., determined as activation data to be unloaded), the activation data will also be unloaded to the second storage device associated with the second processor. At the same time, if the activation data is not included in the target activation data set (i.e., determined as activation data that does not need to be unloaded), it means that the activation data is not suitable for unloading, and therefore it can be stored in the first storage device associated with the first processor for subsequent use in the backward propagation stage.

[0056] In step S330, in the back propagation phase of the training process, for each target activation data in the target activation data set, before using the target activation data for back propagation calculation, the target activation data is reloaded from the second storage device to the first storage device.

[0057] As described above, since each target activation data has been unloaded to the second storage device, when the first processor needs to use the target activation data for back propagation calculation, it is necessary to reload the target activation data from the second storage device associated with the second processor to the first storage device associated with the first processor before using the back propagation calculation of the target activation data for use by the first processor. After the back propagation stage is over, all candidate activation data can be released.

[0058] Optionally, in order to improve the training efficiency, the training time needs to be shortened as much as possible. As mentioned above, the back propagation calculation for each network layer requires corresponding activation data. If the activation data is the target activation data that has been unloaded to the second storage device and is included in the target activation data set, then the target activation data is reloaded to the first storage device before the back propagation calculation using the target activation data. Therefore, it can be considered to load the activation data in parallel during the back propagation calculation. That is, in the back propagation stage, during the back propagation calculation for the nth network layer of the deep learning model (for a certain training sample), the target activation data (the activation data output by the n-2th network layer in the forward propagation stage) required for the back propagation calculation for the n-1th network layer (for the same training sample) and unloaded to the second storage device is reloaded to the first storage device in parallel, wherein n is an integer greater than 2 and less than or equal to the total number of network layers of the deep learning model, and the number n of the network layer gradually increases in the direction of the forward propagation stage. Therefore, the time waiting for reloading the activation data can be shortened, and the training speed can be improved. In addition, the start time of the back-propagation calculation for the nth network layer can be aligned with the start time of reloading the corresponding target activation data, so that the reloading of the target activation data will not be performed too early, thereby reducing the storage pressure of the first storage device.

[0059] For example, Figure 4A A schematic diagram of uninstalling activation data according to an embodiment of the present application is shown.

[0060] like Figure 4A As shown, in the forward propagation stage, activation data is transferred between every two network layers. Figure 4Aexemplarily shows five network layers and four activation data (Act 1-Act 4) for a training sample. These four activation data are registered as objects of the offloading scheduler. Therefore, the offloading scheduler can determine the target activation data to be offloaded as described above, such as Figure 4A The activation data Act 2 and the activation data Act 4 shown in are determined as the target activation data to be unloaded. In the back propagation stage, for the same training sample, the back propagation calculation of the 5th network layer requires the activation data Act 4, so the activation data Act 4 that has been unloaded needs to be reloaded before the back propagation calculation starts. Similarly, the back propagation calculation of the 3rd network layer requires the activation data Act 2, so the activation data Act 2 that has been unloaded needs to be reloaded before the back propagation calculation starts. For the activation data Act 1 and the activation data Act 3, since these two activation data have not been unloaded, they are still stored in the first storage device and can be directly used by the first processor for back propagation calculation. The unloading process of the activation data for other training samples is similar.

[0061] Although not in Figure 4A As shown in FIG. 1 , the process of reloading the activation data Act 2 can be performed in parallel with the back propagation calculation process for the 4th network layer, and the process of reloading the activation data Act 4 needs to be performed before the back propagation calculation for the 5th network layer.

[0062] Optionally, when determining the target activation data set, the target activation data set can be determined from the candidate activation data set to be unloaded based on the expected loading time corresponding to each candidate activation data in the candidate activation data set to be unloaded, wherein the communication status associated with the first processor mentioned above is associated with the expected loading time corresponding to each candidate activation data, because the communication status affects the loading speed of data to the first storage device.

[0063] For example, a trial run can be used to determine the expected loading time of each candidate activation data in the current application scenario, during which the unloading scheduler will attempt to unload all candidate activation data to the second storage device associated with the second processor, and load each candidate activation data to the first storage device associated with the first processor when needed in the back propagation phase, thereby ensuring that the training process can run normally.

[0064] For example, the trial run can be performed by the following operations: in the forward propagation stage of the trial run training process, each candidate activation data in the candidate activation data set to be unloaded is unloaded from the first storage device to the second storage device; and in the reverse propagation stage of the trial run training process, for each candidate activation data in the candidate activation data set to be unloaded, before using the candidate activation data for reverse propagation calculation, the candidate activation data is loaded from the second storage device to the first storage device. Similarly, the reverse propagation calculation and the activation data loading can be performed in parallel. For example, in the reverse propagation stage of the trial run training process, during the reverse propagation calculation for the nth network layer, the candidate activation data required for the reverse propagation calculation for the n-1th network layer is loaded into the first storage device in parallel, where n is an integer greater than 2, and the number n of the network layer gradually increases in the direction of the forward propagation stage.

[0065] In addition, considering that the storage state of the first storage device and the communication state associated with the first processor may change with the training process, a trial run operation can be performed regularly or periodically to more accurately determine the expected loading time corresponding to each candidate activation data, thereby more accurately determining the target activation data set.

[0066] Through trial operation, the expected loading time corresponding to each candidate activation data can be obtained. Therefore, when it is used to determine the target activation data set, a candidate activation data subset can be determined from the candidate activation data set to be unloaded based on the expected loading time corresponding to each candidate activation data, wherein the expected loading time corresponding to each candidate activation data in the candidate activation data subset is greater than the expected computational time of the back propagation calculation performed in parallel with the loading of the candidate activation data. Then, for each candidate activation data in the candidate activation data subset, the time difference between the expected loading time corresponding to the candidate activation data and the expected computational time of the back propagation calculation performed in parallel with the loading of the candidate activation data is determined as the expected idle duration corresponding to the candidate activation data; and the target activation data set is determined based on the expected idle duration corresponding to each candidate activation data in the candidate activation data subset.

[0067] Optionally, each candidate activation data in the candidate activation data subset whose corresponding expected idle time is greater than a predetermined time threshold can be excluded from the candidate activation data set to be unloaded to obtain a target activation data set. For example, if the expected idle time corresponding to the candidate activation data is large, it means that the loading time of the candidate activation data is long, which will cause a long waiting time after the parallel execution of the reverse propagation calculation, so try not to unload such candidate activation data; on the contrary, if the expected idle time corresponding to the candidate activation data is small or there is no expected idle time (that is, the expected loading time corresponding to the candidate activation data is less than or equal to the expected calculation time of the parallel reverse propagation calculation), it means that the loading time of the candidate activation data is small, which will not cause a long waiting time after the parallel execution of the reverse propagation calculation, so try to unload such candidate activation data.

[0068] In addition, the target activation data set can also be determined in combination with the available storage space of the first storage device. For example, the amount of activation data to be uninstalled can be determined based on the size of the available storage space of the first storage device. For example, if the available storage space of the first storage device is large enough, no candidate activation data may be uninstalled, that is, the target activation data set is an empty set; if the available storage space of the first storage device is limited and the storage resource pressure is high, a portion of the candidate activation data may be uninstalled, for example, the candidate activation data that is expected to take a long time to uninstall may be preferentially excluded and the candidate activation data may be uninstalled as little as possible so that the available storage space can be used to the maximum, or all candidate activation data that is expected to take less than or equal to a predetermined threshold may be uninstalled. Alternatively, all candidate activation data that is expected to take less than or equal to a predetermined threshold may be uninstalled regardless of the available storage space of the first storage device.

[0069] Figure 4B A schematic diagram of the trial run process is shown. Figure 4B The main focus is on how the candidate activation data that has been unloaded to the second storage device associated with the second processor during the back-propagation phase during the trial run is reloaded to the first storage device associated with the first processor and the associated timing. Figure 4B Still Figure 2 as well as Figure 4A The deep learning model shown includes 5 neural network layers for illustration.

[0070] like Figure 4B As shown, in the back propagation phase, prefetching can be performed asynchronously to ensure that when a candidate activation data is needed, the candidate activation data has been loaded into the first storage device associated with the first processor. At the same time, this prefetching operation should not be performed too early, otherwise it will bring video memory pressure. For example, Figure 4BAs shown, at the beginning of the back-propagation calculation period of each network layer, a reload (pre-fetch) of candidate activation data is started simultaneously and in parallel.

[0071] For example, in Figure 4B In the process, the back propagation calculation needs to be performed for the 5th layer first. Before that, the candidate activation data (Act 4) output by the 4th layer in the forward propagation stage that has been unloaded to the second storage device needs to be reloaded to the first storage device (i.e. H2D) in advance. This process can be performed immediately after starting the back propagation stage, so that the candidate activation data Act 4 will not be pre-fetched too early.

[0072] In addition, when performing back propagation calculations for the 5th layer, the operation of reloading the candidate activation data (Act 3) output by the 3rd layer of the forward propagation stage that has been unloaded to the second storage device to the first storage device (i.e., H2D) can be synchronously carried out, because the back propagation calculations for the 4th layer require the candidate activation data Act 3.

[0073] In some cases, due to the large number of candidate activation data and / or the influence of communication status, the loading process of the candidate activation data (Act n) output by the nth network layer in the forward propagation phase may take longer than the back propagation calculation of the n+2th network layer in parallel. In this case, the back propagation calculation of the n+1th network layer needs to wait for the loading process of the candidate activation data (Act n) to be completed before it can start, because the back propagation calculation of the n+1th network layer needs to use the candidate activation data (Act n). Figure 3 In the forward propagation phase, the execution of the back propagation calculation for the second network layer can only be executed after the loading process of the candidate activation data (Act 1) output by the first network layer in the forward propagation phase is completed, which results in idle time in the back propagation phase. Therefore, the unloading scheduler can determine the candidate activation data to be unloaded as the target activation data set based on the idle time (which may have a duration of 0 or greater than 0) of the back propagation calculation for each network layer in the back propagation phase and the available storage space of the optional first storage device.

[0074] For example, the unloading scheduler can count the performance overhead (corresponding to the idle time in the back propagation phase) caused by reloading each candidate activation data after unloading it. If the performance overhead of a certain candidate activation data is large, when it is determined that some candidate activation data need to be unloaded, the candidate activation data with high performance overhead (corresponding to a longer idle time) can be unloaded as much as possible. In addition, as mentioned above, whether the candidate activation data needs to be unloaded can be determined based on the size of the available storage space of the first storage device to reasonably disable the unloading behavior of the candidate activation data with high performance overhead.

[0075] Optionally, if the idle time of the back propagation calculations for certain network layers in the back propagation stage is less than a predetermined threshold or there is no idle time, the candidate activation data to be unloaded during the back propagation calculations of these network layers can be determined as the target activation data set, regardless of the size of the available storage space of the first storage device.

[0076] Therefore, according to the reference Figures 3 to 4B The described data processing method for the training process of the deep learning model can determine a relatively reasonable target activation data set that can be unloaded from the candidate activation data based on at least one of the storage state and the communication state associated with the first processor according to the actual application scenario, so that the activation data that needs to be unloaded can be automatically determined in accordance with the current scenario, so that it can be applicable to dynamic changes in storage resources. In addition, since the target activation data is determined by considering the communication state, the activation data with a long loading time can be unloaded as little as possible, so the waiting time can be reduced as much as possible, so that the efficiency of training can be maximized on the basis of the unloading scheme.

[0077] Figure 5 A flow chart of a data processing method in a training process of a deep learning model according to an embodiment of the present application is shown. Figure 5 The data processing method may be Figure 3 The data processing method described above may supplement or replace certain steps.

[0078] As mentioned above, the training process of the deep learning model also includes an optimization phase for updating parameters using an optimizer after the back-propagation phase. The optimizer will generate an optimizer state in the optimization phase, which needs to be used in the next optimization phase of the optimizer.

[0079] like Figure 5 As shown, in step S510, after the current back propagation phase ends, at least a portion of the optimizer state of the optimizer used in the optimization phase is unloaded from the first storage device to the second storage device.

[0080] For example, the optimizer can also be regarded as an algorithm program module running on the first processor or other processors, and the optimizer state therein is only used in the optimization phase. Therefore, after completing the optimization phase of a time step (e.g., corresponding to a batch), the unloading scheduler can unload the current optimizer state to the second storage device (corresponding to the D2H process), and then the optimizer state is always stored on the second storage device in the next forward propagation phase.

[0081] In step S520, during the next back-propagation phase, at an idle period other than a period for loading target activation data in the target activation data set, the at least a portion of the optimizer state is loaded from the second storage device to the first storage device.

[0082] Since the optimizer state unloaded to the second storage device (CPU) is to be used in the next optimization stage to help better adjust the model parameters, the optimizer state needs to be reloaded to the first storage device for use by the first processor (GPU) before the next optimization stage starts, and the next optimization stage is after the next back propagation stage ends. In addition, during the next back propagation stage, the corresponding activation data saved in the forward propagation stage can be gradually released after being used for back propagation calculations, and the pressure on storage resources will be reduced. Therefore, during the next back propagation stage, the optimizer state can be gradually reloaded to the first storage device asynchronously with the back propagation calculations, so that it can be used by the first processor for the next round of optimization update of the model parameters.

[0083] For example, Figure 6 An example process of an optimizer state offloading process is shown.

[0084] The optimizer state may include states corresponding to all weight parameters of the model (e.g., w1_state, w2_state, etc.), which may be registered as objects of the unloading scheduler. The unloading scheduler may write at least a portion of the optimizer state to the second storage device (i.e., D2H process) immediately after completing an optimization phase, keep it at the second storage device during the next forward propagation phase, and reload the optimizer state to the first storage device (i.e., H2D process) during the next backward propagation phase for use in the next optimization phase after the next backward propagation phase.

[0085] Optionally, the offload scheduler can determine the specific loading time of the state corresponding to each weight parameter. Since the expected computation time of the backpropagation calculation at some network layers is longer than the expected loading time of the target activation data loaded in parallel, there will be a certain time interval between the loading processes of two target activation data, so these time intervals can be used to load the states corresponding to all weight parameters.

[0086] Since the calculations in the back-propagation phase have no dependency on the optimizer state, the loading process of the optimizer state does not need to consider the activation data required for each network layer in the back-propagation phase, that is, it does not need to consider the back-propagation calculations. It only needs to ensure that the loading process of the optimizer state does not occupy the communication resources (e.g., timing and bandwidth) required for loading the activation data of each target. Figure 4B ,Although Figure 4B The trial run in the example assumes that each candidate activation data is unloaded, but assumes that candidate activation data Act 3 and candidate activation data Act 2 are target activation data and are thus unloaded to the second storage device. Since the time consumption of the back propagation calculations performed in parallel with the loading of these two target activation data is greater than the loading time consumption of each of the two target activation data, the idle period between the loading processes of activation data Act 3 and activation data Act 2, the idle period between the loading processes of activation data Act 2 and activation data Act 1, and the idle period after the loading of activation data Act 1 is completed can be used to load the optimizer state.

[0087] In addition, combined Figure 5 and Figure 6 The scheme described in the offloading optimizer state to save the storage space of the first processor (eg, GPU) can be used in conjunction with reference Figures 3 to 4B The dynamic unloading activation data scheme is combined with the dynamic unloading activation data scheme to form unloading strategies of different granularities and levels. Figure 5 The method shown may also include: in response to the available storage space of the first storage device associated with the first processor being less than the first storage threshold, determining that in addition to unloading the target activation data set to the second storage device associated with the second processor, at least a portion of the optimizer state of the optimizer used for the training process is also unloaded from the first storage device to the second storage device. Optionally, the method may also include: in response to the available storage space of the first storage device associated with the first processor being greater than or equal to the first storage threshold, determining that only the target activation data set is unloaded to the second storage device. The various data after unloading can be loaded back to the first storage device from the second storage device at an appropriate time as described above.

[0088] pass Figure 5 and Figure 6The data processing method shown in the training process of the deep learning model further considers the unloading of the optimizer state, so that it can form a variety of granularity unloading strategies together with the activation data, so that the appropriate unloading strategy can be selected according to actual needs (for example, the available storage space of the current video memory), thereby reducing the pressure on the storage device while improving the training effect and efficiency.

[0089] In addition, in addition to the activation data and optimizer status output by each network layer, the weight value of the weight parameter of each network layer is also an important factor leading to the storage pressure of the first storage device. Therefore, when the storage resources of the first storage device are extremely limited and the deep learning model is relatively large and thus has a large number of weight parameters, it is also possible to consider unloading the weight value of the weight parameter of each network layer.

[0090] In addition, since the current weight values ​​of the weight parameters of each network layer are used in both the forward propagation stage and the backward propagation stage, after the forward propagation stage is completed, similar to the activation data, the target activation data of certain network layers needed in the backward propagation stage and the corresponding weight values ​​can be unloaded from the first storage device to the second storage device at the same time, and when they are to be used in the backward propagation stage, the target activation data and the corresponding weight values ​​for each of these network layers will be reloaded to the first storage device at the same time for use by the first processor (e.g., GPU) for backward propagation calculations.

[0091] For example, Figure 7 A flow chart of a data processing method in a training process of a deep learning model according to an embodiment of the present application is shown. Figure 7 The data processing method may be Figure 3 and Figure 5 The data processing method described above may supplement or replace certain steps.

[0092] like Figure 7 As shown, in step 710, in the forward propagation stage, each target activation data in the target activation data set and its associated weight value are unloaded from the first storage device to the second storage device.

[0093] The target activation data set here is also as described in step S310 above, and can be determined according to the storage state of the first storage device and the communication state associated with the first processor. The specific determination process can refer to the above description. In other words, each target activation data in the target activation data set and the weight value of the target activation data calculated by the target activation data in the forward propagation calculation can be unloaded to the second storage device together to further reduce the storage pressure of the first storage device.

[0094] In step S720, in the back propagation stage, for each target activation data in the target activation data set, before using the target activation data and its associated weight value for back propagation calculation, the target activation data and its associated weight value are loaded from the second storage device to the first storage device.

[0095] In this way, the first processor can normally utilize the target activation data and its associated weight values ​​to perform back-propagation calculations.

[0096] Optionally, in step S730, in the back propagation stage, for each target activation data in the target activation data set, after the target activation data and its associated weight value are used for back propagation calculation, the target activation data in the first storage device is released but the weight value associated with the target activation data is not released, or the weight value associated with the target activation data is unloaded to the second storage device again.

[0097] Since the current weight value associated with each network layer needs to be updated by the subsequent optimization stage for the next forward propagation stage after the corresponding back-propagation calculation is completed, the weight value can be saved at the first storage device without releasing it after each target activation data and its associated weight value are used for the back-propagation calculation, which can avoid the reloading process and thus improve performance. In this case, the unreleased weight value can be used in subsequent stages. After the updated weight value is used in the next forward propagation stage, the updated weight value is unloaded to the second storage device together with the new activation data, and then reloaded to the first storage device when it is used in the next back-propagation stage. In addition, since the activation data output by each network layer is different, and the activation data output by the same network layer for different training samples is also different, it can be released directly.

[0098] Alternatively, after the target activation data and its associated weight values ​​are used for back propagation calculation, the target activation data may be released, and the associated weight values ​​may be unloaded to the second storage device again, and reloaded to the first storage device before being used in a subsequent stage. This approach can save storage space in the first storage device, for example, Figure 5-Figure 6 The described case of unloading the optimizer state may enable the optimizer state to be better loaded to the first storage device.

[0099] Figure 8 An example process for offloading activation data and its associated weight values ​​is shown.

[0100] Figure 8 and Figure 4BSimilarly, the activation data Act 1-Act 4 for a training sample are replaced with a combination of activation data and weight values ​​(represented by tensor data tensor). Similarly, these four tensor data are registered as objects of the offload scheduler, so the offload scheduler can determine the combination of target activation data and its associated weight values ​​to be offloaded (called target tensor data) as described above, such as Figure 4B . In the back propagation phase, the back propagation calculation of the fifth network layer requires tensor data 4, so the unloaded tensor data 4 needs to be reloaded before the back propagation calculation starts. Similarly, the back propagation calculation of the third network layer requires tensor data 2, so the unloaded tensor data 2 needs to be reloaded before the back propagation calculation starts. As for tensor data 1 and tensor data 3, since these two tensor data have not been unloaded, they are still stored in the first storage device and can be directly used by the first processor for back propagation calculation.

[0101] In addition, since this unloading solution can also be combined with the optimizer state unloading mentioned above, Figure 8 The figure also schematically shows the process of unloading the optimizer state using the unloading scheduler.

[0102] pass Figure 7 and Figure 8 The data processing method shown in the training process of the deep learning model further considers the unloading of weight values ​​associated with the target activation data, and thus can constitute a variety of granularity unloading strategies together with the activation data and the optimizer state, so that the appropriate unloading strategy can be selected according to the actual situation (for example, the available storage space of the current video memory), thereby improving the training effect and efficiency while reducing the pressure on the storage device.

[0103] Therefore, in summary, three different levels of uninstallation strategies can be configured: Uninstallation strategy 1: Only the activation data is uninstalled, and the activation data may be a part of all the activation data.

[0104] Uninstallation strategy 2: Uninstall the optimizer state and one of the weight values ​​associated with the activation data and the activation data at the same time.

[0105] Uninstallation strategy 3: Uninstall the activation data, optimizer status, and weight values ​​associated with the activation data at the same time.

[0106] For example, the offload policy may be determined according to available storage space at a first storage device associated with the first processor.

[0107] For example, in response to the available storage space of the first storage device being greater than or equal to a first storage threshold, it is determined that only the target activation data set (determined in the manner described above) is unloaded to the second storage device; in response to the available storage space of the first storage device being less than the first storage threshold and greater than or equal to the second storage threshold, it is determined that i) the optimizer state of the optimizer used for the training process or the set of weight values ​​associated with the target activation data set, and ii) the target activation data set are unloaded to the second storage device; and in response to the available storage space of the first storage device being less than the second storage threshold, it is determined that the optimizer state of the optimizer used for the training process, the set of weight values ​​associated with the target activation data set, and the target activation data set are unloaded to the second storage device.

[0108] Optionally, in response to the available storage space of the first storage device being greater than or equal to the first storage threshold, the available storage space of the first storage device being greater than or equal to the first storage threshold for a predetermined period of time. A similar explanation also applies to the comparison of the available storage space of the first storage device with other storage thresholds.

[0109] Therefore, in this way, when there are dynamic changes in storage resources, such as when cloud servers or others seize resources, when the system reports OOM (out of memory), the program will not crash directly, but will re-update the unloading strategy to schedule so that training can continue. When it is found that there are a lot of remaining storage resources for a long time, the unloading scheduler will also update the unloading strategy in real time to occupy more storage resources and improve training efficiency.

[0110] On the other hand, an embodiment of the present application also provides a data processing method for use in the training process of a deep learning model.

[0111] Fig. 9 A flow chart of another data processing method for training a deep learning model according to an embodiment of the present application is shown. Similarly, the data processing method can be executed by a first processor equipped with the deep learning model, for example, Figure 1 The GPU shown here can be used to execute the Figure 1 The deep learning model and / or various data generated during the training process can be stored in a first storage device associated with the first processor.

[0112] like Fig. 9As shown, in step S910, in response to the available storage space of the first storage device associated with the first processor carrying the deep learning model being greater than or equal to a first storage threshold, only the target activation data set to be unloaded is unloaded to the second storage device associated with the second processor.

[0113] For example, the target activation data set to be offloaded may include a portion of all activation data to be transferred between every two adjacent network layers in the deep learning model in the forward propagation phase. The portion of activation data may be a fixed activation data set or may be a fixed activation data set as described above. Figure 3 The activation data set determined by the determination method described in FIG. 4 may be an activation data set dynamically determined based on at least one of the storage state of the first storage device and the communication state associated with the first processor.

[0114] In step S920, in response to the available storage space of the first storage device being less than a first storage threshold and greater than or equal to a second storage threshold, i) at least a portion of the optimizer state of the optimizer used for the training process or a set of weight values ​​associated with the target activation data set, and ii) the target activation data set are unloaded to the second storage device.

[0115] In step S930, in response to the available storage space of the first storage device being less than a second storage threshold, the optimizer state used for the training process, the weight value set associated with the target activation data set, and the target activation data set are unloaded to the second storage device.

[0116] In other words, the currently applicable unloading strategy can be determined from the unloading strategies of different levels according to the different situations of available storage space, and the unloading operation from the first storage device to the second storage device can be performed accordingly, and the system can be reloaded from the second storage device to the first storage device at an appropriate time. Therefore, in the process of dynamic changes in storage resources, such as cloud servers or when others seize resources, when the system reports OOM (out of memory), the program will not crash directly, but will re-update the unloading strategy for scheduling so that training can continue. When it is found that there is a lot of remaining storage space for a long time, the unloading scheduler will also update the unloading strategy in real time to occupy more storage space and improve training efficiency.

[0117] According to another aspect of the present application, a data processing device for use in a training process of a deep learning model is also provided.

[0118] Fig.10 FIG. 1000 shows a structural block diagram of a data processing device used in a training process of a deep learning model according to an embodiment of the present application. The device 1000 may include Figure 1The part of the device end shown may include, for example, a first processor and an associated first storage device (such as a memory), or may include Figure 1 Other processors and associated storage devices on the device side are shown.

[0119] like Fig.10 As shown, the apparatus 1000 may include a determination module 1010 , an unloading module 1020 , and a loading module 1030 .

[0120] The determination module 1010 can be configured to determine a target activation data set from candidate activation data sets to be unloaded based on at least one of a storage state of a first storage device associated with a first processor carrying a deep learning model and a communication state associated with the first processor.

[0121] Optionally, the set of candidate activation data to be offloaded includes all activation data to be transferred between every two adjacent network layers in the deep learning model in the forward propagation phase, or a portion of all activation data.

[0122] Optionally, when determining the target activation data set, the determination module 1010 can determine the target activation data set from the candidate activation data set to be unloaded based on the expected loading time corresponding to each candidate activation data, wherein the communication state associated with the first processor is associated with the expected loading time corresponding to each candidate activation data. In addition, the expected loading time corresponding to each candidate activation data can be determined by a trial run. The specific operation of the trial run has been described above, so it will not be repeated here.

[0123] The offloading module 1020 may be configured to offload the target activation data set from the first storage device to the second storage device associated with the second processor during a forward propagation phase of the training process.

[0124] Optionally, the second processor may be Figure 1 The processor on the host side shown is, for example, a central processing unit CPU, and the second storage device may be a storage device at the CPU, for example, a memory.

[0125] The loading module 1030 may be configured to load the target activation data from the second storage device to the first storage device before performing back-propagation calculations using the target activation data for each target activation data in the target activation data set during the back-propagation phase of the training process.

[0126] Optionally, the loading module 1030 may, during the back-propagation phase, in parallel load the target activation data required for back-propagation calculations for the n-1th network layer and unloaded to the second storage device to the first storage device during the back-propagation calculations for the nth network layer of the deep learning model, where n is an integer greater than 2.

[0127] Optionally, the training process includes an optimization phase utilizing the optimizer after the back-propagation phase, and the unloading module 1020 may be configured to unload the optimizer state of the optimizer used for the optimization phase from the first storage device to the second storage device after the back-propagation phase ends, and the loading module 1030 may load the optimizer state from the second storage device to the first storage device during an idle period other than a period for loading the target activation data in the target activation data set during the next back-propagation phase.

[0128] Optionally, the unloading module 1020 may also be configured to unload each of the target activation data sets and their associated weight values ​​from the first storage device to the second storage device in the forward propagation phase of the training process, and the loading module 1030 may also be configured to load the target activation data and their associated weight values ​​from the second storage device to the first storage device in the reverse propagation phase of the training process for each target activation data in the target activation data set before the target activation data and their associated weight values ​​are used for reverse propagation calculations. Optionally, in the reverse propagation phase of the training process, for each target activation data in the target activation data set, after the target activation data and their associated weight values ​​are used for reverse propagation calculations, the target activation data in the first storage device may be released, but the weight value associated with the target activation data is not released, or the weight value associated with the target activation data may be unloaded to the second storage device again.

[0129] Optionally, the determination module 1010 can be configured to automatically determine an appropriate unloading strategy according to different actual situations, for example, in response to the available storage space of the first storage device being greater than or equal to a first storage threshold, determining to unload only the target activation data set to the second storage device; in response to the available storage space of the first storage device being less than the first storage threshold and greater than or equal to a second storage threshold, determining to unload i) at least a portion of the optimizer state of the optimizer used for the training process or the weight value set associated with the target activation data set, and ii) the target activation data set to the second storage device; and in response to the available storage space of the first storage device being less than the second storage threshold, determining to unload the optimizer state of the optimizer used for the training process, the weight value set associated with the target activation data set, and the target activation data set to the second storage device.

[0130] Therefore, according to the reference Fig.10 The described data processing device for the training process of the deep learning model can determine a relatively reasonable target activation data set that can be unloaded from the candidate activation data based on at least one of the storage state and the communication state associated with the first processor according to the actual application scenario, so that the activation data that needs to be unloaded can be automatically determined in accordance with the current scenario, so that it can be suitable for dynamic changes in storage resources. In addition, since the target activation data is determined by considering the communication state, the activation data with a long loading time can be unloaded as much as possible, so the waiting time can be reduced as much as possible, so that the efficiency of training can be guaranteed to the maximum extent on the basis of the unloading scheme. In addition, different levels of unloading strategies can be selected according to different situations of available storage space. Therefore, in the process of dynamic changes in storage resources, such as cloud servers or others occupying resources, when the system reports OOM (out of memory), the program will not crash directly, but re-update the unloading strategy for scheduling, so that the training can continue. When it is found that there is a lot of remaining storage space for a long time, the unloading strategy will also be updated in real time to occupy more storage space and improve training efficiency.

[0131] In addition, in other embodiments, the data processing device 1000 can determine the currently applicable unloading strategy from different levels of unloading strategies according to different situations of available storage space, and perform an unloading operation from the first storage device to the second storage device accordingly, and reload from the second storage device to the first storage device at an appropriate time.

[0132] For example, the determination module 1010 may be configured to determine that only the target activation data set to be unloaded is unloaded to the second storage device associated with the second processor in response to the available storage space of the first storage device associated with the first processor carrying the deep learning model being greater than or equal to the first storage threshold, and the unloading module 1020 may be configured to perform the unloading operation accordingly, and then the loading module 1030 may load the unloaded data back to the first storage device at an appropriate time (for example, in parallel with the backpropagation calculation). Alternatively, the determination module 1010 may be configured to determine that i) at least a portion of the optimizer state of the optimizer used for the training process or a set of weight values ​​associated with the target activation data set, and ii) the target activation data set are unloaded to the second storage device in response to the available storage space of the first storage device being less than the first storage threshold and greater than or equal to the second storage threshold, and the unloading module 1020 may be configured to perform the unloading operation accordingly, and then the loading module 1030 may load the unloaded data back to the first storage device at an appropriate time. Alternatively, the determination module 1010 can be configured to determine that the optimizer state used for the training process, the set of weight values ​​associated with the target activation data set, and the target activation data set will be unloaded to the second storage device in response to the available storage space of the first storage device being less than a second storage threshold, and the unloading module 1020 can be configured to perform the unloading operation accordingly, and then the loading module 1030 can load the unloaded data back to the first storage device at an appropriate time.

[0133] More details on the operations in each module are in the previous reference Figure 3 The contents described are basically the same, so the detailed description of these operations is omitted here.

[0134] In addition, although Fig.10 The above modules are shown in an exemplary manner, but it should be understood that the device 1000 can be divided into more or fewer modules according to different functions, or each module can be divided into further sub-modules, or two or more modules can be merged into one module. In some example embodiments, each module or further divided sub-module can be implemented by electronic hardware (e.g., the first processor or other processors on the device side, such as DSP, ASIC, FPGA or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, etc.), computer software (e.g., can be stored in random access memory (RAM), flash memory, read-only memory (ROM), erasable programmable ROM (EPROM), etc.) or a combination of the two.

[0135] According to another aspect of the present application, a processor is also provided for executing the above-mentioned Figure 3-Figure 9Optionally, the processor may include Figure 1 The first processor at the device end shown may include Figure 1 The processor may have an associated memory so that the computer program or instruction on the memory can be executed to implement the aforementioned reference Figure 3-Figure 9 The data processing method shown. As an example, other processors that perform the above method may include an integrated circuit chip that has the ability to process signals. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, to implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application. The processor may also include a memory or be combined with a memory, and the memory may include a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and may also store a computer executable program or instruction, which, when executed, enables the processor to implement various operations as described above. The internal memory may also store a computer executable program, which, when executed, enables the processor to implement various operations as described above. The non-volatile storage medium may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. It should be noted that memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0136] According to another aspect of the present application, a computer readable storage medium is provided, on which a computer program or instruction is stored. When the computer program or instruction is executed by a processor, the processor executes the above-mentioned Figure 3-Figure 9 The computer readable storage medium may be a non-volatile storage medium as described above.

[0137] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of a code contains at least one executable instruction for realizing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0138] In general, various example embodiments of the present invention may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of embodiments of the present invention are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0139] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the invention belongs. It should also be understood that terms such as those defined in common dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology and should not be interpreted in an idealized or extremely formal sense, unless explicitly defined as such herein.

[0140] The above is an explanation of the present invention and should not be considered as a limitation thereof. Although several exemplary embodiments of the present invention have been described, it will be readily understood by those skilled in the art that many modifications may be made to the exemplary embodiments without departing from the novel teachings and advantages of the present invention. Therefore, all such modifications are intended to be included within the scope of the present invention as defined in the claims. It should be understood that the above is an explanation of the present invention and should not be considered as being limited to the specific embodiments disclosed, and modifications to the disclosed embodiments and other embodiments are intended to be included within the scope of the appended claims. The present invention is defined by the claims and their equivalents.

Claims

1. A data processing method for a deep learning model training process, comprising: Determine a target activation data set from candidate activation data sets to be offloaded based on at least one of a storage state of a first storage device associated with a first processor carrying the deep learning model and a communication state associated with the first processor; During a forward propagation phase of the training process, offloading the target activation data set from the first storage device to a second storage device associated with a second processor; as well as In the back propagation phase of the training process, for each target activation data in the target activation data set, before performing back propagation calculation using the target activation data, the target activation data is loaded from the second storage device to the first storage device.

2. The data processing method according to claim 1, wherein: The candidate activation data set to be offloaded includes activation data to be transferred between every two adjacent network layers in the deep learning model in the forward propagation stage.

3. The data processing method according to claim 1, further comprising: In the back-propagation stage, during the back-propagation calculation for the nth network layer of the deep learning model, the target activation data required for the back-propagation calculation for the n-1th network layer and unloaded to the second storage device are loaded into the first storage device in parallel, where n is an integer greater than 2.

4. The data processing method according to claim 1, wherein: Determining a target activation data set from the candidate activation data set to be uninstalled includes: Based on the expected loading time corresponding to each candidate activation data, a target activation data set is determined from the candidate activation data set to be unloaded, The communication state associated with the first processor is associated with the expected loading time corresponding to each candidate activation data.

5. The data processing method according to claim 4, wherein: The expected loading time corresponding to each candidate activation data is determined by performing a trial run on the deep learning model, and the trial run includes the following operations: In a forward propagation phase of the trial run training process of the trial run, each candidate activation data in the set of candidate activation data to be unloaded is unloaded from the first storage device to the second storage device; In a back propagation phase of the trial run training process of the trial run, for each candidate activation data in the set of candidate activation data to be unloaded, before performing back propagation calculation using the candidate activation data, loading the candidate activation data from the second storage device to the first storage device, Among them, in the back propagation stage of the trial run training process of the trial run, during the back propagation calculation for the nth network layer of the deep learning model, the candidate activation data required for the back propagation calculation for the n-1th network layer is loaded into the first storage device in parallel, wherein n is an integer greater than 2.

6. The data processing method according to claim 5, wherein: Based on the expected loading time corresponding to each candidate activation data, a target activation data set is determined from the candidate activation data set to be unloaded, including: Based on the expected loading time corresponding to each candidate activation data, determine a candidate activation data subset from the set of candidate activation data to be unloaded, wherein the expected loading time corresponding to each candidate activation data in the candidate activation data subset is greater than the expected computation time of a back propagation computation performed in parallel with the loading of the candidate activation data; For each candidate activation data in the candidate activation data subset, determine a time difference between an expected loading time corresponding to the candidate activation data and an expected computation time of a back propagation calculation performed in parallel with the loading of the candidate activation data as an expected idle duration corresponding to the candidate activation data; and The target activation data set is determined based on the expected idle duration corresponding to each candidate activation data in the candidate activation data subset.

7. The data processing method according to claim 6, wherein: Determining the target activation data set based on the expected idle duration corresponding to each candidate activation data in the candidate activation data subset includes: Excluding from the candidate activation data set to be uninstalled each candidate activation data in the candidate activation data subset whose corresponding expected idle time is greater than a predetermined time threshold, so as to obtain the target activation data set; or When the sum of the expected idle durations corresponding to the activation data in the candidate activation data subset is less than or equal to a second predetermined duration threshold, the candidate activation data set to be uninstalled is taken as the target activation data set.

8. The data processing method according to claim 1, wherein the training process includes an optimization phase using an optimizer after the back-propagation phase, The method further comprises: After the back-propagation phase is finished, unloading at least a portion of the optimizer state of the optimizer used in the optimization phase from the first storage device to the second storage device; as well as During a next back-propagation phase, the at least a portion of the optimizer state is loaded from the second storage device to the first storage device during an idle period other than a period for loading the target activation data in the target activation data set.

9. The data processing method according to claim 1 or 8, further comprising: In the forward propagation phase of the training process, each target activation data and its associated weight value in the target activation data set are unloaded from the first storage device to the second storage device; as well as In the back-propagation stage of the training process, for each target activation data in the target activation data set, the target activation data and its associated weight value are loaded from the second storage device to the first storage device before back-propagation calculation is performed using the target activation data and its associated weight value.

10. The data processing method according to claim 9, further comprising: In the back-propagation stage of the training process, for each target activation data in the target activation data set, after the target activation data and its associated weight value are used for back-propagation calculation, the target activation data in the first storage device is released but the weight value associated with the target activation data is not released, or the weight value associated with the target activation data is unloaded to the second storage device again.

11. The data processing method according to claim 1, further comprising: After determining the target activation data set, In response to the available storage space of the first storage device being greater than or equal to a first storage threshold, determining to only unload the target activation data set to the second storage device; In response to the available storage space of the first storage device being less than the first storage threshold and greater than or equal to a second storage threshold, determining to unload at least a portion of optimizer states of the optimizer used for the training process or a set of weight values ​​associated with the target activation data set and the target activation data set to the second storage device; as well as In response to the available storage space of the first storage device being less than a second storage threshold, it is determined that at least a portion of the optimizer state of the optimizer used for the training process, the set of weight values ​​associated with the target activation data set, and the target activation data set are unloaded to the second storage device.

12. The data processing method according to claim 1, wherein: Determining a target activation data set from the candidate activation data set to be uninstalled includes: In response to the available storage space of the first storage device being greater than or equal to a preset threshold, determining that the target activation data set is an empty set; and In response to the available storage space of the first storage device being less than a preset threshold, the target activation data set is determined to be at least a part of the candidate activation data set.

13. The data processing method according to claim 1, wherein: The first processor includes a graphics processing unit, a general-purpose graphics processing unit, or a tensor processing unit; and The second processor includes a central processing unit.

14. A data processing method for a deep learning model training process, comprising: In response to an available storage space of a first storage device associated with a first processor carrying the deep learning model being greater than or equal to a first storage threshold, only the target activation data set to be offloaded is offloaded to a second storage device associated with a second processor; In response to the available storage space of the first storage device being less than the first storage threshold and greater than or equal to a second storage threshold, unloading an optimizer state of an optimizer used for the training process or a set of weight values ​​associated with the target activation data set and the target activation data set to the second storage device; as well as In response to the available storage space of the first storage device being less than a second storage threshold, the optimizer state used for the training process, the set of weight values ​​associated with the target activation data set, and the target activation data set are unloaded to the second storage device.

15. The method according to claim 14, further comprising: The target activation data set is determined from candidate activation data sets to be offloaded based on at least one of a storage state of the first storage device and a communication state associated with the first processor.

16. A data processing device for use in a training process of a deep learning model, comprising: A determination module, configured to determine a target activation data set from candidate activation data sets to be uninstalled based on at least one of a storage state of a first storage device associated with a first processor carrying the deep learning model and a communication state associated with the first processor; an unloading module, configured to unload the target activation data set from the first storage device to a second storage device associated with a second processor during a forward propagation phase of the training process; as well as A loading module is used to load the target activation data from the second storage device to the first storage device before using the target activation data for each target activation data in the target activation data set in the back-propagation phase of the training process and performing back-propagation calculation.

17. A processor configured to execute a computer program or instructions to implement the method according to any one of claims 1 to 14.

18. A computer-readable storage medium having a computer program or instruction stored thereon, wherein when the computer program or instruction is executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 14.

Citation Information

Patent Citations

  • Memory optimization method and system for distributed training of deep learning model

    CN116452404A

  • Heterogeneous multi-core system memory access management method for deep neural network

    CN117591274A

  • Network state self-adaptive target detection computing task unloading scheduling method

    CN118484315A

  • Large language model training acceleration method

    CN118779095A

  • Unloading training optimization method and device, electronic equipment and storage medium

    CN118796514A

Cited By

  • Staggered assembly line parallel training method and device, equipment, storage medium and program product

    CN119376796A