A video memory optimization method, device, computer equipment, storage medium and computer program product

Abstract

The present application discloses a video memory optimization method, apparatus, computer equipment, storage medium, and computer program product, belonging to the field of artificial intelligence technology. In this method, during the distributed training of a deep learning model on multiple chips, when a preset condition is met, the video memory load information of each chip and the video memory usage information of multiple objects on each chip are obtained. The multiple objects correspond to different training tasks performed by this chip. Based on the video memory load information of each chip and the video memory usage information of multiple objects on this chip, the object set of this chip is updated. Based on the object set of each chip, a video memory optimization instruction is sent to this chip to trigger this chip to apply video memory optimization technology to the objects in this object set. In this way, the objects on each chip to which the video memory optimization technology is applied can change with the changes in the training process, the video memory optimization is more matched with the training process, and the video memory optimization is more reasonable, thereby improving the model training effect.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a video memory optimization method, apparatus, computer equipment, storage medium, and computer program product. Background Art

[0002] Typically, when training deep learning models on multiple chips, intermediate activation values ​​and gradient information consume a significant amount of video memory. When video memory resources are insufficient, memory optimization techniques are often required to optimize the training process. However, how to effectively utilize these techniques to improve model training performance has become a pressing technical challenge. Summary of the Invention

[0003] The embodiments of the present application provide a video memory optimization method, apparatus, computer equipment, storage medium, and computer program product for improving the rationality of video memory optimization during deep learning model training, thereby improving the model training effect.

[0004] In a first aspect, an embodiment of the present application provides a method for optimizing video memory, including:

[0005] During distributed training of a deep learning model on multiple chips, when a preset condition is met, obtaining video memory load information of each chip and video memory usage information of multiple objects on the chip, the multiple objects corresponding to multiple training tasks performed by the chip;

[0006] updating an object set of the chip according to the video memory load information of each chip and the video memory usage information of multiple objects on the chip, wherein the object set is used to store objects on the chip to which the video memory optimization technology is applied;

[0007] A video memory optimization instruction is sent to the chip according to the object set of each chip, where the video memory optimization instruction is used to trigger the chip to apply a video memory optimization technology to the objects in the object set.

[0008] In some embodiments, the multiple objects on the chip are multiple functional components or multiple operators of the deep learning model distributed on the chip.

[0009] In some embodiments, updating the object set of each chip according to the video memory load information of each chip and the video memory usage information of multiple objects on the chip includes:

[0010] Determining, based on the video memory load information of each chip and the video memory usage information of multiple objects on the chip, a method for adjusting the video memory optimization intensity of the chip;

[0011] The object set of the chip is updated according to the adjustment method of the video memory optimization strength of the chip.

[0012] In some embodiments, the video memory load information of each chip includes the remaining video memory and the amount of video memory fragmentation, and the video memory usage information of each object is the video memory usage of the chip on which the object resides. Determining, based on the video memory load information of each chip and the video memory usage information of multiple objects on the chip, a method for adjusting the video memory optimization intensity of the chip includes:

[0013] When a first condition is met, determining that the adjustment method is to reduce the video memory optimization intensity, the first condition being that the remaining video memory of the chip is greater than the video memory usage of the target object, the video memory fragmentation of the chip is less than a first threshold, and the target object is the object with the smallest video memory usage in the object set of the chip;

[0014] When a second condition is met, determining that the adjustment method is to increase the video memory optimization intensity, the second condition being that the remaining video memory of the chip is less than a second threshold, the amount of video memory fragmentation of the chip is not less than the first threshold, and the second threshold is less than the video memory usage of the target object;

[0015] When the first condition and the second condition are not met, it is determined that the adjustment method is to maintain the video memory optimization strength.

[0016] In some embodiments, updating the object set of the chip according to the adjustment method of the video memory optimization strength of the chip includes:

[0017] When the adjustment method is to reduce the intensity of video memory optimization, removing an object from the object set;

[0018] When the adjustment method is to increase the video memory optimization strength, adding an object to the object set;

[0019] When the adjustment method is to maintain the video memory optimization strength, the object set is kept unchanged.

[0020] In some embodiments, when a preset condition is met, the training time corresponding to each object on the chip during the most recent L rounds of iterations is also obtained, where L is an integer greater than zero;

[0021] The objects eliminated from the object set are the objects in the object set that take the longest time to train and whose video memory usage is less than the remaining video memory of the chip;

[0022] The objects added to the object set are objects that are not in the object set and have the shortest training time.

[0023] In some embodiments, further comprising:

[0024] Sending a memory optimization type corresponding to each object on each chip to each chip, where the memory optimization type is activation value recalculation or tensor unloading; and

[0025] Each chip applies a video memory optimization technology corresponding to the video memory optimization type of each object in the object set of the chip.

[0026] In some embodiments, the video memory optimization type corresponding to each object on each chip is determined according to the following steps:

[0027] Apply each memory optimization type to each object on each chip.

[0028] Counting the training time of the object and the memory optimization time corresponding to the memory optimization technology after M iterations of training, where M is an integer greater than zero;

[0029] Determining, based on the training time and the memory optimization time, an impact factor of the memory optimization technology on the training task of the object;

[0030] The video memory optimization type corresponding to the video memory optimization technology with the smallest impact factor is determined as the video memory optimization type corresponding to the object.

[0031] In a second aspect, an embodiment of the present application provides a video memory optimization device, comprising:

[0032] An acquisition module is configured to, during distributed training of a deep learning model on multiple chips, acquire video memory load information for each chip and video memory usage information for multiple objects on the chip, the multiple objects corresponding to multiple training tasks performed by the chip, when preset conditions are met;

[0033] An update module, configured to update an object set of the chip based on the video memory load information of each chip and the video memory usage information of multiple objects on the chip, wherein the object set is used to store objects on the chip to which the video memory optimization technology is applied;

[0034] The sending module is used to send a video memory optimization instruction to the chip according to the object set of each chip, and the video memory optimization instruction is used to trigger the chip to apply the video memory optimization technology to the objects in the object set.

[0035] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a chip, and a computer program stored in the memory and executable on the chip, wherein the chip implements any of the above-mentioned video memory optimization methods when executing the computer program.

[0036] In a fourth aspect, an embodiment of the present application provides a storage medium. When a computer program in the storage medium is executed by a processor of a computer device, the computer device can execute any of the above-mentioned video memory optimization methods.

[0037] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements any of the above-mentioned video memory optimization methods when executed by a processor.

[0038] In an embodiment of the present application, during the distributed training of a deep learning model on multiple chips, whenever a preset condition is met, the memory load information of each chip and the memory usage information of multiple objects on each chip can be obtained. The multiple objects correspond to different training tasks performed by this chip. Based on the memory load information of each chip and the memory usage information of multiple objects on this chip, the object set of this chip is updated. The object set is used to store the objects on this chip that need to apply the memory optimization technology. Based on the object set of each chip, a memory optimization instruction is sent to this chip to trigger this chip to apply the memory optimization technology to the objects in this object set. In this way, the objects on each chip that apply the memory optimization technology can change with the changes in the training process, the memory optimization is more compatible with the training process, and the application of the memory optimization technology is more reasonable, thereby improving the model training effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0040] Figure 1 A schematic diagram of a scenario for distributed training of a deep learning model provided in an embodiment of the present application;

[0041] Figure 2 A schematic diagram of the structure of a chip provided in an embodiment of the present application;

[0042] Figure 3 A flowchart of a video memory optimization method provided in an embodiment of the present application;

[0043] Figure 4 A flow chart of a method for updating an object set of each chip provided in an embodiment of the present application;

[0044] Figure 5 A schematic diagram of the structure of a Transformer layer provided in an embodiment of the present application;

[0045] Figure 6 A flowchart of another video memory optimization method provided in an embodiment of the present application;

[0046] Figure 7 A schematic diagram of the structure of an expert component provided in an embodiment of the present application;

[0047] Figure 8 A schematic diagram of the structure of a video memory optimization device provided in an embodiment of the present application;

[0048] Figure 9 A schematic diagram of the hardware structure of a computer device for implementing a video memory optimization method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to improve the rationality of video memory optimization during deep learning model training, thereby improving the model training effect, the embodiments of the present application provide a video memory optimization method, apparatus, computer equipment, storage medium and computer program product.

[0050] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application can be combined with each other if there is no conflict.

[0051] The following is an introduction to the technical terms involved in the embodiments of this application.

[0052] 1. Activation recomputation

[0053] Activation recalculation is a memory optimization technique used during deep learning model training. During deep learning model training, when the model is large or the input sequence is long, intermediate activations can consume significant amounts of memory. Activation recalculation preserves some intermediate activations during the forward propagation phase, discards others, and recalculates the discarded ones during the backward propagation phase, thus reducing memory usage by trading time for storage space.

[0054] 2. Tensor offloading.

[0055] Tensor offloading is another memory optimization technique used during deep learning model training. When training resources are limited, such as limited graphics processing unit (GPU) memory, tensor offloading transfers some tensor data from the GPU to the CPU during the forward propagation phase to alleviate memory pressure and minimize the impact on training speed. During the backward propagation phase, when the offloaded tensor data is needed, it is reloaded from the CPU to the GPU, similarly reducing memory usage by trading time for storage space.

[0056] 3. Unbalanced load training.

[0057] In the process of distributed training of deep learning models based on multiple chips, the load of each chip, such as video memory usage and communication overhead, will be unevenly distributed, resulting in some chips being overloaded and some chips being idle, which will affect the overall training efficiency.

[0058] Take, for example, a deep learning model based on a mixture of experts (MoE). The core of the MoE model is the expert component. The basic unit of data input to the expert component is tokens. Tokens are distributed to different expert components for processing via the router component in the MoE model. In the early stages of training, the number of tokens processed by different expert components varies significantly. This number of tokens varies throughout the training process, but in the later stages of training, the number of tokens processed by each expert component tends to be balanced.

[0059] When different expert components are distributed to different chips, the different numbers of tokens processed by different expert components will cause the activation values ​​of the expert component inputs on different chips to be different, resulting in obvious load imbalance problems between chips; and the continuous change in the number of tokens processed by the same expert component will cause the activation values ​​of the expert network input on the same chip to also change continuously, resulting in load imbalance problems on the same chip due to the change in the number of tokens processed by the expert network.

[0060] In order to alleviate the pressure on video memory, in an embodiment of the present application, during the distributed training of a deep learning model on multiple chips, whenever a preset condition is met, the video memory load information of each chip and the video memory usage information of multiple objects on each chip are obtained. The multiple objects correspond to different training tasks performed by this chip. Based on the video memory load information of each chip and the video memory usage information of multiple objects on this chip, the object set of this chip is updated. The object set is used to store the objects on this chip that need to apply video memory optimization technology. Based on the object set of each chip, a video memory optimization instruction is sent to this chip to trigger this chip to apply video memory optimization technology to the objects in this object set. In this way, the objects on each chip that apply video memory optimization technology can change with the changes in the training process, the video memory optimization is more matched with the training process, and the above-mentioned load imbalance problem can be better dealt with. The application of video memory optimization technology is also more reasonable, thereby improving the training effect of the deep learning model.

[0061] The application scenario of this application is distributed training of deep learning models. The scenario of distributed training of deep learning models is introduced below.

[0062] See also Figure 1 , Figure 1 A schematic diagram of a scenario for distributed training of a deep learning model provided in an embodiment of the present application includes multiple chips 100, and the multiple chips 100 jointly train a deep learning model such as a hybrid expert model.

[0063] Generally, multiple chips 100 can be divided into slave chips and master chips. Each slave chip can deploy some functional components (or functional modules, each functional module may include multiple operators) of a deep learning model to complete the training tasks corresponding to these functional components. The master chip can analyze the memory usage of each slave chip during training and instruct the slave chips to apply memory optimization techniques to which functional modules or operators within the functional modules, thereby better completing model training with limited memory resources.

[0064] See also Figure 2 , Figure 2 A schematic structural diagram of a chip 100 provided in an embodiment of the present application includes at least: a video memory 101 , an on-chip cache 102 and a plurality of computing units 103 .

[0065] The video memory 101 can be high bandwidth memory (HBM) or other types of memory. The on-chip cache 102 is a temporary memory with a smaller capacity than the video memory 101 but a faster data read and write speed than the video memory 101.

[0066] The computing unit 103 can be used to perform training tasks of various functional components, including but not limited to: normalization component, self-attention component, etc., and each functional component may include multiple operators such as matrix multiplication operator, matrix addition operator, matrix inverse operator, etc., each operator corresponds to a training task of smaller granularity.

[0067] The computing unit 103 can read and write data to the video memory 101, and can also read and write data to the on-chip cache 102. For example, the computing unit 103 can read data from the video memory 101 for calculation, or read data from the video memory 101 and then write the read data to the on-chip cache 102, and then read data from the on-chip cache 102 for calculation.

[0068] In addition to the above-mentioned structure, the chip 100 in the embodiment of the present application may also include other structures, which are not specifically limited in the embodiment of the present application.

[0069] In addition, the chip 100 may be a GPU, a general-purpose computing on graphics processing unit (GPGPU), a domain specific architecture (DSA), etc.

[0070] After introducing the application scenarios of the embodiments of the present application, the video memory optimization method proposed in the present application is described below with specific embodiments.

[0071] See also Figure 3 , Figure 3 This is a flowchart of a video memory optimization method provided in an embodiment of the present application, which can be applied to Figure 1 The main chip in the method can also be applied to other devices, and the method includes the following steps.

[0072] In step 301, during the distributed training of a deep learning model on multiple chips, when preset conditions are met, the video memory load information of each chip and the video memory usage information of multiple objects on the chip are obtained, where the multiple objects correspond to the multiple training tasks performed by the chip.

[0073] Multiple chips are like multiple GPUs. Deep learning models can be used to perform various image processing tasks such as image classification, object detection, semantic segmentation, and image generation. Preset conditions include the expiration of a set period and the completion of L rounds of iterative training, where L is an integer greater than zero, such as 1 or 2. Each chip's memory load information, such as remaining memory and memory fragmentation, characterizes the chip's memory load. Furthermore, memory usage information for any object on each chip, such as the object's memory usage on that chip, characterizes the object's memory usage on that chip.

[0074] In step 302, based on the video memory load information of each chip and the video memory usage information of multiple objects on the chip, the object set of the chip is updated. The object set is used to store objects on the chip to which the video memory optimization technology is applied.

[0075] In specific implementation, Figure 4 The process shown in FIG. 4 updates the object set of each chip and includes the following steps.

[0076] In step 3021, a method for adjusting the chip's video memory optimization intensity is determined based on the video memory load information of each chip and the video memory usage information of multiple objects on the chip.

[0077] For each chip, the object with the smallest video memory usage in the object set of this chip can be determined as the target object. If the remaining video memory of this chip is greater than the video memory usage of the target object, and the video memory fragmentation of this chip is less than the first threshold, it means that the video memory of this chip is relatively abundant and less fragmented. In order to improve the training speed, it can be determined that the video memory optimization intensity of this chip should be adjusted by reducing the video memory optimization intensity.

[0078] When the remaining video memory of this chip is not greater than the video memory usage of the target object, if the remaining video memory of this chip is less than the second threshold (the second threshold is less than the video memory usage of the target object), and the amount of video memory fragmentation of this chip is not less than the first threshold, then it means that the video memory of this chip is relatively tight and fragmented. In order to save video memory resources, it can be determined that the video memory optimization intensity of this chip should be adjusted by increasing the video memory optimization intensity.

[0079] When neither of the above two conditions is met, for example, if the remaining video memory of this chip is greater than the second threshold but less than the video memory usage of the target object, it means that the video memory optimization status of this chip is relatively matched with the training stage, and it can be determined that the adjustment method for the video memory optimization strength of this chip is to maintain the video memory optimization strength.

[0080] In step 3022, the object set of the chip is updated according to the adjustment method of the chip's video memory optimization strength.

[0081] In the first case, the way to adjust the chip's video memory optimization strength is to reduce the video memory optimization strength. In this case, an object can be removed from the object set of this chip to improve the chip's video memory utilization and increase the model training speed.

[0082] When pre-set conditions are met, the training time for each object on this chip during the most recent L iterations (including the current iteration) can be obtained. This is the total time it took for this chip to execute the training task for each object during the most recent L iterations, where L is an integer greater than zero, such as 1 or 2. In this case, the object that is eliminated may be the object with the longest training time in the chip's object set and whose memory usage is less than the chip's remaining memory. This maximizes the chip's memory utilization while minimizing training time.

[0083] In the second case, the way to adjust the video memory optimization strength of the chip is to increase the video memory optimization strength. In this case, an object can be added to the object collection of this chip to save video memory resources.

[0084] Furthermore, after obtaining the training time for each object on this chip during the last L iterations, the object to be added can be the one that is not already in the set of objects on this chip and has the shortest training time. This saves memory resources on this chip while minimizing the increase in training time.

[0085] In the third case, the way to adjust the chip's video memory optimization strength is to maintain the video memory optimization strength. In this case, the object set of this chip can be kept unchanged to maintain the matching status of the current video memory resources and the training stage.

[0086] In step 303, a memory optimization instruction is sent to each chip according to the object set of the chip. The memory optimization instruction is used to trigger the chip to apply the memory optimization technology to the objects in the object set.

[0087] The memory optimization instruction may include object information, such as an object identifier, for each object in the object set. The chip applies memory optimization technology to the objects in the object set. Specifically, the chip applies memory optimization technology when executing the training task corresponding to each object in the object set to optimize memory usage during training.

[0088] In some embodiments, upon receiving a memory optimization instruction, each chip may apply activation value recalculation or tensor unloading to each object in the object set of the chip, so that the objects on each chip apply the same memory optimization technique.

[0089] In some embodiments, in order to better optimize video memory, video memory optimization technology of each video memory optimization type can be applied to each object on each chip, and the training time of the object and the video memory optimization time corresponding to this video memory optimization technology are counted after M iterative training. Based on the training time and the video memory optimization time, the impact factor of this video memory optimization technology on the training task of this object is determined. For example, according to the ratio of video memory optimization time to training time and the rule of positive correlation with the impact factor, the impact factor is determined, and then the video memory optimization type corresponding to the video memory optimization technology with the smallest impact factor is determined as the video memory optimization type corresponding to this object, where M is an integer greater than zero, such as 1, 2, etc.

[0090] Specifically, activation value recalculation can be applied to each object on each chip. The first training time and activation value recalculation time (a type of memory optimization time) of this object after M iterative training are counted. Then, according to the rule of positive correlation between the ratio of activation value recalculation time to the first training time and the influencing factor, the influence factor of activation value recalculation on the training task of this object can be determined.

[0091] Furthermore, tensor offloading can be applied to each object on each chip. The second training time and tensor offloading time (another memory optimization time) of this object after M iterative training are counted. Then, the impact factor of tensor offloading on the training task of this object can be determined according to the rule that the ratio of tensor offloading time to the second training time is positively correlated with the impact factor.

[0092] Finally, for each object on each chip, the memory optimization type with the smallest of the two influencing factors is determined as the memory optimization type corresponding to this object, thereby determining the memory optimization technology with the least impact on training for this object.

[0093] Then, the memory optimization type corresponding to each object on this chip can be sent to each chip.

[0094] Subsequently, after receiving the memory optimization instruction, each chip applies the memory optimization technology corresponding to the memory optimization type of each object in its own object set.

[0095] In this way, different memory optimization technologies can be applied to objects on each chip, and each object applies the memory optimization technology that has the least impact on training, which can further improve the memory optimization effect.

[0096] In practical applications, the multiple objects on each chip can be multiple functional components of a deep learning model distributed on the chip, or multiple operators of a deep learning model distributed on the chip. In other words, the solution in this embodiment of the application can perform video memory optimization at both the functional component and operator granularity, thus providing greater flexibility in video memory optimization.

[0097] The following introduces the graphics memory optimization solution of the embodiment of this application by taking the distributed training of the MoE model on multiple GPUs as an example.

[0098] Generally, the MoE model includes multiple Transformer layers. The typical structure of a Transformer layer is as follows: Figure 5 As shown in the figure, it includes a normalize module, a self-attention module, an add and normalize module, a router and permute module (the aforementioned router component is merged into this component), an expert module, an unpermute module, and an add module. Figure 4In , input i represents the i-th input, activation value i represents the activation value of input expert component i (activation value i is obtained by a series of processing on input i), output i represents the output of expert component i, and the value of i ranges from 1 to N, where N is the total number of GPUs.

[0099] When training the MoE model on multiple GPUs, each expert network can be placed on a separate GPU. All other components, excluding the expert network, can be placed equally on each GPU or distributed across all GPUs. The specific distribution method is not detailed here. Furthermore, the total number of components per GPU can be the same or different, depending on how the MoE model is pipelined. This is not detailed here.

[0100] Generally speaking, in the early stages of training, the number of activation values ​​processed by each expert component changes dramatically, the GPU will generate more memory fragments, and the GPU's memory pressure is relatively high. It is suitable to apply memory optimization techniques such as activation value recalculation and tensor unloading to some objects (i.e., some functional components or some operators). As the training progresses (i.e., the number of iterations increases), the model gradually converges, the activation values ​​processed by each expert component will gradually become equal, the GPU's memory fragmentation will gradually decrease, and the memory will gradually be sufficient. You can gradually stop applying memory optimization techniques to some objects to improve the model training speed.

[0101] To this end, the inventors came up with the idea of ​​flexibly adjusting the objects on which memory optimization technology is applied on each GPU as the number of iterations increases, so as to improve the adaptability of memory optimization and the training stage, improve the rationality of memory optimization, and thus improve the overall training effect.

[0102] The following describes the memory optimization process during MoE model training, taking the case where the target of the memory optimization technology on each GPU is the functional component, the preset condition is that one round of iterative training is completed (that is, L=1), and M=1.

[0103] See also Figure 6 , Figure 6 A flowchart of another video memory optimization method provided in an embodiment of the present application includes the following steps.

[0104] In step 601, before model training begins, the user chooses to enable the dynamic activation value recalculation function and the dynamic tensor unloading function.

[0105] Generally, users can enable the dynamic activation value recalculation function and the dynamic tensor unloading function through configuration items, and after enabling them, the solution of the embodiment of the present application can be applied.

[0106] In step 602 , the i-th iteration begins, and i is 0 initially.

[0107] In step 603, when i=0, activation value recalculation is applied to each functional component on each GPU.

[0108] by Figure 4 For example, this step is to Figure 4 The activation values ​​of each functional component of the Transformer layer shown are recalculated to facilitate subsequent analysis of the impact of activation value recalculation on the training task of each functional component.

[0109] In step 604, the first training time and activation value recalculation time of each functional component on each GPU during the iteration process are obtained.

[0110] The first training time of each functional component on each GPU refers to the time it takes for the CPU to execute the training task corresponding to this functional component (after applying activation value recalculation), and the activation value recalculation time of this functional component refers to the time it takes for the CPU to recalculate the activation value while executing the training task corresponding to this functional component.

[0111] In step 605, after the iteration is completed, the first impact factor of each functional component is determined based on the first training time and activation value recalculation time of the functional component.

[0112] Assuming that the first training time of this functional component is T1 and the activation value recalculation time is T2 (T2<T1), then the first impact factor of this functional component (that is, the impact factor of activation value recalculation on the training task of this functional component) recompute_factor = T2×K1 / T1, where K1 is the preset adjustment scale factor.

[0113] In step 606 , i is updated to i+1, and the process returns to step 602 .

[0114] In step 607 , when i=1, tensor offloading is applied to each functional component on each GPU.

[0115] by Figure 4 For example, this step is to Figure 4 The functional components of the Transformer layer shown in the figure apply tensor offloading to facilitate subsequent analysis of the impact of tensor offloading on the training task of each functional component.

[0116] In step 608 , the second training time and tensor unloading time of each functional component on each GPU during the iteration process are obtained.

[0117] The second training time of each functional component on each GPU refers to the time it takes for the CPU to execute the training task corresponding to this functional component (after applying tensor unloading), and the tensor unloading time of this functional component refers to the time it takes for tensor unloading to occur during the CPU's execution of the training task corresponding to this functional component.

[0118] In step 609 , after the iteration is completed, the second impact factor of each functional component is determined based on the second training time and tensor unloading time of the functional component.

[0119] Assuming that the second training time of this functional component is T3 and the tensor offloading time is T4 (T4 < T3), the second impact factor of this functional component (that is, the impact factor of tensor offloading on the training task of this functional component) IO_factor = T4 × K2 / T3, where K2 is the preset adjustment scale factor.

[0120] In step 610 , the smaller video memory optimization type corresponding to the first impact factor and the second impact factor of each functional component is determined as the video memory optimization type corresponding to the functional component, and then the process goes to step 606 .

[0121] That is, for each functional component, if recompute_factor < IO_factor, activation recomputation is determined as the memory optimization type for this functional component; if recompute_factor > IO_factor, tensor offloading is determined as the memory optimization type for this functional component. If recompute_factor = IO_factor, either activation recomputation or tensor offloading is randomly determined as the memory optimization type for this functional component.

[0122] In this way, for each functional component, the video memory optimization technology with the least impact on its training task is selected for video memory optimization, and a more appropriate video memory optimization method can be determined for each functional component, which can reduce the impact on model training while optimizing the video memory.

[0123] In step 611, when i≥2, the memory optimization technology corresponding to the memory optimization type is applied to each functional component in the object set of each GPU. When i=2, the objects in the object set are all objects on this GPU.

[0124] That is, for each functional component in the object set of each GPU, if the memory optimization type of this functional component is activation value recalculation, activation value recalculation is applied to this functional component; if the memory optimization type of this functional component is tensor offloading, tensor offloading is applied to this functional component.

[0125] In step 612 , the remaining amount of video memory and the amount of video memory fragmentation of each GPU during the iteration process, as well as the video memory usage and training time of each functional component on each GPU are obtained.

[0126] The training time of each functional component on each GPU refers to the time it takes for this GPU to execute the training task corresponding to this functional component during this round of iteration.

[0127] In step 613 , after the iteration is completed, it is determined whether the training is completed. If so, the process proceeds to step 619 ; otherwise, the process proceeds to step 614 .

[0128] The training end condition is such as the number of iterations reaching a preset number.

[0129] In step 614 , it is determined whether the remaining video memory of each GPU is greater than the minimum video memory usage in the object set of this GPU, and the video memory fragmentation is less than a first threshold. If so, the process proceeds to step 615 ; otherwise, the process proceeds to step 616 .

[0130] In actual applications, if the remaining video memory of any GPU is greater than the minimum video memory usage in the object set of this GPU, and the amount of video memory fragmentation is less than the first threshold, it means that this GPU has sufficient video memory and less video memory fragmentation. In this case, the video memory optimization efforts for this GPU can be reduced to improve the model training speed.

[0131] In step 615 , it is determined to reduce the memory optimization intensity of the GPU, and an object is removed from the object set of the GPU, and then the process proceeds to step 606 .

[0132] Among them, the objects to be eliminated are, for example, functional modules whose video memory occupancy is less than the remaining video memory of this GPU and whose training takes the longest time.

[0133] In step 616 , it is determined whether the GPU meets the requirement that the remaining amount of video memory is less than the second threshold and the amount of video memory fragmentation is not less than the first threshold. If so, the process proceeds to step 617 ; otherwise, the process proceeds to step 618 .

[0134] In actual applications, if the remaining video memory of this GPU is less than the second threshold and the amount of video memory fragmentation is not less than the first threshold, it means that the video memory of the nth GPU is tight and the video memory fragmentation is large. In this case, the video memory optimization efforts for this GPU can be increased to save the video memory of this GPU and avoid the out of memory (OOM) problem of this GPU.

[0135] In step 617 , it is determined to increase the memory optimization strength for this GPU, and an object is added to the object set of this GPU, and then the process proceeds to step 606 .

[0136] The added object may be an object that is not in the object set of this GPU and has the shortest training time.

[0137] In step 618 , it is determined to maintain the memory optimization strength for this GPU, keep the object set of this GPU unchanged, and then enter step 606 .

[0138] In step 619, the training ends.

[0139] In addition to performing dynamic activation value recalculation and dynamic tensor unloading based on the functional components of the MoE model, dynamic activation value recalculation and dynamic tensor unloading can also be performed based on the operators of the MoE model.

[0140] Taking an expert component of the MoE model as an example, its typical network structure is a feedforward neural network (FFN). A typical operator and network structure information contained in a FFN based on a multilayer perceptron (MLP) is as follows: Figure 7 As shown in the figure, it includes five operators in sequence: matrix multiplication, bias addition, activation function, matrix multiplication, and bias addition. Other functional components of the MoE model can also include multiple operators.

[0141] When performing dynamic activation value recalculation and dynamic tensor unloading in operators, Figure 6 The functional components in the code are updated to operators, and the objects in the object collection are replaced with operators. The other processes are the same and will not be described here.

[0142] Based on the same technical concept, an embodiment of the present application also provides a video memory optimization device. The principle of solving the problem by the video memory optimization device is similar to that of the above-mentioned video memory optimization method. Therefore, the implementation of the video memory optimization device can refer to the implementation of the video memory optimization method, and the repeated parts will not be repeated.

[0143] Figure 8 A schematic diagram of the structure of a video memory optimization device provided in an embodiment of the present application includes:

[0144] An acquisition module 801 is configured to, during distributed training of a deep learning model on multiple chips, acquire video memory load information of each chip and video memory usage information of multiple objects on the chip, where the multiple objects correspond to multiple training tasks performed by the chip, when a preset condition is met;

[0145] An updating module 802 is configured to update an object set of the chip based on the video memory load information of each chip and the video memory usage information of multiple objects on the chip, wherein the object set is used to store objects on the chip to which the video memory optimization technology is applied;

[0146] The sending module 803 is configured to send a memory optimization instruction to each chip according to the object set of the chip, wherein the memory optimization instruction is used to trigger the chip to apply a memory optimization technology to the objects in the object set.

[0147] In some embodiments, the multiple objects on the chip are multiple functional components or multiple operators of the deep learning model distributed on the chip.

[0148] In some embodiments, the update module 802 is specifically configured to:

[0149] Determining, based on the video memory load information of each chip and the video memory usage information of multiple objects on the chip, a method for adjusting the video memory optimization intensity of the chip;

[0150] The object set of the chip is updated according to the adjustment method of the video memory optimization strength of the chip.

[0151] In some embodiments, the video memory load information of each chip includes the remaining video memory and the amount of video memory fragmentation, and the video memory usage information of each object is the video memory usage of the object on the chip where it is located; the updating module 802 is specifically used to:

[0152] When a first condition is met, determining that the adjustment method is to reduce the video memory optimization intensity, the first condition being that the remaining video memory of the chip is greater than the video memory usage of the target object, the video memory fragmentation of the chip is less than a first threshold, and the target object is the object with the smallest video memory usage in the object set of the chip;

[0153] When a second condition is met, determining that the adjustment method is to increase the video memory optimization intensity, the second condition being that the remaining video memory of the chip is less than a second threshold, the amount of video memory fragmentation of the chip is not less than the first threshold, and the second threshold is less than the video memory usage of the target object;

[0154] When the first condition and the second condition are not met, it is determined that the adjustment method is to maintain the video memory optimization strength.

[0155] In some embodiments, the update module 802 is specifically configured to:

[0156] When the adjustment method is to reduce the intensity of video memory optimization, removing an object from the object set;

[0157] When the adjustment method is to increase the video memory optimization strength, adding an object to the object set;

[0158] When the adjustment method is to maintain the video memory optimization strength, the object set is kept unchanged.

[0159] In some embodiments, the acquisition module 801 is further configured to acquire the training time corresponding to each object on each chip during the most recent L rounds of iterations when a preset condition is met, where L is an integer greater than zero;

[0160] The objects eliminated from the object set are the objects in the object set that take the longest time to train and whose video memory usage is less than the remaining video memory of the chip;

[0161] The objects added to the object set are objects that are not in the object set and have the shortest training time.

[0162] In some embodiments, the sending module 803 is further configured to:

[0163] Sending a memory optimization type corresponding to each object on each chip to each chip, where the memory optimization type is activation value recalculation or tensor unloading; and

[0164] Each chip applies a video memory optimization technology corresponding to the video memory optimization type of each object in the object set of the chip.

[0165] In some embodiments, the video memory optimization type corresponding to each object on each chip is determined according to the following steps:

[0166] Apply each memory optimization type to each object on each chip.

[0167] Counting the training time of the object and the memory optimization time corresponding to the memory optimization technology after M iterations of training, where M is an integer greater than zero;

[0168] Determining, based on the training time and the memory optimization time, an impact factor of the memory optimization technology on the training task of the object;

[0169] The video memory optimization type corresponding to the video memory optimization technology with the smallest impact factor is determined as the video memory optimization type corresponding to the object.

[0170] The division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, the functional modules in the embodiments of the present application may be integrated into one processor, or may exist physically separately, or two or more modules may be integrated into one module. The coupling between the modules can be achieved through some interfaces, which are usually electrical communication interfaces, but it is not ruled out that they may be mechanical interfaces or other forms of interfaces. Therefore, the modules described as separate components may or may not be physically separated, and may be located in one place or distributed to different locations of the same or different devices. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software functional modules.

[0171] After introducing the video memory optimization method and apparatus according to an exemplary embodiment of the present application, a computer device according to another exemplary embodiment of the present application is introduced next.

[0172] Based on the same technical concept, the embodiment of the present application provides a computer device, such as Figure 9 As shown, it includes at least one chip 100 and a memory 901 connected to the at least one chip 100. The specific connection medium between the chip 100 and the memory 901 is not limited in the embodiment of the present application. Figure 9 For example, the chip 100 and the memory 901 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, and the like.

[0173] In an embodiment of the present application, the memory 901 stores instructions that can be executed by at least one chip 100. The at least one chip 100 can perform the steps of the above-mentioned cache line scheduling method by executing the instructions stored in the memory 901.

[0174] Among them, chip 100 is the control center of the computer device. It can use various interfaces and lines to connect various parts of the computer device. By running or executing instructions stored in memory 901 and calling data stored in memory 901, it can achieve video memory optimization. Optionally, chip 100 may include one or more processing units. Chip 100 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into chip 100. In some embodiments, chip 100 and memory 901 can be implemented on the same chip. In some embodiments, they can also be implemented separately on independent chips.

[0175] Chip 100 can be a general-purpose processor, such as a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

[0176] The memory 901 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs, and modules. The memory 901 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (RAM), a static random access memory (SRAM), a programmable read-only memory (PROM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic storage device, a magnetic disk, an optical disk, and the like. The memory 901 is any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer device, but is not limited thereto. The memory 901 in the embodiments of the present application may also be a circuit or any other device capable of performing a storage function, used to store program instructions and / or data.

[0177] In an exemplary embodiment, a storage medium is also provided. When a computer program stored in the storage medium is executed by a processor of a computer device, the computer device is capable of performing any of the aforementioned video memory optimization methods. Alternatively, the storage medium may be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0178] In an exemplary embodiment, a computer program product is further provided. When the computer program product is executed by a computer device, the computer device can implement any exemplary method provided in this application.

[0179] It should be noted that although several modules or submodules of the device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, depending on the embodiment of the application, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided and embodied by multiple modules.

[0180] Furthermore, although the operations of the method of the present application are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all illustrated operations must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0181] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0182] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0183] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application also includes these modifications and variations.

Claims

1. A video memory optimization method, characterized in that: include: During distributed training of a deep learning model on multiple chips, whenever a preset condition is met, obtaining video memory load information for each chip, video memory usage information for multiple objects on the chip, and the training time corresponding to each object on the chip during the most recent L iterations, wherein the multiple objects correspond to multiple training tasks performed by the chip, wherein the deep learning model is a hybrid expert model comprising multiple expert networks, each of which is placed on a separate chip, and L is an integer greater than zero; Determining, based on the video memory load information of each chip and the video memory usage information of multiple objects on the chip, a method for adjusting the video memory optimization intensity of the chip; updating an object set of the chip according to an adjustment method for the video memory optimization intensity of the chip, the object set being used to store objects on the chip to which video memory optimization technology is applied, wherein each object corresponds to a video memory optimization type, the video memory optimization type being activation value recalculation or tensor unloading, and the video memory optimization technology corresponding to each object having the smallest impact on the training task of the object among all video memory optimization technologies, and wherein the adjustment method is to remove from the object set objects that have the longest training time and whose video memory usage is less than the remaining video memory of the chip when reducing the video memory optimization intensity, and to add to the object set objects that are not in the object set and have the shortest training time when increasing the video memory optimization intensity; Based on the object set of each chip, a memory optimization indication is sent to the chip, where the memory optimization indication is used to trigger the chip to apply a memory optimization technology to the objects in the object set, wherein the chip applies a memory optimization technology corresponding to the memory optimization type of the object to each object in the object set of the chip.

2. The method according to claim 1, wherein The multiple objects on the chip are multiple functional components or multiple operators of the deep learning model distributed on the chip.

3. The method according to claim 1, wherein The video memory load information of each chip includes the remaining video memory and the amount of video memory fragmentation. The video memory usage information of each object is the video memory usage of the chip on which the object resides. Based on the video memory load information of each chip and the video memory usage information of multiple objects on the chip, a method for adjusting the video memory optimization intensity of the chip is determined, including: When a first condition is met, determining that the adjustment method is to reduce the video memory optimization intensity, the first condition being that the remaining video memory of the chip is greater than the video memory usage of the target object, the video memory fragmentation of the chip is less than a first threshold, and the target object is the object with the smallest video memory usage in the object set of the chip; When a second condition is met, determining that the adjustment method is to increase the video memory optimization intensity, the second condition being that the remaining video memory of the chip is less than a second threshold, the amount of video memory fragmentation of the chip is not less than the first threshold, and the second threshold is less than the video memory usage of the target object; When the first condition and the second condition are not met, it is determined that the adjustment method is to maintain the video memory optimization strength.

4. The method according to claim 1, wherein Use the following steps to determine the memory optimization type for each object on each chip: Apply each memory optimization type to each object on each chip. Counting the training time of the object and the memory optimization time corresponding to the memory optimization technology after M iterations of training, where M is an integer greater than zero; Determining, based on the training time and the memory optimization time, an impact factor of the memory optimization technology on the training task of the object; The video memory optimization type corresponding to the video memory optimization technology with the smallest impact factor is determined as the video memory optimization type corresponding to the object.

5. A video memory optimization device, characterized in that: include: An acquisition module is configured to, during distributed training of a deep learning model on multiple chips, acquire, whenever a preset condition is met, video memory load information of each chip, video memory usage information of multiple objects on the chip, and the training time corresponding to each object on the chip during the most recent L iterations, wherein the multiple objects correspond to multiple training tasks performed by the chip, where the deep learning model is a hybrid expert model comprising multiple expert networks, each of which is placed on a separate chip, and L is an integer greater than zero; An update module is configured to determine an adjustment method for the video memory optimization intensity of the chip based on the video memory load information of each chip and the video memory usage information of multiple objects on the chip; and update an object set of the chip based on the adjustment method for the video memory optimization intensity of the chip, the object set being used to store objects on the chip to which video memory optimization technology is applied, wherein each object corresponds to a video memory optimization type, the video memory optimization type being activation value recalculation or tensor unloading, and the video memory optimization technology corresponding to each object having the smallest impact on the training task of the object among all video memory optimization technologies; the adjustment method is to remove from the object set objects whose training time is the longest and whose video memory usage is less than the remaining video memory of the chip when reducing the video memory optimization intensity; and the adjustment method is to add to the object set objects that are not in the object set and whose training time is the shortest when increasing the video memory optimization intensity; A sending module is used to send a video memory optimization instruction to the chip according to the object set of each chip, and the video memory optimization instruction is used to trigger the chip to apply the video memory optimization technology to the objects in the object set, wherein the chip applies the video memory optimization technology corresponding to the video memory optimization type of the object to each object in the object set of the chip.

6. A computer device comprising a memory, a chip, and a computer program stored in the memory and executable on the chip, wherein: When the chip executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

7. A storage medium, characterized in that: When the computer program in the storage medium is executed by a processor of a computer device, the computer device can perform the method according to any one of claims 1 to 4.

8. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 4 when executed by a processor.

Images