Video memory optimization method and device, computer equipment, storage medium and computer program product

Abstract

The invention discloses a video memory optimization method and device, computer equipment, a storage medium and a computer program product, and belongs to the technical field of artificial intelligence, in the method, in the process of distributed training of a deep learning model on a plurality of chips, when a preset condition is met, video memory load information of each chip is obtained, and the video memory load information of each chip is obtained; the method comprises the following steps of: updating an object set of a chip according to video memory load information of each chip, video memory use information of a plurality of objects on each chip and different training tasks executed by the plurality of objects corresponding to the chip according to the video memory load information of each chip and the video memory use information of the plurality of objects on the chip; and sending a video memory optimization instruction to the chip to trigger the chip to apply a video memory optimization technology to the objects in the object set. Therefore, the object applying the video memory optimization technology on each chip can change along with the change of the training process, the video memory optimization is more matched with the training process, and the video memory optimization is more reasonable, so that the model training effect can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and particularly to a video memory optimization method, device, computer device, storage medium, and computer program product. Background Art

[0002] Generally, during the process of training a deep learning model on multiple chips, intermediate activation values, gradient information, etc. will occupy a large amount of video memory. In the case of insufficient video memory resources, it is usually necessary to use video memory optimization technology to optimize the training process. How to reasonably use video memory optimization technology to improve the model training effect has become a technical problem to be solved urgently. Summary of the Invention

[0003] Embodiments of this application provide a video memory optimization method, device, computer device, storage medium, and computer program product, so as to improve the rationality of video memory optimization during the training process of a deep learning model, thereby improving the model training effect.

[0004] In a first aspect, an embodiment of this application provides a video memory optimization method, including: During the process of distributed training of a deep learning model on multiple chips, when a preset condition is met, obtain the video memory load information of each chip and the video memory usage information of multiple objects on the chip, where the multiple objects correspond to multiple training tasks executed by the chip; Update the object set of the chip according to the video memory load information of each chip and the video memory usage information of multiple objects on the chip, where the object set is used to save the objects on the chip to which video memory optimization technology is applied; Send a video memory optimization instruction to the chip according to the object set of each chip, where the video memory optimization instruction is used to trigger the chip to apply video memory optimization technology to the objects in the object set.

[0005] In some embodiments, the multiple objects on the chip are multiple functional components or multiple operators of the deep learning model distributed on the chip.

[0006] In some embodiments, updating the object set of the chip according to the video memory load information of each chip and the video memory usage information of multiple objects on the chip includes: Determine the adjustment method of the video memory optimization intensity of the chip according to the video memory load information of each chip and the video memory usage information of multiple objects on the chip; Update the object set of the chip according to the adjustment method of the video memory optimization intensity of the chip.

[0007] In some embodiments, the video memory load information of each chip includes the remaining video memory and the amount of video memory fragmentation, and the video memory usage information of each object is the video memory usage of the chip on which the object resides. Determining, based on the video memory load information of each chip and the video memory usage information of multiple objects on the chip, a method for adjusting the video memory optimization intensity of the chip includes: When a first condition is met, determining that the adjustment method is to reduce the video memory optimization intensity, the first condition being that the remaining video memory of the chip is greater than the video memory usage of the target object, the video memory fragmentation of the chip is less than a first threshold, and the target object is the object with the smallest video memory usage in the object set of the chip; When a second condition is met, determining that the adjustment method is to increase the video memory optimization intensity, the second condition being that the remaining video memory of the chip is less than a second threshold, the amount of video memory fragmentation of the chip is not less than the first threshold, and the second threshold is less than the video memory usage of the target object; When the first condition and the second condition are not met, it is determined that the adjustment method is to maintain the video memory optimization strength.

[0008] In some embodiments, updating the object set of the chip according to the adjustment method of the video memory optimization strength of the chip includes: When the adjustment method is to reduce the intensity of video memory optimization, removing an object from the object set; When the adjustment method is to increase the video memory optimization strength, adding an object to the object set; When the adjustment method is to maintain the video memory optimization strength, the object set is kept unchanged.

[0009] In some embodiments, when a preset condition is met, the training time corresponding to each object on the chip during the most recent L rounds of iterations is also obtained, where L is an integer greater than zero; The objects eliminated from the object set are the objects in the object set that take the longest time to train and whose video memory usage is less than the remaining video memory of the chip; The objects added to the object set are objects that are not in the object set and have the shortest training time.

[0010] In some embodiments, it further includes: Sending a memory optimization type corresponding to each object on each chip to each chip, where the memory optimization type is activation value recalculation or tensor unloading; and Each chip applies a video memory optimization technology corresponding to the video memory optimization type of each object in the object set of the chip.

[0011] In some embodiments, the video memory optimization type corresponding to each object on each chip is determined according to the following steps: Apply the video memory optimization technology of each video memory optimization type to each object on each chip; After M iterations of training are completed, count the training time of the object and the video memory optimization time corresponding to the video memory optimization technology, where M is an integer greater than zero; According to the training time and the video memory optimization time, determine the influence factor of the video memory optimization technology on the training task of the object; Determine the video memory optimization type corresponding to the video memory optimization technology with the smallest influence factor as the video memory optimization type corresponding to the object.

[0012] In a second aspect, an embodiment of the present application provides a video memory optimization device, including: An acquisition module, configured to obtain the video memory load information of each chip and the video memory usage information of multiple objects on the chip when a preset condition is met during the process of distributed training of a deep learning model on multiple chips, where the multiple objects correspond to multiple training tasks executed by the chip; An update module, configured to update the object set of the chip according to the video memory load information of each chip and the video memory usage information of multiple objects on the chip, where the object set is used to store the objects to which the video memory optimization technology is applied on the chip; A sending module, configured to send a video memory optimization instruction to the chip according to the object set of each chip, where the video memory optimization instruction is used to trigger the chip to apply the video memory optimization technology to the objects in the object set.

[0013] In a third aspect, an embodiment of the present application provides a computer device, including a memory, a chip, and a computer program stored on the memory and executable on the chip. When the chip executes the computer program, any of the above video memory optimization methods is implemented.

[0014] In a fourth aspect, an embodiment of the present application provides a storage medium. When the computer program in the storage medium is executed by a processor of a computer device, the computer device can execute any of the above video memory optimization methods.

[0015] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, any of the above video memory optimization methods is implemented.

[0016] In the embodiments of the present application, during the process of distributed training of a deep learning model on multiple chips, whenever a preset condition is met, the video memory load information of each chip and the video memory usage information of multiple objects on each chip can be obtained. The multiple objects correspond to different training tasks executed by this chip. According to the video memory load information of each chip and the video memory usage information of multiple objects on this chip, the object set of this chip is updated. The object set is used to save the objects on this chip that need to apply the video memory optimization technology. According to the object set of each chip, a video memory optimization instruction is sent to this chip to trigger this chip to apply the video memory optimization technology to the objects in this object set. In this way, the objects to which the video memory optimization technology is applied on each chip can change with the change of the training process, the video memory optimization is more matched with the training process, and the application of the video memory optimization technology is more reasonable. Therefore, the model training effect can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation of the present application. In the drawings: Figure 1 FIG. is a schematic diagram of a scenario for distributed training of a deep learning model provided by an embodiment of the present application; Figure 2 FIG. is a schematic diagram of the structure of a chip provided by an embodiment of the present application; Figure 3 FIG. is a flowchart of a video memory optimization method provided by an embodiment of the present application; Figure 4 FIG. is a flowchart of a method for updating the object set of each chip provided by an embodiment of the present application; Figure 5 FIG. is a schematic diagram of the structure of a single Transformer layer provided by an embodiment of the present application; Figure 6 FIG. is a flowchart of another video memory optimization method provided by an embodiment of the present application; Figure 7 FIG. is a schematic diagram of the structure of an expert component provided by an embodiment of the present application; Figure 8 FIG. is a schematic diagram of the structure of a video memory optimization device provided by an embodiment of the present application; Figure 9 FIG. is a schematic diagram of the hardware structure of a computer device for implementing the video memory optimization method provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0018] In order to improve the rationality of video memory optimization during the training process of deep learning models, and thus improve the model training effect, the embodiments of the present application provide a video memory optimization method, device, computer device, storage medium, and computer program product.

[0019] The following describes the preferred embodiments of the present application with reference to the accompanying drawings of the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. And without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0020] First, the technical terms involved in the embodiments of the present application will be introduced below.

[0021] 1. Activation recomputation.

[0022] Activation recomputation is a video memory optimization technique used during the training of deep learning models. During the training process of deep learning models, when the model scale is large or the input sequence is long, the intermediate activation values will occupy a large amount of video memory resources. Activation recomputation saves some intermediate activation values in the forward propagation stage, discards some intermediate activation values, and recomputes the discarded intermediate activation values in the backward propagation stage, thereby reducing video memory usage by trading time for storage space.

[0023] 2. Tensor offloading.

[0024] Tensor offloading is another video memory optimization technique used during the training of deep learning models. In the case where training resources are limited, such as when the video memory of a graphics processing unit (GPU) is limited, tensor offloading transfers some tensor data during the training process from the GPU to the CPU in the forward propagation stage to relieve the video memory pressure and minimize the impact on the training speed as much as possible. When accessing the offloaded tensor data during the backward propagation stage, the tensor data is reloaded from the CPU to the GPU again, also reducing video memory usage by trading time for storage space.

[0025] 3. Unbalanced load training.

[0026] During the process of training a deep learning model in a distributed manner based on multiple chips, the loads of each chip, such as video memory occupancy and communication overhead, will be unevenly distributed, resulting in a situation where some chips are overloaded while some chips are idle, which will affect the overall training efficiency.

[0027] Take the deep learning model as an example of the mixture of experts (MoE) model. The core of the MoE model is the expert component. The basic data unit input to the expert component is tokens, and the tokens are assigned to different expert components for processing through the router component in the MoE model. In the initial stage of training, there are significant differences in the number of tokens processed by different expert components. During the training process, the number of tokens processed by each expert component will also change continuously. In the later stage of training, the number of tokens processed by each expert component tends to be balanced.

[0028] When different expert components are distributed on different chips, the different number of tokens processed by different expert components will result in different sizes of activation values input to the expert components on different chips, thus causing an obvious load imbalance problem between the chips; and the continuous change in the number of tokens processed by the same expert component will cause the activation values input to the expert network on the same chip to also change continuously, thus causing the same chip to also have a load imbalance problem due to the change in the number of tokens processed by the expert network.

[0029] In order to relieve the video memory pressure, in the embodiments of the present application, during the process of distributed training of a deep learning model on multiple chips, whenever a preset condition is met, the video memory load information of each chip and the video memory usage information of multiple objects on each chip are obtained. The multiple objects correspond to different training tasks executed by this chip. According to the video memory load information of each chip and the video memory usage information of multiple objects on this chip, the object set of this chip is updated. The object set is used to save the objects on this chip that need to apply the video memory optimization technology. According to the object set of each chip, a video memory optimization instruction is sent to this chip to trigger this chip to apply the video memory optimization technology to the objects in this object set. In this way, the objects applying the video memory optimization technology on each chip can change with the change of the training process, the video memory optimization is more matched with the training process, the above-mentioned load imbalance problem can be better addressed, and the application of the video memory optimization technology is more reasonable. Therefore, the training effect of the deep learning model can be improved.

[0030] The application scenario of the present application is the distributed training of a deep learning model. First, the scenario of distributed training of a deep learning model will be introduced below.

[0031] See Figure 1 , Figure 1 FIG. is a schematic diagram of a scenario for distributed training of a deep learning model provided by an embodiment of the present application, including multiple chips 100, and the multiple chips 100 jointly train a deep learning model such as a mixture of experts model.

[0032] Generally, multiple chips 100 can be divided into slave chips and a master chip. Some functional components (or functional modules, and each functional component includes multiple operators) of the deep learning model can be deployed on each slave chip to complete the training tasks corresponding to these functional components. The master chip can analyze the video memory usage of each slave chip during the training process and instruct the slave chips to apply video memory optimization techniques to which functional modules or which operators in the functional modules, so as to better complete model training when the video memory resources are limited.

[0033] See Figure 2 , Figure 2 FIG. 1 is a schematic structural diagram of a chip 100 provided by an embodiment of the present application, which at least includes: a video memory 101, an on-chip cache 102, and multiple computing units 103.

[0034] The video memory 101 can be a high bandwidth memory (HBM for short), or other types of memories. The on-chip cache 102 is a temporary memory with a smaller capacity than the video memory 101, but a faster data read / write speed than the video memory 101.

[0035] The computing units 103 can be used to execute the training tasks of various functional components. The various functional components include but are not limited to: a normalization component, a self-attention component, etc. Moreover, each functional component can include multiple operators such as a matrix multiplication operator, a matrix addition operator, a matrix inverse operator, etc., and each operator corresponds to a smaller-grained training task.

[0036] The computing units 103 can read and write data to / from the video memory 101, and can also read and write data to / from the on-chip cache 102. For example, the computing units 103 can read data from the video memory 101 for calculation, or read data from the video memory 101, then write the read data to the on-chip cache 102, and then read data from the on-chip cache 102 for calculation.

[0037] In addition to including the above structure, the chip 100 in the embodiment of the present application can also include other structures, and the embodiment of the present application does not make specific limitations thereon.

[0038] In addition, the chip 100 can be: a GPU, a general-purpose computing on graphics processing units (GPGPU), a domain specific architecture (DSA), etc.

[0039] After introducing the application scenarios of the embodiments of the present application, the video memory optimization method proposed by the present application will be described below with specific embodiments.

[0040] See Figure 3 , Figure 3 which is a flowchart of a video memory optimization method provided by an embodiment of this application. This method can be applied to the main chip in Figure 1 or other devices, and the method includes the following steps.

[0041] In step 301, during the process of training a deep learning model distributively on multiple chips, when a preset condition is met, obtain the video memory load information of each chip and the video memory usage information of multiple objects on this chip. The multiple objects correspond to multiple training tasks executed on this chip.

[0042] Among them, the multiple chips are such as multiple GPUs. The deep learning model can be used to execute various image processing tasks such as image classification, object detection, semantic segmentation, image generation, etc. The preset conditions are such as the arrival of a set period, the completion of L rounds of iterative training, etc., where L is an integer greater than zero such as 1, 2, etc. The video memory load information of each chip, such as the remaining video memory and the amount of video memory fragmentation, is used to characterize the video memory load situation of this chip; the video memory usage information of any object on each chip, such as the video memory usage of the object on this chip, is used to characterize the video memory occupancy of the object on this chip.

[0043] In step 302, according to the video memory load information of each chip and the video memory usage information of multiple objects on this chip, update the object set of this chip. The object set is used to save the objects applying the video memory optimization technology on this chip.

[0044] Specifically, during implementation, the object set of each chip can be updated according to the process shown in Figure 4 , including the following steps.

[0045] In step 3021, according to the video memory load information of each chip and the video memory usage information of multiple objects on the chip, determine the adjustment method of the video memory optimization intensity for the chip.

[0046] For each chip, the object with the smallest video memory usage in the object set of this chip can be determined as the target object. If the remaining video memory of this chip is greater than the video memory usage of the target object and the amount of video memory fragmentation of this chip is less than the first threshold, it means that the video memory of this chip is relatively rich and has less fragmentation. In order to improve the training speed, it can be determined that the adjustment method of the video memory optimization intensity for this chip is to reduce the video memory optimization intensity.

[0047] When the remaining video memory of this chip is not greater than the video memory usage of the target object, if the remaining video memory of this chip is less than the second threshold (the second threshold is less than the video memory usage of the target object) and the amount of video memory fragmentation of this chip is not less than the first threshold, it indicates that the video memory of this chip is relatively tight and there is more fragmentation. To save video memory resources, it can be determined that the adjustment method for the video memory optimization intensity of this chip is to increase the video memory optimization intensity.

[0048] In the case where neither of the above two conditions is met, such as when the remaining video memory of this chip is greater than the second threshold but less than the video memory usage of the target object, it indicates that the video memory optimization situation of this chip is relatively matched with the training stage. It can be determined that the adjustment method for the video memory optimization intensity of this chip is to maintain the video memory optimization intensity.

[0049] In step 3022, update the object set of the chip according to the adjustment method for the video memory optimization intensity of the chip.

[0050] In the first case, the adjustment method for the video memory optimization intensity of the chip is to decrease the video memory optimization intensity. In this case, an object can be removed from the object set of this chip to improve the video memory utilization rate of this chip and improve the model training speed.

[0051] When the preset conditions are met, it is also possible to obtain the training time corresponding to each object on this chip during the most recent L rounds of iteration (including the current round of iteration), that is, the total training time of this chip for executing the training tasks corresponding to each object during the most recent L rounds of iteration. L is a positive integer, such as 1, 2, etc. In this case, the object to be removed can be the object with the longest time-consuming and the video memory occupancy less than the remaining video memory of this chip in the object set of this chip. In this way, the video memory utilization rate of this chip can be maximally improved, and the training duration can be reduced as much as possible.

[0052] In the second case, the adjustment method for the video memory optimization intensity of the chip is to increase the video memory optimization intensity. In this case, an object can be added to the object set of this chip to save video memory resources.

[0053] Moreover, when the training time corresponding to each object on this chip during the most recent L rounds of iteration has been obtained, the object to be added can be the object that is not in the object set of this chip and has the shortest time-consuming. In this way, the video memory resources of this chip can be saved, and the increase in training duration can be minimized.

[0054] In the third case, the adjustment method for the video memory optimization intensity of the chip is to maintain the video memory optimization intensity. In this case, the object set of this chip can be kept unchanged to maintain the matching state between the current video memory resources and the training stage.

[0055] In step 303, according to the object set of each chip, a video memory optimization instruction is sent to this chip, and the video memory optimization instruction is used to trigger this chip to apply video memory optimization technology to the objects in this object set.

[0056] Among them, the object information of each object in the object set, such as the object identifier, can be carried in the video memory optimization instruction. This chip applies video memory optimization technology to the objects in this object set, that is, this chip applies video memory optimization technology when executing the training tasks corresponding to each object in this object set to optimize the video memory usage during the training process.

[0057] In some embodiments, after each chip receives the video memory optimization instruction, it can apply activation value recomputation or tensor offloading to each object in the object set of this chip. In this way, the same video memory optimization technology is applied to the objects on each chip.

[0058] In some embodiments, in order to better optimize the video memory, video memory optimization technology of each type of video memory optimization can be applied to each object on each chip, and the training time consumption of this object and the video memory optimization time consumption corresponding to this type of video memory optimization technology are counted after M iterative trainings are completed. According to the training time consumption and the video memory optimization time consumption, the influence factor of this type of video memory optimization technology on the training task of this object is determined. For example, according to the rule that the ratio of the video memory optimization time consumption to the training time consumption is positively correlated with the influence factor, the influence factor is determined, and then the type of video memory optimization corresponding to the video memory optimization technology with the smallest influence factor is determined as the type of video memory optimization corresponding to this object, where M is an integer greater than zero, such as 1, 2, etc.

[0059] Specifically, activation value recomputation can be applied to each object on each chip, and the first training time consumption of this object and the activation value recomputation time consumption (a type of video memory optimization time consumption) are counted after M iterative trainings are completed. Then, according to the rule that the ratio of the activation value recomputation time consumption to the first training time consumption is positively correlated with the influence factor, the influence factor of activation value recomputation on the training task of this object is determined.

[0060] Moreover, tensor offloading can be applied to each object on each chip, and the second training time consumption of this object and the tensor offloading time consumption (another type of video memory optimization time consumption) are counted after M iterative trainings are completed. Then, according to the rule that the ratio of the tensor offloading time consumption to the second training time consumption is positively correlated with the influence factor, the influence factor of tensor offloading on the training task of this object is determined.

[0061] Finally, for each object on each chip, the type of video memory optimization with the smallest of the two influence factors is determined as the type of video memory optimization corresponding to this object, so as to determine the video memory optimization technology with the least influence on the training for this object.

[0062] Then, the video memory optimization type corresponding to each object on each chip can be sent to each chip.

[0063] Subsequently, after each chip receives the video memory optimization indication, for each object in its own object set, it can apply the video memory optimization technology corresponding to the video memory optimization type of this object.

[0064] In this way, the objects on each chip can apply different video memory optimization technologies, and each object applies the video memory optimization technology with the least impact on training, which can further improve the video memory optimization effect.

[0065] In practical applications, multiple objects on each chip can be multiple functional components of a deep learning model distributed on this chip, or multiple operators of a deep learning model distributed on this chip. That is to say, the solution of the embodiment of the present application can perform video memory optimization at the granularity of functional components, and can also perform video memory optimization at the granularity of operators. In this way, the flexibility of video memory optimization is also relatively good.

[0066] Next, taking the distributed training of the MoE model on multiple GPUs as an example, the video memory optimization solution of the embodiment of the present application will be introduced.

[0067] Generally, the MoE model includes multiple Transformer layers. A typical structure of a Transformer layer is as Figure 5 shown, which successively includes a normalization component (normalize module), a self-attention component (self-attention module), an add and normalization component (add and normalize module), a router and a tokens rearrangement component (router and permute module) (the aforementioned router component is merged in this component), an expert component (expert module), an inverse tokens rearrangement component (unpermute module), and an add component (add module). Figure 4 Among them, input i represents the i-th input, activation value i represents the activation value input to expert component i (activation value i is obtained by performing a series of processes on input i), output i represents the output of expert component i, and i takes values from 1 to N, where N is the total number of GPUs.

[0068] When the MoE model is distributedly trained on multiple GPUs, each expert network can be placed on one GPU, and other components except the expert network can be placed once on each GPU, or can be scattered on all GPUs. The specific distribution form will not be elaborated here. In addition, the total number of components corresponding to one GPU can be the same or different, depending on the pipelining segmentation method of the MoE model, which will not be elaborated here.

[0069] Generally, in the initial stage of training, the number of activation values processed by each expert component changes drastically. The GPU will generate more video memory fragmentation, and the video memory pressure on the GPU is relatively high. It is suitable to apply video memory optimization techniques such as activation value recomputation and tensor offloading to some objects (i.e., some functional components or some operators). As the training progresses (i.e., the number of iterations increases), the model gradually converges, the activation values processed by each expert component will gradually tend to be equal, the video memory fragmentation on the GPU will gradually decrease, and the video memory will gradually be sufficient. Some objects can gradually stop applying video memory optimization techniques to improve the model training speed.

[0070] Therefore, the inventors thought of flexibly adjusting the objects to which the video memory optimization technique is applied on each GPU as the number of iterations increases, so as to improve the adaptability of video memory optimization to the training stage, improve the rationality of video memory optimization, and further improve the overall training effect.

[0071] The following takes the object to which the video memory optimization technique is applied on each GPU as a functional component, and the preset condition is that one round of iterative training is completed (i.e., L = 1) and M = 1 as an example to illustrate the video memory optimization process in the training process of the MoE model.

[0072] See Figure 6 , Figure 6 which is a flowchart of another video memory optimization method provided by an embodiment of the present application, including the following steps.

[0073] In step 601, before the model training starts, the user selects to enable the dynamic activation value recomputation function and the dynamic tensor offloading function.

[0074] Generally, the user can enable the dynamic activation value recomputation function and the dynamic tensor offloading function through configuration items. After enabling, the solution of the embodiment of the present application can be applied.

[0075] In step 602, at the beginning of the i-th iteration, initially i = 0.

[0076] In step 603, when i = 0, activation value recomputation is applied to each functional component on each GPU.

[0077] Taking Figure 4 as an example, this step applies activation value recomputation to each functional component of the Transformer layer shown in Figure 4 so as to analyze the influence factor of activation value recomputation on the training task of each functional component subsequently.

[0078] In step 604, the first training time and the activation value recomputation time of each functional component on each GPU during the iteration process are obtained.

[0079] Among them, the first training time consumption of each functional component on each GPU refers to the time consumption for this CPU to execute the training task corresponding to this functional component (after the application activation value recomputation), and the activation value recomputation time consumption of this functional component refers to the time consumption for activation value recomputation during the process of this CPU executing the training task corresponding to this functional component.

[0080] In step 605, after the iteration ends, according to the first training time consumption and the activation value recomputation time consumption of each functional component, determine the first influence factor of this functional component.

[0081] Suppose the first training time consumption of this functional component is T1 and the activation value recomputation time consumption is T2 (T2 < T1), then the first influence factor of this functional component (i.e., the influence factor of activation value recomputation on the training task of this functional component) recompute_factor = T2 × K1 / T1, where K1 is a preset adjustment ratio factor.

[0082] In step 606, update i to i + 1 and return to step 602.

[0083] In step 607, when i = 1, apply tensor offloading to each functional component on each GPU.

[0084] Take Figure 4 as an example, this step applies tensor offloading to each functional component of the Transformer layer shown in Figure 4 in order to subsequently analyze the influence factor of tensor offloading on the training task of each functional component.

[0085] In step 608, obtain the second training time consumption and the tensor offloading time consumption of each functional component on each GPU during the iteration process.

[0086] Among them, the second training time consumption of each functional component on each GPU refers to the time consumption for this CPU to execute the training task corresponding to this functional component (after the application of tensor offloading), and the tensor offloading time consumption of this functional component refers to the time consumption for tensor offloading during the process of this CPU executing the training task corresponding to this functional component.

[0087] In step 609, after the iteration ends, according to the second training time consumption and the tensor offloading time consumption of each functional component, determine the second influence factor of this functional component.

[0088] Suppose the second training time consumption of this functional component is T3 and the tensor offloading time consumption is T4 (T4 < T3), then the second influence factor of this functional component (i.e., the influence factor of tensor offloading on the training task of this functional component) IO_factor = T4 × K2 / T3, where K2 is a preset adjustment ratio factor.

[0089] In step 610, determine the corresponding video memory optimization type with the smaller value between the first influence factor and the second influence factor of each functional component as the video memory optimization type corresponding to this functional component, and then proceed to step 606.

[0090] That is, for each functional component, if recompute_factor < IO_factor, then determine the activation value recomputation as the video memory optimization type of this functional component; if recompute_factor > IO_factor, then determine the tensor offloading as the video memory optimization type of this functional component. If recompute_factor = IO_factor, then either the activation value recomputation or the tensor offloading can be randomly determined as the video memory optimization type of this functional component.

[0091] In this way, by selecting the video memory optimization technology with the least impact on its training task for each functional component for video memory optimization, a more suitable video memory optimization method can be determined for each functional component, reducing the impact on model training while optimizing the video memory.

[0092] In step 611, when i ≥ 2, apply the video memory optimization technology corresponding to the video memory optimization type to each functional component in the object set of each GPU. When i = 2, the objects in the object set are all the objects on this GPU.

[0093] That is, for each functional component in the object set of each GPU, if the video memory optimization type of this functional component is the activation value recomputation, then apply the activation value recomputation to this functional component; if the video memory optimization type of this functional component is the tensor offloading, then apply the tensor offloading to this functional component.

[0094] In step 612, obtain the remaining video memory, video memory fragmentation amount of each GPU during the iteration process, as well as the video memory usage amount and training time consumption of each functional component on each GPU.

[0095] Among them, the training time consumption of each functional component on each GPU refers to the time consumption of this GPU in executing the training task corresponding to this functional component during this round of iteration.

[0096] In step 613, after the iteration ends, determine whether the training is over. If so, proceed to step 619; otherwise, proceed to step 6 + 14.

[0097] Among them, the training end condition is, for example, that the number of iterations reaches the preset number.

[0098] In step 614, it is determined whether the remaining video memory of each GPU is greater than the minimum video memory usage in the object set of this GPU and the video memory fragmentation is less than the first threshold. If so, step 615 is entered; otherwise, step 616 is entered.

[0099] In practical applications, if the remaining video memory of any GPU is greater than the minimum video memory usage in the object set of this GPU and the video memory fragmentation is less than the first threshold, it means that the video memory of this GPU is surplus and the video memory fragmentation is less. In this case, the video memory optimization effort for this GPU can be reduced to improve the model training speed.

[0100] In step 615, it is determined to reduce the video memory optimization effort for this GPU, and an object is removed from the object set of this GPU, and then step 606 is entered.

[0101] Among them, the removed object is, for example, a functional module with video memory occupancy less than the remaining video memory of this GPU and the longest training time.

[0102] In step 616, it is determined whether the remaining video memory of this GPU is less than the second threshold and the video memory fragmentation is not less than the first threshold. If so, step 617 is entered; otherwise, step 618 is entered.

[0103] In practical applications, if the remaining video memory of this GPU is less than the second threshold and the video memory fragmentation is not less than the first threshold, it means that the video memory of the nth GPU is tight and the video memory fragmentation is more. In this case, the video memory optimization effort for this GPU can be increased to save the video memory of this GPU and avoid the out of memory (OOM) problem of this GPU.

[0104] In step 617, it is determined to increase the video memory optimization effort for this GPU, and an object is added to the object set of this GPU, and then step 606 is entered.

[0105] Among them, the added object can be an object that is not in the object set of this GPU on this GPU and has the shortest training time.

[0106] In step 618, it is determined to maintain the video memory optimization effort for this GPU, keep the object set of this GPU unchanged, and then step 606 is entered.

[0107] In step 619, the training ends.

[0108] In addition to performing dynamic activation value recomputation and dynamic tensor offloading in units of the functional components of the MoE model, dynamic activation value recomputation and dynamic tensor offloading can also be performed in units of the operators of the MoE model.

[0109] Taking an expert component of the MoE model as an example, its typical network structure is a feedforward neural network (FFN). A typical FFN based on a multilayer perceptron (MLP) includes typical operators and network structure information as follows Figure 7 shown, successively including five operators: matrix multiplication, bias addition, activation function, matrix multiplication, and bias addition. Other functional components of the MoE model can also include multiple operators.

[0110] When performing dynamic activation value recalculation and dynamic tensor offloading in units of operators, update the Figure 6 functional components in to operators, and replace the objects in the object set with operators. The other processes are the same and will not be elaborated here.

[0111] Based on the same technical concept, the embodiment of the present application also provides a video memory optimization device. The principle of the video memory optimization device to solve problems is similar to the above video memory optimization method. Therefore, the implementation of the video memory optimization device can refer to the implementation of the video memory optimization method, and the repeated parts will not be elaborated.

[0112] Figure 8 The following is a schematic structural diagram of a video memory optimization device provided by the embodiment of the present application, including: An acquisition module 801, configured to obtain the video memory load information of each chip and the video memory usage information of multiple objects on the chip when a preset condition is met during the process of distributed training of a deep learning model on multiple chips. The multiple objects correspond to multiple training tasks executed by the chip; An update module 802, configured to update the object set of the chip according to the video memory load information of each chip and the video memory usage information of multiple objects on the chip. The object set is used to save the objects on the chip that apply the video memory optimization technology; A sending module 803, configured to send a video memory optimization instruction to the chip according to the object set of each chip. The video memory optimization instruction is used to trigger the chip to apply the video memory optimization technology to the objects in the object set.

[0113] In some embodiments, the multiple objects on the chip are multiple functional components or multiple operators of the deep learning model distributed on the chip.

[0114] In some embodiments, the update module 802 is specifically configured to: Determine the adjustment method of the video memory optimization intensity for the chip according to the video memory load information of each chip and the video memory usage information of multiple objects on the chip; Update the object set of the chip according to the adjustment method of the video memory optimization intensity for the chip.

[0115] In some embodiments, the video memory load information of each chip includes the remaining video memory and the amount of video memory fragmentation, and the video memory usage information of each object is the amount of video memory used by the object on the chip where it is located; the update module 802 is specifically configured to: When the first condition is met, it is determined that the adjustment method is to reduce the video memory optimization strength. The first condition is that the remaining video memory of the chip is greater than the video memory usage of the target object, and the amount of video memory fragmentation of the chip is less than the first threshold. The target object is the object with the smallest video memory usage in the object set of the chip; When the second condition is met, it is determined that the adjustment method is to increase the video memory optimization strength. The second condition is that the remaining video memory of the chip is less than the second threshold, and the amount of video memory fragmentation of the chip is not less than the first threshold. The second threshold is less than the video memory usage of the target object; When the first condition and the second condition are not met, it is determined that the adjustment method is to maintain the video memory optimization strength.

[0116] In some embodiments, the update module 802 is specifically configured to: When the adjustment method is to reduce the video memory optimization strength, remove an object from the object set; When the adjustment method is to increase the video memory optimization strength, add an object to the object set; When the adjustment method is to maintain the video memory optimization strength, keep the object set unchanged.

[0117] In some embodiments, the acquisition module 801 is further configured to, when a preset condition is met, acquire the training time consumed by each object on each chip during the most recent L rounds of iteration, where L is an integer greater than zero; The object removed from the object set is the object with the longest training time consumed and a video memory usage less than the remaining video memory of the chip; The object added to the object set is an object that is not in the object set and has the shortest training time consumed.

[0118] In some embodiments, the sending module 803 is further configured to: Send the video memory optimization type corresponding to each object on each chip to each chip. The video memory optimization type is activation value recomputation or tensor offloading; and Each chip applies the video memory optimization technology corresponding to the video memory optimization type of each object in the object set of the chip.

[0119] In some embodiments, the video memory optimization type corresponding to each object on each chip is determined according to the following steps: Apply the video memory optimization technology of each type of video memory optimization to each object on each chip; After M iterations of training are completed, count the training time of the object and the video memory optimization time corresponding to the video memory optimization technology, where M is an integer greater than zero; According to the training time and the video memory optimization time, determine the influence factor of the video memory optimization technology on the training task of the object; Determine the type of video memory optimization corresponding to the video memory optimization technology with the smallest influence factor as the type of video memory optimization corresponding to the object.

[0120] The division of modules in the embodiments of the present application is illustrative. It is only a logical function division. In actual implementation, there may be other division methods. In addition, each functional module in the embodiments of the present application can be integrated in one processor, or can exist separately physically, or two or more modules can be integrated in one module. The coupling between each module can be realized through some interfaces, and these interfaces are usually electrical communication interfaces, but it does not exclude the possibility of being mechanical interfaces or other forms of interfaces. Therefore, the modules described as separate components may or may not be physically separated, and can be located in one place, or distributed to different positions of the same or different devices. The above integrated modules can be implemented in the form of hardware or in the form of software function modules.

[0121] After introducing the video memory optimization method and device of the exemplary embodiment of the present application, next, a computer device according to another exemplary embodiment of the present application will be introduced.

[0122] Based on the same technical concept, the embodiments of the present application provide a computer device, as Figure 9 shown, including at least one chip 100 and a memory 901 connected to at least one chip 100. In the embodiments of the present application, the specific connection medium between the chip 100 and the memory 901 is not limited, Figure 9 taking the connection between the chip 100 and the memory 901 through a bus as an example. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0123] In the embodiments of the present application, the memory 901 stores instructions executable by at least one chip 100. By executing the instructions stored in the memory 901, at least one chip 100 can execute the steps of the above cache line scheduling method.

[0124] Among them, the chip 100 is the control center of the computer device. It can connect various parts of the computer device through various interfaces and circuits, and optimize the video memory by running or executing the instructions stored in the memory 901 and calling the data stored in the memory 901. Optionally, the chip 100 may include one or more processing units. The chip 100 may integrate an application processor and a modem processor. Among them, the application processor mainly processes the operating system, user interface, application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above modem processor may not be integrated into the chip 100. In some embodiments, the chip 100 and the memory 901 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on independent chips.

[0125] The chip 100 may be a general-purpose processor, such as a digital signal processor, an application specific integrated circuit (ASIC), a field programmable gate array, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application may be directly embodied as being executed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0126] As a non-volatile computer-readable storage medium, the memory 901 can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. The memory 901 may include at least one type of storage medium. For example, it may include flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (RAM), a static random access memory (SRAM), a programmable read only memory (PROM), a read only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic memory, a magnetic disk, an optical disk, and so on. The memory 901 is any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by the computer device, but is not limited thereto. The memory 901 in the embodiments of the present application may also be a circuit or any other device capable of implementing a storage function, for storing program instructions and / or data.

[0127] In an exemplary embodiment, a storage medium is further provided. When the computer program in the storage medium is executed by a processor of a computer device, the computer device can execute any of the above video memory optimization methods. Optionally, the storage medium may be a non-transitory computer-readable storage medium. For example, the non-transitory computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0128] In an exemplary embodiment, a computer program product is further provided. When the computer program product is executed by a computer device, the computer device can implement any of the exemplary methods provided in this application.

[0129] It should be noted that although several modules or sub-modules of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more of the above-described modules can be embodied in one module. Conversely, the features and functions of one module described above can be further divided and embodied by multiple modules.

[0130] In addition, although the operations of the method of this application are described in a specific order in the drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the illustrated operations must be performed to achieve the desired result. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step for execution, and / or one step may be decomposed into multiple steps for execution.

[0131] Those skilled in the art should understand that the embodiments of this application can be provided as a method, a system, or a computer program product. Therefore, this application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0132] Although the preferred embodiments of this application have been described, those skilled in the art can make additional changes and modifications once they learn the basic creative concept. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0133] Obviously, those skilled in the art can make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalent technologies, this application also includes these modifications and variations.

Claims

1. A video memory optimization method, characterized in that, Including: During the process of distributed training of a deep learning model on multiple chips, when a preset condition is met, obtain the video memory load information of each chip and the video memory usage information of multiple objects on the chip, where the multiple objects correspond to multiple training tasks executed by the chip; Update the object set of the chip according to the video memory load information of each chip and the video memory usage information of multiple objects on the chip, where the object set is used to store the objects on the chip that apply video memory optimization technology; Send a video memory optimization instruction to the chip according to the object set of each chip, where the video memory optimization instruction is used to trigger the chip to apply video memory optimization technology to the objects in the object set.

2. The method according to claim 1, wherein The multiple objects on the chip are multiple functional components or multiple operators of the deep learning model distributed on the chip.

3. The method according to claim 1, wherein Updating the object set of the chip according to the video memory load information of each chip and the video memory usage information of multiple objects on the chip includes: Determine the adjustment method of the video memory optimization intensity for the chip according to the video memory load information of each chip and the video memory usage information of multiple objects on the chip; Update the object set of the chip according to the adjustment method of the video memory optimization intensity for the chip.

4. The method according to claim 3, wherein The video memory load information of each chip includes the remaining video memory and the amount of video memory fragmentation. The video memory usage information of each object is the amount of video memory used by the object on the chip where it is located. Determining the adjustment method of the video memory optimization intensity for the chip according to the video memory load information of each chip and the video memory usage information of multiple objects on the chip includes: When the first condition is met, determine that the adjustment method is to reduce the video memory optimization intensity. The first condition is that the remaining video memory of the chip is greater than the video memory usage of the target object, and the amount of video memory fragmentation of the chip is less than the first threshold, where the target object is the object with the smallest video memory usage in the object set of the chip; When the second condition is met, determine that the adjustment method is to increase the video memory optimization intensity. The second condition is that the remaining video memory of the chip is less than the second threshold, and the amount of video memory fragmentation of the chip is not less than the first threshold, where the second threshold is less than the video memory usage of the target object; When the first condition and the second condition are not met, determine that the adjustment method is to maintain the video memory optimization intensity.

5. The method according to claim 3, wherein Updating the object set of the chip according to the adjustment method of the video memory optimization intensity for the chip includes: When the adjustment method is to reduce the video memory optimization intensity, remove an object from the object set; When the adjustment method is to increase the video memory optimization intensity, add an object to the object set; When the adjustment method is to maintain the video memory optimization intensity, keep the object set unchanged.

6. The method according to claim 5, wherein When the preset condition is met, the training time consumed by each object on the chip during the most recent L rounds of iteration is also obtained, where L is an integer greater than zero; The object removed from the object set is the object in the object set with the longest training time and a video memory usage less than the remaining video memory of the chip. The object added to the object set is an object that is not in the object set and has the shortest training time.

7. The method according to any one of claims 1 to 6, characterized in that It further includes: Sending the video memory optimization type corresponding to each object on each chip to each chip, where the video memory optimization type is activation value recomputation or tensor offloading; And Each chip applies the video memory optimization technology corresponding to the video memory optimization type of each object in the object set of the chip.

8. The method according to claim 7, characterized in that, Determine the video memory optimization type corresponding to each object on each chip according to the following steps: Apply the video memory optimization technology of each video memory optimization type to each object on each chip; Statistically calculate the training time of the object and the video memory optimization time corresponding to the video memory optimization technology after M iterations of training end, where M is an integer greater than zero; Determine the influence factor of the video memory optimization technology on the training task of the object according to the training time and the video memory optimization time; Determine the video memory optimization type corresponding to the video memory optimization technology with the smallest influence factor as the video memory optimization type corresponding to the object.

9. A video memory optimization device, characterized in that, It includes: An acquisition module, configured to, during the process of distributed training of a deep learning model on multiple chips, when a preset condition is met, acquire the video memory load information of each chip and the video memory usage information of multiple objects on the chip, where the multiple objects correspond to multiple training tasks executed by the chip; An update module, configured to update the object set of the chip according to the video memory load information of each chip and the video memory usage information of multiple objects on the chip, where the object set is used to store the objects to which the video memory optimization technology is applied on the chip; A sending module, configured to send a video memory optimization instruction to the chip according to the object set of each chip, where the video memory optimization instruction is used to trigger the chip to apply the video memory optimization technology to the objects in the object set.

10. A computer device, comprising a memory, a chip, and a computer program stored on the memory and executable on the chip, characterized in that, When the chip executes the computer program, it implements the steps of the method according to any one of claims 1-8.

11. A storage medium, characterized in that, When the computer program in the storage medium is executed by the processor of the computer device, the computer device can execute the method according to any one of claims 1-8.

12. A computer program product, characterized in that, It includes a computer program, and when the computer program is executed by a processor, it implements the method according to any one of claims 1-8.

Images