Deep learning model training method and video memory optimization device

Abstract

The invention discloses a deep learning model training method and a video memory optimization device. The method comprises the following steps: in a first training stage, acquiring video memory life cycle information and video memory size information of each tensor in the stage; tensor classification is carried out according to the video memory life cycle information, and a video memory allocation scheme is generated according to a tensor classification result and the video memory size information; and in the second training stage, carrying out video memory allocation based on the video memory allocation scheme. According to the method and the device, video memory allocation is planned in advance by utilizing predictable allocation mode priori knowledge in model training, so that video memory fragments are effectively reduced on the basis of not influencing the training performance, and the resource use efficiency in the training process is further improved. In addition, the generation efficiency of the video memory allocation scheme can be improved through spatial aggregation of temporary tensors grouped according to the size and tensor merging before aggregation, and efficient video memory allocation is achieved on dense and MoE models by means of the video memory optimization scheme through introduction of a static and dynamic video memory allocator.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, and particularly relates to a deep learning model training method and a GPU memory optimization device. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, deep learning models have been widely applied in the fields of image recognition, speech recognition, natural language processing, etc. To improve the expression ability and reasoning performance of the model, the scale of neural networks is continuously expanding, and the number of model parameters is continuously growing. In recent years, as an important development direction of deep learning, the parameter scale of large language models (LLM) has gradually expanded from hundreds of millions to hundreds of millions or even tens of billions, and the training process has put higher requirements on computing resources and GPU memory capacity.

[0003] Under this background, the training of LLM is usually difficult to complete on a single GPU. To solve the problem of insufficient memory, two types of optimization techniques are mainly used in research and engineering practice: one is model parallelism technology, including tensor parallelism and pipeline parallelism, which divides the parameters or computing tasks of the model to multiple GPUs for execution, thereby sharing the memory load; the other is non-parallel memory optimization technology, such as activation recomputation, which re-computes intermediate results in the backpropagation phase to save storage space; and tensor offload, which moves temporarily unused tensors out of GPU memory and recalls them when necessary.

[0004] Although the above-mentioned techniques effectively alleviate the memory pressure, they also bring new challenges, especially in terms of memory allocation. These techniques break the default memory management logic of deep learning frameworks. Taking the deep learning framework PyTorch as an example, its memory allocation mechanism is based on the "best-fit" strategy, which relies on the relative stability of the life cycle and size pattern of memory requests. After using strategies such as activation recomputation, tensor offload, and tensor reuse, the memory application and release behavior of tensors becomes dynamic and uncertain, making it difficult for the original memory scheduling strategy to remain efficient, resulting in an increase in memory fragmentation, a decrease in memory utilization, and an impact on the resource usage efficiency and system stability of the training process. SUMMARY

[0005] To this end, the disclosure provides a deep learning model training method and a GPU optimization device for deep learning model training. The GPU optimization device uses the predictable allocation mode prior knowledge in the model training to plan the GPU allocation in advance, thereby effectively reducing the GPU fragmentation without affecting the training performance, and further improving the resource utilization efficiency in the training process.

[0006] According to a first aspect of the disclosure, a deep learning model training method is provided, including: obtaining GPU behavior information in a first training stage of the deep learning model, the GPU behavior information including GPU life cycle information and GPU size information of each tensor in the first training stage, wherein the first training stage includes at least three iteration cycles; generating a GPU allocation scheme according to the GPU behavior information, wherein the tensors are classified according to the GPU life cycle information, and the GPU allocation scheme is generated according to the tensor classification result and the GPU size information; and performing GPU allocation based on the GPU allocation scheme in a second training stage of the deep learning model.

[0007] Optionally, generating the GPU allocation scheme according to the GPU behavior information includes: classifying tensors with a GPU life cycle less than a first life cycle threshold as spatially aggregable tensors; grouping the spatially aggregable tensors according to the GPU size information; and spatially aggregating multiple spatially aggregable tensors with non-overlapping GPU life cycles in the same group to generate a first GPU allocation sub-scheme for GPU multiplexing of the spatially aggregable tensors.

[0008] Optionally, generating the GPU allocation scheme according to the GPU behavior information includes: classifying tensors with a GPU life cycle greater than a second life cycle threshold as non-spatially aggregable tensors, and generating a second GPU allocation sub-scheme for allocating continuous GPU to the non-spatially aggregable tensors according to the GPU size information of the non-spatially aggregable tensors.

[0009] Optionally, classifying tensors with a GPU life cycle less than a first life cycle threshold as spatially aggregable tensors includes: merging multiple specific type tensors with a GPU life cycle less than a first life cycle threshold and belonging to the same third life cycle threshold interval as one virtual tensor, wherein other non-virtual tensors that do not belong to the specific type are non-virtual tensors, and grouping the spatially aggregable tensors according to the GPU size information includes: combining the multiple virtual tensors and the multiple non-virtual tensors to form a tensor set; and grouping the tensors in the tensor set according to the GPU size information of each tensor in the tensor set.

[0010] Optionally, the first memory allocation sub-scheme for memory multiplexing of the spatially aggregable tensors comprises: sorting the spatially aggregated groups in descending order; and sequentially allocating each group in the sorted order, wherein the memory allocation operation for the current group comprises: searching for an unoccupied space region in the allocated memory space that can accommodate the current group; in response to finding an unoccupied space region that meets the requirements, allocating the current group to the unoccupied space region; and in response to not finding an unoccupied space region that meets the requirements, allocating the current group to the unallocated memory space.

[0011] Optionally, the memory allocation based on the memory allocation scheme comprises: performing memory allocation for a current memory request by a static allocator based on the memory allocation scheme; and in response to detecting that the current memory request does not conform to the memory allocation scheme, performing memory allocation for the current memory request by a dynamic allocator.

[0012] Optionally, the memory lifetime information of the tensor is determined based on a memory application node and a memory release node of the tensor, and the memory allocation by the static allocator based on the memory allocation scheme comprises: allocating a complete memory space based on the memory allocation scheme; the static allocator reads the pointers and sizes of the static memory requests in the memory allocation scheme; and in response to receiving a memory request corresponding to the memory application node and the memory size information, returning the corresponding pointer of the memory request in the memory space.

[0013] Optionally, the deep learning model is a Mixture of Experts (MoE) model comprising a dynamic layer, and the memory allocation based on the memory allocation scheme comprises: in response to receiving a memory request for the dynamic layer, performing memory allocation by a static allocator based on the memory allocation scheme; and in response to the static allocator not finding available memory, performing memory allocation for the memory request of the dynamic layer by a dynamic allocator.

[0014] Optionally, the first training phase corresponds to the first N training iteration periods when the model starts training, and N is an integer not less than 3, and the second training phase corresponds to a complete training process when the model restarts.

[0015] Optionally, the GPU allocation scheme includes a first iteration period GPU allocation scheme and a second iteration period GPU allocation scheme, and the GPU allocation based on the GPU allocation scheme in the second training phase of the deep learning model includes: in the first iteration period of the restarted complete training process, performing GPU allocation based on the first iteration period GPU allocation scheme; and in each iteration period after the first iteration period of the restarted complete training process, performing GPU allocation based on the second iteration period GPU allocation scheme.

[0016] According to a second aspect of the present disclosure, a deep learning model training method is provided, including: recording GPU behavior information of first N training iteration periods of the deep learning model, the GPU behavior information including GPU life cycle information and GPU size information of each tensor in the first N training iteration periods, where N is an integer greater than or equal to 3; generating a GPU allocation scheme according to the recorded GPU behavior information; and in M training iteration periods after the first N training iteration periods of the deep learning model, performing GPU allocation based on the GPU allocation scheme, where M is an integer greater than N, and the M training iteration periods after the first N training iteration periods correspond to M training iteration periods after the first N training iteration periods in a complete training process of the deep learning model or M training iteration periods in a restarted complete training process of the deep learning model. Generating the GPU allocation scheme according to the recorded GPU behavior information includes: classifying tensors into parameter tensors, optimizer tensors and activation value tensors according to the GPU life cycle information; generating a resident GPU allocation sub-scheme according to the GPU size information of the parameter tensors and the optimizer tensors; grouping the activation value tensors according to the GPU size information of the activation value tensors, and performing spatial aggregation on multiple activation value tensors in the same group with non-overlapping life cycles to generate a spatial aggregation GPU allocation sub-scheme; and merging the resident GPU allocation sub-scheme and the spatial aggregation GPU allocation sub-scheme to generate the GPU allocation scheme.

[0017] According to a third aspect of the present disclosure, a GPU optimization apparatus for deep learning model training is provided, including a runtime component and an offline analysis component, where the runtime component includes: a GPU allocator configured to record GPU behavior information in a GPU behavior recording mode, the GPU behavior information including GPU life cycle information and GPU size information of each tensor, and the GPU behavior recording mode lasts for at least three iteration periods, and the offline analysis component includes: a GPU behavior analysis module configured to classify tensors according to the GPU life cycle information; and a GPU scheme generation module configured to generate a GPU allocation scheme according to the tensor classification result and the GPU size information, and the GPU allocator is further configured to perform GPU allocation based on the GPU allocation scheme in a GPU scheme allocation mode.

[0018] Optionally, the GPU allocator comprises a dynamic allocator configured to allocate GPU memory in the GPU behavior recording mode, and a static allocator configured to allocate GPU memory based on the GPU allocation scheme in the GPU scheme allocation mode.

[0019] Optionally, the GPU allocator further comprises a semi-dynamic allocator configured to, when the deep learning model is an expert-mixed model comprising a dynamic layer, in response to receiving a GPU memory request for the dynamic layer, cause the static allocator to allocate GPU memory based on the GPU allocation scheme, and in response to the static allocator failing to find available GPU memory, cause the dynamic allocator to allocate GPU memory for the GPU memory request of the dynamic layer.

[0020] Optionally, the runtime component further comprises an error detection and recovery module configured to detect whether a current GPU memory request conforms to the GPU allocation scheme in the GPU scheme allocation mode, wherein the GPU allocator is further configured to, in response to detecting that the current GPU memory request does not conform to the GPU allocation scheme, allocate GPU memory for the current GPU memory request by the dynamic allocator.

[0021] Optionally, the offline analysis component further comprises an abnormal GPU detection module configured to diagnose and debug abnormal conditions in GPU memory usage based on a GPU analysis report containing tensor classification information generated by the GPU behavior analysis module, and the GPU scheme generation module generates a GPU allocation scheme according to the GPU analysis report obtained through abnormal GPU detection.

[0022] Thus, the deep learning model training method and GPU optimization device of the present disclosure improve the generation efficiency of the GPU allocation scheme by grouping very non-resident tensors by size and aggregating space before tensor merging, and by introducing a static and dynamic GPU allocator, the GPU optimization scheme of the present disclosure can achieve efficient GPU allocation on Dense models and MoE models. BRIEF DESCRIPTION OF DRAWINGS

[0023] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings, in which like reference characters designate the same components throughout the figures thereof.

[0024] Figure 1 A schematic flowchart of a deep learning model training method according to one embodiment of the present disclosure is shown.

[0025] Figure 2 A typical Transformer structure and the differences in FFN modules between Dense models and MoE models are shown.

[0026] Figure 3 A constituent schematic diagram of a video memory optimization device according to one embodiment of the present disclosure is shown.

[0027] Figure 4 A specific application flow of a video memory optimization method in a deep learning model training process according to one embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0028] Preferred embodiments of the present disclosure will be described in greater detail below with reference to the accompanying drawings. Although preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure is more thoroughly and completely conveyed to those skilled in the art, and the scope of the present disclosure is fully conveyed to those skilled in the art.

[0029] As used in the specification and the appended claims of the present application, the singular forms "a," "the," "said," and "the" are intended to include the plural forms as well, e.g., "one," "the," "said," and "the" are intended to include "one or more," unless the context clearly indicates otherwise. The term "including" and variations thereof as used herein are meant to be open-ended and to mean "including, but not limited to." The term "or" as used herein is meant to be "and / or," unless the context clearly indicates otherwise. The term "based on" means "based, at least in part, on" unless expressly specified otherwise. The term "one example embodiment" and "an embodiment" means "at least one example embodiment." The term "another embodiment" means "at least one additional embodiment." The terms "first," "second," and the like can refer to different or identical objects. Also, the terms "comprising," "having," containing," and the like can be used interchangeably with "including" and vice versa.

[0030] Since the parallel strategy introduced in large model training changes the scheduling and life cycle of tensors in the original training graph, there is a mismatch with the default video memory management mechanism in mainstream deep learning frameworks (such as the "best fit" strategy of PyTorch). Some systems choose to directly call the native video memory allocation and release interfaces provided by the underlying computing platform to obtain greater flexibility. Such interfaces are usually provided by the device driver layer or runtime and are used to dynamically allocate and release memory blocks in the physical memory of the computing device. For example, on a GPU platform, video memory management can be accomplished by calling native allocation functions (such as cudaMalloc and cudaFree of CUDA). However, native calls frequently produce kernel context switches and lack a reuse mechanism, resulting in large allocation and recovery overheads, which significantly affects training efficiency in the iterative training process.

[0031] To solve the above problems, the prior art solution such as GMLake proposes a bottom-layer virtual memory stitching (VMS) mechanism, which combines multiple non-continuous physical GPU fragments into a logically continuous virtual address space by calling the underlying virtual memory management interface (VMM), so that the program can use continuous GPU logically, and the fragment reuse capability is improved.

[0032] In actual operation, GMLake sets the fraglimit parameter to regulate the minimum fragment unit participating in stitching. When the parameter is set small, the fragment GPU can be used to the maximum extent, but frequent stitching operations will cause system overhead to rise; if it is set larger, smaller fragments cannot be used, which affects the GPU resource utilization. This makes it difficult for GMLake to balance between running speed and GPU efficiency, limiting its applicability in high-intensity training tasks.

[0033] As can be seen from the above, there is a lack of GPU optimization mechanism in the prior art that takes into account efficient operation and GPU utilization, making it difficult to meet the actual needs of large-scale deep learning model training.

[0034] Therefore, the present disclosure proposes a GPU optimization mechanism, and can be specifically implemented as a deep learning model training method and a GPU optimization device for training. Through observation of most deep learning models, the tensor allocation involved in each iteration during training has strong repeatability and predictability, and the GPU lifetime pattern remains relatively stable between iterations. Therefore, the present disclosure records the GPU behavior information in the previous iteration, analyzes and extracts the predictable allocation pattern priori knowledge, and plans the GPU allocation scheme for GPU allocation in each iteration period in subsequent training, thereby effectively reducing GPU fragments, significantly improving resource use efficiency in the training process while ensuring that the training performance is not affected.

[0035] Figure 1 A schematic flowchart of a deep learning model training method according to an embodiment of the present disclosure is shown. The GPU optimization mechanism of the present disclosure is applicable to various deep learning models whose GPU lifetime pattern remains relatively stable between iterations during training, and is particularly suitable for scenarios where the input data size remains consistent in each iteration, including the case where the input size is consistent after padding operation. Based on this, the present solution can fully utilize the predictability of GPU allocation behavior to achieve efficient GPU planning and reuse. In the following description, the training process of LLM will be mainly used as an example to illustrate the training method of the present disclosure. The GPU optimization mechanism of the present disclosure can also be applicable to other deep learning models, such as CNN convolutional neural networks, as long as the input data size remains consistent in each iteration.

[0036] At step S110, in the first training phase of the deep learning model, memory behavior information is obtained, which includes memory lifetime information and memory size information of each tensor in the first training phase. Since the memory lifetime pattern remains relatively stable between iterations, the first training phase needs to include at least three iteration periods. The main reason is that the first iteration process often involves model parameter initialization, optimizer state allocation, cache mechanism activation, and preheating of the memory allocator, etc. which only appear in the first round, and the memory usage pattern of the dense model will tend to be stable from the second iteration period, and the memory behavior of each iteration period remains consistent in theory. Therefore, the first training phase can correspond to the beginning of model training, for example, the first N iterations of model training, N can be an integer not less than 2. In some embodiments, in order to verify the consistency of the memory behavior of each iteration period from the second iteration period, N can be set to 3 or even larger. However, in order to improve the overall efficiency of training as much as possible, N is usually not set too large, for example, N is usually not greater than 5 (such as equal to 4 or 5). That is, in an embodiment, the first training phase corresponds to the first N iterations of model training, where N is at least equal to 3, and in a preferred embodiment, N is 3, 4 or 5.

[0037] In this first training phase, the memory behavior information of each tensor can be recorded. Specifically, the memory application behavior and the memory release behavior of each tensor can be recorded, thereby obtaining the memory lifetime information of the tensor. The memory application behavior of a tensor includes the memory size information and the memory application node when the tensor applies for memory. The memory release behavior of a tensor can correspond to the memory release node of the tensor. If a tensor is not released at the end of the first training phase, it can be considered that the memory release node of the tensor is located after the end of the first training phase (for example, corresponding to the case of parameter tensor and optimizer tensor).

[0038] It should be understood that since the purpose of obtaining memory behavior information is to generate a memory allocation scheme similar to the subsequent iterations of the memory allocation pattern, the memory application node and the memory release node of a tensor can be the model execution sub-phase corresponding to the application and release of the tensor in the first training phase, rather than simply the behavior execution time. Specifically, the memory application node reflects which computing node or module (such as a certain layer of a certain forward or backward operation) first applies for memory in the training process, and the memory release node reflects which computing node or module (such as a certain layer of a certain forward or backward operation) is released in the training process, thereby forming a memory lifetime description closely related to the model structure and training logic.

[0039] At step S120, a GPU memory allocation scheme is generated according to the GPU memory behavior information. The tensor classification can be performed according to the GPU memory lifetime information, and the GPU memory allocation scheme can be generated according to the tensor classification result and the GPU memory size information. Specifically, the tensors can be classified into spatially aggregable tensors and non-spatially aggregable tensors according to the GPU memory lifetime information of each tensor contained in the GPU memory behavior information.

[0040] Here, the spatial aggregation refers to mapping multiple tensors with non-overlapping lifetimes to the same GPU memory region, thereby realizing the reuse of GPU memory space in the time axis to reduce the total GPU memory occupation and reduce fragmentation, that is, to perform GPU memory reuse. The spatially aggregable tensors have short GPU memory lifetime information (corresponding to tensors with GPU memory lifetime less than the first lifetime threshold), and thus an allocation sub-scheme involving GPU memory reuse can be generated for the spatially aggregable tensors. Specifically, the spatially aggregable tensors can be grouped according to the GPU memory size information, and multiple spatially aggregable tensors with non-overlapping GPU memory lifetimes in the same group can be spatially aggregated to generate a first GPU memory allocation sub-scheme for the spatially aggregable tensors.

[0041] The non-spatially aggregable tensors have long GPU memory lifetime information (corresponding to tensors with GPU memory lifetime greater than the second lifetime threshold, where the second lifetime threshold is not less than the first lifetime threshold), and thus a second GPU memory allocation sub-scheme not involving GPU memory reuse can be generated for the non-spatially aggregable tensors. Since the second GPU memory allocation sub-scheme does not involve GPU memory reuse, it can allocate contiguous GPU memory for the non-spatially aggregable tensors. The two GPU memory allocation sub-schemes can be combined to generate the final GPU memory allocation scheme.

[0042] The spatially aggregable tensor and the non-spatially aggregable tensor are the results of classifying the tensors in the training process based on the GPU life cycle from the perspective of whether the GPU memory can be multiplexed. If the tensors are classified according to their use in the model training process, the tensors can be classified into parameter tensors, optimizer tensors, and activation value tensors. The parameter tensors are used to store the learnable parameters of the model, and the GPU memory thereof is allocated at the beginning of the training and continuously resides in the GPU memory until the end of the training, the life cycle covers the entire training process, and belongs to the long life cycle tensor. The optimizer tensors are used to save the state information of the optimizer, such as the momentum term, the gradient accumulation term, etc., and the GPU memory thereof is allocated after the end of the first training iteration, and continuously exists in the subsequent training process until the end of the training. Therefore, the life cycle starts from the end of the first iteration and runs through the entire remaining training phase, and also belongs to the long life cycle tensor. In contrast, the activation value tensors are intermediate calculation results temporarily generated in each forward propagation and backward propagation process, and the GPU memory thereof is dynamically applied and released within each training iteration period, the life cycle only covers a single iteration process, and belongs to the short life cycle tensor. Because each has a clear life cycle characteristic as described above, the tensors in the training process can also be distinguished according to the GPU life cycle information into parameter tensors, optimizer tensors, and activation value tensors. And because the life cycle of the parameter tensors and the optimizer tensors is long, and the life cycle of the activation value tensors is short, in some embodiments, the parameter tensors and the optimizer tensors can be considered to correspond to the non-spatially aggregable tensors, and the activation value tensors correspond to the spatially aggregable tensors. Further, the tensors can also be classified into permanent tensors and non-permanent tensors from the perspective of the GPU residence time of the tensors. According to the life cycle characteristics as described above, in some embodiments, the parameter tensors and the optimizer tensors can be considered to correspond to the permanent tensors, and the activation value tensors correspond to the temporary tensors. Regardless of which of the above three classification methods, the tensor classification can be based on the GPU life cycle information. It should be understood that the first training phase needs to cover the first iteration period of the training, in order to distinguish the parameter tensors and the optimizer tensors through the different GPU application nodes (the GPU application node of the parameter tensors is at the beginning of the first iteration period, and the GPU application node of the optimizer tensors is at the end of the first iteration period).

[0043] After the GPU allocation information is generated, the GPU allocation based on the GPU allocation scheme can be performed in the second training phase of the deep learning model at step S130.

[0044] In this context, the "first training phase" and the "second training phase" are both training phases, i.e. phases in which the model parameters are optimized by forward propagation, loss calculation and back propagation using training data. Since the memory allocation prior knowledge extracted in the "first training phase" needs to be utilized, the "second training phase" is obviously located after the "first training phase". The second training phase can include M complete iteration cycles. In order to make full use of the extracted memory allocation prior knowledge, M is an integer greater than N. In one embodiment, the "first training phase" and the "second training phase" are located in the same complete training process, the first training phase for obtaining memory allocation behavior information corresponds to the first few iteration cycles (e.g. the first 3-5 iteration cycles) after the start of model training, and the second training phase corresponds to all iteration cycles from the end of the first training phase to the completion of model training. In another embodiment, the "first training phase" and the "second training phase" can not be located in the same complete training process. At this time, the first training phase corresponds to the first N training iteration cycles after the start of model training, N is an integer not less than 3, and the second training phase corresponds to at least part of a complete training process restarted, for example, the first M iterations of a restarted complete training process, and preferably corresponds to a restarted complete training process.

[0045] Since there is already a memory allocation scheme generated based on the prior knowledge of the previous iteration cycles, in this second training phase, static allocation can be performed on the memory application. Here, "static" allocation is a concept opposite to "dynamic" allocation. "Static" allocation refers to determining the allocation position and size of all tensors in the memory in advance before model training or inference, and allocating the memory resources required for the entire execution process at one time. "Dynamic" allocation refers to dynamically applying and releasing memory resources according to runtime requirements during model execution. The memory of the tensor is allocated only when it is actually needed, and is released after use.

[0046] In the execution of static memory allocation based on the memory allocation scheme, a complete memory space can be allocated in advance at the beginning of the second training phase of the model, and memory allocation can be performed for each memory request according to the manner specified by the memory allocation scheme during the execution process. Specifically, during the execution process of the second training phase, the corresponding memory request in the memory allocation scheme can be determined based on the memory application behavior (i.e. memory application node and memory size information) of each memory request, and the corresponding memory can be allocated for it.

[0047] Since the memory allocation of the model training is repeatable between the iteration cycles, the memory allocation scheme generated in step S130 needs to cover the memory allocation of each tensor in at least one complete iteration cycle. When the second training phase includes the first iteration cycle and multiple subsequent iteration cycles, since the memory allocation behaviors of the first iteration cycle and the subsequent iteration cycles are different (the first iteration cycle involves the memory allocation of the parameter tensor during the training initialization, the memory allocation of the optimizer tensor at the end of the first iteration cycle, and optionally the memory allocation of a small number of temporary tensors during the training initialization, and the subsequent iteration cycles do not involve the above contents), the memory allocation scheme generated in step S130 needs to be able to guide the allocation of the first iteration cycle and the multiple subsequent iteration cycles of the second training phase.

[0048] Thus, the training method of the present disclosure can significantly reduce the memory fragmentation rate in subsequent training by analyzing the memory behavior in the iteration cycle, extracting the predictable allocation pattern as prior information to plan the memory allocation scheme for subsequent training, improving the memory utilization efficiency and overall resource utilization in the training process without affecting the training performance.

[0049] Since one iteration cycle of LLM training involves a large number of activation value tensors, in order to improve the efficiency of spatial aggregation, the spatially aggregable tensors can be grouped according to the memory size of each spatially aggregable tensor, and then the multiple spatially aggregable tensors in the same group whose memory life cycles do not overlap are spatially aggregated to generate a first memory allocation sub-scheme for memory multiplexing of the spatially aggregable tensors. In one embodiment, spatially aggregable tensors of the same memory size can be grouped into a group. In other embodiments, spatially aggregable tensors with similar memory sizes (e.g., within a predetermined memory size range) can also be grouped into a group. The multiple tensors in the same group whose memory life cycles do not overlap are memory multiplexed. Different algorithms can be used to determine the spatial aggregation manner of the tensors in the same group. In one embodiment, a group can be divided into multiple layers according to the life cycle of the tensors, and the life cycles of all tensors in each layer do not overlap, so they can be mapped to the same memory region. The layering process uses a greedy algorithm, which aims to minimize the number of layers while ensuring that the life cycles of the tensors in the same layer do not overlap, thereby maximizing the spatial multiplexing efficiency of the memory.

[0050] The space-aggregable tensors can be selected and combined to further reduce the calculation difficulty of space aggregation in the group. At this time, the tensors with a memory lifetime less than the first lifetime threshold are classified as space-aggregable tensors, including: combining a plurality of specific type tensors with a memory lifetime less than the first lifetime threshold and belonging to the same third lifetime threshold interval as each other into a virtual tensor, and other space-aggregable tensors that do not belong to the specific type are regarded as non-virtual tensors, and grouping the space-aggregable tensors according to the memory size information includes: combining the plurality of virtual tensors and the plurality of non-virtual tensors to form a tensor set; and grouping the tensors in the tensor set according to the memory size information of each tensor in the tensor set.

[0051] The third lifetime threshold interval can correspond to a micro-batch. In the training process of a deep learning model, a training iteration usually corresponds to a complete parameter update process, that is, the forward propagation, backward propagation and gradient update of a batch of training samples are completed. However, in the training scene of a large model or limited memory, it is often impossible to process all samples in the entire training batch at one time. Therefore, the micro-batch mechanism can be used to further divide the training batch into several smaller sub-batches. Each micro-batch independently performs forward propagation and backward propagation, and accumulates the gradient on multiple micro-batches, and then performs a unified parameter update operation after all micro-batches are processed. That is, a training iteration can be composed of multiple micro-batches, which are processed in turn, and finally collectively complete an effective model weight update, so as to relieve the memory pressure and consider the training stability without changing the global batch size.

[0052] In the same micro-batch, the activation value tensors can be divided into two categories according to the life cycle: (1) the tensors that apply memory in the forward propagation of the micro-batch and release memory in the backward propagation of the micro-batch, i.e., the tensors across the forward and backward propagation; (2) the tensors that apply and release memory within the forward or backward propagation of the micro-batch, i.e., the forward or backward internal tensors. Here, since all the tensors across the forward and backward propagation in the same micro-batch follow the life cycle characteristic of continuous application and continuous release, these tensors can be regarded as a large virtual tensor. In such embodiments, all the tensors across the forward and backward propagation in the same micro-batch correspond to the aforementioned multiple specific type tensors. The memory space required by a virtual tensor is equal to the theoretical peak of all the tensors across the forward and backward propagation in the micro-batch, i.e., equal to the sum of the memory sizes of these tensors. The other space-aggregable tensors that do not belong to the specific type can include the forward or backward internal tensors, and correspond to the aforementioned non-virtual tensors. Thus, by merging the tensors across the forward and backward propagation in the same micro-batch, the scale of the tensors that need to be arranged in the group space aggregation is reduced, and thus the computational difficulty of the group space aggregation is reduced. For example, if an iteration includes four micro-batches, each micro-batch includes 100 tensors across the forward and backward propagation and 500 forward or backward internal tensors. If no merging operation is performed, 2400 (2400 = (500 + 100) x 4) tensors need to be grouped and calculated for group space aggregation. If the merging operation is performed, 100 tensors across the forward and backward propagation in each micro-batch are merged into 1 virtual tensor. At this time, 4 virtual tensors and 2000 (2000 = 500 x 4) non-virtual tensors, i.e., a total of 2004 tensors, can be grouped and calculated for group space aggregation, thereby reducing the computational complexity compared to the case of 2400 tensors. It should be understood that this example is only given to illustrate the effect of the merging calculation, and in actual operation, much more tensors across the forward and backward propagation and forward or backward internal tensors are usually involved.

[0053] In one embodiment, the memory regions corresponding to each group of hierarchical results can be arranged sequentially or randomly. In another embodiment, the memory allocation requirement can be further reduced by gap filling in the allocated memory space. In this case, generating the first memory allocation sub-scheme further includes: sorting the plurality of groups aggregated by space in descending order; and sequentially arranging each group according to the sorted order, wherein the memory arrangement operation for the current group includes: searching for an unoccupied space region capable of accommodating the current group in the allocated memory space; arranging the current group in the unoccupied space region in response to finding an unoccupied space region meeting the requirement; and arranging the current group in the unallocated memory space in response to not finding an unoccupied space region meeting the requirement.

[0054] The memory optimization mechanism of the present disclosure can be implemented with the participation of a dynamic allocator and a static allocator. The dynamic allocator can be used for dynamic memory allocation in the first training stage. In this case, the dynamic allocator can have consistent or inconsistent implementation with the default dynamic allocator of the deep learning framework (for example, the dynamic allocator using the "best fit" strategy). If OOM (Out of Memory) occurs when using the dynamic allocator in the first training stage, the native memory allocation and release interface provided by the underlying computing platform (for example, cudaMalloc and cudaFree of CUDA) can be used for dynamic memory allocation. The static allocator can be used for memory allocation based on the memory allocation scheme in the second training stage. In this case, the memory allocation by the static allocator based on the memory allocation scheme can include: allocating a complete memory space based on the memory allocation scheme; the static allocator reading the pointers and sizes of the static memory requests in the memory allocation scheme; and in response to receiving a memory request corresponding to the memory application node and the memory size information, returning the memory pointer corresponding to the memory request in the memory space. In some embodiments, the static allocator can allocate a complete memory space based on the memory allocation scheme. In other embodiments, the static allocator can allocate a complete memory space based on the memory allocation scheme.

[0055] As mentioned earlier, due to the high repeatability of memory allocation between iterations, in the second training stage, when an actual memory request is received, the previous memory request corresponding to the actual memory request in the memory allocation scheme is searched for, and the memory pointer allocated according to the previous memory request is provided to the actual memory request. The memory application node and the memory size information of the actual memory request can be used to search for the corresponding previous memory request in the memory allocation scheme.

[0056] In one embodiment, an error detection and recovery mechanism can be introduced for the second training phase. That is, when a GPU allocation error is detected, dynamic GPU allocation is performed by the dynamic allocator. At this time, the GPU allocation based on the GPU allocation scheme includes: performing GPU allocation by the static allocator based on the GPU allocation scheme; and in response to detecting that the current GPU request does not conform to the GPU allocation scheme, performing GPU allocation for the current GPU request by the dynamic allocator. After the current GPU request is processed, GPU allocation based on the GPU allocation scheme can still be performed by the static allocator.

[0057] In the design of deep learning models, common structures include two types of dense models and MoE models. In dense models, all network modules are activated and participate in computation in each forward propagation, and the calculation path is fixed. MoE models use a sparse activation mechanism, dynamically selecting a portion of expert subnetworks for activation through a gating network, so that only a few experts are calculated in each forward propagation, significantly reducing the consumption of computing resources while maintaining large-scale parameter capacity. Expert subnetworks are usually composed of different feedforward neural network (FFN) modules, with similar structure but independent parameters. This architecture is particularly suitable for LLMs containing Transformer structures. For ease of understanding, Figure 2 The typical Transformer structure and the differences between FFN modules in Dense and MoE models are shown.

[0058] As shown in the figure, a Transformer structure is composed of self-attention, layer normalization, feedforward neural network (FFN), and layer normalization modules. For dense models, the FFN structure is always activated in each forward propagation, and all inputs are processed by the same FFN module. In MoE models, the FFN part is replaced by multiple parallel expert subnetworks, i.e., multiple feedforward sublayers (e.g., FFN1 and FFN2 in the figure), and a gating network (router) dynamically selects a portion of experts for activation according to the input content. For example, in the example of Figure 2 FFN1 is activated by the router. The layers in the MoE model that dynamically select the execution path through the routing mechanism can be referred to as dynamic layers, such as Figure 2 the feedforward selection structure on the right.

[0059] The memory behavior of the MoE model has non-determinism in the training process due to the introduction of the "expert selection" mechanism based on input dynamic activation. The training method of the present disclosure can be adapted to the training process of the MoE model by introducing a semi-dynamic mechanism. At this time, the memory allocation based on the memory allocation scheme can include: in response to receiving a memory request of the dynamic layer, allocating memory by the static allocator based on the memory allocation scheme; and in response to the static allocator not finding available memory, allocating memory for the memory request of the dynamic layer by the dynamic allocator. In this way, the planned memory layout can be preferentially utilized while flexibly responding to the uncertain memory demand of the dynamic layer caused by sparse activation during operation, thereby ensuring resource utilization while taking into account the flexible response and efficient scheduling of the dynamic layer memory demand in the MoE model.

[0060] In one embodiment, the present disclosure can also be implemented as a deep learning model training method, comprising: recording memory behavior information of the first N training iteration periods of the deep learning model, the memory behavior information comprising tensor memory life cycle information and memory size information in the first N training iteration periods, wherein N is an integer not less than 2, and preferably an integer not less than 3; generating a memory allocation scheme according to the recorded memory behavior information; and in the next M training iteration periods of the deep learning model, allocating memory for tensors based on the memory allocation scheme, wherein M is an integer greater than N. The next M training iteration periods correspond to: M training iteration periods located after the first N training iteration periods in a complete training process of the deep learning model, or M training iteration periods in a restarted complete training process, and the next M training iteration periods preferably continue until the end of training. Wherein generating a memory allocation scheme according to the recorded memory behavior information comprises: classifying tensors into parameter tensors, optimizer tensors and activation value tensors according to the memory life cycle information; generating a resident memory allocation sub-scheme according to the memory size information of the parameter tensors and the optimizer tensors; grouping the activation value tensors according to the memory size information of the activation value tensors, and spatially aggregating multiple activation value tensors with non-overlapping life cycles in the same group to generate a spatial aggregation memory allocation sub-scheme; and merging the resident memory allocation sub-scheme and the spatial aggregation memory allocation sub-scheme to obtain the memory allocation scheme.

[0061] As mentioned above, the first training phase corresponds to the first N training iteration cycles of the model, where N is an integer no less than 3, and the second training phase corresponds to a complete training process of the model restarted. At this time, the memory allocation scheme includes a first iteration cycle memory allocation scheme and a second iteration cycle memory allocation scheme. The first iteration cycle memory allocation scheme can be a memory allocation scheme generated according to the memory allocation behavior of the first iteration cycle of the model training included in the first training phase. The second iteration cycle memory allocation scheme can be a memory allocation scheme generated according to the memory allocation behavior of the second iteration cycle (or the third, fourth, or fifth iteration cycle, because the memory allocation behavior of each iteration cycle after the first iteration cycle is consistent, or at least largely consistent) of the model training included in the first training phase. At this time, in the second training phase of the deep learning model, the memory allocation based on the memory allocation scheme includes: in the first iteration cycle of the complete training process restarted, the memory allocation based on the first iteration cycle memory allocation scheme; and in each iteration cycle after the first iteration cycle of the complete training process restarted, the memory allocation based on the second iteration cycle memory allocation scheme. In addition, in the implementation of generating the first memory allocation sub-scheme (equivalent to the spatial aggregation memory allocation sub-scheme in some embodiments) for the space-aggregable tensor and the second memory allocation sub-scheme (equivalent to the resident memory allocation sub-scheme in some embodiments) for the non-space-aggregable tensor as above, the first iteration cycle memory allocation scheme needs to include the second memory allocation sub-scheme and the first memory allocation sub-scheme, and the second iteration cycle memory allocation scheme does not include the second memory allocation sub-scheme but needs to include the first memory allocation sub-scheme. When the allocation of the activation value tensor in the first iteration cycle and the subsequent iteration cycles is the same or sufficiently similar, the first and second iteration cycle memory allocation schemes can reuse the same first memory allocation sub-scheme. When the allocation of the activation value tensor in the first iteration cycle and the subsequent iteration cycles is sufficiently different, the first and second iteration cycle memory allocation schemes can use different first memory allocation sub-schemes.

[0062] The present disclosure can implement the deep learning model training method as above by using a memory optimization device. Figure 3 The composition diagram of the memory optimization device according to one embodiment of the present disclosure is shown. As shown in the figure, the memory optimization device includes a runtime component 100 and an offline analysis component 200.

[0063] The runtime component 100 is responsible for memory allocation and management during the training of a deep learning model, and mainly includes a memory allocator 110. The memory allocator 110 is responsible for memory allocation during model training, and supports two working modes: a memory behavior recording mode and a memory scheme allocation mode. The memory allocator 110 records memory behavior information in the memory behavior recording mode. As described above, the memory behavior information includes memory lifetime information and memory size information of each tensor, and the memory behavior recording mode lasts at least three iteration periods. The duration of the memory behavior recording mode corresponds to the first training phase described above. The memory allocator 110 allocates memory based on the memory allocation scheme generated by the offline analysis component 200 in the memory scheme allocation mode. The duration of the memory scheme allocation mode corresponds to the second training phase described above, for example, a complete training process that is restarted.

[0064] In order to record memory behavior and allocate memory according to a memory scheme, the memory allocator 110 can include a dynamic allocator 111 capable of allocating memory in the memory behavior recording mode, and a static allocator 112 capable of allocating memory based on a memory allocation scheme in the memory scheme allocation mode.

[0065] In the memory behavior recording state, the memory allocator 110 defaults to the dynamic allocator 111 for memory allocation and recording of memory behavior information, and when the model memory usage approaches the GPU limit, the memory allocator 110 can use the native memory application and release interface provided by the underlying computing platform (e.g., GPU) to allocate memory and record memory behavior information. In one embodiment, the dynamic allocator 111 can use the same implementation as the memory allocator of the deep learning framework used for model training, and dynamically apply memory blocks at runtime according to memory requests.

[0066] In the memory scheme allocation mode, the memory allocator 110 reads the memory allocation scheme and initializes the memory space of the static allocator 112, and issues memory requests. The memory allocator 110 can allocate a complete memory space to the static allocator 112 according to the memory allocation scheme, and the static allocator 112 reads the pointer and size of each static memory request in the memory allocation scheme, and returns the memory pointer in its memory space when it receives a memory request. In one implementation, the static allocator 112 performs cudaMalloc() once at initialization to apply a complete memory space for accommodating all memory required by parameter tensors, optimizer tensors, and activation value tensors. The parameter tensors and optimizer tensors are allocated to consecutive positions in this memory space according to the memory allocation scheme (e.g., the second memory allocation sub-scheme described above).

[0067] To optimize the training of the MoE model, the memory allocator 110 can further include a semi-dynamic allocator 113 to be responsible for the application of dynamic memory in the training of the MoE model. The memory allocator 110 can dispatch the memory request of the dynamic layer in the MoE model to the semi-dynamic allocator 113, and the semi-dynamic allocator 113 can preferentially find available free memory blocks from the static allocator 112 upon receiving the memory request, and if no available free memory block is found, the memory request is forwarded to the dynamic allocator 111. It should be understood that the proportion of the dynamic layer in the MoE model is small, and therefore in the training process of the MoE model, the memory allocation is still mainly performed by the static allocator 112 according to the memory allocation scheme.

[0068] In the illustrated implementation, the runtime component 100 further includes an error detection and recovery module 120. The error detection and recovery module 120 is configured to detect whether the current memory request conforms to the memory allocation scheme in the memory allocation mode according to the memory scheme. The memory allocator 110 can perform memory allocation for the current memory request by the dynamic allocator in response to detecting that the current memory request does not conform to the memory allocation scheme.

[0069] The error detection and recovery module 120 can be responsible for checking the correctness of the static allocator 112 in the allocation process to avoid the occurrence of memory stepping behavior, and can use the dynamic allocator 111 to perform memory allocation behavior that conforms to the user's expectation when an error is detected. For example, in the training process of a dense model, theoretically, the memory behavior of all subsequent iteration periods after the second iteration period is the same as that of the second iteration period. At this time, if the error detection and recovery module 120 detects that the current memory request does not conform to the memory allocation scheme, it can be considered that an error is detected, and the memory request is allocated to the dynamic allocator 111 for processing, but the static allocator 112 is still used to process subsequent memory requests.

[0070] The offline analysis component 200 is responsible for analyzing the memory behavior of the model training and generating an efficient memory allocation scheme. The offline analysis component 200 includes a memory behavior analysis module 210 configured to classify tensors according to the memory behavior information, and a memory scheme generation module 220 configured to generate a memory allocation scheme based on the classification result.

[0071] The memory behavior analysis module 210 can analyze the life cycle characteristics of tensors, such as the memory life cycle information of tensors, according to the model training memory behavior recorded in the memory behavior recording mode of the memory allocator 110. According to the life cycle characteristics, the parameter tensors, optimizer tensors and activation value tensors in model training can be accurately identified. The parameter tensors and optimizer tensors are resident tensors in the memory, and can correspond to the aforementioned non-spatially aggregable tensors. The activation value tensors can be used for memory reuse due to their short life cycle, and thus correspond to the spatially aggregable tensors. When virtual vector merging of activation value tensors across forward and backward propagation needs to be performed to reduce the subsequent intra-group spatial aggregation calculation complexity, the activation value tensors can also be further classified into mergable tensors (for example, the same micro-batch is merged into a virtual tensor) and non-mergable tensors (corresponding to the aforementioned non-virtual tensors) according to the life cycle characteristics. The memory behavior analysis module 210 can output a memory analysis report according to the results of the tensor classification.

[0072] The memory scheme generation module 220 generates an efficient memory allocation scheme based on the memory analysis report. This module can first arrange the requests of parameter tensors, optimizer tensors and other resident memory closely (corresponding to the generation of the aforementioned resident memory allocation sub-scheme), and then obtain the memory arrangement of activation value tensors through the spatial aggregation algorithm described in the present disclosure (corresponding to the generation of the aforementioned spatial aggregation memory allocation sub-scheme), and finally generate a complete memory allocation scheme for subsequent model training.

[0073] In the example shown, the offline analysis component 200 also includes an abnormal memory detection module 230 for diagnosing and debugging abnormal situations in the memory usage process based on the memory analysis report generated by the memory behavior analysis module 210, which contains tensor classification information. The abnormal memory detection module 230 can perform error detection on the memory analysis report to determine whether the current model training is abnormal, thereby identifying potential problems in the training framework, such as memory leakage, unexpected re-computation configuration, etc. Subsequently, memory debugging can be performed to eliminate the detected errors.

[0074] The following will describe one specific implementation example of the memory optimization device of the present disclosure in executing the training method. Figure 4 , in the execution of the training method. Figure 4 The specific application flow of the memory optimization method according to one embodiment of the present disclosure in the deep learning model training process is shown.

[0075] As shown in the figure, when starting the model training task, the memory tool is loaded. At this time, the memory tool that needs to be loaded corresponds to the runtime component of the present disclosure, for example, as described above in connection with Figure 3The runtime component 100 in the memory optimization apparatus. The loading of the runtime component of the present disclosure can be achieved by utilizing the memory management interface of the deep learning framework. For example, when the underlying running environment is based on the CUDA architecture and the training task is built based on the PyTorch framework, the memory allocator of the present disclosure (e.g., the memory allocator 110 in the runtime component 100 shown) can be loaded to the system by utilizing the torch.cuda.memory.CUDAPluggableAllocator interface, and the native PyTorch memory allocator can be replaced by the memory allocator 110 through the torch.cuda.memory.change_current_allocator interface, so as to enable the memory tool to be seamlessly embedded in the existing PyTorch training process and fully exert its performance optimization function. Figure 3 The memory allocator 110 in the runtime component 100 shown) is loaded to the system, and the native PyTorch memory allocator is replaced by the memory allocator 110 through the torch.cuda.memory.change_current_allocator interface, so as to enable the memory tool to be seamlessly embedded in the existing PyTorch training process and fully exert its performance optimization function.

[0076] When training the model for the first time, the memory behavior recording mode of the memory allocator 110 is entered first, and a short iteration cycle of training task is performed. The short iteration cycle can be 3 to 5 training iterations, i.e., the first training stage corresponds to the first N iterations after the start of model training, where N is 3, 4 or 5. After the end of the model training of the short iteration cycle, the memory allocator 110 will save the memory request records in the training process. The memory request records can include records of memory application behaviors and memory release behaviors. The memory application behavior corresponds to the memory size information and the memory application node, and the memory release behavior corresponds to the memory release node. If a tensor is not released at the end of the memory behavior recording mode, it can be considered that the memory release node of the tensor is located after the end of the memory behavior recording mode.

[0077] The memory request records are then output to the offline analysis component of the present disclosure, for example, the offline analysis component 200 in the memory optimization apparatus described above in combination with Figure 3 The memory behavior analysis module can analyze the memory request records to output a memory analysis report. The abnormal memory detection module can detect the memory analysis report and generate an abnormal detection report. Based on the memory analysis report and the abnormal detection report, the memory scheme generation module is called again, and the memory allocation scheme for the subsequent training of the model can be generated.

[0078] Subsequently, the memory allocator of the present disclosure can be set to the memory scheme allocation mode, and the model training is performed in combination with the memory allocation scheme obtained in the previous step. At this time, the model training should be in the second training stage described above, and preferably corresponds to a complete training process that is restarted. In this training stage, the error detection and recovery module (e.g., the error detection and recovery module 120 in the runtime component 100 shown) included in the runtime component can be utilized to detect and recover errors in the training process. Figure 3The error detection and recovery module 120 in the memory management module 110 detects the current memory request does not conform to the memory allocation scheme, and allocates the memory request to the dynamic allocator for processing to avoid the occurrence of memory stepping behavior.

[0079] The deep learning model training method and the memory optimization device for training according to the present disclosure have been described in detail above with reference to the accompanying drawings. The present disclosure effectively reduces memory fragmentation and improves resource use efficiency in the training process by using prior knowledge of predictable allocation patterns in model training to plan memory allocation in advance without affecting training performance. The generation efficiency of the memory allocation scheme is improved by grouping non-resident tensors by size and merging tensors before aggregation, and the memory optimization scheme of the present disclosure can achieve efficient memory allocation on both dense models and MoE models by introducing a static and dynamic memory allocator.

[0080] In addition, the method according to the present disclosure can also be implemented as a computer program or computer program product, which includes computer program code instructions for executing the above steps defined in the above method of the present disclosure.

[0081] Alternatively, the present disclosure can also be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) having stored executable code (or computer program, or computer instruction code) which, when executed by a processor of an electronic device (or computing device, server, etc.), causes the processor to perform the steps of the above method according to the present disclosure.

[0082] Those skilled in the art will also appreciate that the various example logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of the two.

[0083] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0084] Embodiments of the application have been described above with the understanding that such embodiments are exemplary, but are not exhaustive, and are not limited to the embodiments disclosed. Many modifications and variations of the described embodiments are possible in light of the above teachings. The choice of terms to be used in the description is intended to best describe the principles of the embodiments, practical application, or improvement over the technology in the art, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for training a deep learning model, comprising: obtaining memory behavior information in a first training phase of the deep learning model, the memory behavior information comprising memory lifetime information and memory size information of tensors in the first training phase, wherein the first training phase comprises at least three iteration cycles; generating a memory allocation scheme according to the memory behavior information, wherein tensor classification is performed according to the memory lifetime information, and the memory allocation scheme is generated according to the tensor classification result and the memory size information; and performing memory allocation based on the memory allocation scheme in a second training phase of the deep learning model.

2. The method of claim 1, wherein, The generating of the memory allocation scheme according to the memory behavior information comprises: classifying tensors with a memory lifetime less than a first lifetime threshold as spatially aggregable tensors; grouping the spatially aggregable tensors according to the memory size information; and performing spatial aggregation on a plurality of spatially aggregable tensors with non-overlapping memory lifetimes in a same group to generate a first memory allocation sub-scheme for memory multiplexing of the spatially aggregable tensors.

3. The method of claim 2, wherein, The generating of the memory allocation scheme according to the memory behavior information comprises: classifying tensors with a memory lifetime greater than a second lifetime threshold as non-spatially aggregable tensors; and generating a second memory allocation sub-scheme for allocating continuous memory to the non-spatially aggregable tensors according to the memory size information of the non-spatially aggregable tensors.

4. The method of claim 2, wherein, The classifying of the tensors with a memory lifetime less than a first lifetime threshold as spatially aggregable tensors comprises: merging a plurality of specific type tensors with a memory lifetime less than the first lifetime threshold and belonging to a same third lifetime threshold interval as a virtual tensor, wherein other spatially aggregable tensors not belonging to the specific type are non-virtual tensors, and The grouping of the spatially aggregable tensors according to the memory size information comprises: combining the plurality of virtual tensors and the plurality of non-virtual tensors to form a tensor set; and grouping the tensors in the tensor set according to the memory size information of each tensor in the tensor set.

5. The method of claim 2, wherein, The performing of spatial aggregation on a plurality of spatially aggregable tensors with non-overlapping memory lifetimes in a same group to generate a first memory allocation sub-scheme for memory multiplexing of the spatially aggregable tensors comprises: sorting the plurality of groups after spatial aggregation in descending order; sequentially performing memory arrangement on each group in the sorted order, wherein the memory arrangement operation for a current group comprises: searching for an unoccupied space region capable of accommodating the current group in the allocated memory space; in response to finding an unoccupied space region meeting the requirement, arranging the current group in the unoccupied space region; in response to not finding an unoccupied space region meeting the requirement, arranging the current group in the unallocated memory space.

6. The method of claim 1, wherein, The performing of memory allocation based on the memory allocation scheme comprises: performing memory allocation for a current memory request by a static allocator based on the memory allocation scheme; and in response to detecting that the current memory request does not conform to the memory allocation scheme, performing memory allocation for the current memory request by a dynamic allocator.

7. The method of claim 1, wherein, The memory lifetime information of the tensor is determined based on a memory application node and a memory release node of the tensor, and the memory allocation by the static allocator based on the memory allocation scheme comprises: allocating a complete memory space based on the memory allocation scheme; the static allocator reads in the pointers and sizes of the static memory requests in the memory allocation scheme; and in response to receiving a memory request corresponding to the memory application node and the memory size information, returning the corresponding pointer of the memory request in the memory space.

8. The method of claim 1, wherein, The deep learning model is an expert hybrid MoE model including a dynamic layer, and the memory allocation based on the memory allocation scheme comprises: in response to receiving a memory request of the dynamic layer, performing memory allocation by the static allocator based on the memory allocation scheme; in response to the static allocator not finding available memory, performing memory allocation for the memory request of the dynamic layer by the dynamic allocator.

9. The method of claim 1, wherein, The first training phase corresponds to the first N training iteration periods when the model starts training, N being an integer not less than 3, and the second training phase corresponds to a complete training process restarted by the model.

10. The method of claim 9, wherein the memory allocation scheme comprises a first iteration period memory allocation scheme and a second iteration period memory allocation scheme, and the memory allocation based on the memory allocation scheme in the second training phase of the deep learning model comprises: in the first iteration period of the complete training process restarted, performing memory allocation based on the first iteration period memory allocation scheme; and in each iteration period after the first iteration period of the complete training process restarted, performing memory allocation based on the second iteration period memory allocation scheme.

11. A deep learning model training method, comprising: recording memory behavior information of the first N training iteration periods of the deep learning model, the memory behavior information comprising memory lifetime information and memory size information of each tensor in the first N training iteration periods, wherein N is an integer not less than 3; generating a memory allocation scheme according to the recorded memory behavior information; and in the last M training iteration periods of the deep learning model, performing memory allocation based on the memory allocation scheme, wherein M is an integer greater than N, and the last M training iteration periods correspond to: M training iteration periods after the first N training iteration periods in a complete training process of the deep learning model, or M training iteration periods in a complete training process restarted, wherein generating a memory allocation scheme according to the recorded memory behavior information comprises: classifying tensors into parameter tensors, optimizer tensors and activation value tensors according to the memory lifetime information; generating a resident memory allocation sub-scheme according to the memory size information of the parameter tensors and the optimizer tensors; grouping the activation value tensors according to the memory size information of the activation value tensors, and spatially aggregating multiple activation value tensors in the same group with non-overlapping lifetimes to generate a spatial aggregation memory allocation sub-scheme; and The resident memory allocation sub-scheme and the spatial aggregation memory allocation sub-scheme are combined to generate the memory allocation scheme.

12. A memory optimization apparatus for deep learning model training, comprising a runtime component and an offline analysis component, wherein the runtime component comprises: a memory allocator configured to record memory behavior information in a memory behavior recording mode, the memory behavior information comprising memory lifetime information and memory size information of each tensor, wherein the memory behavior recording mode lasts for at least three iteration periods, the offline analysis component comprises: a memory behavior analysis module configured to classify tensors according to the memory lifetime information; a memory scheme generation module configured to generate a memory allocation scheme according to tensor classification results and the memory size information, and the memory allocator is further configured to perform memory allocation based on the memory allocation scheme in a memory scheme allocation mode.

13. The video memory optimization device of claim 12, wherein, the memory allocator comprises: a dynamic allocator configured to perform memory allocation in the memory behavior recording mode; and a static allocator configured to perform memory allocation based on the memory allocation scheme in the memory scheme allocation mode.

14. The video memory optimization device of claim 13, wherein, the memory allocator further comprises: a semi-dynamic allocator configured to, when the deep learning model is an expert hybrid model comprising a dynamic layer, cause the static allocator to perform memory allocation based on the memory allocation scheme in response to receiving a memory request of the dynamic layer, and cause the dynamic allocator to perform memory allocation for the memory request of the dynamic layer in response to the static allocator failing to find available memory.

15. The video memory optimization device of claim 13, wherein, the runtime component further comprises: an error detection and recovery module configured to detect whether a current memory request conforms to the memory allocation scheme in the memory scheme allocation mode, wherein the memory allocator is further configured to cause the dynamic allocator to perform memory allocation for the current memory request in response to detecting that the current memory request does not conform to the memory allocation scheme.

16. The video memory optimization device of claim 12, wherein, the offline analysis component further comprises: an abnormal memory detection module configured to diagnose and debug abnormal situations in memory usage based on a memory analysis report containing tensor classification information generated by the memory behavior analysis module, and the memory scheme generation module generates a memory allocation scheme according to the memory analysis report obtained through abnormal memory detection.

Images