Method, apparatus, electronic device and product for determining memory

Abstract

Embodiments of this disclosure relate to methods, apparatus, electronic devices, and products for determining memory usage. The method includes acquiring a set of memory usage information by initiating a set of parallel training sessions related to a target model, wherein each parallel training session in the set of parallel training sessions is configured using predetermined parallel parameters. The method also includes determining memory modeling parameters for the target model based on the set of memory usage information. Furthermore, the method includes determining multiple memory usage peaks of the target model in different parallel training sessions corresponding to different parallel parameters based on the memory modeling parameters. Thus, this method can model the memory usage of a target model using memory usage information from a set of parallel training sessions without knowing the specific network structure, thereby determining the maximum memory usage of the target model under different parallel configurations. This provides an important basis for automatic tuning of parallel training and avoids training interruptions or failures due to insufficient memory.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more particularly to methods, apparatus, electronic devices, and computer program products for determining memory. Background Technology

[0002] Large models have played an increasingly important role in the field of artificial intelligence in recent years. With the improvement of computing power and the accumulation of massive amounts of data, the scale of deep learning models continues to expand, enabling them to handle more complex tasks. Through training on massive amounts of data, large models have achieved significant progress in many fields such as natural language processing, image recognition, and machine translation.

[0003] Parallel training, as a key technology for improving the training efficiency of large models, is becoming increasingly important. By distributing model training tasks across multiple computing nodes and utilizing distributed computing resources, training efficiency can be improved. Parallel training can not only handle larger datasets and more complex network structures, but also effectively address storage and computing power bottlenecks. With the development of hardware technology and the maturity of distributed computing frameworks, future large model training will increasingly rely on parallelization strategies. Summary of the Invention

[0004] Embodiments of this disclosure provide a method, apparatus, electronic device, and computer program product for determining memory.

[0005] According to a first aspect of this disclosure, a method for determining memory usage is provided. The method includes acquiring a set of memory usage information by initiating a set of parallel training sessions related to a target model, wherein each parallel training session in the set is configured using predetermined parallel parameters. The method also includes determining memory modeling parameters for the target model based on the set of memory usage information. Furthermore, the method includes determining multiple memory usage peaks of the target model in different parallel training sessions corresponding to different parallel parameters, based on the memory modeling parameters. This method can model the memory usage of a target model using memory usage information from a set of parallel training sessions without knowing the specific network structure, and remains robust and accurate when the network structure changes. Thus, this method can provide memory prediction for memory usage issues under different parallel configurations, providing an important basis for automatic tuning of parallel training and avoiding training interruptions or failures due to insufficient memory.

[0006] In some embodiments of this disclosure, the target model includes an input portion, an intermediate portion, and an output portion. The input portion relates to an embedding operation performed at the beginning of the target model, the intermediate portion includes one or more hidden layers, and the output portion is the part of the target model following the last hidden layer in the intermediate portion. This structural partitioning of the target model facilitates the determination of the memory footprint of each part, contributing to improved accuracy in memory modeling.

[0007] In some embodiments of this disclosure, a set of memory usage information is obtained by initiating a set of parallel training related to the target model, including: generating sub-models of the target model by pruning the target model; and obtaining a set of memory usage information using performance analysis tools by initiating a set of parallel training for the sub-models. Therefore, by pruning the target model to generate sub-models and obtaining the memory usage information of the sub-models, computational overhead can be reduced, and the efficiency and accuracy of memory data acquisition can be improved.

[0008] In some embodiments of this disclosure, each piece of memory usage information in a set of memory usage information includes static memory usage information and dynamic memory usage information. The static memory usage information is related to the weights, gradients, and optimizer of the sub-model, while the dynamic memory usage information is related to the activation values ​​of the sub-model. In this way, by distinguishing between static and dynamic memory usage information, the different sources of memory usage can be clearly identified, which helps to perform more refined memory modeling.

[0009] In some embodiments of this disclosure, memory modeling parameters include static memory parameters and dynamic memory parameters. Determining the memory modeling parameters for a target model includes: determining static memory parameters based on a set of static memory usage information from a set of memory usage information; and determining dynamic memory parameters based on a set of dynamic memory usage information from a set of memory usage information. Therefore, by determining the static and dynamic memory parameters separately, memory usage from different sources can be accurately reflected, achieving more refined memory modeling.

[0010] In some embodiments of this disclosure, each of the predetermined parallel parameters includes at least one of tensor parallel parameters, pipeline parallel parameters, data parallel parameters, sequence parallel parameters, and expert parallel parameters. This allows for the support of multiple parallel parameters and adaptation to different parallel training configurations.

[0011] In some embodiments of this disclosure, determining static memory parameters includes: determining the memory footprint of the input and output portions of the sub-model based on the static memory differences of the sub-model across different computational acceleration units, wherein the static memory differences are determined based on a set of static memory footprint information; determining the pipeline memory footprint of the sub-model affected by pipeline parameters based on the static memory differences of the sub-model across different pipeline parallelism parameters; and determining the non-segmented tensors of the sub-model unaffected by tensor parallelism parameters based on the static memory differences of the sub-model across different tensor parallelism parameters. Therefore, by modeling static memory through the static memory differences of multiple parts, static memory can be modeled more accurately.

[0012] In some embodiments of this disclosure, in response to the intermediate portion of the target model including a hybrid expert layer, determining the static memory parameters further includes: based on the static memory differences of the submodel under parallel parameters with different numbers of experts, determining the expert tensors belonging to the expert portion and the non-expert tensors not belonging to the expert portion in each hidden layer of the intermediate portion of the submodel. Thus, static memory modeling for expert models can be achieved.

[0013] In some embodiments of this disclosure, determining dynamic memory parameters includes: constructing a nonlinear regression model of the dynamic memory of a sub-model during parallel training; the nonlinear regression modeling includes multiple sub-regression models related to the forward peak memory, loss-enabling memory, loss-peak memory, backward-enabling memory, backward-peak memory, and optimizer peak memory of the sub-model; and determining the model parameters of the multiple sub-regression models as dynamic memory parameters based on a set of dynamic memory occupancy information. Thus, dynamic memory is accurately modeled using multiple nonlinear regression models.

[0014] In some embodiments of this disclosure, determining multiple memory usage peaks of the target model in different parallel training sessions corresponding to different parallel parameters includes: determining the static memory usage and dynamic memory usage peaks of the target model in parallel training sessions corresponding to given parallel parameters; and determining the corresponding memory usage peaks of the target model in the corresponding parallel training sessions based on the static memory usage and dynamic memory usage peaks. Therefore, by combining the static memory usage and dynamic memory usage peaks, the total memory requirements of the target model under different parallel configurations can be determined.

[0015] In some embodiments of this disclosure, determining the static memory footprint of the target model during corresponding parallel training includes: determining the memory footprint of each tensor in the hidden layer of the target model based on the tensor parallel parameters in the given parallel parameters and the non-splitting tensor in the static memory parameters; determining the memory footprint of the hidden layer of the target model based on the memory footprint of each tensor in the hidden layer, as well as the expert tensor and the non-expert tensor; and determining the static memory footprint based on the memory footprint of the hidden layer of the target model, the pipeline parameters in the given parallel parameters, the memory footprint of the input part, and the memory footprint of the output part. Therefore, by combining the static memory footprint of different parts of the model, the static memory footprint of the target model can be accurately predicted.

[0016] In some embodiments of this disclosure, determining the peak dynamic memory usage of the target model during corresponding parallel training includes: determining the peak forward memory, peak loss memory, peak backward memory, and peak optimizer memory of the target model during corresponding parallel training based on dynamic memory parameters; and determining the maximum value among the peak forward memory, peak loss memory, peak backward memory, and peak optimizer memory as the peak dynamic memory usage. Thus, based on the dynamic memory parameters, multiple peak dynamic memory usage values ​​are determined, thereby determining the peak dynamic memory usage of the target model during parallel training.

[0017] In some embodiments of this disclosure, automatic parallelization of the target model is performed based on multiple memory usage peaks of the target model during different parallel training sessions corresponding to different parallel parameters. This allows for dynamic adjustment of the parallel configuration according to memory resource requirements, avoiding training interruptions due to insufficient memory and improving the efficiency and stability of parallel training.

[0018] According to a second aspect of this disclosure, an apparatus for determining memory usage is provided. The apparatus includes a memory usage acquisition module configured to acquire a set of memory usage information by initiating a set of parallel training sessions related to a target model, wherein each parallel training session in the set of parallel training sessions is configured using predetermined parallel parameters. The apparatus also includes a modeling parameter determination module configured to determine memory modeling parameters for the target model based on the set of memory usage information. Furthermore, the apparatus includes a peak usage determination module configured to determine multiple memory usage peaks of the target model in different parallel training sessions corresponding to different parallel parameters, based on the memory modeling parameters.

[0019] In a third aspect of this disclosure, an electronic device is provided. The electronic device includes at least one processor and a memory. The memory is coupled to the at least one processor and has instructions stored thereon. When executed by the at least one processor, the instructions cause the electronic device to perform the method according to a first aspect of this disclosure.

[0020] In a fourth aspect of this disclosure, a computer-readable storage medium is provided on which a computer program is stored. The computer program is executed by a processor to implement the method described according to a first aspect of this disclosure.

[0021] In a fifth aspect of this disclosure, a computer program product is provided, comprising computer-executable instructions. When executed by a processor, the instructions implement some or all of the steps of the method described according to a first aspect of this disclosure.

[0022] Understandably, the apparatus for determining memory according to the second aspect, the electronic device according to the third aspect, the computer storage medium according to the fourth aspect, or the computer program product according to the fifth aspect are all used to execute the method provided in the first aspect. Therefore, the explanations or descriptions regarding the first aspect also apply to the second, third, fourth, and fifth aspects. Furthermore, the beneficial effects achieved by the second, third, fourth, and fifth aspects can be referred to the beneficial effects in the corresponding methods, and will not be repeated here.

[0023] The summary section is intended to present the chosen concepts in a simplified form, which will be further described in the detailed description below. The summary section is not intended to identify key or principal features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description

[0024] The above and other features, advantages and aspects of the embodiments of this disclosure will become more apparent when taken in conjunction with the accompanying drawings and the following detailed description.

[0025] In the accompanying drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0026] Figure 1 A schematic diagram of an example environment in which the apparatus and / or methods according to embodiments of the present disclosure may be implemented is shown;

[0027] Figure 2 A flowchart of a method for determining memory according to an embodiment of the present disclosure is shown;

[0028] Figure 3A A schematic diagram illustrating the process of static memory modeling according to an embodiment of the present disclosure is shown;

[0029] Figure 3B An example diagram illustrating the process of predicting static memory according to an embodiment of the present disclosure is shown;

[0030] Figure 4 A schematic diagram illustrating a process for dynamic memory modeling and prediction according to embodiments of the present disclosure is provided.

[0031] Figure 5 A schematic diagram of an apparatus for determining memory according to an embodiment of the present disclosure is shown;

[0032] Figure 6 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0033] In all the accompanying figures, the same or similar reference numerals denote the same or similar elements. Detailed Implementation

[0034] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0035] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0036] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this subject matter pertains. It will be further understood that terms such as those defined in commonly used dictionaries shall be interpreted as having the meaning consistent with their meaning in the context of the specification and in the relevant art, and shall not be interpreted in an idealized or overly formal form unless otherwise explicitly defined herein. As used herein, the statement of “connecting” or “coupling” two or more parts together shall mean that these parts are directly joined together or joined through at least one intermediate component.

[0037] As mentioned earlier, parallel training, as a key technology for improving the training efficiency of large models, has become increasingly important. With the increasing number of configurable parameters for parallel training of large models, such as data parallelism (DP), tensor parallelism (TP), pipeline parallelism (PP), sequence parallelism (CP), zero-redundant optimizer (ZeRO), virtual pipeline parallelism (VPP), expert parallelism (EP), micro-batch size (MBS), and recomputation, the impact of various configurations on memory and performance has become increasingly complex, making manual tuning increasingly difficult. Therefore, the industry has begun to explore automatic parallelization methods, attempting to have the program autonomously predict the performance under various parallelization configurations and recommend the best parallelization configuration. During the execution of automatic parallelization, accurately predicting the memory usage under each parallelization configuration and ensuring sufficient memory resources has become a critical issue.

[0038] In related technologies, Bayesian search requires repeated configuration trials and must use resources of the same scale as real training, resulting in high resource overhead. As large-scale training tasks increase, the number of configuration combinations grows exponentially, gradually rendering Bayesian search infeasible. Theoretical modeling methods rely on assumptions about the network structure, but actual network structures often differ from these assumptions, leading to inaccurate modeling results. There is a discrepancy between theoretical modeling and actual memory usage, making it difficult to accurately reflect the actual situation. Therefore, existing technologies cannot quickly and accurately predict memory usage during model training under different parallel configurations.

[0039] To address this, embodiments of this disclosure propose a scheme for determining memory usage. First, a set of parallel training exercises related to the target model is initiated, and a corresponding set of memory usage information is obtained, wherein each parallel training exercise in this set is configured with different parallel parameters. Then, by comparing the memory usage information, the impact of different parallel configurations on memory usage is evaluated to generate a model of memory usage, thereby enabling the prediction of memory usage under different parallel configurations.

[0040] Therefore, through the embodiments of this disclosure, the memory usage of the target model can be modeled using a set of memory usage information from parallel training without needing to know the specific network structure, and it remains robust and accurate when the network structure changes. Thus, this method can provide memory prediction for memory usage issues under different parallel configurations, providing an important basis for automatic tuning of parallel training and avoiding training interruptions or failures due to insufficient memory.

[0041] Figure 1 A schematic diagram of an example environment in which the apparatus and / or methods according to embodiments of the present disclosure may be implemented is shown. Figure 1 As shown, the example environment 100 may include a computing device 110, which may be a user terminal, mobile device, computer, etc., or it may be a computing system, a single server, a distributed server, or a cloud-based server.

[0042] The computing device 110 can initiate a set of parallel training 130 related to the target model 120, each of which is configured with predetermined parallel parameters. In some embodiments, the computing device 110 can initiate a set of parallel training for the target model 120. In some embodiments, the computing device 110 can initiate a set of parallel training for sub-models pruned from the target model 120. In some embodiments, the target model 120 can be a large model that may include multiple Transform layers. In some embodiments, the target model 120 may employ a Mixture of Experts (MoE) model. In some embodiments, each parallel training is configured with predetermined parallel parameters. For example, one parallel training parameter may be TP=4, PP=4, DP=1, CP=1, EP=1, num_experts=1, and another parallel parameter may be TP=8, PP=1, DP=2, CP=1, EP=1, num_experts=2, where num_experts represents the number of experts included in each layer of the target model 120. It should be understood that when the value of num_experts is 1, the target model is a non-MoE model, that is, each layer contains only a unified subnetwork for computation, without involving a sparse expert selection mechanism.

[0043] The computing device 110 can acquire a set of memory usage information 140 through a set of parallel training 130. In some embodiments, the set of memory usage information 140 can indicate the memory usage of the target model 120 on a set of computing acceleration units during the corresponding parallel training process. The computing acceleration units may include, but are not limited to, graphics processing units (GPUs), tensor processing units (TPUs), neural processing units (NPUs), or artificial intelligence (AI) accelerator cards. In some embodiments, performance analysis tools (profiling) can be used to acquire the memory usage information of the target model during parallel training.

[0044] The computing device 110 can determine memory modeling parameters 150 for the target model 120 based on a set of memory usage information 140. For example, by comparing the memory usage information 140, the impact of each parallel configuration on memory can be determined, thereby modeling the memory usage. The computing device 110 can determine the peak memory usage 170 of the target model 120 during different parallel training sessions corresponding to different parallel parameters 160, based on the memory modeling parameters 150. For example, in automatic parallelization of the model, the peak memory usage for the recommended parallel parameters can be calculated, and it can be ensured that this peak memory usage does not exceed the memory of the computing acceleration unit, preventing training interruption or failure due to insufficient memory.

[0045] It should be understood that the architecture and functionality in example environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure. Embodiments of this disclosure can also be applied to other environments with different structures and / or functionalities.

[0046] The following will combine Figures 2 to 6 The process according to embodiments of this disclosure is described in detail. For ease of understanding, the specific data mentioned in the following description are exemplary and not intended to limit the scope of this disclosure. It is understood that the embodiments described below may also include additional actions not shown and / or actions shown may be omitted, and the scope of this disclosure is not limited in this respect.

[0047] Figure 2 A flowchart of a method 200 for determining memory according to an embodiment of the present disclosure is shown. Example method 200 may be, for example, by... Figure 1 The computing device 110 shown performs the operation. It should be understood that method 200 may also include additional actions not shown, and the scope of this application is not limited in this respect. The following is in conjunction with... Figure 1The example environment 100 is used to describe method 200 in detail. At box 202, a set of memory usage information is obtained by initiating a set of parallel training exercises related to the target model, where each parallel training exercise in the set is configured using predetermined parallel parameters. For example, combined with... Figure 1 The computing device 110 can obtain a set of memory usage information 140 by initiating a set of parallel training 130 related to the target model 120, wherein each parallel training in the set of parallel training is configured using predetermined parallel parameters.

[0048] At box 204, memory modeling parameters for the target model are determined based on a set of memory usage information. For example, combined with... Figure 1 The description states that computing device 110 can determine memory modeling parameters 150 for a target model 120 based on a set of memory usage information 140. At box 206, based on the memory modeling parameters, multiple memory usage peaks of the target model during different parallel training sessions corresponding to different parallel parameters are determined. For example, combining... Figure 1 The computing device 110 can determine multiple memory usage peaks 170 of the target model 120 in different parallel training corresponding to different parallel parameters 160, based on memory modeling parameters 150.

[0049] Therefore, according to the method 200 of the embodiments of this disclosure, without knowing the specific network structure, the memory usage of the target model can be modeled using a set of memory usage information from a set of parallel training data, and it remains robust and accurate when the network structure changes. Thus, when facing complex memory usage problems under different parallel configurations, it can provide memory prediction results, assisting the automatic tuning system in quickly selecting parallel configurations with sufficient resources, ensuring stable operation of automatic parallelism, and avoiding training interruptions or failures due to insufficient memory.

[0050] As mentioned earlier, memory usage information can be used to model the memory consumption of a target model during parallel training. It should be understood that the memory consumption described in this article refers to the memory consumption on computationally accelerator units during model training, such as the video memory usage on a GPU. In model training, memory consumption can be divided into static memory consumption and dynamic memory consumption. Static memory consumption is related to the weights, gradients, and optimizer state of the target model and remains constant during training. Dynamic memory consumption is related to the activation values ​​of the target model and fluctuates during training. The following will combine... Figure 3A Describe the process of static memory modeling, and combine it with Figure 4 Describe the process of dynamic memory modeling.

[0051] Figure 3AA schematic diagram of a static memory modeling process 300A according to an embodiment of the present disclosure is shown. Figure 3A As shown, at box 302, memory usage information of the target model during parallel training can be obtained. In some embodiments, the target model may include an input part, an intermediate part, and an output part. The input part relates to the embedding operation performed at the beginning of the target model, the intermediate part includes one or more hidden layers, and the output part is the portion of the target model after the last hidden layer in the intermediate part. For example, the target model may be a large model, the input and output parts may include embedding layers, and the intermediate part may include multiple Transformer layers with identical structures. In some embodiments, a set of parallel training operations as shown in Table 1 may be performed, where each parallel training operation is configured with different parallel parameters.

[0052] Table 1 Example Parallel Parameters

[0053] Parallel training Parallel parameters A TP=4,PP=4,DP=1,CP=1,EP=1,num_experts=1 B TP=4,PP=1,DP=4,CP=1,EP=1,num_experts=1 C TP=8,PP=1,DP=2,CP=1,EP=1,num_experts=1 D TP=8,PP=1,DP=2,CP=1,EP=1,num_experts=2 E TP=8,PP=1,DP=1,CP=2,EP=1,num_experts=1

[0054] As mentioned earlier, `num_experts` represents the number of experts contained in each layer of the target model. It should be understood that when `num_experts` is 1, the target model is a non-MoE model, meaning each layer contains only one unified sub-network for computation, without involving a sparse expert selection mechanism. In some embodiments, performance analysis tools can be used to collect memory usage information. Performance analysis tools, also known as profiling tools, are used to collect and analyze key data and performance metrics during model training and / or inference, including but not limited to memory usage, task scheduling information, operator call counts, and time consumption statistics. In some embodiments, the target model can be pruned to have the same PP value as the pipeline parallel model to generate a sub-model, and parallel training of this sub-model can be initiated to facilitate modeling computation. For example, if PP = 4, the number of hidden layers in the middle part of the target model can be set to 4 to obtain a sub-model of the target model.

[0055] At box 304, the memory footprint of the input and output parts of the target model can be determined. For example, in parallel training A, the number of hidden layers in the middle part of the target model is set to 4, and the pipeline parallelism parameter (PP) is set to 4. In this configuration, the 4 hidden layers are evenly distributed across 4 computational acceleration units (such as GPUs or other acceleration units). Since the first_embedding (the network for embedding operations in the input part) exists only before the first layer, the difference between the static memory footprint of the first card and the memory footprint of either of the two middle cards is the memory footprint of the input part. Similarly, the last_layernorm_and_embedding (the network related to the output part) exists only after the last hidden layer in the middle part, so the difference between the static memory footprint of the last card and the static memory footprint of either of the two middle cards is the memory footprint of the output part.

[0056] At box 306, the memory footprint of the target model affected by pipeline parallelism parameters can be determined. For example, in parallel training A and parallel training B, by setting the pipeline parallelism parameters to PP=4 and PP=1 respectively, parallel training of the pruned sub-models is initiated, and the collected static memory information is compared and analyzed. With the PP=1 configuration, all layers of the model are placed on the same computational unit. Some parameters (such as the output embedding) may reuse the embedding parameters from the first layer, resulting in a reduction in the final static memory footprint. These reduced parameters are called the pipeline memory footprint, also known as pp_affected. By comparing the memory footprint information for PP=4 and PP=1, the specific memory amount of pp_affected can be determined.

[0057] For example, assuming PP=4, the input, output, and hidden layers of the model are all allocated to different computational units. In this case, the first_embedding of the input and the last_layernorm_and_embedding of the output will each occupy memory separately. However, when PP=1, the output embedding may directly reuse the input embedding, resulting in a reduction in static memory usage. By comparing the memory usage difference of the final computational unit under the two configurations of PP=4 and PP=1, the memory usage affected by pipeline parallelism parameters can be determined.

[0058] At box 308, unsplit tensors in the target model that are unaffected by tensor parallel parameters can be identified. For example, in parallel training B and parallel training C, parallel training of the pruned sub-models is initiated by setting the tensor parallel parameters to TP=4 and TP=8 respectively, and the collected static memory usage information is compared and analyzed. Under tensor parallel configuration, the model's weight parameters are usually split according to the number of tensor parallel parameters. Therefore, when TP=8, the memory usage should theoretically be half that when TP=4. However, some tensors do not change with tensor parallel splitting; these tensors are called tp_unaffected, i.e., unsplit tensors unaffected by tensor parallelism. If the size of some tensors remains consistent under both TP=8 and TP=4 configurations, these tensors can be identified as unsplit tensors.

[0059] At box 310, if the target model is a hybrid expert MoE model, determine the expert tensors and non-expert tensors in the hidden layers of the target model. If each feedforward network layer in the Transformer layer of the target model is replaced with a MoE layer, the target model is a MoE model. For example, in parallel training C and parallel training D, the corresponding static memory usage information can be collected by setting the parallel parameters (e.g., TP, PP) to the same value, setting the number of experts parameter to 1 in one group and 2 in the other group. In the above configuration, the difference in static memory usage of each layer in the two sets of memory usage information can reflect the additional memory usage brought by an expert. For example, the difference between the two sets of static memory usage is the expert tensor belonging to the expert part. Removing the expert tensors belonging to the expert part can determine the non-expert tensors that do not belong to the expert part, and the size of each tensor can be determined based on the static memory usage information. In addition, it should be understood that for non-MoE models, since there is no MoE layer, each tensor in the hidden layer is a non-expert tensor, and there is no need to further distinguish expert parameters. Therefore, for non-MoE models, it is not necessary to run parallel training D in Table 1.

[0060] However, it should be understood that the input and output parts of the sub-model and the target model are consistent, differing only in the number of hidden layers in the intermediate parts. Therefore, the memory usage of the input part, the memory usage of the output part, and the memory usage per unit layer in the intermediate parts, determined during the memory modeling process for the sub-model, can be directly used to predict the overall memory usage of the target model. This pruning method not only reduces the computational resources and time costs required for memory modeling but also ensures the transferability and applicability of the modeling results. Thus, through the static memory modeling process 300A, the static memory modeling process for the target model is completed. The following will combine... Figure 3BThis describes the static memory parameters obtained from static memory modeling and predicts the static memory usage of the target model under different parallel configurations.

[0061] Figure 3B An example diagram of a process 300B for predicting static memory according to an embodiment of the present disclosure is shown. For example, when performing automatic parallelization of a target model, it is necessary to predict memory usage under a given parallelization configuration to determine whether memory is sufficient for that given parallelization configuration. As previously described, predicting this memory usage requires considering both static and dynamic memory usage simultaneously. The given parallelization parameters described herein may include one or more of tensor parallelization parameters, pipeline parallelization parameters, data parallelization parameters, sequence parallelization parameters, and expert parallelization parameters.

[0062] like Figure 3B As shown in box 320, the memory footprint of each tensor in the hidden layers of the target model can be determined based on the tensor parallelism parameter and the unsplit tensor. For example, for each tensor in one or more hidden layers in the middle part of the target model, it is first adjusted according to the size of the tensor parallelism parameter: for tensors affected by the tensor parallelism parameter, their memory footprint is split according to the tensor parallelism parameter (TP), that is, the memory footprint of the original tensor is divided by the value of the tensor parallelism parameter; for unsplit tensors (tp_unaffected) that are not affected by the tensor parallelism parameter, their memory footprint remains unchanged.

[0063] At box 322, the memory footprint of each hidden layer can be determined based on the memory footprint of each tensor, the expert tensor, and the non-expert tensor. For example, for the MoE model, the memory footprint of each hidden layer can be determined by a combination of the non-expert memory footprint and the expert memory footprint, as shown in formula (1):

[0064] per_layer=(per_layer_wo_experts + per_experts * num_experts / EP) (1)

[0065] The non-expert memory usage is represented by per_layer_wo_experts, and the expert memory usage is represented by per_experts×num_experts / EP, where num_experts is the number of experts, EP is the expert parallelism parameter, and per_layer is the memory usage per hidden layer. Then, the memory usage per computation acceleration unit can be determined using formula (2):

[0066] per_unit = per_layer * num_layers / PP (2)

[0067] Where num_layers represents the number of hidden layers in the middle part of the target model, PP is the pipeline parallelism parameter, and per_unit is the memory usage per computational acceleration unit, such as the video memory usage per GPU card. Furthermore, if the target model is a non-MoE model, each hidden layer has a uniform network structure and does not involve a sparse expert selection mechanism, then there is no need to consider the expert memory usage; only the memory usage of the non-expert parts in the hidden layers needs to be calculated.

[0068] At box 324, the static memory footprint can be determined based on the memory footprint of the hidden layers, the memory footprint of the input and output sections, and the pipeline parallelism parameters. For example, with pipeline parallelism (PP) enabled, the hidden layers of the target model are divided into multiple pipeline stages, and each pipeline stage's computational acceleration unit needs to allocate a corresponding amount of hidden layer memory. Furthermore, the memory footprint of the input and output sections needs to be adjusted according to the pipeline parallelism parameters (PP) to allocate them to specific pipeline stages.

[0069] For example, when PP>1, the memory usage of the hidden layers is allocated to the computation acceleration units of each pipeline stage according to formula (2). Simultaneously, the memory usage of the input portion is added to the computation acceleration unit of the first pipeline stage, while the memory usage of the output portion is added to the computation acceleration unit of the last pipeline stage. When pipeline parallelism is not enabled (i.e., PP=1), the memory usage of all hidden layers is allocated to the same computation acceleration unit. In this case, the memory usage of the input and output portions needs to be added to the computation acceleration unit. Furthermore, as mentioned earlier, with PP=1, all layers of the model are placed on the same computation acceleration unit. Some parameters (e.g., the embedding of the output portion) may reuse the embedding parameters from the first layer, resulting in a reduction in the static memory usage at the end, i.e., the pipeline memory usage. Therefore, with PP=1, since pipeline parallelism is not enabled, the computation acceleration unit also needs to include the pipeline memory usage.

[0070] Figure 4 A schematic diagram illustrating a process 400 for dynamic memory modeling and prediction according to embodiments of the present disclosure is provided. During the training of the target model, peaks in dynamic memory may occur at multiple locations. For example... Figure 4As shown, forward_peak 402 represents the peak memory usage of the target model during the forward propagation phase; loss_start 404 represents the memory usage at the start of loss function computation; loss_peak 406 represents the peak memory usage during the loss function computation phase; backward_start 408 represents the memory usage at the start of the backpropagation phase; backward_peak 410 represents the peak memory usage during the backpropagation phase; and optimizer_peak 412 represents the peak memory usage during the optimizer update phase. Furthermore, the dashed line 414 represents the static memory usage of the target model during parallel training.

[0071] refer to Figure 4 The dynamic memory usage changes primarily correspond to the various stages of the target model's training process, including forward propagation, loss calculation, backpropagation, and optimizer update. In the graph, the vertical axis represents memory usage, and the horizontal axis represents training time. During the forward propagation stage, the target model calculates intermediate results layer by layer, and the dynamic memory usage increases as the calculation progresses. When the peak of the forward propagation stage is reached, corresponding to a peak memory usage of 402, this peak reflects the highest point of memory usage by the target model during forward propagation. Afterward, memory usage decreases, entering the next stage, the loss calculation stage.

[0072] During the loss calculation phase, dynamic memory usage gradually decreases from the forward propagation phase to a minimum of 404 memory slots when the loss function is initially calculated. During loss function calculation, dynamic memory usage increases again, reaching a peak of 406 memory slots, because loading relevant parameters and intermediate results is required. After loss calculation is complete, the training process enters the backpropagation phase. When backpropagation is initiated, memory usage decreases to a minimum of 408 memory slots. As backpropagation performs layer-by-layer calculations, memory usage gradually increases and reaches its peak at 410 memory slots.

[0073] After backpropagation, the training process enters the optimizer update phase. During this phase, memory usage increases again, reaching a peak of 412 memory for the optimizer. This is because the optimizer requires additional memory to store gradients and optimizer states during parameter updates. Furthermore, the dashed line 414 represents the static memory usage of the target model during parallel training. This dashed line runs throughout the entire training process, indicating that static memory usage is a fixed portion on top of dynamic memory usage, primarily determined by static data such as model weights, gradients, and optimizer states.

[0074] In some embodiments, dynamic memory can be modeled in five parts, as shown in Table 2:

[0075] Table 2 Dynamic Memory Modeling

[0076] Part One loss_start-forward_start Part Two forward_peak-loss_start Part Three loss_peak-loss_start Part Four backward_peak-loss_start Part Five optimizer_peak–forward_start

[0077] In each section, dynamic memory usage data is collected using the performance analysis tool Profiling, and modeling is performed based on a nonlinear regression model according to the following formula (3):

[0078] y = a + b / CP + c / (CP * TP) (3)

[0079] Where CP represents the sequence parallel parameters, TP represents the tensor parallel parameters, and a, b, and c are model parameters obtained by regression from a set of dynamic memory usage data. Each part corresponds to a nonlinear regression sub-model, which is modeled according to formula (3). For example, for the first part (loss_start-forward_start), the corresponding regression parameters a, b, and c are calculated by collecting dynamic memory data. Similarly, the second part (forward_peak-loss_start) can describe the trend of memory change from the forward propagation to the loss calculation stage, and the corresponding regression parameters for this stage can be obtained by fitting the data. For example, the three coefficients a, b, and c can be calculated using the dynamic memory usage data of parallel training B, parallel training C, and parallel training E shown in Table 1.

[0080] In some embodiments, four possible dynamic memory peaks of the target model under different parallel configurations can be predicted based on dynamic memory modeling parameters. For example, each part shown in Table 2 can be calculated according to formula (3), such as the first part (loss_start-forward_start) and the second part (forward_peak-loss_start). By adding the two together, forward_peak-forward_start can be determined, where forward_start represents the memory usage at the beginning of the forward propagation phase, which is the previously determined static memory usage, and thus forward_peak can be obtained. Similarly, loss_peak, backward_peak, and optimizer_peak can be determined. Then, the maximum value among these four peaks is the peak dynamic memory usage.

[0081] As mentioned earlier, the static memory usage of the model during training remains relatively constant and does not change significantly with different training phases. This is because static memory is primarily used to store model weights, gradients, and optimizer states, and is independent of the dynamic training process. Peak dynamic memory reflects memory fluctuations at various stages of model training, such as memory usage during forward propagation, loss calculation, backpropagation, and optimizer update. By determining the peak dynamic memory usage and adding it to the static memory usage, the maximum memory required for training can be obtained. In some embodiments, the sum of static memory and peak dynamic memory is the estimated maximum memory usage. For example, by superimposing the static memory usage and peak dynamic memory usage, the maximum memory requirement of the target model under a specific parallel configuration can be determined. It should be understood that the memory usage described in this paper refers to the memory usage on computationally accelerated units during model training, such as GPU memory usage. By comparing the predicted maximum memory usage with the GPU memory capacity, it can be determined whether the memory requirements are met: if the maximum memory usage is less than the GPU memory capacity, it means that the target model has sufficient memory under the corresponding parallel parameter configuration and can perform parallel training; otherwise, if the memory is insufficient, the parallel parameters cannot be recommended for parallel training during automatic parallelization.

[0082] For example, in some embodiments, the automatic parallel strategy can generate a set of candidate parallel parameter configurations, such as tensor parallel parameters (TP), pipelined parallel parameters (PP), data parallel parameters (DP), sequence parallel parameters (CP), and expert parallel parameters (EP). For each set of candidate parallel parameter configurations, based on embodiments of this disclosure, the static memory and peak dynamic memory of the target model under that set of parallel configurations can be calculated, and the maximum memory usage can be determined. If the maximum memory usage is less than the video memory capacity of the computing acceleration unit (such as a GPU), it indicates that memory resources are sufficient under this configuration, and the target model can be trained normally; otherwise, if the maximum memory usage is greater than the video memory capacity, it indicates that memory resources are insufficient, and the automatic parallel strategy needs to adjust the parallel parameters. For example, the memory usage of a single computing acceleration unit can be reduced by increasing the tensor parallel parameters (TP) or pipelined parallel parameters (PP), or the data parallel parameters (DP) and micro-batch size (MBS) can be adjusted to find new feasible configurations. Through this method, parallel configurations with sufficient memory resources can be efficiently screened, thereby ensuring the stability of the training process.

[0083] Figure 5A schematic diagram of an apparatus 500 for determining memory usage according to an embodiment of the present disclosure is shown. The apparatus 500 includes a memory usage acquisition module 502 configured to acquire a set of memory usage information by initiating a set of parallel training sessions related to a target model, wherein each parallel training session in the set of parallel training sessions is configured using predetermined parallel parameters. The apparatus 500 also includes a modeling parameter determination module 504 configured to determine memory modeling parameters for the target model based on the set of memory usage information. Furthermore, the apparatus includes a peak usage determination module 506 configured to determine multiple memory usage peaks of the target model in different parallel training sessions corresponding to different parallel parameters, based on the memory modeling parameters.

[0084] In some embodiments of this disclosure, the target model includes an input portion, an intermediate portion, and an output portion. The input portion relates to an embedding operation performed at the beginning of the target model, the intermediate portion includes one or more hidden layers, and the output portion is the portion of the target model after the last hidden layer in the intermediate portion.

[0085] In some embodiments of this disclosure, the memory usage acquisition module 502 includes: a sub-model generation module configured to generate a sub-model of the target model by pruning the target model; and a second memory usage acquisition module configured to acquire a set of memory usage information by initiating a set of parallel training of the sub-model and using performance analysis tools.

[0086] In some embodiments of this disclosure, each memory usage information in a set of memory usage information includes static memory usage information and dynamic memory usage information, wherein the static memory usage information is related to the weights, gradients and optimizers of the sub-model, and the dynamic memory usage information is related to the activation values ​​of the sub-model.

[0087] In some embodiments of this disclosure, the memory modeling parameters include static memory parameters and dynamic memory parameters, and the modeling parameter determination module 504 includes: a static parameter determination module configured to determine static memory parameters based on a set of static memory usage information from a set of memory usage information; and a dynamic parameter determination module configured to determine dynamic memory parameters based on a set of dynamic memory usage information from a set of memory usage information.

[0088] In some embodiments of this disclosure, each of the predetermined parallel parameters includes at least one of tensor parallel parameters, pipeline parallel parameters, data parallel parameters, sequence parallel parameters, and expert parallel parameters.

[0089] In some embodiments of this disclosure, the static parameter determination module includes: an input / output determination module configured to determine the memory usage of the input portion and the memory usage of the output portion of the sub-model based on the static memory differences of the sub-model on different computing acceleration units, wherein the static memory differences are determined based on a set of static memory usage information; a pipeline usage determination module configured to determine the pipeline memory usage of the sub-model affected by the pipeline parameters based on the static memory differences of the sub-model under different pipeline parallel parameters; and a non-segmented tensor determination module configured to determine the non-segmented tensors in the sub-model that are not affected by the tensor parallel parameters based on the static memory differences of the sub-model under different tensor parallel parameters.

[0090] In some embodiments of this disclosure, in response to the intermediate portion of the target model including a hybrid expert layer, the static parameter determination module further includes: an expert parameter determination module configured to determine, based on the static memory difference of the submodel under parallel parameters with different numbers of experts, the expert tensors belonging to the expert portion and the non-expert tensors not belonging to the expert portion in each hidden layer of the intermediate portion of the submodel.

[0091] In some embodiments of this disclosure, the dynamic parameter determination module includes: a regression model determination module configured to construct a nonlinear regression model of the dynamic memory of the sub-model during parallel training, wherein the nonlinear regression modeling includes multiple sub-regression models related to the forward peak memory, loss enable memory, loss peak memory, backward enable memory, backward peak memory, and optimizer peak memory of the sub-model; and a regression parameter determination module configured to determine the model parameters of the multiple sub-regression models as dynamic memory parameters based on a set of dynamic memory occupancy information.

[0092] In some embodiments of this disclosure, the peak memory usage determination module 506 includes: a static and dynamic memory usage determination module, configured to determine the static memory usage and dynamic memory usage peak of the target model in parallel training corresponding to the given parallel parameters based on the given parallel parameters; and a memory usage determination module, configured to determine the corresponding memory usage peak of the target model in the corresponding parallel training based on the static memory usage and dynamic memory usage peak.

[0093] In some embodiments of this disclosure, the static / dynamic memory usage determination module includes: a tensor memory determination module configured to determine the memory usage of each tensor in the hidden layer of the target model based on the tensor parallel parameters in the given parallel parameters and the non-splitting tensor in the static memory parameters; a hidden layer memory determination module configured to determine the memory usage of the hidden layer of the target model based on the memory usage of each tensor in the hidden layer, as well as expert tensors and non-expert tensors; and a static memory determination module configured to determine the static memory usage based on the memory usage of the hidden layer of the target model, the pipeline parameters in the given parallel parameters, the memory usage of the input portion, and the memory usage of the output portion.

[0094] In some embodiments of this disclosure, the static and dynamic occupancy determination module further includes: a dynamic peak candidate determination module, configured to determine the forward peak memory, loss peak memory, reverse peak memory, and optimizer peak memory of the target model in the corresponding parallel training based on dynamic memory parameters; and a dynamic peak determination module, which determines the maximum value among the forward peak memory, loss peak memory, reverse peak memory, and optimizer peak memory as the dynamic memory occupancy peak.

[0095] In some embodiments of this disclosure, the apparatus 500 further includes an automatic parallel execution module configured to perform automatic parallel execution of the target model based on multiple memory usage peaks of the target model in different parallel training corresponding to different parallel parameters.

[0096] Figure 6 A block diagram of an electronic device 600 according to an embodiment of the present disclosure is shown. As shown, the electronic device 600 includes a processing unit 601, which can perform various appropriate actions and processes according to computer program instructions stored in random access memory (RAM) 603 and / or read-only memory (ROM) 602 or computer program instructions loaded from storage unit 608 into RAM 603 and / or ROM 602. Various programs and data required for the operation of the electronic device 600 may also be stored in RAM 603 and / or ROM 602. The processing unit 601 and RAM 603 and / or ROM 602 are interconnected via bus 604. An input / output (I / O) interface 605 is also connected to bus 604.

[0097] Multiple components in electronic device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of displays, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows electronic device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0098] Processing unit 601 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processing unit 601 include, but are not limited to, CPUs, GPUs, NPUs, various special-purpose AI computing chips, various processing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processing unit 601 performs the various methods and processes described above, such as method 200. For example, in some embodiments, method 200 may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 600 via RAM and / or ROM and / or communication unit 609. When the computer program is loaded into RAM and / or ROM and executed by processing unit 601, one or more actions of the various methods and processes described above may be performed. Alternatively, in other embodiments, processing unit 601 may be configured to perform one or more actions of the methods and processes described above by any other suitable means (e.g., by means of firmware).

[0099] This disclosure can be a method, apparatus, system, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0100] A computer-readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer-readable storage medium can be, for example—but not limited to—an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), and any suitable combination thereof. The computer-readable storage medium as used herein is not to be construed as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0101] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0102] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0103] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0104] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0105] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0106] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0107] Unless otherwise expressly indicated by the context, the singular form of words used herein and in the appended claims includes the plural form, and vice versa. Thus, when referring to the singular, the plural form of the corresponding term is generally included. Where the term “example” is used herein, particularly when it follows a set of terms, the “example” is merely exemplary and illustrative and should not be considered exclusive or pervasive.

[0108] Further aspects and scope of adaptation become apparent from the description provided herein. It should be understood that various aspects of this application may be implemented individually or in combination with at least one other aspect. It should also be understood that the descriptions and specific embodiments herein are for illustrative purposes only and are not intended to limit the scope of this application.

[0109] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for determining memory, comprising: By initiating a set of parallel training related to the target model, a set of memory usage information is obtained, wherein each of the parallel training sessions in the set of parallel training sessions is configured using predetermined parallel parameters; Based on the set of memory usage information, memory modeling parameters for the target model are determined; as well as Based on the memory modeling parameters, the peak memory usage of the target model in different parallel training corresponding to different parallel parameters is determined.

2. The method of claim 1, wherein the target model includes an input portion, an intermediate portion, and an output portion, the input portion relating to an embedding operation performed at the beginning of the target model, the intermediate portion including one or more hidden layers, and the output portion being the portion of the target model following the last hidden layer in the intermediate portion.

3. The method according to claim 2, wherein obtaining a set of memory usage information by initiating a set of parallel training related to the target model includes: By cropping the target model, a sub-model of the target model is generated; as well as By initiating a set of parallel training sessions for the sub-model, the memory usage information of the set is obtained using performance analysis tools.

4. The method according to claim 3, wherein each memory usage information in the set of memory usage information includes static memory usage information and dynamic memory usage information, and the static memory usage information is related to the weights, gradients, and optimizers of the sub-model, and the dynamic memory usage information is related to the activation values ​​of the sub-model.

5. The method of claim 4, wherein the memory modeling parameters include static memory parameters and dynamic memory parameters, and determining the memory modeling parameters for the target model includes: The static memory parameters are determined based on a set of static memory usage information from the set of memory usage information. as well as The dynamic memory parameters are determined based on a set of dynamic memory usage information from the set of memory usage information.

6. The method of claim 5, wherein each of the predetermined parallel parameters comprises at least one of tensor parallel parameters, pipeline parallel parameters, data parallel parameters, sequence parallel parameters, and expert parallel parameters.

7. The method of claim 6, wherein determining the static memory parameters comprises: Based on the static memory differences of the sub-model on different computing acceleration units, the memory usage of the input part and the memory usage of the output part of the sub-model are determined, wherein the static memory differences are determined based on the set of static memory usage information; Based on the static memory differences of the sub-model under different pipeline parallel parameters, the pipeline memory usage in the sub-model affected by the pipeline parameters is determined. Based on the static memory differences of the sub-model under different tensor parallel parameters, the unsegmented tensors in the sub-model that are not affected by the tensor parallel parameters are determined.

8. The method of claim 7, wherein the intermediate portion of the target model includes a hybrid expert layer, wherein determining the static memory parameters further includes: Based on the static memory differences of the sub-model under parallel parameters with different numbers of experts, the expert tensors belonging to the expert part and the non-expert tensors not belonging to the expert part in each hidden layer of the intermediate part of the sub-model are determined.

9. The method of claim 8, wherein determining the dynamic memory parameters comprises: Construct a nonlinear regression model of the dynamic memory of the sub-model in parallel training. The nonlinear regression modeling includes multiple sub-regression models related to the forward peak memory, loss-enabling memory, loss-peak memory, backward-enabling memory, backward-peak memory, and optimizer peak memory of the sub-model. as well as Based on the set of dynamic memory usage information, the model parameters of the multiple sub-regression models are determined as the dynamic memory parameters.

10. The method of claim 9, wherein determining the plurality of memory usage peaks of the target model in different parallel training corresponding to different parallel parameters comprises: Based on the given parallel parameters, determine the static memory usage and peak dynamic memory usage of the target model during parallel training corresponding to the given parallel parameters; as well as Based on the static memory usage and the dynamic memory usage peak, the corresponding memory usage peak of the target model in the corresponding parallel training is determined.

11. The method of claim 10, wherein determining the static memory footprint of the target model in the corresponding parallel training comprises: Based on the tensor parallel parameters in the given parallel parameters and the unsegmented tensor in the static memory parameters, determine the memory usage of each tensor in the hidden layer of the target model; The memory usage of the hidden layer of the target model is determined based on the memory usage of each tensor of the hidden layer, as well as the expert tensor and the non-expert tensor. as well as The static memory usage is determined based on the memory usage of the hidden layer of the target model, the pipeline parameters in the given parallel parameters, the memory usage of the input part, and the memory usage of the output part.

12. The method of claim 11, wherein determining the peak dynamic memory usage of the target model during the corresponding parallel training comprises: Based on the dynamic memory parameters, the forward peak memory, loss peak memory, backward peak memory, and optimizer peak memory of the target model in the corresponding parallel training are determined. as well as The maximum value among the forward peak memory, the loss peak memory, the reverse peak memory, and the optimizer peak memory is determined as the dynamic memory usage peak.

13. The method according to claim 1, further comprising: Based on the multiple memory usage peaks of the target model during different parallel training corresponding to different parallel parameters, automatic parallelization of the target model is performed.

14. An apparatus for determining memory, comprising: The memory usage acquisition module is configured to acquire a set of memory usage information by initiating a set of parallel training related to the target model, wherein each of the parallel training sessions in the set of parallel training sessions is configured using predetermined parallel parameters; The modeling parameter determination module is configured to determine memory modeling parameters for the target model based on the set of memory usage information. as well as The peak memory usage determination module is configured to determine multiple peak memory usage values ​​of the target model in different parallel training sessions corresponding to different parallel parameters, based on the memory modeling parameters.

15. An electronic device comprising: At least one processor; as well as A memory coupled to the at least one processor and having instructions stored thereon, which, when executed by the at least one processor, cause the electronic device to perform the method according to any one of claims 1-13.

16. A computer program product tangibly stored on a non-transient computer-readable medium and comprising machine-executable instructions for performing the method according to any one of claims 1-13.

Images