Recalculating training strategy generation method, electronic device, and computer program product
By determining an adaptive recomputation training strategy for each in-memory resource in a heterogeneous cluster, the problem of the inability to optimize the differences between heterogeneous in-memory resources in existing technologies is solved, thereby improving model training efficiency and throughput and balancing the performance of in-memory resources.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HANGZHOU HIGH-TECH ZONE (BINJIANG) INSTITUTE OF BLOCKCHAIN & DATA SECURITY
- Filing Date
- 2026-03-19
- Publication Date
- 2026-06-26
AI Technical Summary
Existing recomputation strategies cannot differentiate and optimize for the differences in storage and computing power of heterogeneous storage and computing resources, resulting in reduced training efficiency in heterogeneous environments.
By determining the cluster information of the heterogeneous cluster and the model feature information of the model to be trained, an adaptive recomputation training strategy is determined for each in-memory resource, including recomputation time information and performance requirements. The computing units are divided and it is determined whether to perform recomputation. The training strategy is optimized to balance the performance of in-memory resources.
It improves the utilization rate of heterogeneous clusters, enhances model training efficiency and throughput, balances the computational and storage overhead of computing resources, and alleviates storage bottlenecks.
Smart Images
Figure CN121859973B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of artificial intelligence technology, and in particular relates to a method for generating recomputational training strategies, electronic devices, and computer program products. Background Technology
[0002] In training large models, as the model size continues to expand, storage requirements surge, making storage a significant bottleneck restricting training efficiency. To mitigate the impact of storage on training efficiency, recomputation strategies are widely adopted. These strategies primarily replace storage with computation, saving storage by discarding activation values during the forward pass and recomputing activations during the backward pass.
[0003] However, existing recomputation strategies are all based on the assumption that multiple computing resources are homogeneous, such as using a uniform activation / discard strategy for multiple computing resources. Existing recomputation strategies cannot differentiate for the storage and computing power differences of heterogeneous computing resources. In heterogeneous environments, this uniform recomputation strategy typically uses low-performance computing resources as an upper limit reference factor, leading to a more aggressive overall recomputation scheme, significantly increasing the computational burden and ultimately reducing training efficiency.
[0004] Therefore, there is an urgent need for a recomputation strategy that can adapt to the heterogeneous characteristics of in-memory computing resources. Summary of the Invention
[0005] In view of this, embodiments of this application provide a method for generating recomputation training strategies, an electronic device, and a computer program product, which are used to determine adaptive recomputation training strategies for each storage and computing resource in a heterogeneous cluster. By adopting the recomputation training strategies corresponding to each storage and computing resource for model training, the efficiency and throughput of model training are improved.
[0006] The first aspect of this application provides a method for generating a recomputable training strategy, including:
[0007] Determine the cluster information of the heterogeneous cluster and the model feature information of the model to be trained;
[0008] For each of the aforementioned in-memory computing resources, based on the cluster information and the model feature information, the recomputation time information and recomputation performance requirements are determined; the recomputation time information is used to characterize the time required for a computing unit corresponding to the same in-memory computing resource to perform recomputation; the computing unit corresponds to the activation tensor in the model to be trained;
[0009] If the recomputation performance requirement is not greater than the performance limit of the storage resources, determine the recomputation time information characteristic value;
[0010] Based on the recomputation time information feature values corresponding to each computing resource, a recomputation training strategy is determined for the model to be trained.
[0011] In some implementations of the first aspect, the model feature information includes the segmentation information of the neural network layers in the model to be trained; the cluster information includes the neural network layers and performance information corresponding to each of the computing resources; the recomputation time information is determined through the following steps:
[0012] Based on the performance information, determine the time estimate for each computing unit according to the corresponding neural network layer.
[0013] Obtain the recalculation variables corresponding to each computing unit; the recalculation variables are used to identify whether the corresponding computing unit enables recalculation.
[0014] The recalculation time information is determined by summing the products of each of the time estimates and the corresponding recalculated variables.
[0015] In some implementations of the first aspect, the cluster information includes the total number of stages in the training pipeline set for the model to be trained, and the stage values corresponding to different computing resources, wherein the stage values are used to characterize the stage in the training pipeline; the model feature information includes the storage requirements and fixed storage consumption corresponding to each computing unit; the recomputation performance requirements are determined through the following steps:
[0016] Based on the recomputation variables and storage requirements corresponding to each computing unit, the storage requirement information of each neural network layer is determined; the storage requirement information is used to characterize the storage requirements of computing units that do not perform recomputation.
[0017] The difference between the total number of stages and the corresponding stage information of the storage resources is determined as the stage difference value;
[0018] The recalculation performance requirement is calculated by multiplying the storage requirement information and the stage difference, and summing the product with the fixed storage consumption.
[0019] In some implementations of the first aspect, the performance limit includes storage capacity; determining the recomputation time information feature value when the recomputation performance requirement is not greater than the performance limit of the storage resources includes:
[0020] With the requirement that the recomputation performance should not exceed the storage capacity of the in-memory computing resources as a constraint, and with the goal of minimizing the recomputation time information, linear programming is performed to determine the linear programming result;
[0021] The linear programming result is determined as a time information feature value;
[0022] The time information feature value includes the parameter values of the recomputation variables corresponding to each computing unit of the storage and computing resources.
[0023] In some implementations of the first aspect, the method further includes:
[0024] Based on the aforementioned recomputation training strategy, the execution time of each in-memory resource is determined;
[0025] When the difference between the execution times meets the preset conditions, the number of neural network layers corresponding to at least two storage resources is adjusted, and the recomputation training strategy is updated.
[0026] In some implementations of the first aspect, the recalculated time information is Time(R);
[0027]
[0028] in, For computing unit Time estimate, For computing unit Recalculated variables, This serves as the identifier for the computing unit.
[0029] In some implementations of the first aspect, the recomputation performance requirement is Mem(R);
[0030]
[0031] Where Const is the fixed storage consumption, P is the total number of stages, and S is the stage value. For computing unit Storage requirements for the generated activation tensor For computing unit The recalculated variables.
[0032] In some implementations of the first aspect, the method further includes:
[0033] Obtain training data;
[0034] The heterogeneous cluster is invoked to perform parallel training on the model to be trained based on the cluster information, the model feature information, the recomputation training strategy, and the training data.
[0035] A second aspect of this application provides a recomputation training strategy generation apparatus located in a heterogeneous cluster, the apparatus comprising:
[0036] The information acquisition module is used to determine the cluster information of the heterogeneous cluster and the model feature information of the model to be trained;
[0037] The recomputation time information and performance requirement determination module is used to determine the recomputation time information and recomputation performance requirements for each of the aforementioned in-memory resources, based on the cluster information and the model feature information; the recomputation time information is used to characterize the time required for a computing unit corresponding to the same in-memory resource to perform recomputation; the computing unit corresponds to the activation tensor in the model to be trained;
[0038] The recomputation time information feature value determination module is used to determine the recomputation time information feature value when the recomputation performance requirement is not greater than the performance limit of the storage and computing resources.
[0039] The recomputation training strategy generation module is used to determine the recomputation training strategy for the model to be trained based on the recomputation time information feature values corresponding to each computing resource.
[0040] A third aspect of this application provides an electronic device including a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the electronic device implements the recomputation training strategy generation method as described in the first aspect above.
[0041] A fourth aspect of this application provides a computer program product, including a computer program that, when run, causes the recomputation training strategy generation method as described in the first aspect above to be executed.
[0042] A fifth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the recomputation training strategy generation method as described in the first aspect above.
[0043] Compared with the prior art, the embodiments of this application have the following beneficial effects:
[0044] In this embodiment, the cluster information of the heterogeneous cluster and the model feature information of the model to be trained are determined. For each in-memory resource in the heterogeneous cluster, based on the cluster information and model feature information, the recomputation time information and recomputation performance requirements are determined. The recomputation time information is used to characterize the time required for the computing unit corresponding to the same in-memory resource to perform recomputation. The computing unit corresponds to the activation tensor in the model to be trained. When the recomputation performance requirement is not greater than the performance limit of the in-memory resource, the recomputation time information feature value is determined. Based on the recomputation time information feature value corresponding to each in-memory resource, the recomputation training strategy for the model to be trained is determined. For scenarios that require model training using a heterogeneous cluster, the smallest granularity of recomputation computing units can be divided for the neural network layers corresponding to each in-memory resource in the heterogeneous cluster. The recomputation training strategy for training the model for each in-memory resource is determined by the recomputation time information and recomputation performance requirements, that is, the computing units that need to be recomputed are determined. Different in-memory resources correspond to different recomputation training strategies, so that the recomputation training strategy can balance the performance of the corresponding in-memory resources, improve the utilization rate of the heterogeneous cluster, and thus improve the throughput of the training model and improve the training efficiency of the model. Attached Figure Description
[0045] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0046] Figure 1 This is a schematic diagram of a recomputation training strategy generation method provided in an embodiment of this application;
[0047] Figure 2 This is a schematic diagram of a computational unit of a neural network layer provided in an embodiment of this application;
[0048] Figure 3 This is a schematic diagram of a computational unit for another neural network layer provided in an embodiment of this application;
[0049] Figure 4 This is a schematic diagram of a heterogeneous sensing adaptive recomputation system provided in an embodiment of this application;
[0050] Figure 5 This is a schematic diagram illustrating a model training experiment comparison provided in an embodiment of this application;
[0051] Figure 6 This is a schematic diagram of a recomputation training strategy generation device provided in an embodiment of this application;
[0052] Figure 7 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation
[0053] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.
[0054] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.
[0055] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.
[0056] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."
[0057] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0058] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.
[0059] One of the inventive concepts of this application is to achieve fine-grained control over recomputation behavior by dividing each neural network layer of the model into multiple computational units, supporting the determination on demand whether each computational unit needs to be recomputed. For the heterogeneous in-memory resources contained in the heterogeneous cluster used to train the model, corresponding computational units that need to be recomputed and those that do not are determined, resulting in corresponding recomputation strategies for each heterogeneous in-memory resource. This improves the adaptability of different in-memory resources to their corresponding recomputation strategies, balances the storage and computational overhead of in-memory resources, and can also balance the computational pressure of different in-memory resources in a heterogeneous environment, improving the utilization rate of in-memory resources, alleviating storage bottlenecks in model training, and improving model training efficiency, especially the training efficiency of large artificial intelligence models.
[0060] The technical solution of this application will be described below through specific embodiments.
[0061] Reference Figure 1 This illustration shows a schematic diagram of a recomputation training strategy generation method provided in an embodiment of this application, applied to a heterogeneous cluster. The heterogeneous cluster contains two or more different in-memory computing resources, which have both storage and computing capabilities, i.e., they can handle storage tasks and computing tasks. For example, in-memory computing resources can be GPUs or in-memory computing chips. Different in-memory computing resources may differ in one or more dimensions, such as hardware architecture, computing performance (e.g., floating-point operation performance), storage performance, bandwidth, etc.
[0062] The specific steps included in this application embodiment are as follows:
[0063] Step 101: Determine the cluster information of the heterogeneous cluster and the model feature information of the model to be trained.
[0064] Cluster information can include information that characterizes a single in-memory resource (e.g., performance information of a certain in-memory resource) and information that characterizes the relationships between in-memory resources (e.g., cluster information between multiple in-memory resources).
[0065] Multiple computational units can be defined for the activation tensors in the model to be trained, with each computational unit serving as the smallest unit of recomputation (i.e., the granularity of recomputation is based on the computational unit). The model to be trained includes multiple neural network layers, each with multiple computational units. Each computational unit is bound to a corresponding activation tensor, serving as the basic decision-making object for whether to retain (store) or recompile it. Activation tensors refer to intermediate computation results that have dependencies during the backpropagation phase and need to be directly read or recovered through recomputation. Computation results that are only used momentarily during the forward propagation and have no need for reuse during backpropagation are not included in the scope of the computational units established in this embodiment.
[0066] Taking a trainable model based on the Transformer architecture as an example, the computational units can cover major neural network layers such as attention layers and feedforward layers, supporting flexible combinations and policy switching. For instance, if an activation tensor needs to be used in backpropagation, its parent computational unit can be preferentially retained to avoid redundant computation; conversely, it can be discarded to save GPU memory. Through the recomputation mechanism, each computing resource can adaptively select a recomputation strategy based on its own computing capabilities, achieving an optimal trade-off between memory saving and computational cost. (See reference...) Figure 2 This illustration shows a schematic diagram of a computational unit for a neural network layer according to an embodiment of this application. In some examples, the neural network layer in the model to be trained includes, for example... Figure 2 The attention layers shown can be arranged according to... Figure 2 The method shown targets each activation tensor of the attention layer. Figure 2 Separate calculation units are set up for X1, Q, K, V, X2, and X3. (Refer to...) Figure 3 This illustration shows a schematic diagram of the computational unit of another neural network layer provided in an embodiment of this application. In some examples, the neural network layer in the model to be trained includes, for example... Figure 3 The feedforward layer shown can be configured according to... Figure 3 The method shown applies to each activation tensor of the feedforward layer ( Figure 3 Separate computation units are set up for X1, X2, X3, and X4. The above method of dividing computation units is highly versatile and can be seamlessly adapted to mainstream Transformer models such as GPT, BERT, and LLaMA.
[0067] Understandably, the trainable model in the Transformer architecture can include other neural network layers besides the attention layer and feedforward layer. Figure 2 , Figure 3 As an example only, in actual implementations, computational units can be set in the attention layer, feedforward layer, or other neural network layers.
[0068] It should be noted that the computational units corresponding to the activation tensors can be set according to the actual model structure to adapt to different model structures, including CNN, MoE, etc.
[0069] Model feature information can include information related to the model to be trained, such as the segmentation information of the neural network layers in the model. The segmentation information includes the neural network layers processed by different computing resources.
[0070] Step 102: For each storage and computing resource in the heterogeneous cluster, determine the recomputation time information and recomputation performance requirements based on the cluster information and model feature information.
[0071] Among them, recomputation time information is used to characterize the time required for a memory computing resource to perform recomputation; recomputation performance requirements are used to characterize the performance required for a memory computing resource to perform recomputation.
[0072] Based on cluster information and model feature information, the recomputation time information and recomputation performance requirements corresponding to each heterogeneous in-memory computing resource can be determined.
[0073] Understandably, before determining the recomputation training strategy, it is not determined whether each computing unit needs to be recomputed. Therefore, a recomputation variable can be set, which corresponds to two different values. One value is used to indicate that the corresponding storage unit needs to be recomputed, and the other value is used to indicate that the corresponding storage unit does not need to be recomputed.
[0074] The computation time and performance requirements for recomputation correspond to the computational units that need to be recomputed. When the computational units that need to be recomputed are different, the recomputation time information and recomputation performance requirements are different for different computational resources.
[0075] Step 103: Determine the recomputation time information characteristic value if the recomputation performance requirement does not exceed the performance limit of the storage resources.
[0076] This performance limit is a limit related to the storage capacity of in-memory computing resources. One of the objectives of this application's embodiments is to determine a recomputation training strategy for training a model to be trained. As can be seen from the above, this application's embodiments aim to balance the computational and storage capabilities of in-memory computing resources. It is understood that for activation tensors that need to be reused, two methods can be used: recomputation or storage. Recomputation requires additional computation time compared to storage, while storage requires more storage capacity than recomputation. To avoid errors occurring during training when the storage capacity required by in-memory computing resources exceeds the storage capacity of the in-memory computing resources themselves, this approach is necessary.
[0077] Step 104: Determine the recomputation training strategy for the model to be trained based on the recomputation time information feature values corresponding to each storage resource.
[0078] Given the recomputation time feature value, the target computation unit (the target computation unit is the computation unit that needs to be recomputed) can be identified, and the recomputation training strategy corresponding to the model to be trained can be determined.
[0079] For example, storage resource 0 corresponds to computing units 0-99. Given the recomputation time characteristic value of storage resource 0, computing units 0-50 are identified as target computing units, and computing units 51-99 as non-target computing units. Therefore, the recomputation training strategy for storage resource 1 can be determined as follows: when training the model, recomputation is performed on computing units 0-50, but not on computing units 51-99.
[0080] Since different computing units correspond to different in-memory resources, the recomputation training strategies for different in-memory resources are different. For example, the number of target units in the recomputation training strategies for different in-memory resources is different.
[0081] As an example, the recomputation time information feature value can be the minimum recomputation information value. That is, under the condition that the recomputation performance requirement is not greater than the performance limit of the storage resources, the minimum recomputation time information value is determined, the target computing unit corresponding to the minimum recomputation time information value is determined, and then the recomputation training strategy is determined.
[0082] In this embodiment, the cluster information of the heterogeneous cluster and the model feature information of the model to be trained are determined. For each in-memory resource in the heterogeneous cluster, the recomputation time information and recomputation performance requirements are determined based on the cluster information and model feature information. The recomputation time information is used to characterize the time required for the computing unit corresponding to the same in-memory resource to perform recomputation. The computing unit corresponds to the activation tensor in the model to be trained. When the recomputation performance requirement is not greater than the performance limit of the in-memory resource, the recomputation time information feature value is determined. Based on the recomputation time information feature value corresponding to each in-memory resource, the recomputation training strategy for the model to be trained is determined. For scenarios that require model training using a heterogeneous cluster, the smallest granularity of recomputation computing units can be divided for the neural network layers corresponding to each in-memory resource in the heterogeneous cluster. The recomputation training strategy for training the model for each in-memory resource is determined by the recomputation time information and recomputation performance requirements, that is, the computing units that need to be recomputed are determined. Different in-memory resources correspond to different recomputation training strategies, so that the recomputation training strategy can balance the performance of the corresponding in-memory resources, improve the utilization rate of the heterogeneous cluster, and thus provide the throughput of the training model and improve the training efficiency of the model.
[0083] In some implementations of this application, the model feature information includes the segmentation information of the neural network layers in the model to be trained; the cluster information includes the neural network layers and performance information corresponding to each of the computing resources; the recomputation time information is determined by the following steps: according to the corresponding neural network layer, determine the time estimate corresponding to each computing unit based on the performance information; obtain the recomputation variable corresponding to each computing unit; the recomputation variable is used to identify whether the corresponding computing unit enables recomputation; determine the recomputation time information based on the sum of the products of each time estimate and the corresponding recomputation variable.
[0084] The segmentation information is used to characterize the neural network layers executed by different in-memory resources when training the model to be trained. Based on the segmentation information, one or more neural network layers corresponding to each in-memory resource can be determined.
[0085] For each computing unit corresponding to a storage resource, the estimated computation time is determined based on the performance information of the storage resource. As an example, the execution time of some key operators can be analyzed offline, and the estimation of computing units of different matrix sizes can be obtained by scaling the computational load using the proportional relationship between the computational load and the matrix dimension.
[0086] Since different storage resources correspond to different neural network layers, and different neural network layers contain different computing units, it is necessary to obtain multiple recomputable variables based on the corresponding neural network layer for each storage resource.
[0087] The recalculation variable is used to characterize whether the computing unit is recalculated. The value of the recalculation variable can be 0 or 1. "0" means that no recalculation is required, and "1" means that recalculation is required (that is, the computing unit corresponding to the recalculation variable "1" is the target computing unit). The recalculation time information is determined as the sum of the products of each time estimate and the corresponding recalculation variable.
[0088] In some implementations of this application, the recalculation time information is Time(R);
[0089]
[0090] in, For computing unit Time estimate, For computing unit Recalculated variables, This is the identifier for the computation unit (i.e., the smallest recomputable computation unit that generates the activation tensor).
[0091] The identifier of the computing unit is , Used to distinguish different technical units. The time estimate and recalculation variable corresponding to the same computing unit can be determined by "i". For example, t1 and t2 are the time estimates for computing units u1 and u2, respectively, and r1 and r2 are the recalculation variables for computing units u1 and u2, respectively.
[0092] In some implementations of this application, the cluster information includes the total number of stages in the training pipeline set for the model to be trained, and the stage values corresponding to different computing resources, wherein the stage values are used to characterize the stage in the training pipeline; the model feature information includes the storage requirements and fixed storage consumption corresponding to each computing unit; the recomputation performance requirements are determined by the following steps: determining the storage requirement information of each neural network layer based on the recomputation variables and storage requirements corresponding to each computing unit; the storage requirement information is used to characterize the storage requirements of computing units that do not perform recomputation; determining the difference between the total number of stages and the stage information corresponding to the computing resources as the stage difference value; and summing the product of the storage requirement information and the stage difference value with the fixed storage consumption to obtain the recomputation performance requirements.
[0093] A training pipeline can be configured, and the total number of stages can be determined based on the in-memory computing resources in the heterogeneous cluster and actual needs. The total number of stages represents the number of in-memory computing resources used to train the model to be trained. After determining the total number of stages and the expected in-memory computing resources, the total number of stages is determined for each in-memory computing resource. The stage value starts from 0 and increments by 1. For example, if 3 GPUs are needed to train the model to be trained, the total number of stages is 3. In the training pipeline, the stages of the 3 GPUs are ordered as GPU-A, GPU-B, and GPU-C, respectively. Then, the stage value of GPU-A is 0, the stage value of GPU-B is 1, and the stage value of GPU-C is 2.
[0094] Storage requirement refers to the storage space needed for the activation tensor corresponding to the storage computation unit. Since different computation units correspond to different activation tensors, the storage requirements for different computation units are different. As an example, storage requirement = activation tensor shape (batch size × sequence length × hidden layer dimension, etc.) × data type bytes (e.g., 2 bytes / element for FP16).
[0095] Fixed storage consumption refers to a fixed storage capacity that does not change with the recalculation training strategy. As an example, fixed storage consumption = basic fixed overhead (weights + gradients + optimizer states) + runtime framework (i.e., model training framework) + model input data.
[0096] The recomputation performance requirements are determined based on the total number of stages, the stage values of each in-memory computing resource, the fixed storage consumption, and the storage requirements of each computing unit. Specifically, the storage requirement information is determined based on the recomputation variables and storage requirements corresponding to each computing unit; the difference between the total number of stages and the stage information corresponding to the in-memory computing resource is determined as the stage difference; the product of the storage requirement information and the stage difference is calculated, and the sum of this product and the fixed storage consumption is calculated as the recomputation performance requirement.
[0097] Recomputation performance requirements are used to characterize the storage performance requirements (i.e., storage capacity requirements) of each in-memory resource under different recomputation variables. Storage requirement information is determined based on the computing units corresponding to the same in-memory resource. Since different in-memory resources may correspond to different neural network layers, and different neural network layers may have different numbers of computing units, the storage requirement information for a given in-memory resource is determined based on the computing units of all neural network layers corresponding to that resource.
[0098] In some other implementations of this application, the recomputation performance requirement is Mem(R);
[0099]
[0100] Where Const is the fixed storage consumption, P is the total number of stages, and S is the stage value. For computing unit Storage requirements for the generated activation tensor For computing unit Recalculated variables, For the storage requirements mentioned above, n represents the computing unit corresponding to the same storage resource. quantity.
[0101] (PS) represents the stage difference mentioned above. It should be noted that in a parallel processing scenario of the training pipeline, the different activation tensor fractions to be stored at each stage of the training pipeline will lead to different storage requirements for in-memory resources. In this scenario, when a heterogeneous cluster processes multiple micro-batches simultaneously, this embodiment adopts a 1F1B (One-Forward-One-Backward) scheduling strategy. Before backpropagation begins, multiple activation copies corresponding to all micro-batches that have not yet completed backpropagation must be retained, and each activation copy is of the same size. The number of copies of storage requirement information stored by in-memory resources corresponding to different pipeline stages is the stage difference. To address the different storage performance pressures on in-memory resources corresponding to different stages due to different stage differences, this embodiment can adjust the partitioning information (adjusting the neural network layers processed by different in-memory resources) and adaptively adjust the computational units that do not require recomputation, i.e., (1- It balances the heavy computing performance requirements at different stages to make it compatible with its own storage performance.
[0102] In some implementations of this application, the performance limit includes storage capacity; step 104 includes: performing linear programming with the recomputation performance requirement not exceeding the storage capacity of the in-memory computing resources as a constraint, and minimizing the recomputation time information as the objective, and determining the linear programming result; determining the linear programming result as a time information feature value; wherein the time information feature value includes the parameter values of the recomputation variables corresponding to each computing unit corresponding to the in-memory computing resources.
[0103] Since different in-memory computing resources correspond to different computing units, it is necessary to perform linear programming for each in-memory computing resource, with the constraint that the recomputation performance should not exceed the storage capacity of the in-memory computing resource, and with the objective of minimizing the recomputation time information. The recomputation time information corresponding to the linear programming result is then determined as the time information feature value. The time information feature value includes the parameter value of the recomputation variable corresponding to each computing unit of the in-memory computing resource. The parameter value can be either "0" or "1" as mentioned above, corresponding to the recomputation variable.
[0104] As an example, combining the recomputation time information TIME(R) and the recomputation performance requirement Mem(R), an optimization model is constructed with the objective of minimizing TIME(R) and the constraint that Mem(R) is no greater than the storage capacity:
[0105]
[0106]
[0107] st (subject to) represents the constraint condition. For storage and computing resource identification, For storage resources Storage capacity.
[0108] Linear programming is performed based on the optimization model to obtain the linear programming result, thereby obtaining the minimum recomputation time information under the condition of satisfying storage capacity, and then obtaining the parameter values of each recomputation variable under the condition of the minimum recomputation time information.
[0109] In some implementations of the embodiments of this application, the recomputation training strategy generation method further includes: determining the execution time of each in-memory resource based on the recomputation training strategy; when the difference between the execution times meets a preset condition, adjusting the number of neural network layers corresponding to at least two in-memory resources, and updating the recomputation training strategy.
[0110] Based on empirical data and static performance information of each storage resource, the partitioning information for the neural network layers corresponding to each storage resource can be determined, or the partitioning information for the number of neural network layers corresponding to each storage resource can be distributed as evenly as possible. After obtaining the recomputation training strategy, it is verified whether the partitioning of model layers is the optimal partitioning scheme.
[0111] Specifically, the latest recomputation training strategy can be adopted, calling upon various in-memory computing resources to process a certain amount of model input data to determine the execution time of each in-memory computing resource. When the difference between the execution times meets a preset condition (e.g., the difference between the maximum and minimum execution times is greater than a preset threshold), the number of neural network layers corresponding to at least two in-memory computing resources is adjusted. As can be seen from the above, since the recomputation time of each in-memory computing resource is at its minimum when the corresponding recomputation training strategy is adopted, the in-memory computing resource corresponding to the maximum execution time is the training bottleneck of the current recomputation training strategy. Because its execution time is significantly greater than that of other in-memory computing resources, the recomputation time corresponding to different in-memory computing resources can be balanced by adjusting the number of neural network layers corresponding to at least two in-memory computing resources. This balances the load of different in-memory computing resources, achieving finer-grained load balancing and dynamic neural network layer partitioning, thereby improving the throughput and utilization of model training.
[0112] After adjusting the neural network layers corresponding to different computing resources, steps 101-104 above can be repeated to iteratively update the recomputation training strategy until the iteration conditions are met. The iteration conditions include the failure to meet preset conditions between the various execution times and the number of times the recomputation training strategy is updated reaching a preset number.
[0113] For example, the model to be trained contains a 12-layer neural network, divided into 3 pipeline stages (Stage 0 / 1 / 2). The GPU (in-memory computing resources) computing power corresponding to the 3 pipeline stages is weak, medium, and strong, respectively. The initial partitioning of the 12-layer neural network is uniform, i.e., [4, 4, 4]. Through performance evaluation (determining the execution time of each in-memory resource), Stage 0 is found to be the bottleneck. In the iteration, the algorithm migrates layer 1 from Stage 0 to Stage 2, forming a partitioning of [3, 4, 5]. After re-evaluation, the maximum computation time of each stage is closer, and there is no obvious bottleneck. The algorithm determines that it has converged and outputs the partitioning result of this layer.
[0114] In some implementations of the embodiments of this application, the recomputation training strategy generation method further includes: acquiring training data; and calling the heterogeneous cluster to perform parallel training on the model to be trained based on the cluster information, the model feature information, the recomputation training strategy, and the training data.
[0115] After determining the recomputation training strategy, a heterogeneous cluster can be used to train the model to be trained. Based on actual needs, the training data for model training is determined. The heterogeneous cluster is then invoked to perform parallel training of the model to be trained, using cluster information, model feature information, the recomputation training strategy, and the training data. This allows different computing resources to execute according to their respective recomputation training strategies when training the model using a heterogeneous cluster, achieving a balance between the computational and storage performance of each computing resource and improving training efficiency.
[0116] Reference Figure 4 This diagram illustrates a heterogeneous sensing adaptive recomputation system provided in an embodiment of this application. Based on the above method embodiment, a system such as... Figure 4 The heterogeneous awareness adaptive recomputation system (software system) shown is suitable for heterogeneous GPU clusters composed of GPUs. The system includes:
[0117] The training configuration input module 401 is used to configure various parameters related to training, including training cluster topology, parallel dimension, data batch size, optimizer parameters, mixed precision training, etc.
[0118] The heterogeneous performance acquisition module 402 is used to systematically acquire and collect key performance indicator data generated during the training of large-scale models in a heterogeneous GPU environment. It provides support for customizing personalized recomputation strategies for each GPU. This module covers two dimensions: storage and computation. Specifically, it includes static hardware parameters such as GPU memory capacity and memory bandwidth, as well as dynamic computational performance obtained through evaluation of actual operator execution processes, such as the computational efficiency of matrix multiplication, attention mechanisms, activation functions, and normalization operations. Based on the operator execution status of each layer of the model and combined with the current training topology, the system can construct an accurate performance evaluation model for each GPU (the accurate performance evaluation model is used to evaluate the GPU's recomputation performance, including the aforementioned recomputation time information TIME(R) and the recomputation performance requirement Mem(R)).
[0119] The adaptive recomputation strategy search module 403 is used in heterogeneous GPU clusters to automatically search for the optimal recomputation strategy based on the computing power and memory capacity of each GPU, minimizing the overall recomputation overhead under memory constraints. This module divides the model into fine-grained computational units and constructs an optimization model (including the aforementioned precise performance evaluation model and the constraints of each GPU) with the objective of minimizing recomputation time and memory constraints. It adaptively allocates activation retention and recomputation strategies in the search space using integer linear programming (ILP) or heuristic search methods. Unlike homogeneous methods with a unified strategy, this module can generate customized recomputation configurations for each GPU, thereby achieving efficient collaboration of computing resources in a heterogeneous environment and improving the throughput and stability of the training system.
[0120] The heterogeneous parallel partitioning strategy search module 404 is used to jointly optimize activation tensor, sequence, and pipeline parallel partitioning strategies based on the FLOPS, memory, and bandwidth of each GPU. Through heuristic algorithms and strategy table construction, the module achieves fine-grained load balancing and dynamically adjusts layer partitioning to improve the system throughput and resource utilization of heterogeneous training.
[0121] Based on the differences in memory pressure and performance bottlenecks faced by different GPUs at various stages of the pipeline, this system searches for heterogeneous recomputation strategies to ultimately maximize system throughput under memory constraints.
[0122] The following experiment further illustrates the embodiments of this application. In this experiment, a large model training performance simulator that supports heterogeneous GPUs is used to model the differences in computing power, memory, and bandwidth of heterogeneous GPUs. It supports simulation of computation, storage, and communication. The simulator can analyze the computation and waiting time of each GPU in detail and output key indicators such as peak memory usage and training latency, which significantly improves the efficiency of training strategy evaluation, with low prediction error and reliable accuracy.
[0123] This experiment selects a full recalculation strategy as comparison strategy 1 and an existing adaptive recalculation strategy as comparison strategy 2. The recalculation training strategy obtained according to the embodiments of this application is used as the target strategy. Strategy 1 only retains the activation values of each layer boundary, discarding all other activations. It requires recalculating the entire forward process before backpropagation at each layer, and all GPUs perform the same recalculation operation. Strategy 2 supports selective recalculation between different pipeline stages, but it cannot customize different strategies for GPUs within the same Tensor Parallelism (TP) group, making it difficult to adapt to heterogeneous training requirements.
[0124] The evaluation metric used is training throughput (the number of tokens trained per second).
[0125] Experiments were conducted using a large model training performance simulator and three large models of different sizes: Model 1 (1.8 billion model parameters), Model 2 (6 billion model parameters), and Model 3 (13.7 billion model parameters).
[0126] Reference Figure 5 The diagram illustrates a comparative model training experiment provided in an embodiment of this application. The experimental results are as follows: Figure 5 As shown, by Figure 5As can be seen, the recomputation training strategy obtained in this application embodiment achieves optimal training throughput under various model loads, with an average improvement of 16% compared to comparison strategy 2 and 27% compared to comparison strategy 1. The core advantage of the target strategy is that it supports the activation of heterogeneous GPUs within the TP group to use differentiated recomputation strategies. The recomputation intensity can be flexibly adjusted according to the performance of each GPU, avoiding the "bottleneck effect" caused by uniform configuration, and significantly improving system efficiency and throughput performance.
[0127] It should be noted that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0128] Reference Figure 6 This diagram illustrates a recomputation training strategy generation device provided in an embodiment of this application, located in a heterogeneous cluster. Specifically, it may include: an information acquisition module 601, a recomputation time information and performance requirement determination module 602, a recomputation time information feature value determination module 603, and a recomputation training strategy generation module 604, wherein:
[0129] The information acquisition module 601 is used to determine the cluster information of the heterogeneous cluster and the model feature information of the model to be trained.
[0130] The recomputation time information and performance requirement determination module 602 is used to determine the recomputation time information and recomputation performance requirements for each of the in-memory computing resources based on the cluster information and the model feature information; the recomputation time information is used to characterize the time required for the computing unit corresponding to the same in-memory computing resource to perform recomputation; the computing unit corresponds to the activation tensor in the model to be trained;
[0131] The recomputation time information feature value determination module 603 is used to determine the recomputation time information feature value when the recomputation performance requirement is not greater than the performance limit of the storage and computing resources.
[0132] The recomputation training strategy generation module 604 is used to determine the recomputation training strategy for the model to be trained based on the recomputation time information feature values corresponding to each computing resource.
[0133] In some implementations of this application, the model feature information includes the segmentation information of the neural network layers in the model to be trained; the cluster information includes the neural network layers and performance information corresponding to each of the computing resources; the recomputation time information is determined by the following sub-modules:
[0134] The time estimation determination submodule is used to determine the time estimation value corresponding to each computing unit according to the performance information based on the corresponding neural network layer.
[0135] The recalculation variable determination submodule is used to obtain the recalculation variables corresponding to each of the computing units; the recalculation variables are used to identify whether the corresponding computing unit enables recalculation.
[0136] The recalculation time information determination submodule is used to determine the recalculation time information based on the sum of the products of each of the time estimates and the corresponding recalculation variables.
[0137] In some implementations of this application, the cluster information includes the total number of stages in the training pipeline set for the model to be trained, and the stage values corresponding to different computing resources, wherein the stage values are used to characterize the stage in the training pipeline; the model feature information includes the storage requirements and fixed storage consumption corresponding to each computing unit; the recomputation performance requirements are determined by the following sub-modules:
[0138] The storage requirement information determination submodule is used to determine the storage requirement information of each neural network layer based on the recomputation variables and storage requirements corresponding to each computing unit; the storage requirement information is used to characterize the storage requirements of computing units that do not perform recomputation.
[0139] The stage difference determination submodule is used to determine the difference between the total number of stages and the stage information corresponding to the storage resources as the stage difference;
[0140] The recomputation performance requirement determination submodule is used to calculate the recomputation performance requirement by summing the product of the storage requirement information and the stage difference with the fixed storage consumption.
[0141] In some implementations of this application, the performance limit includes storage capacity; the recomputation time information feature value determination module 603 includes:
[0142] The linear programming execution submodule is used to perform linear programming with the constraint that the recomputation performance requirement is no greater than the storage capacity of the computing resources and the objective of minimizing the recomputation time information, and to determine the linear programming result.
[0143] The time information feature value determination submodule is used to determine the linear programming result as time information feature values;
[0144] The time information feature value includes the parameter values of the recomputation variables corresponding to each computing unit of the storage and computing resources.
[0145] In some implementations of the embodiments of this application, the apparatus further includes:
[0146] The execution time determination module is used to determine the execution time of each computing resource based on the recomputation training strategy.
[0147] The recomputation training strategy update module is used to adjust the number of neural network layers corresponding to at least two storage resources and update the recomputation training strategy when the difference between the execution times meets a preset condition.
[0148] In some implementations of this application, the recalculation time information is Time(R);
[0149]
[0150] in, For computing unit Time estimate, For computing unit Recalculated variables, This serves as the identifier for the computing unit.
[0151] In some implementations of this application, the recomputation performance requirement is Mem(R);
[0152]
[0153] Where Const is the fixed storage consumption, P is the total number of stages, and S is the stage value. For computing unit Storage requirements for the generated activation tensor For computing unit The recalculated variables.
[0154] In some implementations of the embodiments of this application, the apparatus method includes:
[0155] The training data acquisition module is used to acquire training data;
[0156] The training module is used to call the heterogeneous cluster to perform parallel training on the model to be trained based on the cluster information, the model feature information, the recomputation training strategy, and the training data.
[0157] This application provides a recomputation training strategy generation device. By using this device, the steps in the aforementioned method embodiments can be implemented.
[0158] As the apparatus embodiments are basically similar to the method embodiments, they are described in a relatively simple manner. For relevant details, please refer to the description in the method embodiment section.
[0159] Reference Figure 7 The diagram illustrates an electronic device according to an embodiment of this application. Figure 7As shown, the electronic device 700 in this embodiment includes: a processor 710, a memory 720, and a computer program 721 stored in the memory 720 and executable on the processor 710. When the processor 710 executes the computer program 721, it implements the steps in the various embodiments of the recomputation training strategy generation method described above, for example... Figure 1 Steps 101 to 104 are shown. Alternatively, when the processor 710 executes the computer program 721, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 6 The functions of modules 601 to 604 are shown.
[0160] For example, the computer program 721 can be divided into one or more modules / units, which are stored in the memory 720 and executed by the processor 710 to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which can be used to describe the execution process of the computer program 721 in the electronic device 700. For example, the computer program 721 can be divided into an information acquisition module, a recomputation time information and performance requirement determination module, a recomputation time information feature value determination module, and a recomputation training strategy generation module, with the specific functions of each module as follows:
[0161] The information acquisition module is used to determine the cluster information of the heterogeneous cluster and the model feature information of the model to be trained;
[0162] The recomputation time information and performance requirement determination module is used to determine the recomputation time information and recomputation performance requirements for each of the aforementioned in-memory resources, based on the cluster information and the model feature information; the recomputation time information is used to characterize the time required for a computing unit corresponding to the same in-memory resource to perform recomputation; the computing unit corresponds to the activation tensor in the model to be trained;
[0163] The recomputation time information feature value determination module is used to determine the recomputation time information feature value when the recomputation performance requirement is not greater than the performance limit of the storage and computing resources.
[0164] The recomputation training strategy generation module is used to determine the recomputation training strategy for the model to be trained based on the recomputation time information feature values corresponding to each computing resource.
[0165] The electronic device 700 can be a desktop computer, cloud server, or other computing device. The electronic device 700 may include, but is not limited to, a processor 710 and a memory 720. Those skilled in the art will understand that... Figure 7This is merely one example of electronic device 700 and does not constitute a limitation on electronic device 700. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device 700 may also include input / output devices, network access devices, buses, etc.
[0166] The processor 710 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.
[0167] The memory 720 can be an internal storage unit of the electronic device 700, such as a hard disk or memory of the electronic device 700. The memory 720 can also be an external storage device of the electronic device 700, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the electronic device 700. Furthermore, the memory 720 can include both internal and external storage units of the electronic device 700. The memory 720 is used to store the computer program 721 and other programs and data required by the electronic device 700. The memory 720 can also be used to temporarily store data that has been output or will be output.
[0168] This application also discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the recomputation training strategy generation method as described in the foregoing embodiments.
[0169] This application also discloses a computer program product, including a computer program, which, when run, causes the recomputation training strategy generation method as described in the foregoing embodiments to be executed.
[0170] The embodiments described above are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.
Claims
1. A method for generating recomputable training strategies, characterized in that, Applied to heterogeneous clusters; the method includes: The cluster information of the heterogeneous cluster and the model feature information of the model to be trained are determined; the model to be trained includes multiple neural network layers, and each neural network layer is provided with multiple computing units; For each computing resource in the heterogeneous cluster, based on the cluster information and the model feature information, the recomputation time information and recomputation performance requirements are determined; the recomputation time information is used to characterize the time required for the computing unit corresponding to the same computing resource to perform recomputation; the computing unit corresponds to the activation tensor in the model to be trained; If the recomputation performance requirement is not greater than the performance limit of the storage resources, determine the recomputation time information characteristic value; Based on the recomputation time information feature values corresponding to each computing resource, a recomputation training strategy is determined for the model to be trained. The cluster information includes the total number of stages in the training pipeline set for the model to be trained, and the stage values corresponding to different computing resources, wherein the stage values are used to characterize the stage in the training pipeline; the model feature information includes the storage requirements and fixed storage consumption corresponding to each computing unit; the recomputation performance requirements are determined through the following steps: Based on the recomputation variables and storage requirements corresponding to each computing unit, the storage requirement information of each neural network layer is determined; the storage requirement information is used to characterize the storage requirements of computing units that do not perform recomputation; the recomputation variables are used to identify whether the corresponding computing unit enables recomputation. The difference between the total number of stages and the corresponding stage value of the storage resource is defined as the stage difference. The recalculation performance requirement is calculated by multiplying the storage requirement information and the stage difference, and summing the product with the fixed storage consumption.
2. The method according to claim 1, characterized in that, The model feature information includes the segmentation information of the neural network layers in the model to be trained; the cluster information includes the neural network layers and performance information corresponding to each of the computing resources; the recomputation time information is determined through the following steps: Based on the performance information, determine the time estimate for each computing unit according to the corresponding neural network layer. Obtain the recalculated variables corresponding to each of the aforementioned computing units; The recalculation time information is determined by summing the products of each of the time estimates and the corresponding recalculated variables.
3. The method according to claim 1, characterized in that, The performance limit includes storage capacity; determining the recomputation time information feature value when the recomputation performance requirement is not greater than the performance limit of the storage resources includes: With the requirement that the recomputation performance should not exceed the storage capacity of the in-memory computing resources as a constraint, and with the goal of minimizing the recomputation time information, linear programming is performed to determine the linear programming result; The linear programming result is determined as a time information feature value; The time information feature value includes the parameter values of the recomputation variables corresponding to each computing unit of the storage and computing resources.
4. The method according to claim 1 or 3, characterized in that, The method further includes: Based on the aforementioned recomputation training strategy, the execution time of each computing resource is determined; When the difference between the execution times meets the preset conditions, the number of neural network layers corresponding to at least two storage resources is adjusted, and the recomputation training strategy is updated.
5. The method according to claim 2, characterized in that, The recalculation time information is Time(R); in, For computing unit Time estimate, For computing unit Recalculated variables, This serves as the identifier for the computing unit.
6. The method according to claim 1, characterized in that, The recomputation performance requirement is Mem(R); Where Const is the fixed storage consumption, P is the total number of stages, and S is the stage value. For computing unit Storage requirements for the generated activation tensor For computing unit The recalculated variables.
7. The method according to any one of claims 1-3, characterized in that, The method further includes: Obtain training data; The heterogeneous cluster is invoked to perform parallel training on the model to be trained based on the cluster information, the model feature information, the recomputation training strategy, and the training data.
8. An electronic device, characterized in that, The device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device performs the method as described in any one of claims 1-7.
9. A computer program product, characterized in that, Includes a computer program, which, when run, causes the method as described in any one of claims 1-7 to be performed.