Model training methods, devices, equipment, and media based on dynamic recomputation scheduling

Abstract

This application discloses a model training method, apparatus, device, and medium based on dynamic recomputation scheduling. The method includes: obtaining the current training strategy of the model to be trained; the current training strategy includes the micro-batch data size and a recomputation strategy used to indicate the scheduling status of the recomputation component; based on the current training strategy, after training the model to be trained for a preset number of rounds, obtaining memory usage information, recomputation benefit information, and training performance information; determining whether the memory usage information meets preset conditions; if not, adjusting the current training strategy according to the recomputation benefit information to obtain a new current training strategy, and then returning to continue training for the preset number of rounds. If yes, based on the training performance information, and based on the target training strategy that meets the preset conditions and has the highest training efficiency, continuing iterative training until the preset iteration conditions are met. Thus, by introducing a dynamic recomputation adjustment mechanism, a dynamic balance between memory pressure and computational overhead is achieved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a model training method, apparatus, device and medium based on dynamic recomputation scheduling. Background Technology

[0002] With the rapid development of large language models (LLMs), the number of model parameters has expanded to hundreds of billions or even trillions in distributed training, which has placed unprecedented demands on the memory capacity of computing accelerator cards (GPUs).

[0003] In distributed training under a pipelined parallel strategy, significant differences exist in the model's layer structure, parameter count, and computational complexity, making it difficult to achieve a balanced allocation of computing resources and flexible distribution of GPU memory. To alleviate insufficient GPU memory, existing systems often introduce recomputation mechanisms, trading additional computation time for GPU memory space.

[0004] However, introducing recomputation mechanisms can lead to significant time overhead if over-reliance on recomputation occurs, reducing model training efficiency. Conversely, limiting recomputation to reduce computation time can exacerbate memory pressure and restrict model scaling.

[0005] Therefore, how to improve the utilization of video memory while taking into account computational overhead and ensuring model training efficiency is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of this, one aspect of this application provides a model training method based on dynamic recomputation scheduling, the method comprising: Obtain the current training strategy of the model to be trained; the current training strategy includes the micro-batch data size and the recomputation strategy; the recomputation strategy is used to indicate the scheduling status of the recomputation component; Based on the current training strategy, the model to be trained is trained for a preset number of rounds; After the preset number of training rounds, memory usage information, recomputation benefit information, and training performance information are obtained; the recomputation benefit information is used to characterize the degree of memory saving and computation time consumption after the recomputation component is scheduled; the training performance information is used to characterize the training efficiency of the model to be trained under the current training strategy. Determine whether the video memory usage information meets preset conditions; If not, adjust the current training strategy according to the recalculation benefit information to obtain a new current training strategy, and return to the step of training the model to be trained for a preset number of rounds based on the current training strategy; If so, based on the training performance information, the strategy with the highest training efficiency is determined from the current training strategies that meet the preset conditions as the target training strategy; and based on the target training strategy, the model to be trained is iteratively trained until the preset iteration conditions are met.

[0007] Optionally, obtaining the current training strategy of the model to be trained includes: Obtain the model structure and parallel strategy of the model to be trained; Based on the model structure and the parallel strategy, determine the model layer allocation information of the model to be trained in each parallel stage; Based on the model layer allocation information, the micro-batch data size and the recomputation strategy are initialized to obtain the current training strategy; wherein, the initialized micro-batch data size is a preset initial value, and the initialized recomputation strategy is used to indicate that each of the recomputation components is in the enabled state.

[0008] Optionally, the model structure includes a multi-layer Transformer structure; the recomputation components include a core attention component, a multi-head attention component, a feedforward network component, and a layer normalization component; The recomputation benefit information includes the memory savings, recomputation time, and recomputation benefit value corresponding to each recomputation component; the recomputation benefit value is used to characterize the memory savings of each recomputation component per unit recomputation time.

[0009] Optionally, after the preset number of training rounds, recalculation benefit information is obtained, including: Based on the model layer allocation information, determine the number of Transformer layers allocated in the parallel phase corresponding to the current training strategy; Determine the amount of single-layer memory saved by the recomputation component in a single-layer Transformer structure; The amount of memory saved is determined based on the amount of memory saved per single layer and the number of Transformer layers; During the reverse computation process of the preset number of training rounds, the average recomputation time introduced when each of the recomputation components is scheduled is statistically analyzed to obtain the recomputation time. Based on the amount of video memory saved and the time spent on recomputation, the efficiency reduction of video memory per unit time is determined, and the benefit value of recomputation is obtained.

[0010] Optionally, the video memory usage information includes the average video memory usage ratio; after the preset number of training rounds, the video memory usage information is obtained, including: Obtain the available video memory of the computing accelerator card; After each training iteration of the preset number of training rounds, the actual memory usage of the computing acceleration card is obtained; wherein, the actual memory usage includes the memory usage of model parameters, the memory usage of gradients, the memory usage of optimizer states, and the memory usage of activation values. The average video memory usage ratio is determined based on the actual video memory usage and the available video memory.

[0011] Optionally, the preset condition includes that the absolute value of the difference between the average video memory usage ratio and the preset video memory threshold is less than a preset value; wherein, the preset video memory threshold is less than the upper limit of the video memory usage ratio that represents video memory removal.

[0012] Optionally, the current training strategy may be adjusted based on the recalculation benefit information, including: If the average video memory usage ratio is less than or equal to a specified multiple of the preset video memory threshold, the size of the micro-batch data is increased while maintaining the current recalculation strategy; the specified multiple is less than 1. If the average video memory usage ratio is greater than a specified multiple of the preset video memory threshold, the following steps are performed: Increase the size of the micro-batch data; Based on the recalculation revenue information, the target recalculation component is determined; The scheduling state of the target recompile component is adjusted to reduce the actual video memory usage, resulting in an adjusted recompile strategy.

[0013] Optionally, increasing the size of the micro-batch data includes: If the average video memory usage ratio is less than or equal to a specified multiple of the preset video memory threshold, the product of the micro-batch data size and the preset multiple is rounded up, and the preset multiple is greater than 1. If the average video memory usage ratio is greater than a specified multiple of the preset video memory threshold, the size of the micro-batch data is summed with the preset increment.

[0014] Another aspect of this application provides a model training apparatus based on dynamic recomputation scheduling, the apparatus comprising: The strategy acquisition module is used to acquire the current training strategy of the model to be trained; the current training strategy includes the micro-batch data size and the recomputation strategy; the recomputation strategy is used to indicate the scheduling status of the recomputation component. An iterative training module is used to train the model to be trained for a preset number of rounds based on the current training strategy; The information acquisition module is used to acquire memory usage information, recomputation benefit information, and training performance information after the preset number of training rounds; the recomputation benefit information is used to characterize the degree of memory saving and computation time consumption after the recomputation component is scheduled; the training performance information is used to characterize the training efficiency of the model to be trained under the current training strategy. The condition determination module is used to determine whether the video memory usage information meets preset conditions; if not, the strategy adjustment module is called; if yes, the target processing module is called. The strategy adjustment module is used to adjust the current training strategy according to the recalculation benefit information to obtain a new current training strategy, and return to the step of training the model to be trained for a preset number of rounds based on the current training strategy. The target processing module is used to determine the strategy with the highest training efficiency as the target training strategy from the current training strategies that meet the preset conditions according to the training performance information; and to continue iteratively training the model to be trained until the preset iteration conditions are met based on the target training strategy.

[0015] Another aspect of this application provides an electronic device including a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor executes the computer program to implement the steps of the model training method based on dynamic recomputation scheduling.

[0016] Another aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the model training method based on dynamic recomputation scheduling.

[0017] The model training method, apparatus, device, and medium based on dynamic recomputation scheduling provided in this application have the following beneficial effects: Real-time monitoring of memory usage, recomputation benefits, and training performance during model training; and adaptive balance between memory pressure and computational overhead through the introduction of a dynamic recomputation adjustment mechanism. This avoids the dilemma of "excessive recomputation leading to decreased training efficiency" or "limited recomputation leading to insufficient memory" caused by fixed recomputation strategies. Furthermore, in multiple rounds of iterative training with a preset number of rounds, the final target strategy is selected from strategies that meet preset conditions, ensuring maximum training efficiency while maintaining memory safety, thereby improving resource utilization efficiency and model convergence speed in distributed training. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating a model training method based on dynamic recalculation scheduling provided in an embodiment of this application.

[0019] Figure 2This is a schematic diagram of a training framework provided in an embodiment of this application.

[0020] Figure 3 This is a schematic diagram illustrating the principle of a model training method based on dynamic recomputation scheduling provided in an embodiment of this application.

[0021] Figure 4 This is a schematic diagram illustrating the model training effect under different recomputation scheduling strategies provided in an embodiment of this application.

[0022] Figure 5 This is a schematic diagram illustrating the model training effect under different recomputation scheduling strategies, as provided in another embodiment of this application.

[0023] Figure 6 This is a schematic diagram of a model training device based on dynamic recalculation scheduling provided in an embodiment of this application.

[0024] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0025] The attached diagram is labeled as follows: 20 is the configuration parsing module, 21 is the model training module, 22 is the video memory analysis module, 23 is the performance tuning module, 24 is the recomputation scheduling module, 60 is the strategy acquisition module, 61 is the iterative training module, 62 is the information acquisition module, 63 is the condition determination module, 64 is the strategy adjustment module, 65 is the target processing module, 70 is the memory, 71 is the processor, 72 is the display screen, 73 is the input / output interface, 74 is the communication interface, 75 is the power supply, 76 is the communication bus, 701 is the computer program, 702 is the operating system, and 703 is the data. Detailed Implementation

[0026] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0027] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0028] Figure 1 The flowchart of a model training method based on dynamic recomputation scheduling provided in this application embodiment is as follows: It can be understood that in the training of large language models (LLM), it is necessary to improve the training throughput under limited GPU memory conditions and avoid memory waste or excessive computation time caused by static recomputation configuration. Therefore, it is necessary to perform joint collection of memory, recomputation benefits and training performance in the early stage of model training.

[0029] Figure 2 This is a schematic diagram of a training framework provided in an embodiment of this application. It should be noted that the execution entity of the training method provided in this application can be a server with the training framework deployed, a training node in a GPU cluster, or a distributed training system composed of multiple training nodes. For example... Figure 2 As shown, the server or training node executes the configuration parsing module 20, model training module 21, memory analysis module 22, performance tuning module 23, and recomputation scheduling module 24 in the training framework through the processor. Each module can be implemented in the form of software program, hardware logic, or a combination of software and hardware.

[0030] Based on the above training framework, such as Figure 1 As shown, the model training method provided in this application includes: S10: Obtain the current training strategy of the model to be trained; the current training strategy includes the micro-batch data size and the recomputation strategy; the recomputation strategy is used to indicate the scheduling status of the recomputation component; In specific embodiments, the model to be trained may include, but is not limited to, the GPT (Generative Pre-trained Transformer) series, the BERT model, large models based on the Transformer architecture, LLaMA (Large Language Model Meta AI), and LLM models such as Tongyi Qianwen. This application does not limit the model to be trained. In an optional embodiment, the LLM may be a neural network model with a parameter count of billions, tens of billions, or higher, and employing a multi-layer Transformer structure.

[0031] The acquired current training strategy may include, but is not limited to, the micro-batch data size and the recomputation strategy. The micro-batch data size represents the number of samples input to the training pipeline in a single forward and backward propagation. The recomputation strategy indicates the scheduling status of the recomputation component, i.e., whether it is enabled or disabled. When the recomputation strategy indicates that the recomputation component is enabled, the corresponding recomputation component performs recomputation.

[0032] In a specific embodiment, Figure 2 After obtaining relevant information about the model to be trained, the configuration parsing module 20 sets the current training strategy, namely, the micro-batch data size and recomputation strategy for the training task. This application does not limit the initial training parameters. However, to achieve near-full memory utilization without OOM (Out of Memory) and maximize training throughput by continuously adjusting the recomputation strategy, a conservative micro-batch data size and recomputation strategy can be used as the initial training strategy. Specifically, a smaller micro-batch data size is set, while recomputation components with higher recomputation strategies are enabled to perform recomputation.

[0033] S11: Based on the current training strategy, perform a preset number of training rounds on the model to be trained; S12: After training for a preset number of rounds, obtain memory usage information, recomputation benefit information, and training performance information; the recomputation benefit information is used to characterize the degree of memory saving and computation time consumption after the recomputation component is scheduled; the training performance information is used to characterize the training efficiency of the model to be trained under the current training strategy. Figure 3 This is a schematic diagram illustrating the principle of a model training method based on dynamic recomputation scheduling provided in an embodiment of this application. In an optional embodiment, the preset number of rounds can be set to M. The M rounds of training can be used as a warm-up training window to observe the memory usage and performance of the current training strategy in a real hardware environment. Figure 2 After the model training module 21 completes M rounds of training, the memory analysis module 22 obtains the memory usage information and transmits it to the performance tuning module 23 so that the performance tuning module 23 can analyze and obtain the recomputation benefit information and training performance information.

[0034] Among them, the video memory usage information is used to reflect whether the GPU video memory is close to the target usage range. This allows us to observe the GPU video memory usage after M rounds of training, so as to adjust the scheduling status of the recomputation component based on the video memory usage information.

[0035] The recomputation benefit information is used to characterize the degree of memory saving and computation time consumption after the recomputation component is scheduled. In a specific embodiment, the recomputation benefit information can be used to analyze how much GPU memory can be saved and how much additional computation time will be caused by the scheduling of the recomputation component in the current training strategy during the M rounds of training.

[0036] Training performance information is used to characterize the training efficiency of the model under the current training strategy, such as the number of samples processed per second or the number of tokens processed per second. In a specific embodiment, the training performance information can be used to analyze the impact of the currently adopted training strategy, i.e., the set micro-batch data size and recomputation strategy, on the training efficiency of the model.

[0037] It should be noted that, in one optional embodiment, the model to be trained can be a large model in the field of satellite technology, and the training dataset for training the model can be a text dataset including questions and answers. For example, when the trained model is input into a professional technical question in the field of satellite technology, it can directly obtain the corresponding technical answer.

[0038] S13: Determine whether the video memory usage information meets the preset conditions; if not, proceed to step S14; if yes, proceed to step S15. S14: Adjust the current training strategy based on the recalculation benefit information to obtain a new current training strategy, and return to step S11; S15: Based on the training performance information, determine the strategy with the highest training efficiency from the current training strategies that meet the preset conditions as the target training strategy; and based on the target training strategy, continue iterative training of the model to be trained until the preset iteration conditions are met.

[0039] Furthermore, such as Figure 3 As shown, based on the acquisition of memory usage information, recomputation benefit information and training performance information, in order to achieve dynamic adjustment of the recomputation strategy and achieve dynamic balance between memory pressure and computational overhead, the performance tuning module 23 determines whether the collected memory usage information meets the preset conditions. If the preset conditions are not met, it indicates that the current training strategy still has insufficient memory utilization, inappropriate recomputation configuration, or that the micro-batch data size still has room for adjustment.

[0040] At this point, based on the recomputation benefit information, namely the current memory usage and computation time, the current training strategy is adjusted to obtain a new current training strategy. After obtaining the new current training strategy, M rounds of iterative training continue. At this time, the recomputation scheduling module 24 schedules the recomputation components based on the new current training strategy, that is, iteratively executes steps 11 to S13 until the memory usage information meets the preset conditions.

[0041] In another optional embodiment, if the video memory usage information meets the preset conditions, it indicates that several training strategies have entered the preset video memory utilization range. At this time, based on the training performance information, the strategy with the highest training efficiency is selected from the current training strategies that meet the preset conditions as the target training strategy, thereby ensuring that the subsequent model training is carried out with the highest training efficiency. That is, the model training module 21 continues to iterate and train based on the target training strategy until the preset iteration conditions are met.

[0042] In a specific embodiment, based on the adjusted current training strategy, M rounds of iterative training are continued, gradually adjusting and optimizing the strategy combination, and recording the results in the global file `config`. Simultaneously, the overall training performance data (denoted as `TGS`) is recorded, and the average performance at each M-round interval is recorded in the `TGS[index]` array. The `index` is a key index for each optimized strategy combination and the current training performance, enabling the rapid identification of the optimal strategy combination and performance.

[0043] See Figure 3 The performance tuning module 23 obtains the latest memory usage information, recomputation benefit information, and training performance information for the M rounds. If the memory usage information does not meet the preset conditions, it will revise the recomputation strategy and adjust the fusion scheme of micro-batch data size. If the preset conditions are met, the performance tuning module 23 immediately compares the performance TGS[index] array and selects the strategy combination of the index group with the highest performance TGS. The model training module 21 will maintain this optimal recomputation scheduling strategy and micro-batch data size until the total amount of data set for the training task is completed. The training performance information includes, but is not limited to, training throughput, training speed, number of samples processed per unit time, time consumed per iteration, or other performance indicators used to evaluate training efficiency.

[0044] It is worth noting that, in one optional embodiment, the preset iteration condition may be reaching a preset total number of iterations. In this case, the preset number of rounds M in the above embodiment is less than the preset total number of iterations. In another optional embodiment, the preset iteration condition may also be the convergence of the loss function. This application does not limit the preset iteration condition.

[0045] In summary, the model training method of this application first starts training with conservative parameters, then collects memory and performance data during the warm-up phase, then dynamically adjusts the recomputation component and micro-batch size until the memory utilization rate approaches the target threshold, i.e., meets the preset conditions, and finally selects the configuration with the highest training performance and fixes it to complete the subsequent training of the model.

[0046] In one alternative embodiment, within the field of Earth sciences, the method provided in this application can be used for training large-scale meteorological models, large-scale remote sensing interpretation models, or earthquake waveform recognition models. For example, when training multi-year satellite cloud images using Transformer, the above method can improve GPU memory utilization, allowing larger micro-batch data to enter the training pipeline, thereby shortening the training cycle for the same amount of data.

[0047] Figure 4 This is a schematic diagram illustrating the model training effect under different recomputation scheduling strategies provided in an embodiment of this application. Figure 5This application provides a schematic diagram illustrating the model training effect under different recomputation scheduling strategies, as another embodiment of the present application. To make the technical solution of this application clearer to those skilled in the art, the following will be combined with... Figure 4 and Figure 5 Please provide an explanation.

[0048] In one alternative embodiment, such as Figure 4 As shown, when training the same model, if recomputation is not enabled, the computation time is low, i.e., the computation is fast, but the GPU memory usage is too high, and the micro-batch data size cannot be increased, which may lead to insufficient memory / out of memory (OOM). If recomputation is fully enabled, GPU memory is saved, but it will lead to serious computation time consumption, slowing down training, i.e., reducing training efficiency. At this time, the model training method of dynamic recomputation scheduling provided in this application can be used to dynamically enable partial recomputation, achieving a dynamic balance between GPU memory and computation time.

[0049] In another alternative embodiment, such as Figure 5 As shown, when training the same model, if all recomputation components are fully computed throughout the entire training process (i.e., all recomputation components are enabled), it will introduce significant recomputation time consumption. If selective recomputation is used during training, i.e., randomly selecting a few recomputation components to recompile (i.e., randomly selecting recomputation components to enable), it can alleviate some of the recomputation time consumption, but the memory usage remains very low; that is, a balance cannot be achieved between computation time and memory usage. See also... Figure 5 If the dynamic recomputation provided in this application is adopted, that is, the size of the micro-batch data and the mechanical energy of the recomputation strategy are dynamically adjusted according to the real-time monitoring of memory usage, recomputation benefits and training performance, the memory usage can be maximized, while reducing the additional recomputation time, thus achieving a dynamic balance between memory usage and computation time.

[0050] Therefore, the model training method based on dynamic recomputation scheduling adopted in this application monitors memory usage, recomputation benefits, and training performance in real time during model training. By introducing a dynamic recomputation adjustment mechanism, it adaptively balances memory pressure and computational overhead. This avoids the dilemma of "excessive recomputation leading to decreased training efficiency" or "limited recomputation leading to insufficient memory" caused by a fixed recomputation strategy. Furthermore, under multiple rounds of iterative training with a preset number of rounds, the final target strategy is selected from strategies that meet preset conditions, ensuring maximum training efficiency while maintaining memory safety, thereby improving the resource utilization efficiency and model convergence speed of distributed training.

[0051] In one optional embodiment, obtaining the current training strategy of the model to be trained includes: Obtain the model structure and parallel strategy of the model to be trained; Based on the model structure and parallel strategy, determine the model layer allocation information of the model to be trained in each parallel stage; Based on the model layer allocation information, the micro-batch data size and recomputation strategy are initialized to obtain the current training strategy; wherein, the initialized micro-batch data size is a preset initial value, and the initialized recomputation strategy is used to indicate that all recomputation components are in the enabled state.

[0052] like Figure 3 As shown in the specific embodiment, when obtaining the current training strategy of the model to be trained, it is done by... Figure 2 The configuration parsing module 20 shown obtains the basic parameters of the model to be trained, including but not limited to the model structure and parallel strategy. The model structure reflects the network layer types, number of layers, parameter sizes, and inter-layer connections of the model to be trained. The parallel strategy represents one or more combinations of pipeline parallelism (PP), data parallelism (DP), tensor parallelism (TP), and expert parallelism (EP) employed by the training framework.

[0053] It should be noted that PP refers to the training method of dividing the model layers into different pipeline stages, DP refers to the training method of distributing the same model copy across different data shards, TP refers to the training method of splitting the computation of a single-layer tensor across multiple devices, and EP refers to the training method of distributing the expert network across different devices in a hybrid expert model.

[0054] After obtaining the model structure and parallel strategy of the model to be trained, the next step is to determine the model layer allocation information in each parallel stage, that is, to determine the number of model layers corresponding to different parallel stages. For ease of understanding, the following explanation uses a model structure based on multiple Transformers and a parallel strategy of PP as an example.

[0055] For example, if the model to be trained includes Layered Transformer structure, PP stage number is The number of Transformer layers allocated to each pipeline stage can be denoted as: The number of Transformer layers ΔL can be calculated using formula (1): (1) in, The number of Transformer layers allocated to each PP stage. The number of Transformer layers in the model to be trained. This represents the number of stages in a pipeline operating in parallel.

[0056] Furthermore, based on the model layer allocation information, the micro-batch data size and recomputation strategy are initialized to obtain the current training strategy. In an optional embodiment, the micro-batch data size and recomputation strategy can be obtained by parsing the global config configuration through the framework initialization function. Initializing these two parameters allows for subsequent tuning to adjust the training to achieve optimal performance.

[0057] Specifically, such as Figure 3 As shown, the initialized micro-batch size can be set to a preset initial value of 1. The initialized recompute strategy is used to indicate that all recompute components are enabled (i.e., recompute-params:full). It is understandable that the true peak memory usage has not yet been determined in the early stages of model training. Therefore, a conservative micro-batch size can be used and recompute can be fully enabled to reduce the probability of memory overflow due to excessively large activation values.

[0058] Therefore, the method provided in this application initializes the micro-batch data size to 1 and initializes the recalculation strategy to be fully enabled, ensuring that the micro-batch data size is not set too large at the beginning, but rather the most conservative micro-batch data size is used first, while recalculation is fully enabled to minimize activation value usage and avoid OOM.

[0059] In one alternative embodiment, the model structure includes a multi-layer Transformer structure; the recomputation components include a core attention component, a multi-head attention component, a feedforward network component, and a layer normalization component. The recomputation benefit information includes the memory savings, recomputation time, and recomputation benefit value for each recomputation component; the recomputation benefit value is used to characterize the memory savings of each recomputation component per unit recomputation time.

[0060] In a specific embodiment, the Transformer structure includes attention computation, a feedforward network, residual connections, and layer normalization, making it suitable for training large-scale language models and multimodal large models. Correspondingly, the computationally recalculated components include a core attention component, a multi-head attention component, a feedforward network component, and a layer normalization component. In an optional embodiment, the core attention component can be denoted as Attn_core, the multi-head attention component as MHA, the feedforward network component as MLP, and the layer normalization component as LayerNorm.

[0061] The recomputation benefit information includes the memory savings, recomputation time, and recomputation benefit value for each recomputation component. Memory savings represent the amount of active memory saved by enabling or retaining a particular recomputation component. Recomputation time represents the average time introduced by re-executing the forward computation during the reverse phase of the corresponding recomputation component. The recomputation benefit value characterizes the degree of memory savings per unit recomputation time for each recomputation component.

[0062] Table 1 is a schematic table of recalculated revenue information provided in the embodiments of this application. The following will be explained in conjunction with Table 1.

[0063] Table 1. Schematic diagram of a recalculation of revenue information

[0064] As shown in Table 1, the model to be trained includes four core recomputation components. The memory savings of each component's single-layer GPU can be recorded in the array Compute_Units[i], where i is a natural number from 0 to 3. As shown in Table 1, the corresponding relationships are as follows: Attn_core is recorded in the array Compute_Units[0], MHA is recorded in the array Compute_Units[1], MLP is recorded in the array Compute_Units[2], and LayerNorm is recorded in the array Compute_Units[3].

[0065] In one alternative embodiment, the relationship between the amount of video memory saved can be derived by training input and other basic parameters as Compute_Units[0]>Compute_Units[1]>Compute_Units[2]>Compute_Units[3].

[0066] In another optional embodiment, referring to Table 1, the additional computation time introduced by each type of recomputation component in the reverse engineering phase is statistically analyzed. Specifically, based on the performance tuning module 23's analysis of the recomputation component time data in the first M rounds of reverse engineering, the average time of the four recomputation components is calculated and denoted as follows: See Table 1. The average time consumption for different recomputed components is as follows: , , as well as In an alternative embodiment, the average time taken by these four recomputational components is sorted as follows: > > > .

[0067] Based on the above embodiments, as an optional embodiment, after a preset number of training rounds, obtaining recalculation profit information includes: Based on the model layer allocation information, determine the number of Transformer layers allocated in the parallel phase corresponding to the current training strategy; Determine the amount of single-layer memory saved by the recomputed component in a single-layer Transformer structure; The amount of memory saved is determined based on the amount of memory saved per single layer and the number of Transformer layers; During the reverse computation process of training in a preset number of rounds, the average recomputation time introduced when each recomputation component is scheduled is statistically analyzed to obtain the recomputation time. Based on the amount of video memory saved and the time spent on recomputation, the efficiency reduction of video memory per unit time is determined, and the benefit value of recomputation is obtained.

[0068] In a specific embodiment, when obtaining recalculation benefit information after a preset number of training rounds, the number of Transformer layers allocated in the parallel stage corresponding to the current training strategy is first determined according to the model layer allocation information and formula (1). Then, the amount of single-layer memory saved by the recomputed components in a single-layer Transformer structure is determined.

[0069] Furthermore, in an alternative implementation, based on the single-layer memory savings and the number of Transformer layers recorded in Table 1 above... To determine the amount of video memory saved, refer to formula (2): (2) in, For the first The total memory savings that individual computing components can achieve in the current PP stage, or per PP stage. The total memory saved by the Transformer layer, of which... . For the first The amount of single-layer video memory that can be saved by individual computational components in a single-layer Transformer.

[0070] It is worth noting that, in specific embodiments, since the micro-batch data size and recalculation strategy are mainly related to the activation value memory, in one optional embodiment, It can also be directly expressed as the first Individual weight calculation components can save activation value memory in a single-layer Transformer.

[0071] Furthermore, by combining "saving video memory" and "increased computation time," the efficiency reduction per unit time of video memory for each component is determined. Specifically, this is based on the amount of video memory saved per single layer. The average time spent is denoted as The amount of video memory that can be reduced per unit time for each recomputation component is derived, that is, the efficiency of video memory reduction per unit time is determined, see formula (3): (3) in, For the first The recalculation component recalculates the revenue value.

[0072] In a specific embodiment, when the revenue value is recalculated The longer the time, the more it represents the first The more video memory a heavy computing component can retrieve per unit of time.

[0073] In one alternative embodiment, the amount of memory savings can be based on a single layer of video memory. Average time consumed is denoted as and recalculation of revenue value A recomputation efficiency table is constructed to allow for the scheduling of recomputation components during subsequent training strategy adjustments. Specifically, when it is necessary to reduce actual GPU memory usage, the target recomputation component in the recomputation efficiency table will be called first, rather than blindly adding recomputation to the entire layer.

[0074] In one optional embodiment, the video memory usage information includes the average video memory usage ratio; after a preset number of training rounds, the video memory usage information is obtained, including: Obtain the available video memory of the computing accelerator card; After each training iteration of the preset number of training rounds, the actual GPU memory usage of the computing acceleration card is obtained; the actual GPU memory usage includes the GPU memory usage of model parameters, gradients, optimizer state, and activation values. The average video memory usage ratio is determined based on the actual video memory usage and the available video memory.

[0075] In a specific embodiment, the GPU memory analysis module 22, which initiates the training framework, first obtains the available GPU memory. Simultaneously, the model training module 21 continuously acquires and monitors the actual GPU memory usage after each training iteration of a preset number of rounds. In an optional embodiment, the actual memory usage includes model parameter memory usage, gradient memory usage, optimizer state memory usage, and activation value memory usage. It should be noted that the model parameter memory usage, gradient memory usage, and optimizer state memory usage are fixed values ​​under the current training configuration, while the activation value memory usage varies with the micro-batch data size and recalculation strategy.

[0076] After obtaining the original available video memory and the current actual video memory usage, in one optional implementation, the average video memory usage ratio is calculated according to formula (4): (4) in, This represents the average video memory usage ratio. This refers to the memory usage of the model parameters. This refers to the amount of memory used for the gradient. This refers to the memory usage of the optimizer's state. This is the activation value for video memory usage. This represents the amount of available video memory.

[0077] It should be noted that, in the specific embodiment, a preset number of rounds is performed. Calculating the average memory usage ratio can avoid memory spikes in a single training round, which would lead to frequent scheduling of computationally intensive components. In other words, using the average value can reduce the impact of single-round peak jitter on policy judgment.

[0078] Therefore, the average GPU memory usage during the warm-up training phase was determined. The data usage of each component is analyzed, followed by the analysis of memory utilization and performance comparison before and after optimization. Further optimization of the recomputation component scheduling strategy and increase of micro-batch data size are then implemented. The current optimized scheme (i.e., the new current training strategy) is then fixed and trained for M iterations. The overall training task performance and average GPU memory usage are then continuously collected. .

[0079] Based on the above embodiments, as an optional embodiment, the preset conditions include that the absolute value of the difference between the average video memory usage ratio and the preset video memory threshold is less than a preset value; wherein, the preset video memory threshold is less than the upper limit of the video memory usage ratio that characterizes video memory removal.

[0080] For details, please refer to formula (5): (5) in, To preset the video memory threshold, This is the default value.

[0081] In a specific embodiment, the average video memory usage ratio With preset memory threshold The absolute value of the difference between them is less than the preset value. Used to characterize average video memory usage ratio Approaching the preset video memory threshold For example, close to the preset video memory threshold. It is 95%. In fact, in specific embodiments, the average video memory usage ratio is... Achieving an accurate 95% is difficult. To avoid OutOfMemoryError (OOM), the average video memory usage ratio should be kept to a minimum. If the difference falls within a specified range, meaning the absolute value of the difference is less than a preset value, it indicates that the video memory usage information meets the preset conditions. In other words, the preset video memory threshold... It can be set to 95%, which is to reserve a 5% safety margin to prevent the peak memory usage of a single card from triggering an OOM (Out of Memory) error.

[0082] Based on the above embodiments, as an optional embodiment, the current training strategy is adjusted according to the recalculation benefit information, including: If the average video memory usage ratio is less than or equal to a specified multiple of the preset video memory threshold, increase the micro-batch data size while maintaining the current recalculation strategy; the specified multiple is less than 1. If the average video memory usage ratio is greater than a specified multiple of the preset video memory threshold, perform the following steps: Increase the size of micro-batch data; Based on the recalculation revenue information, the target recalculation components are identified; The scheduling status of the target recompile component is adjusted to reduce the actual memory usage, resulting in the adjusted recompile strategy.

[0083] In a specific embodiment, when adjusting the current training strategy based on recalculation gain information, the relationship between the average memory usage ratio and a specified multiple of a preset memory threshold can be determined. Specifically, as follows... Figure 3 As shown, it is determined whether the average video memory usage ratio is less than or equal to a specified multiple of a preset video memory threshold. In an optional embodiment, the specified multiple can be 1 / 2.

[0084] If the average video memory usage ratio is less than or equal to a specified multiple of the preset video memory threshold, it indicates that the video memory is still relatively idle. Performance tuning module 23 will prioritize increasing the micro-batch data size and maintain the current recalculation strategy, i.e., see [link to relevant documentation]. Figure 3When the video memory utilization is low, there is no need to perform complex recalculation scheduling; instead, the size of the micro-batch data should be increased first. As an optional embodiment, increasing the size of the micro-batch data in this case includes: multiplying the micro-batch data size by a preset multiple and rounding up. The preset multiple is greater than 1. In an optional embodiment, the preset multiple can be 1.5. See formula (6): (6) in, This refers to the adjusted micro-batch data size. This refers to the size of the micro-batch data before adjustment.

[0085] Simultaneously, after updating the micro-batch data size, the average video memory usage ratio is immediately updated. In an optional embodiment, the update of the average video memory usage ratio is shown in formula (7): (7) in, This is the updated average video memory usage ratio. The average video memory usage ratio before the update. This is the sum of the increase in video memory for the recomputing components. In a specific embodiment, after the estimated increase in the size of the micro-batch data, the video memory increase for each additional micro-batch data size is calculated. Specifically, the video memory analysis module 22 will obtain the video memory usage of the current micro-batch data, which is the sum of the increase in video memory for the four recomputing components mentioned above.

[0086] See Figure 3 In another optional embodiment, if the average video memory usage ratio is greater than a specified multiple of a preset video memory threshold, the performance tuning module 23 will formulate a new recomputation scheduling strategy and a fusion scheme with the micro-batch data size. Specifically, on the one hand, the micro-batch data size is increased, and on the other hand, the target recomputation component is determined based on the recomputation benefit information, and the scheduling state of the target recomputation component is adjusted to reduce the actual video memory usage, thus obtaining the adjusted recomputation strategy.

[0087] Understandably, when the average video memory usage ratio exceeds a specified multiple of the preset video memory threshold, it indicates that the video memory is no longer idle, but has not yet reached the preset condition. At this point, simply increasing the micro-batch data size may approach OOM (Out of Memory), so it is necessary to simultaneously adjust the recalculation component to free up video memory. That is, by enabling or adjusting part of the recalculation, video memory is freed up, and then the freed-up video memory is used to increase the micro-batch data size, thereby improving throughput.

[0088] In one optional embodiment, increasing the micro-batch data size includes summing the micro-batch data size with a preset increment. For example, the preset increment can be set to 2, i.e. It should be noted that the preset increment can be set according to actual business needs, and this application does not impose any restrictions on this.

[0089] Similarly, the average video memory usage ratio needs to be updated. The update of the average video memory usage ratio is shown in formula (8): (8) in, Used to characterize the amount of memory loss caused by adjusting the recomputed components.

[0090] Therefore, the method provided in this application avoids introducing unnecessary complex scheduling in the low memory range and releases memory by using recomputation components in the medium memory range, thereby gradually improving training throughput while preventing OOM.

[0091] The scheme that integrates recomputation scheduling strategy and optimizes micro-batch data size can achieve efficient utilization of GPU memory (target 95% utilization), increase the amount of single input data in each iteration, reduce the latency of single batch data computation and transmission in multi-stage pipelines, accelerate the efficiency of training the same amount of data for the entire task, and avoid single GPU memory overflow through dynamic recomputation component scheduling, effectively reduce the time overhead of all recomputations, and improve the training performance of large models.

[0092] In the above embodiments, the model training method based on dynamic recomputation scheduling has been described in detail. This application also provides an embodiment of a model training device based on dynamic recomputation scheduling.

[0093] Figure 6 This is a schematic diagram of the structure of a model training device based on dynamic recomputation scheduling provided in an embodiment of this application, as shown below. Figure 6 As shown, the device includes: The strategy acquisition module 60 is used to acquire the current training strategy of the model to be trained; the current training strategy includes the micro-batch data size and the recomputation strategy; the recomputation strategy is used to indicate the scheduling status of the recomputation component; Iterative training module 61 is used to train the model to be trained for a preset number of rounds based on the current training strategy; The information acquisition module 62 is used to acquire memory usage information, recomputation benefit information, and training performance information after a preset number of training rounds. The recomputation benefit information is used to characterize the degree of memory saving and computation time consumption after the recomputation component is scheduled. The training performance information is used to characterize the training efficiency of the model to be trained under the current training strategy. The condition determination module 63 is used to determine whether the video memory usage information meets the preset conditions; if not, the strategy adjustment module 64 is called; if yes, the target processing module 65 is called. The strategy adjustment module 64 is used to adjust the current training strategy based on the recalculation benefit information to obtain a new current training strategy, and return the steps of training the model to be trained for a preset number of rounds based on the current training strategy. The target processing module 65 is used to determine the strategy with the highest training efficiency from the current training strategies that meet the preset conditions based on the training performance information, and to continue iteratively training the model to be trained until the preset iteration conditions are met.

[0094] Therefore, by integrating a perceptual recalculation scheduling technique with dynamic adjustment of micro-batch data size, available GPU memory space is calculated during training, and intermediate activation values ​​are selectively recalculated. This reduces GPU memory usage and computational latency, alleviating the redundant overhead caused by static recalculation mechanisms in traditional training methods. Simultaneously, combined with a micro-batch adaptive adjustment mechanism, the amount of micro-batch data processed per iteration is effectively increased under the same GPU memory conditions, enhancing the efficiency of GPU computing units and thus improving the computational throughput of a single GPU in each forward and backward propagation. Achieving a highly efficient balance between reducing GPU memory usage efficiency and increasing computational throughput not only significantly optimizes GPU memory utilization but also provides stable and scalable acceleration support for large model training tasks.

[0095] Furthermore, the model training apparatus based on dynamic recomputation scheduling provided in this application embodiment also includes: The target acquisition module is used to acquire the model structure and parallel strategy of the model to be trained; The layered information acquisition module is used to determine the model layer allocation information of the model to be trained in each parallel stage based on the model structure and parallel strategy. The initialization module is used to initialize the micro-batch data size and recomputation strategy based on the model layer allocation information to obtain the current training strategy. The initialized micro-batch data size is a preset initial value, and the initialized recomputation strategy is used to indicate that all recomputation components are in the enabled state.

[0096] The model structure includes a multi-layer Transformer structure; the recomputation components include a core attention component, a multi-head attention component, a feedforward network component, and a layer normalization component; the recomputation benefit information includes the memory savings, recomputation time, and recomputation benefit value corresponding to each recomputation component; the recomputation benefit value is used to characterize the degree of memory savings per unit recomputation time of each recomputation component.

[0097] The quantity determination module is used to determine the number of Transformer layers allocated in the parallel stage corresponding to the current training strategy based on the model layer allocation information. The single-layer memory saving determination module is used to determine the single-layer memory saving of recomputed components in a single-layer Transformer structure. The target memory saving determination module is used to determine the memory saving amount based on the memory saving amount per single layer and the number of Transformer layers; The statistics module is used to calculate the average recalculation time introduced when each recalculation component is scheduled during the reverse computation process of training in a preset number of rounds, and to obtain the recalculation time. The benefit value determination module is used to determine the reduction in video memory efficiency per unit time based on the amount of video memory saved and the time spent on recalculation, and to obtain the benefit value of recalculation.

[0098] The available video memory acquisition module is used to obtain the available video memory of the computing accelerator card; among which, the video memory usage information includes the average video memory usage ratio; The actual memory usage acquisition module is used to acquire the actual memory usage of the computing acceleration card after each training iteration of a preset number of training rounds; wherein, the actual memory usage includes the memory usage of model parameters, gradient memory usage, optimizer state memory usage, and activation value memory usage. The average video memory usage ratio determination module is used to determine the average video memory usage ratio based on the actual video memory usage and the available video memory.

[0099] The preset conditions include that the absolute value of the difference between the average video memory usage ratio and the preset video memory threshold is less than a preset value; wherein the preset video memory threshold is less than the upper limit of the video memory usage ratio that represents video memory removal.

[0100] The first adjustment module is used to increase the size of the micro-batch data while maintaining the current recalculation strategy when the average video memory usage ratio is less than or equal to a specified multiple of a preset video memory threshold; the specified multiple is less than 1. The second adjustment module is used to increase the size of the micro-batch data when the average video memory usage ratio is greater than a specified multiple of the preset video memory threshold; determine the target recalculation component based on the recalculation benefit information; adjust the scheduling status of the target recalculation component to reduce the actual video memory usage, and obtain the adjusted recalculation strategy.

[0101] The first increase module is used to multiply the micro-batch data size by the preset multiple and round up when the average video memory usage ratio is less than or equal to a specified multiple of the preset video memory threshold. The preset multiple is greater than 1. The second increment module is used to sum the size of the micro-batch data with the preset increment when the average video memory usage ratio is greater than a specified multiple of the preset video memory threshold.

[0102] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 7As shown, the electronic device includes: a memory 70 for storing computer programs; The processor 71 is used to implement the steps of the model training method based on dynamic recomputation scheduling as described in the above embodiments when executing a computer program.

[0103] The electronic devices provided in this embodiment may include, but are not limited to, laptops or desktop computers.

[0104] The processor 71 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 71 may be implemented using at least one of the following hardware forms: Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 71 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 71 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 71 may also include an Artificial Intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0105] The memory 70 may include one or more computer-readable storage media, which may be non-transitory. The memory 70 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 70 is used to store at least the following computer program 701, which, after being loaded and executed by the processor 71, is capable of implementing the relevant steps of the model training method based on dynamic recomputation scheduling disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 70 may also include an operating system 702 and data 703, etc., and the storage method may be temporary storage or permanent storage. The operating system 702 may include Windows, Unix, Linux, etc. The data 703 may include, but is not limited to, the relevant data involved in the model training method based on dynamic recomputation scheduling.

[0106] In some embodiments, the electronic device may further include a display screen 72, an input / output interface 73, a communication interface 74, a power supply 75, and a communication bus 76.

[0107] Those skilled in the art will understand that Figure 7 The structures shown do not constitute a limitation on electronic devices and may include more or fewer components than those shown.

[0108] The electronic device provided in this application includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the model training method based on dynamic recomputation scheduling in the above embodiments.

[0109] It should be noted that although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

Claims

1. A model training method based on dynamic recomputation scheduling, characterized in that, The method includes: Obtain the current training strategy of the model to be trained; the current training strategy includes the micro-batch data size and the recomputation strategy; the recomputation strategy is used to indicate the scheduling status of the recomputation component; Based on the current training strategy, the model to be trained is trained for a preset number of rounds; After the preset number of training rounds, memory usage information, recomputation benefit information, and training performance information are obtained; the recomputation benefit information is used to characterize the degree of memory saving and computation time consumption after the recomputation component is scheduled; the training performance information is used to characterize the training efficiency of the model to be trained under the current training strategy. Determine whether the video memory usage information meets preset conditions; If not, adjust the current training strategy according to the recalculation benefit information to obtain a new current training strategy, and return to the step of training the model to be trained for a preset number of rounds based on the current training strategy; If so, based on the training performance information, the strategy with the highest training efficiency is determined from the current training strategies that meet the preset conditions as the target training strategy; and based on the target training strategy, the model to be trained is iteratively trained until the preset iteration conditions are met.

2. The model training method based on dynamic recomputation scheduling as described in claim 1, characterized in that, The process of obtaining the current training strategy of the model to be trained includes: Obtain the model structure and parallel strategy of the model to be trained; Based on the model structure and the parallel strategy, determine the model layer allocation information of the model to be trained in each parallel stage; Based on the model layer allocation information, the micro-batch data size and the recomputation strategy are initialized to obtain the current training strategy; wherein, the initialized micro-batch data size is a preset initial value, and the initialized recomputation strategy is used to indicate that each of the recomputation components is in the enabled state.

3. The model training method based on dynamic recomputation scheduling as described in claim 2, characterized in that, The model structure includes a multi-layer Transformer structure; the recomputation components include a core attention component, a multi-head attention component, a feedforward network component, and a layer normalization component; The recomputation benefit information includes the memory savings, recomputation time, and recomputation benefit value corresponding to each recomputation component; the recomputation benefit value is used to characterize the memory savings of each recomputation component per unit recomputation time.

4. The model training method based on dynamic recomputation scheduling as described in claim 3, characterized in that, After training for the preset number of rounds, the recalculation benefit information is obtained, including: Based on the model layer allocation information, determine the number of Transformer layers allocated in the parallel phase corresponding to the current training strategy; Determine the amount of single-layer memory saved by the recomputation component in a single-layer Transformer structure; The amount of memory saved is determined based on the amount of memory saved per single layer and the number of Transformer layers; During the reverse computation process of the preset number of training rounds, the average recomputation time introduced when each of the recomputation components is scheduled is statistically analyzed to obtain the recomputation time. Based on the amount of video memory saved and the time spent on recomputation, the efficiency reduction of video memory per unit time is determined, and the benefit value of recomputation is obtained.

5. The model training method based on dynamic recomputation scheduling as described in claim 1, characterized in that, The video memory usage information includes the average video memory usage ratio; After training for the preset number of rounds, obtain the video memory usage information, including: Obtain the available video memory of the computing accelerator card; After each training iteration of the preset number of training rounds, the actual memory usage of the computing acceleration card is obtained; wherein, the actual memory usage includes the memory usage of model parameters, the memory usage of gradients, the memory usage of optimizer states, and the memory usage of activation values. The average video memory usage ratio is determined based on the actual video memory usage and the available video memory.

6. The model training method based on dynamic recomputation scheduling as described in claim 5, characterized in that, The preset conditions include that the absolute value of the difference between the average video memory usage ratio and the preset video memory threshold is less than a preset value; wherein, the preset video memory threshold is less than the upper limit of the video memory usage ratio that represents video memory removal.

7. The model training method based on dynamic recomputation scheduling as described in claim 6, characterized in that, Based on the recalculation benefit information, the current training strategy is adjusted, including: If the average video memory usage ratio is less than or equal to a specified multiple of the preset video memory threshold, the size of the micro-batch data is increased while maintaining the current recalculation strategy; the specified multiple is less than 1. If the average video memory usage ratio is greater than a specified multiple of the preset video memory threshold, the following steps are performed: Increase the size of the micro-batch data; Based on the recalculation revenue information, the target recalculation component is determined; The scheduling state of the target recompile component is adjusted to reduce the actual video memory usage, resulting in an adjusted recompile strategy.

8. The model training method based on dynamic recomputation scheduling as described in claim 7, characterized in that, Increasing the size of the micro-batch data includes: If the average video memory usage ratio is less than or equal to a specified multiple of the preset video memory threshold, the product of the micro-batch data size and the preset multiple is rounded up, and the preset multiple is greater than 1. If the average video memory usage ratio is greater than a specified multiple of the preset video memory threshold, the size of the micro-batch data is summed with the preset increment.

9. A model training device based on dynamic recomputation scheduling, characterized in that, The device includes: The strategy acquisition module is used to acquire the current training strategy of the model to be trained; the current training strategy includes the micro-batch data size and the recomputation strategy; the recomputation strategy is used to indicate the scheduling status of the recomputation component. An iterative training module is used to train the model to be trained for a preset number of rounds based on the current training strategy; The information acquisition module is used to acquire memory usage information, recomputation benefit information, and training performance information after the preset number of training rounds; the recomputation benefit information is used to characterize the degree of memory saving and computation time consumption after the recomputation component is scheduled; the training performance information is used to characterize the training efficiency of the model to be trained under the current training strategy. The condition determination module is used to determine whether the video memory usage information meets preset conditions; if not, the strategy adjustment module is called; if yes, the target processing module is called. The strategy adjustment module is used to adjust the current training strategy according to the recalculation benefit information to obtain a new current training strategy, and return to the step of training the model to be trained for a preset number of rounds based on the current training strategy. The target processing module is used to determine the strategy with the highest training efficiency as the target training strategy from the current training strategies that meet the preset conditions according to the training performance information; and to continue iteratively training the model to be trained until the preset iteration conditions are met based on the target training strategy.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the model training method based on dynamic recomputation scheduling as described in any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the model training method based on dynamic recomputation scheduling as described in any one of claims 1 to 8.

Images