Large language model breakpoint continuous training method, system and device and storage medium
By using real-time monitoring and encrypted storage, the system automates the resuming of training from breakpoints for large language models, solving the problem of training interruption, improving training efficiency and quality, and ensuring data security and the rational use of resources.
Patent Information
- Application Number
- CN202512011731.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-29
- Publication Date
- 2026-04-07
AI Technical Summary
Existing large language model training methods lack a robust breakpoint continuation mechanism, requiring manual intervention after training is interrupted. This process is cumbersome and prone to errors. Furthermore, traditional methods do not fully consider dynamic information changes and resource utilization, resulting in low training efficiency.
The system monitors the training status in real time, saves the current model status information, builds a training continuation initialization environment under breakpoint conditions, uses encrypted and compressed storage of key parameters, dynamically adjusts computing resources, and achieves automated and precise training continuation operations.
It improved the consistency and stability of training, shortened the optimization and construction cycle, reduced resource waste, ensured the security and integrity of data, and improved training quality.
Smart Images
Figure CN121808393A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of large language model training technology, specifically to a method, system, device, and storage medium for resuming training from breakpoints in a large language model. Background Technology
[0002] Most existing large language model training methods lack a robust breakpoint resumption mechanism. When the training process is unexpectedly interrupted, manual intervention is usually required to restart the training. This is not only cumbersome, but also difficult to accurately locate the previous training breakpoint. Furthermore, errors are prone to occur during the manual recovery process, such as incorrect model parameter loading or disordered data batch order, which can lead to poor training results or even failure to converge.
[0003] In addition, some traditional backup and recovery methods do not fully consider the dynamic information changes during the training process and the rational use of resources, resulting in low training efficiency and seriously affecting the research and development progress and application promotion of large language models.
[0004] Therefore, this application proposes a method, system, device, and storage medium for resuming training from breakpoints in a large language model to solve the aforementioned technical problems. Summary of the Invention
[0005] The main objective of this invention is to provide a method, system, device, and storage medium for resuming training from breakpoints in a large language model, in order to solve the technical problems mentioned in the background art.
[0006] The present invention solves the above-mentioned technical problems by adopting the following technical solutions: A method for resuming training from breakpoints in a large language model includes: S1. Monitor training status information in real time during the training of large language models; S2. When a preset breakpoint condition is detected, save the current model training state information; S3. After the breakpoint, receive the training continuation instruction, build the training continuation initialization environment according to the saved training progress and specified model training parameters, restore the model parameters to the parameter state at the breakpoint according to the model training state information, and locate the starting position of the unprocessed data batch. S4. In the initial training environment, continue training the large language model according to the original training configuration until the predetermined training objective is achieved or the breakpoint condition occurs again.
[0007] Preferably, during the large language model training process in step S1, a set of monitoring functions are used to periodically collect and analyze model training status information; The training status information includes, but is not limited to, model parameter update status, training rounds, loss value changes, and data batch processing progress. The sampling frequency of the monitoring function is adaptively adjusted according to the size of the training dataset and the training complexity, and the range is once every [X] minutes, where [X] is a positive integer and [X] ≥ [0.5]. Preferably, the preset breakpoint conditions in step S2 are: the training time reaches a preset duration threshold, the model loss value fails to converge for multiple consecutive rounds, or an external interruption signal occurs. The training progress and specified model training parameters are stored in the form of an encrypted compressed package to a local storage device or a remote cloud storage server. The specified model training parameters include the number of training rounds, the current model parameter status, and the identifier of the unprocessed data batch.
[0008] Preferably, in step S3, when constructing the training initialization environment, computing resources are allocated and optimized simultaneously, specifically including: Based on the estimated computing resources required for the remaining training tasks, the utilization of GPU and CPU and memory allocation are dynamically adjusted to ensure that the training process can run efficiently under the specified hardware resource configuration.
[0009] A system for resuming training from breakpoints in a large language model, used to execute any of the above-described methods for resuming training from breakpoints in a large language model, includes: The training monitoring module is used to collect training status information in real time during the training of a large language model and transmit the collected information to a set of data processing components for analysis. The data processing components include a data storage module, a training continuation initialization module, and data processing components. The data storage module, connected to the training monitoring module, is used to receive and store the training progress and specified model training parameters when a preset breakpoint condition occurs. The training continuation initialization module is connected to the data storage module. After receiving the training continuation instruction, it is used to build the training continuation initialization environment based on the parameters stored in the data storage module, restore the model parameter state, and locate the starting position of the data batch. The training execution module, connected to the retraining initialization module, is used to continuously train the large language model according to the original training configuration in the retraining initialization environment. The early warning module is used to send an early warning message to a designated terminal when the training monitoring module detects that the training status information is close to the preset breakpoint condition threshold.
[0010] Preferably, the data storage module is equipped with a data backup unit for performing off-site backups of specified parameters stored according to a preset period.
[0011] Preferably, the training execution module provides real-time feedback of the current training progress and status information to the training monitoring module during the training process.
[0012] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method described above.
[0013] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described above.
[0014] As can be seen from the above technical solution, the present invention provides a method and system for resuming training from breakpoints in a large language model. Compared with the prior art, the present invention has the following advantages: 1. This invention can effectively solve the problem of training interruption caused by unexpected interruptions during the training of large language models, realize automated and precise retraining operations, improve training efficiency and quality, thereby significantly shortening the optimization and construction cycle of large language models and reducing the waste of time and resources caused by training interruptions.
[0015] 2. This invention improves the continuity and stability of training by real-time monitoring of the training status and timely saving of key parameters when preset breakpoint conditions are met, and by quickly restoring the training environment based on these parameters during training continuation.
[0016] 3. This invention uses encrypted compression to store key parameters and sets up a data backup mechanism to ensure data security and integrity. Even in the face of local storage failure or other emergencies, it can effectively prevent data loss, thereby ensuring the reliability of data during the training of large language models.
[0017] 4. In the initialization process of continuous training, this invention can make full use of hardware resources and improve the efficiency of continuous training by rationally allocating and optimizing computing resources. At the same time, it forms a closed-loop monitoring and adjustment mechanism by providing real-time feedback on training progress and status information, which helps to grasp the training effect in a timely manner and flexibly adjust the training strategy, thereby further improving the training quality of large language models.
[0018] It should be understood that the descriptions in this section are not intended to identify key or essential features of embodiments of the invention, nor are they intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Of course, implementing any product of the invention does not necessarily require achieving all of the advantages described above simultaneously. Attached Figure Description
[0019] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1This is a schematic diagram of the overall operation process of the present invention; Figure 2 This is a schematic diagram of the data processing flow in the interrupted training process of the present invention. Detailed Implementation
[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0021] For details in the embodiments, please refer to Figure 1 and Figure 2 .
[0022] like Figure 1 and Figure 2 As shown in the figure, the large language model breakpoint continuation training method proposed in this embodiment of the invention includes the following steps: S1. Training status monitoring.
[0023] After the training of the large language model is started, the training status is tracked in real time through a monitoring function embedded in the training code. This monitoring function will periodically collect information such as model parameter updates, training rounds, changes in loss values, and data batch processing progress (the collection frequency is adaptively adjusted according to the size and complexity of the training dataset, generally once every [X] minutes, where [X] is a positive integer and [X]≥[0.5]).
[0024] In one embodiment, during the training of a language generation model based on a large-scale text dataset, the monitoring function can accurately record detailed status information such as the minute changes in the model weight parameters after each round of training, the total number of training rounds completed, whether the loss value shows a downward trend, and the relative position of the data batch being processed in the entire dataset. The comprehensive collection of this information provides a solid data foundation for accurately determining whether to enter the breakpoint retraining process and how to continue training.
[0025] S2. Breakpoint condition judgment and data saving.
[0026] When a preset breakpoint condition is detected, a data saving operation is automatically triggered. The preset breakpoint conditions mainly cover the following situations: First, when the training time reaches a preset duration threshold, such as setting the longest continuous training time to 120 hours, the system will force the system to enter the breakpoint saving phase even if the training has not been completed after reaching this duration. Second, if the model loss value fails to converge for multiple consecutive rounds (e.g., 5 consecutive rounds), that is, the loss value no longer decreases significantly but instead fluctuates greatly or increases, it indicates that the model may be trapped in a local optimum or that there is an anomaly in the training. Third, it receives external interrupt signals (such as hardware fault alarms, network connection disconnection notifications, video memory overflow (OOM), or computing power suddenly drops to 0%, etc.).
[0027] Once any of the above conditions are met, it can be determined that the training process of the large language model cannot continue or is ineffective. The system will immediately save the current training progress and related key parameters. At this time, the key parameters include the number of training rounds, the current model parameter status (accurate to the value of each learnable weight parameter), and the identifier of the unprocessed data batch (recording the data batch number and corresponding sample information that has not yet started processing).
[0028] It should be noted that, since the parameters in the interruption / failure point state and the normal training state are different in the big oracle model, the purpose of this application is to collect information before the failure state and use it to continue training under the adjustment of the reconstructed training scenario. In addition, since a sudden interruption in the normal training state of the big oracle model will result in the loss of core parameters such as model weight parameters and optimizer state, and the training environment is not usually stored, this application also sets an interruption judgment condition before the interruption, which can store the parameters required for continued training in a timely manner. Therefore, it can reconstruct the original training requirements more quickly, conveniently and efficiently, and facilitate the complete operation of the overall training.
[0029] The data is stored here using encrypted compression on a local storage device or a remote cloud storage server. The encryption algorithm used is the national standard SM3. The resulting compressed file is managed using object storage to ensure the security and integrity of the data and prevent the data from being stolen or damaged during storage or transmission.
[0030] At this point, the use of encrypted compression to store key parameters and the setting of a data backup mechanism ensure the security and integrity of the data. Even in the face of local storage failure or other emergencies, data loss can be effectively prevented, thereby ensuring the reliability of data during the training of large language models.
[0031] S3. Continuing training initialization environment construction.
[0032] When a user decides to resume interrupted training, the system receives the resume training instruction and constructs the resume training initialization environment based on the previously saved training progress and related key parameters.
[0033] In the specific implementation process, firstly, the model parameters are accurately restored based on the saved model parameter status, so that they return to the parameter configuration at the breakpoint; then, the corresponding batch start position in the dataset is quickly located based on the batch identifier of the unprocessed data.
[0034] Meanwhile, when building the initialization environment for continued training, the system will reallocate and optimize computing resources, including dynamically adjusting GPU, CPU utilization and memory allocation based on the estimated computing resources required for the remaining training tasks.
[0035] In one embodiment, if the remaining training tasks are large and the computational speed requirements are high, the system will automatically increase the GPU utilization to an appropriate level (e.g., from the original 45% to 56%), while optimizing the memory allocation strategy to ensure sufficient memory space for subsequent training processes, thereby enabling the continued training process to run efficiently under appropriate hardware resource configuration.
[0036] At this point, by rationally allocating and optimizing computing resources during the retraining initialization process, hardware resources can be fully utilized to improve retraining efficiency.
[0037] S4. Continued training execution.
[0038] In the initial training environment, the large language model continues to be trained according to the original training configuration (including optimization algorithm, learning rate setting, data preprocessing method, etc.) until the predetermined training goal is achieved (such as reaching a specific accuracy or recall index, or completing the predetermined training rounds or the predetermined training time) or the breakpoint condition occurs again.
[0039] This invention achieves complete reproducibility of the training process by fully preserving the training state information at the breakpoint (including model weight parameters, optimizer state, learning rate scheduler state, etc.), ensuring that the convergence trajectory of the model after retraining remains strictly consistent with that of the uninterrupted training. It avoids the accumulation of parameter deviations caused by floating-point operation errors, communication delays, and other factors during recalculation, and does not depend on specific parallel strategies or the existence of non-faulty nodes. It is applicable to various distributed training architectures, making the training results more complete, realistic, and stable.
[0040] In summary, this method improves the overall continuity and stability of training by monitoring the training status in real time, saving key parameters in a timely manner when the preset breakpoint conditions are met, and quickly restoring the training environment according to the original training configuration when resuming training by acquiring these key parameters.
[0041] On the other hand, the present invention also discloses a large language model breakpoint continuation training system for executing the large language model breakpoint continuation training method of the above embodiments, comprising: (1) Training monitoring module, responsible for collecting training status information in real time during the training of large language models.
[0042] This module achieves this function by integrating a dedicated monitoring function into the training code. The monitoring function periodically (according to the adaptive acquisition frequency mentioned above) obtains information such as model parameter updates, training epochs, changes in loss values, and data batch processing progress, and transmits this information to the data processing unit for analysis and processing in a timely manner.
[0043] In a specific embodiment, in a distributed training scenario, the training monitoring module can run monitoring functions independently on each training node, and then aggregate the collected information to the central control node for comprehensive analysis, so as to fully grasp the status of the entire training process.
[0044] In addition, the module also has an early warning function. When the training status information is detected to be close to the preset breakpoint condition threshold (such as the training time being close to 95% of the preset time limit, or the model loss value being close to the non-convergence judgment standard for multiple consecutive rounds), an early warning message will be sent to the user terminal (which can be a local computer terminal or a mobile terminal device) to remind the user to prepare in advance, such as checking whether the hardware device is normal and whether the network connection is stable.
[0045] (2) Data storage module, used to receive and securely store training progress and related key parameters when a preset breakpoint condition occurs.
[0046] This module is closely connected to the training and monitoring module. Once the monitoring module triggers the save command, it will store the key parameters in the form of an encrypted compressed package. It adopts an advanced data backup mechanism to perform off-site backups of the stored key parameters regularly (e.g., at 2:00 AM every day). The backup location can be selected from data centers or cloud storage service providers in different regions. This can effectively prevent data loss due to local storage device failures (such as hard drive damage, server crashes, etc.).
[0047] Meanwhile, the data storage module also has a set of data verification functions, which are used to verify the integrity and accuracy of the data each time it is saved and read, to ensure that the data used for continued training is reliable.
[0048] (3) The training continuation initialization module is used to construct the training continuation initialization environment based on the key parameters obtained from the data storage module after receiving the user's training continuation instruction.
[0049] In the specific implementation process, the module first accurately restores the model parameter state, enabling the model to return to the instantaneous state before the interruption; then it quickly locates the starting position of the unprocessed data batch; and finally, it rationally allocates and optimizes computing resources.
[0050] In one embodiment, the module can dynamically adjust the utilization of GPU and CPU and memory allocation based on the size and complexity of the remaining training tasks, ensuring that the training process can run efficiently under appropriate hardware resource configuration.
[0051] (4) Training execution module, used to perform continuous training operations on the large language model according to the original training configuration in the retraining initialization environment.
[0052] During training, this module can provide real-time feedback on the current training progress and status information to the training monitoring module, forming a closed-loop monitoring and adjustment mechanism. This mechanism helps to promptly grasp the training effect and flexibly adjust the training strategy, thereby further improving the training quality of the large language model.
[0053] In one embodiment, the module sends information such as the current training round, loss value, and model parameter update magnitude back to the training monitoring module every 30 seconds, so as to understand the training effect in a timely manner and further adjust the training strategy as needed (such as fine-tuning the learning rate, changing the optimization algorithm, etc.).
[0054] In summary, the technical methods and systems adopted in this application can effectively solve the problem of training interruption caused by unexpected interruptions during the training of large language models, realize automated and precise retraining operations, improve training efficiency and quality, thereby significantly shortening the optimization and construction cycle of large language models and reducing the waste of time and resources caused by training interruptions.
[0055] Furthermore, it should be noted that this application is not only applicable to large language models in actual use. Since the cached content is parameters and instantaneous data, the technology of this application is also applicable to multimodal large models such as audio and images. Any application of the technical points adopted in this application to other multimodal large models within the spirit and principles of this invention should also be included in the protection scope of this application.
[0056] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method described above.
[0057] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described above.
[0058] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the large language model breakpoint continuation training methods described above.
[0059] It is understood that the system provided in the embodiments of the present invention corresponds to the method provided in the embodiments of the present invention, and the explanation, examples and beneficial effects of the relevant content can be referred to the corresponding parts of the above methods.
[0060] This application also provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, communication interface, and memory communicate with each other via the communication bus. Memory, used to store computer programs; The processor, when executing programs stored in memory, implements the above-mentioned large language model breakpoint continuation training method.
[0061] The communication bus mentioned in the above-mentioned electronic devices can be a standard bus for interconnecting peripheral components or an extended industrial standard structure bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc.
[0062] The communication interface is used for communication between the aforementioned electronic devices and other devices.
[0063] The memory may include random access memory or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.
[0064] The processors mentioned above can be general-purpose processors, including central processing units, network processors, etc.; they can also be digital signal processors, application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0065] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, an optical medium, or a semiconductor medium, etc.
[0066] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
[0067] Furthermore, it should be noted that if any directional indication (such as up, down, left, right, front, back, etc.) is involved in the embodiments of the present invention, the directional indication is only used to explain the relative positional relationship and movement of each component in a specific posture. If the specific posture changes, the directional indication will also change accordingly.
[0068] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the meaning of "and / or" throughout the text includes three parallel solutions; for example, "A and / or B" includes solution A, solution B, or a solution where both A and B are satisfied simultaneously. Furthermore, in the embodiments of this invention, "multiple" refers to two or more. Moreover, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.
Claims
1. A method for resuming training from breakpoints in a large language model, characterized in that, include: S1. Monitor training status information in real time during the training of large language models; S2. When a preset breakpoint condition is detected, save the current model training state information; S3. After the breakpoint, receive the training continuation instruction, build the training continuation initialization environment according to the saved training progress and specified model training parameters, restore the model parameters to the parameter state at the breakpoint according to the model training state information, and locate the starting position of the unprocessed data batch. S4. In the initial training environment, continue training the large language model according to the original training configuration until the predetermined training objective is achieved or the breakpoint condition occurs again.
2. The method for resuming training from breakpoints in a large language model as described in claim 1, characterized in that, During the large language model training process in step S1, a set of monitoring functions are used to periodically collect and analyze model training status information. The training status information includes, but is not limited to, model parameter update status, training rounds, loss value changes, and data batch processing progress. The sampling frequency of the monitoring function is adaptively adjusted according to the size and complexity of the training dataset, ranging from once every [X] minutes, where [X] is a positive integer and [X] ≥ [0.5].
3. The method for resuming training from breakpoints in a large language model as described in claim 1, characterized in that, The preset breakpoint conditions in step S2 are: the training time reaches a preset duration threshold, the model loss value fails to converge for multiple consecutive rounds, or an external interruption signal occurs. The training progress and specified model training parameters are stored in the form of encrypted compressed packages to local storage devices or remote cloud storage servers. The specified model training parameters include the number of training rounds, the current model parameter status, and the identifier of unprocessed data batches.
4. The method for resuming training from breakpoints in a large language model as described in claim 1, characterized in that, In step S3, during the construction of the training initialization environment, computing resources are allocated and optimized simultaneously, specifically including: Based on the estimated computing resources required for the remaining training tasks, the utilization of GPU and CPU and memory allocation are dynamically adjusted to ensure that the training process can run under the specified hardware resource configuration.
5. A system for resuming training from breakpoints in a large language model, characterized in that, The method for executing the large language model breakpoint continuation training method according to any one of claims 1-4 includes: The training monitoring module is used to collect training status information in real time during the training of a large language model and transmit the collected information to a set of data processing components for analysis. The data processing components include a data storage module, a training continuation initialization module, and a data processing component. The data storage module, connected to the training monitoring module, is used to receive and store the training progress and specified model training parameters when a preset breakpoint condition occurs. The training continuation initialization module is connected to the data storage module. After receiving the training continuation instruction, it is used to build the training continuation initialization environment according to the parameters stored in the data storage module, restore the model parameter state, and locate the starting position of the data batch. The training execution module, connected to the retraining initialization module, is used to continuously train the large language model according to the original training configuration in the retraining initialization environment.
6. The large language model breakpoint continuation training system as described in claim 5, characterized in that, It also includes an early warning module, which is used to send an early warning message to a designated terminal when the training monitoring module detects that the training status information is close to the preset breakpoint condition threshold.
7. The large language model breakpoint continuation training system as described in claim 5, characterized in that, The data storage module is equipped with a data backup unit, which is used to perform off-site backup of specified parameters stored according to a preset period.
8. The large language model breakpoint continuation training system as described in claim 5, characterized in that, The training execution module provides real-time feedback on the current training progress and status information to the training monitoring module during the training process.
9. A computer-readable storage medium, characterized in that, The device stores a computer program that, when executed by a processor, causes the processor to perform the steps of the method as described in any one of claims 1 to 4.
10. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the method as described in any one of claims 1 to 4.