A model training method and device

Abstract

The embodiment of the application provides a model training method and device, after training precision is abnormal, the model can be continuously trained according to the precision parameter cache in the local memory, the training parameters in the remote storage system are avoided to be loaded for continuous training, the training interruption duration can be shortened, and the effective training duration is improved. The method comprises the following steps: obtaining a first intermediate result, the first intermediate result is an output result when a first model is subjected to i-th round iteration training; when the first intermediate result does not meet the precision requirement, the first model is continuously trained according to the precision parameter cache in the local memory, the precision parameter cache comprises first model parameters, and the first model parameters are model parameters of the first model that meet the precision requirement in the training process.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more particularly to a model training method and apparatus. Background Technology

[0002] With the rapid development of artificial intelligence (AI) and large AI models, the parameter scale of these models is constantly expanding. However, the computing power and memory of single-machine devices are limited, and single-machine environments cannot effectively support the training of ultra-large models. Therefore, a distributed training strategy using large-scale clusters is proposed to train large AI models.

[0003] However, as the scale of large-scale distributed training clusters continues to expand, they also face many new challenges. For example, training failures such as hardware and / or software malfunctions of cluster nodes, network communication interruptions, and model training accuracy issues can all lead to training interruptions, affecting the continuity and stability of training and wasting significant computing resources. To address the training interruption problem caused by failures in large-scale distributed training clusters, existing solutions isolate devices and restart scheduling upon detecting a training process failure. Specifically, after receiving a failure notification, the training cluster scheduler stops the training cluster, isolates the failed node, reallocates container resources for rescheduling, restarts a new training container, and begins a new training process to continue training.

[0004] However, currently, a uniform recovery mechanism is used for different training failures. Some training failures are caused by random factors and do not require device isolation and rescheduling, such as accuracy anomalies. For these types of failures, the solution is still to restart and reschedule the training cluster, which leads to long recovery times and wasted computing resources. Summary of the Invention

[0005] This application provides a model training method and apparatus, which allows the model to continue training based on the accuracy parameters cached in local memory after training accuracy becomes abnormal. This avoids loading training parameters from a remote storage system for continued training, thereby shortening the training interruption time and increasing the effective training time.

[0006] In a first aspect, this application provides a model training method, comprising: obtaining a first intermediate result, the first intermediate result being the output result of training a first model in the i-th iteration; subsequently, when the first intermediate result does not meet the accuracy requirements, training the first model again according to the accuracy parameter cache in local memory, wherein the accuracy parameter cache includes first model parameters, the first model parameters being model parameters that meet the accuracy requirements during the training process of the first model.

[0007] In this embodiment, when an abnormal training accuracy is detected, training of the first model can be stopped, avoiding device isolation and rescheduling. Since the current training process does not need to exit when accuracy is abnormal, rescheduling is unnecessary, saving cluster scheduling time. Furthermore, training of the first model can continue by loading the accuracy parameter cache from local memory, avoiding the need to load training parameters stored in a remote storage system, saving remote fetching and loading time. This shortens training interruption time, increases effective training time, and reduces wasted computing resources.

[0008] In one possible implementation, before continuing to train the first model based on the precision parameter cache in local memory, the method may further include: obtaining the first model parameters when the second intermediate result meets the precision requirement and the j-th iteration round meets the update cycle of the precision parameter cache, wherein the second intermediate result is the output result of training the first model in the j-th iteration round; and storing the first model parameters in the precision parameter cache.

[0009] In one possible implementation, before continuing to train the first model based on the precision parameter cache in local memory, the method may further include: initializing the second model parameters of the first model to obtain a second model, wherein the second model parameters are the model parameters updated when training the first model.

[0010] In this embodiment of the application, after the training of the first model is stopped, the second model parameters of the first model are iteratively updated during the model training process, which may affect the resuming of the model from the breakpoint. Therefore, in order to avoid affecting the subsequent training of the model, the second model parameters can be initialized.

[0011] In one possible implementation, the aforementioned process of continuing to train the first model based on the precision parameter cache in local memory may include: determining training parameters based on the precision parameter cache, wherein the training parameters include one or more of the following: training parameters, training iteration steps, and training data sample retraining positions; the training parameters are model parameters for continuing to train the second model; and the training iteration steps are used to indicate the number of iterations for continuing to train the second model; and continuing to train the second model based on the training parameters.

[0012] In one possible implementation, training the second model continues based on the training parameters, which may include: training the second model continues based on the parameters of the first model, the number of training iteration steps, and the training data sample training position.

[0013] Secondly, this application provides a model training apparatus, comprising: an acquisition module for acquiring a first intermediate result, the first intermediate result being the output result of the i-th iteration training of a first model; and a training module for continuing to train the first model according to a precision parameter cache in local memory when the first intermediate result does not meet the precision requirements, the precision parameter cache including first model parameters, the first model parameters being model parameters that meet the precision requirements during the training process of the first model.

[0014] In one possible implementation, before continuing to train the first model based on the precision parameter cache in local memory, the aforementioned acquisition module is further configured to acquire the first model parameters when the second intermediate result meets the precision requirement and the j-th iteration round meets the update cycle of the precision parameter cache, wherein the second intermediate result is the output result of training the first model in the j-th iteration round; the storage module is configured to store the first model parameters in the precision parameter cache.

[0015] In one possible implementation, before continuing to train the first model based on the precision parameter cache in local memory, the apparatus may further include: an initialization module for initializing the second model parameters of the first model to obtain a second model, wherein the second model parameters are the model parameters updated during the training of the first model.

[0016] In one possible implementation, the aforementioned training module is specifically used to: determine the continuing training parameters based on the accuracy parameter cache, wherein the continuing training parameters include one or more of the following: training parameters, training iteration steps, and continuing training positions of training data samples; the training parameters are model parameters for continuing training of the second model, and the training iteration steps are used to indicate the number of iterations for continuing training of the second model; and continue training the second model based on the continuing training parameters.

[0017] In one possible implementation, the aforementioned training module is specifically used to: continue training the second model based on the first model parameters, the number of training iteration steps, and the training data sample continuation position.

[0018] Thirdly, this application provides a model training apparatus, which includes a processor for executing a program, causing the aforementioned model training apparatus to perform the method described in the first aspect or any implementation thereof as claimed in the first aspect.

[0019] Optionally, the model training apparatus further includes a memory, with the processor coupled to the memory; the memory is used to store programs.

[0020] Fourthly, embodiments of this application provide a computer-readable storage medium. The computer-readable storage medium stores computer instructions; when the computer instructions are executed on a computer, the computer causes the computer to perform the method as described in the first aspect or any possible implementation thereof.

[0021] Fifthly, embodiments of this application provide a computer program product. This computer program product includes a computer program or instructions that, when executed on a computer, cause the computer to perform the method as described in the first aspect or any possible implementation thereof.

[0022] The technical effects of the second to fifth aspects or any of the possible implementations thereof can be found in the first aspect or the technical effects of the relevant possible implementations thereof, and will not be repeated here. Attached Figure Description

[0023] Figure 1 A structural diagram illustrating the main framework of artificial intelligence;

[0024] Figure 2 A schematic diagram of a system framework provided for an embodiment of this application;

[0025] Figure 3 Another system framework diagram provided for an embodiment of this application;

[0026] Figure 4 A schematic flowchart illustrating a model training method provided in an embodiment of this application;

[0027] Figure 5 A diagram illustrating the interaction between the TPSC module and the training framework via an external interface;

[0028] Figure 6 This is a schematic diagram showing the TPSC module deployed within the training framework.

[0029] Figure 7 A flowchart comparing in-situ process recovery and process restart recovery when accuracy is abnormal;

[0030] Figure 8 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. Detailed Implementation

[0031] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0032] The method provided in this application can be applied to artificial intelligence (AI) scenarios. AI is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. Artificial intelligence studies the design principles and implementation methods of various intelligent machines, enabling them to have the functions of perception, reasoning, and decision-making. Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and fundamental AI theories.

[0033] First, the overall workflow of the artificial intelligence system is described; please refer to [link / reference]. Figure 1 , Figure 1 The diagram illustrates a structural framework for artificial intelligence (AI). The framework is further elaborated below along two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that AI brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed through technological means) to the industrial ecosystem of the system.

[0034] (1) Infrastructure

[0035] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0036] (2) Data

[0037] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0038] (3) Data processing

[0039] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0040] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0041] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0042] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0043] (4) General ability

[0044] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0045] (5) Smart Products and Industry Applications

[0046] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, smart cities, etc.

[0047] To facilitate understanding, some concepts involved in this application will be introduced first.

[0048] 1. Large-scale artificial intelligence model

[0049] Large-scale artificial intelligence models, or simply large models, refer to a class of AI models constructed from artificial neural networks with parameter scales reaching tens of billions, hundreds of billions, or even trillions. Large-scale models mainly include: large language models, large vision models, multimodal large models, and large models for basic science. Large-scale models have been widely applied in multiple fields, including search engines, intelligent agents, related vertical industries, and basic science.

[0050] 2. Large Language Model (LLM)

[0051] LLM (Language Modeling) refers to a language model with hundreds of millions (or more) parameters trained on massive amounts of text data. It's a deep learning-based natural language processing model. These models can process large amounts of text data to learn the grammatical and semantic rules of natural language. LLMs can be applied to text generation, machine translation, question answering systems, text summarization, and sentiment analysis, offering advantages such as strong generative capabilities, high adaptability, accurate prediction, and scalability. For example, in movie recommendation scenarios, large language models can generate descriptions of movie scenes, including genre, main actors, and plot, enabling the system to better recommend similar films. Large language models can also generate recommendation reasons; for instance, e-commerce websites can use large language models to generate reasons for recommending products, such as product quality, price, and features, allowing users to better understand the value of the products.

[0052] 3. Training Framework

[0053] A training framework is a collection of software tools and libraries used in machine learning and deep learning to build, train, and optimize models. Common training frameworks can be categorized into general-purpose deep learning frameworks (such as TensorFlow and MindSpore), domain-specific frameworks, distributed training frameworks (such as Megatron-LM and DeepSpeed), and mobile / edge computing frameworks.

[0054] 4. Training container

[0055] A training container is a technical solution that uses containerization technologies (such as Docker and Kubernetes) to package machine learning / deep learning training tasks and their runtime environment (code, dependencies, data, configuration, etc.) into an independent, portable unit to achieve environment consistency, rapid deployment, and resource isolation. It is primarily used to simplify training process management and support efficient model training across platforms (local, cloud, cluster).

[0056] 5. Model training iteration step

[0057] A step refers to the process of updating parameters using a batch of data during model training, i.e., one iteration of training. In a step, the model processes one batch of data, performing forward propagation, calculating the loss, backpropagation, and updating the parameters. The number of steps is typically equal to the total number of samples in the training dataset divided by the batch size. For example, if the training set has 10,000 samples and the batch size is 100, then each epoch contains 100 steps.

[0058] 6. Number of training epochs

[0059] An epoch refers to the process by which a model traverses the entire training dataset once during training. In one epoch, the model sees each training sample once and performs one complete forward and backward propagation. An epoch is the process by which the model performs a complete traversal of the entire training set, and it typically involves multiple steps.

[0060] 7. On-the-spot recovery training

[0061] In-situ training recovery means that training can continue on the training node before the failure without needing to be rescheduled to a new node.

[0062] To make the technical solution of this application clearer and easier to understand, the system architecture of this application will be described below with reference to the accompanying drawings.

[0063] See Figure 2 This application provides a system architecture 200. For example... Figure 2 As shown, the system architecture 200 may include a computing device cluster 210, which includes at least one computing device. The computing device may be a server, such as a cloud server, central server, edge server, or local server in a local data center. In some possible embodiments, the computing device may also be a desktop computer, laptop computer, or other terminal device.

[0064] Specifically, the multiple computing units (also called accelerator cards) used for training large artificial intelligence models can be distributed in any one or more computing devices in the aforementioned system architecture. The accelerator cards can be graphics processing units (GPUs), neural network processing units (NPUs), Ascend processors, etc., and are not limited here.

[0065] It is worth noting that, Figure 2The system architecture shown is merely an example and is not intended to limit its specific implementation to this example. For example, other possible system architectures may also include a client that provides the computing device with the training dataset and configuration information for the large AI model (such as termination conditions for model training).

[0066] In practical implementation, the model training method proposed in this application can be deployed as independent software on a server, interacting with the training framework through an external interface, or it can be deployed as program code within the training framework, interacting with the training framework through an internal interface. The training framework can be the MindSpore AI framework.

[0067] like Figure 3 As shown, Figure 3 These are different implementations of the model training method proposed in the embodiments of this application. For example... Figure 3 As shown in (a) of the diagram, the accuracy anomaly fault recovery architecture may include a training cluster scheduler 310, a training monitoring agent 320, a training framework 330, and a training process state control (TPSC) module 340. The training cluster scheduler 310 is used to start the training container and initiate the training monitoring agent and the training process. The training monitoring agent is used to monitor the running status of the training process. The training framework 330 is used to run the training process. The TPSC module 340 is used to implement the model training method proposed in this application to perform accuracy anomaly detection, control the training to stop after detecting an accuracy anomaly, and start resuming training after repairing the resuming training parameters. The training framework 330 calls the TPSC module 340 through an external interface to implement the model training method proposed in this application.

[0068] The TPSC module 340 includes several sub-modules: runtime configuration 341, precision parameter cache management 342, precision anomaly detection 343, and training execution control and state repair 344. Runtime configuration 341 configures TPSC runtime parameters, such as the callback function for training state recovery and the precision threshold. Precision parameter cache management 342 saves training parameters to the precision parameter cache when training accuracy is normal or loads the precision parameter cache during fault recovery. Precision anomaly detection 343 detects whether there are anomalies in model training accuracy. Training execution control and state repair 344 controls training to stop, repair the training state, and start resuming training after detecting and capturing precision anomalies. For example, it cleans up invalid training states after abnormal stoppage, repairs training parameters through the precision cache, repairs the number of steps for resuming training, and repairs the location of the resuming training dataset. Furthermore, as... Figure 3As shown in (b), another accuracy anomaly fault recovery architecture is provided. The TPSC module 340 can be placed in the training framework 330 and connected to the training framework 330 through an internal interface.

[0069] Currently, as the parameter scale of large-scale artificial intelligence models continues to expand, single-machine environments cannot effectively handle their training. Therefore, distributed training clusters are adopted for distributed training of large-scale artificial intelligence models. However, model training failures will lead to training interruptions, affecting the continuity and stability of model training. Therefore, existing solutions, upon detecting a training failure, will isolate the device, reallocate training containers for rescheduling, and restart a new training process to continue training. However, different training failures have different characteristics; using the same recovery process for all would lead to a waste of computing resources.

[0070] To address the current problem, this application proposes a model training method that, when abnormal training accuracy is detected, can continue training the model by loading the correct training parameters from local memory, thereby enabling breakpoint continuation training under large model accuracy abnormality faults.

[0071] The method flow provided in this application will be described below in conjunction with the aforementioned system architecture.

[0072] See Figure 4 The flowchart of a model training method provided in this application is as follows.

[0073] Step 401: Obtain the first intermediate result;

[0074] First, after training the first model begins, the first intermediate result of the model training can be obtained in each iteration step. This first intermediate result is the output result of the i-th iteration step when training the first model. For example, the first intermediate result can be the global gradient norm or the loss norm, etc.

[0075] The first model can be a large AI model such as a multimodal large model, a large language model, or a large vision model; no specific limitation is made here.

[0076] Step 402: When the first intermediate result does not meet the accuracy requirements, stop training the first model;

[0077] After obtaining the first intermediate result of the i-th iteration step, it can be determined whether the first intermediate result meets the accuracy requirements (or whether the accuracy is abnormal) based on the accuracy threshold. Not meeting the accuracy requirements (i.e., abnormal accuracy) can be defined as the first intermediate result being greater than the accuracy threshold, or greater than or equal to the accuracy threshold. Similarly, meeting the accuracy requirements (i.e., correct accuracy) can be defined as the first intermediate result being less than or equal to the accuracy threshold, or less than the accuracy threshold. The specific conditions for meeting the accuracy requirements are not limited.

[0078] One possible implementation is that when the first intermediate result is greater than the accuracy threshold, it indicates an accuracy anomaly, and training of the first model can be stopped. In this embodiment, stopping training means pausing the training process, but not exiting the training process.

[0079] In this embodiment, when an accuracy anomaly is detected, the training of the first model is stopped instead of exiting the training process, so that subsequent cluster rescheduling is not required, saving cluster scheduling time. This can shorten the training interruption time, increase the effective training time, and reduce the waste of computing resources caused by accuracy anomalies.

[0080] The accuracy threshold can be set manually or based on a sliding window algorithm, taking the average of the first intermediate result over K iterations as the accuracy threshold; the specific method is not limited here. For example, if the first intermediate result is the global gradient norm of the 6th iteration, the average of the global gradient norms of the 3rd to 5th iterations can be taken as the accuracy threshold.

[0081] One possible implementation is that when the second intermediate result is less than or equal to the accuracy threshold, indicating that the training accuracy is correct, it can be determined whether the j-th iteration is the update cycle of the accuracy parameter cache. Here, the second intermediate result is the output result of the j-th iteration training of the first model, and the update cycle indicates the iteration in which the accuracy parameter cache is updated. For example, if the accuracy parameter cache is updated every 10 iterations, then the update cycle T = 10i, and the 10th iteration, the 20th iteration, ..., the 10ith iteration will update the accuracy parameter cache.

[0082] Subsequently, when the second intermediate result is less than or equal to the accuracy threshold and the update cycle of the accuracy parameter cache is satisfied in the j-th iteration, the first model parameters are obtained. These first model parameters are the model parameters of the first model that are less than or equal to the accuracy threshold during training. The first model parameters are then stored in the accuracy parameter cache in local memory. For example, when the second intermediate result is less than or equal to the accuracy threshold, and the second intermediate result is an intermediate result of the 20th iteration, the update cycle condition of the accuracy parameter cache is satisfied, and the first model parameters of the 20th iteration can be stored in the accuracy parameter cache.

[0083] Step 403: Based on the accuracy parameter cache, continue training the first model.

[0084] In this embodiment of the application, after detecting abnormal training accuracy and stopping the training of the first model, the first model can continue to be trained according to the accuracy parameter cache in local memory.

[0085] In existing solutions, the checkpoint CKPT is stored in a remote storage system. This checkpoint CKPT is used to store training parameters with correct accuracy, while the accuracy parameter cache is a temporary storage in local memory used to store model parameters with correct accuracy. Instead of loading the checkpoint CKPT to repair training parameters, loading the accuracy parameter cache in local memory saves loading time, thereby shortening training interruption time and reducing the waste of computing resources caused by abnormal training accuracy.

[0086] One possible implementation is that after stopping training the first model, the parameters of the second model of the first model can be initialized to obtain the second model. Here, the second model parameters are the model parameters updated during the training of the first model, i.e., the model parameters that have changed compared to the initial model parameters. In this case, the second model is a continuation of training the first model; the first and second models are iterative updates of the same model, and the second model and the first model have the same model structure.

[0087] One possible implementation involves determining retraining parameters based on the accuracy parameter cache after initializing the second model parameters. These retraining parameters include one or more of the following: training parameters, training iteration step number, and retraining position of training data samples. They may also include necessary retraining parameters such as random states. The training parameters are the model parameters for continuing training the second model, the training step number indicates the iteration rounds for continuing training the second model, and the retraining position of training data samples indicates the batch of training data in the training data samples. Subsequently, the second model is trained again based on the retraining parameters.

[0088] Specifically, the first model parameters in the precision parameter cache that meet the precision requirements (or are accurately accurate) can be used as training parameters. The number of training steps can be determined based on the iteration epoch corresponding to these first model parameters. For example, if the first model parameters are the model parameters for the m-th epoch, then the number of training steps is the m-th iteration. Since training data samples are typically divided into multiple batches based on batch size, with each batch containing a certain amount of sample data, and one step corresponds to the training of one batch, the number of steps is usually equal to the total number of training data samples divided by the batch size, which can be expressed as steps per epoch = total number of samples / batch size. Therefore, based on the number of training steps, the position of the step in the current iteration epoch within the training data samples can be determined, thus identifying the position for continued training. Based on this position, the training batch data for the m-th iteration epoch can be determined.

[0089] Subsequently, the second model can be trained based on the parameters of the first model, the number of training steps, and the position of the training data samples for further training. This process is repeated in a new round of iterative training until training is complete.

[0090] The model training method proposed in this application can be connected to the training framework as an independent software component via an external interface to achieve rapid recovery of model training in scenarios of abnormal accuracy. The program code of the model training method proposed in this application can also be deployed within the training framework, which uses an internal interface to achieve rapid recovery of model training in scenarios of abnormal accuracy. The following descriptions illustrate deployment outside and within the training framework, respectively.

[0091] like Figure 5 As shown, Figure 5This diagram illustrates the interaction between the TPSC module and the training framework via an external interface. Before training the model, the training framework registers a callback function with the TPSC module. This callback function controls the training execution and parameter recovery of the training framework. Subsequently, for each iteration, the training framework reports the intermediate results of each iteration. For example, this intermediate result can be the global gradient norm or the loss norm; the specific choice is not limited here. After obtaining the global gradient norm, a precision threshold can be calculated. For instance, the global gradient norm values ​​of the previous 5 iterations can be taken, and their average value can be used as the precision threshold. Alternatively, this precision threshold can be manually set based on experience. Then, based on the calculated precision threshold, it is determined whether the global gradient norm of the current iteration is greater than the precision threshold. When the global gradient norm is greater than the precision threshold, the training of the model can be stopped by calling the callback interface cb_stop_train provided by the training framework. The training process does not need to be exited; only its execution needs to be stopped. The `cb_clean_train` interface is then called back to clean up the training state, including temporary states during the training process or temporary states of the GPU / NPU. Next, the precision parameter cache is loaded, and the `cb_repair_train` interface is called back to repair the training parameters (e.g., weight parameters, optimizer parameters), training steps, and the position for resuming training on the training data samples, based on the correct precision parameter cache. Finally, the process returns to the training flow to resume training on the model from breakpoints. After completing the current iteration, a new iteration will begin until training is complete.

[0092] The precision parameter cache is obtained by calling the cb_get_train_param interface when the global gradient norm is less than or equal to the precision threshold and the training epoch meets the precision parameter cache update cycle. The training parameters are the model training parameters for this training epoch, and the precision parameter cache is used to store these training parameters.

[0093] like Figure 6 As shown, Figure 6This diagram illustrates the deployment of the TPSC module within the training framework. The program code implementing the model training method is included in the training framework's program code. No callback access to the training framework is required, therefore, no callback function registration is necessary; the internal interfaces provided by the training framework can be called directly. The global gradient norm for each iteration is obtained. Then, based on a calculated precision threshold or a manually set precision threshold, it is determined whether there is a precision anomaly. Specifically, a precision anomaly can be determined by whether the obtained global gradient norm is greater than the precision threshold. When the global gradient norm for the i-th iteration is greater than the precision threshold, it indicates a precision anomaly. At this point, the training execution control and state repair module is entered. The training stop interface of the framework is called to stop model training. Subsequently, the training state cleanup interface of the framework is called to initialize the model parameters updated during model training, i.e., to clean up the training state, including the temporary state of the training process and the temporary state of the GPU / NPU. When the global gradient norm of the j-th iteration is less than or equal to the accuracy threshold, the accuracy is considered correct, and the system enters the accuracy parameter cache management. Subsequently, it can be determined whether the j-th iteration has reached the accuracy parameter cache update cycle. If the j-th iteration meets the accuracy parameter cache update cycle, the framework's training parameter acquisition interface is called to obtain the model parameters with correct accuracy for the j-th iteration and store them in the accuracy parameter cache. Then, the framework's training parameter loading interface is called to load the accuracy parameter cache, update the step count and training data sample continuation training position, repair the continuation training parameters, and then perform breakpoint continuation training based on the continuation training parameters. After completing the training of the current iteration, a new iteration training will begin until training ends.

[0094] In the embodiments of this application, such as Figure 7 As shown, Figure 7 This diagram compares in-situ process recovery and process restart recovery when accuracy anomalies occur. The model training method proposed in this application performs in-situ process recovery upon detecting accuracy anomalies, primarily applicable to rapid recovery scenarios for large model training accuracy anomalies. For the in-situ process recovery scheme, a recovery process is initiated after detecting an accuracy anomaly. By stopping the training model instead of rescheduling the training task, the training process does not need to exit, thus avoiding cluster rescheduling as required by the process restart recovery scheme, saving cluster scheduling time. Subsequently, the training state can be cleared instead of restarting the training process, avoiding time-consuming processes such as communication domain connection establishment and graph compilation, reducing computational resource waste. Furthermore, loading the accuracy parameter cache in local memory can replace loading the remote CKPT, saving time spent fetching and loading the remote CKPT. In addition, since the training process is not restarted, there is no need to re-establish communication connections.

[0095] The method flow provided in this application has been described above. The apparatus provided in this application will now be described based on the aforementioned method flow.

[0096] See Figure 8 The structural schematic diagram of a model training device provided in this application is as follows.

[0097] The acquisition module 801 is used to acquire the first intermediate result, which is the output result of the i-th iteration training of the first model;

[0098] The training module 802 is used to continue training the first model based on the precision parameter cache in local memory when the first intermediate result does not meet the precision requirements. The precision parameter cache includes the first model parameters, which are the model parameters that meet the precision requirements during the training process.

[0099] In one possible implementation, before continuing to train the first model based on the precision parameter cache in local memory, the aforementioned acquisition module 801 is further configured to acquire the first model parameters when the second intermediate result meets the precision requirements and the j-th iteration round meets the update cycle of the precision parameter cache, wherein the second intermediate result is the output result of the j-th iteration training of the first model; and the storage module 803 is configured to store the first model parameters in the precision parameter cache.

[0100] In one possible implementation, before continuing to train the first model based on the precision parameter cache in local memory, the device may further include: an initialization module 804 for initializing the second model parameters of the first model to obtain a second model, wherein the second model parameters are the model parameters updated when training the first model.

[0101] In one possible implementation, the aforementioned training module 802 is specifically used to: determine the continuing training parameters based on the accuracy parameter cache, wherein the continuing training parameters include one or more of the following: training parameters, training iteration steps, and continuing training positions of training data samples; the training parameters are model parameters for continuing training of the second model, and the training iteration steps are used to indicate the number of iterations for continuing training of the second model; and continue training the second model based on the continuing training parameters.

[0102] In one possible implementation, the aforementioned training module 802 is specifically used to: continue training the second model based on the first model parameters, the number of training iteration steps, and the training data sample continuation position.

[0103] This application also provides a computer-readable storage medium storing a program that, when run on a computer, causes the computer to perform the aforementioned actions. Figure 4 The steps in the method described in the illustrated embodiment.

[0104] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned actions. Figure 4 The method steps described in the illustrated embodiment.

[0105] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0106] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, the specific working process of the systems, devices and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0107] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0108] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0109] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0110] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0111] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0112] Finally, it should be noted that the above are only specific embodiments of this application, but the protection scope of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application.

Claims

1. A model training method, characterized in that, include: Obtain the first intermediate result, which is the output result of the i-th iteration training of the first model; When the first intermediate result does not meet the accuracy requirement, the first model continues to be trained according to the accuracy parameter cache in the local memory. The accuracy parameter cache includes the first model parameters, which are the model parameters of the first model that meet the accuracy requirement during the training process.

2. The method according to claim 1, characterized in that, Before continuing training of the first model based on the precision parameter cache in local memory, the method further includes: When the second intermediate result meets the accuracy requirement and the j-th iteration round meets the update cycle of the accuracy parameter cache, the first model parameters are obtained, and the second intermediate result is the output result of the j-th iteration training of the first model; The first model parameters are stored in the precision parameter cache.

3. The method according to claim 1 or 2, characterized in that, Before continuing training of the first model based on the precision parameter cache in local memory, the method further includes: The second model parameters of the first model are initialized to obtain the second model, where the second model parameters are the model parameters updated when training the first model.

4. The method according to claim 3, characterized in that, The step of continuing to train the first model based on the precision parameter cache in local memory includes: Based on the accuracy parameter cache, the training continuation parameters are determined. The training continuation parameters include one or more of the following: training parameters, training iteration steps, and training data sample continuation positions. The training parameters are the model parameters for continuing to train the second model, and the training iteration steps are used to indicate the iteration rounds for continuing to train the second model. The second model is then trained again based on the training parameters.

5. The method according to claim 4, characterized in that, The step of continuing to train the second model according to the training parameters includes: The second model is trained again based on the first model parameters, the number of training iteration steps, and the training data sample continuation position.

6. A model training device, characterized in that, include: The acquisition module is used to acquire a first intermediate result, which is the output result of the i-th iteration training of the first model; The training module is used to continue training the first model based on the accuracy parameter cache in local memory when the first intermediate result does not meet the accuracy requirement. The accuracy parameter cache includes the first model parameters, which are the model parameters of the first model that meet the accuracy requirement during the training process.

7. The apparatus according to claim 6, characterized in that, Before continuing training the first model based on the precision parameter cache in local memory. The acquisition module is further configured to acquire the first model parameters when the second intermediate result meets the accuracy requirement and the j-th iteration round meets the update cycle of the accuracy parameter cache, wherein the second intermediate result is the output result of the j-th iteration training of the first model; A storage module is used to store the first model parameters into the precision parameter cache.

8. The apparatus according to claim 6 or 7, characterized in that, Before continuing to train the first model based on the precision parameter cache in local memory, the device further includes: An initialization module is used to initialize the second model parameters of the first model to obtain a second model, wherein the second model parameters are the model parameters updated when training the first model.

9. The apparatus according to claim 8, characterized in that, The training module is specifically used for: Based on the accuracy parameter cache, the training continuation parameters are determined. The training continuation parameters include one or more of the following: training parameters, training iteration steps, and training data sample continuation positions. The training parameters are the model parameters for continuing to train the second model, and the training iteration steps are used to indicate the iteration rounds for continuing to train the second model. The second model is then trained again based on the training parameters.

10. The apparatus according to claim 9, characterized in that, The training module is specifically used for: The second model is trained again based on the first model parameters, the number of training iteration steps, and the training data sample continuation position.

11. A model training device, characterized in that, include: A processor for executing a program that causes the model training apparatus to perform the method as described in any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that, Includes instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 5.

13. A computer program product containing instructions, characterized in that, When it is run on a computer, it causes the computer to perform the method as described in any one of claims 1 to 5.

Images