Method and apparatus for saving checkpoint file for training task, device, and medium

Abstract

The present application provides a method and apparatus for saving a checkpoint file for a training task, a device, and a medium. The method comprises: during synchronous training and saving of a large language model, deconstructing a checkpoint file corresponding to a training task of the current iteration of the large language model to obtain a plurality of pieces of data to be saved, and saving said data into a shared memory; and in an asynchronous service storage process, restoring, from the shared memory, the plurality of pieces of said data corresponding to the training task of the current iteration into the checkpoint file and storing the checkpoint file in a shared storage device. In the present application, a checkpoint file is deconstructed into a plurality of pieces of data to be saved and saved in a shared memory, then the plurality of pieces of said data are extracted from the shared memory, merged into the checkpoint file and stored in the shared storage device, which avoids storing the checkpoint file in a shared storage device after coupling the checkpoint file with a training framework in a training process, and also eliminates the need for deep understanding of a training framework mechanism to the save checkpoint file, thereby providing wide applicability.

Description

Checkpoint file saving method, device and equipment for training task and medium

[0001] Cross-reference to related applications

[0002] The present application claims priority to the Chinese patent application No. 202411377340.6, filed on September 30, 2024, and entitled "Checkpoint file saving method, device and equipment for training task and medium", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0003] The present application relates to the field of artificial intelligence, in particular to a checkpoint file saving method, device and equipment for training task and medium. BACKGROUND

[0004] In the evolution of artificial intelligence, the rise of large models marks a significant leap in computing power and data scale. These models, known for their massive number of parameters, have rapidly grown from early millions to current billions or even tens of billions. This scale of expansion has enabled large models to exhibit unprecedented capabilities in complex tasks such as language understanding, image and video analysis. They not only push the boundaries of technology, but also provide new perspectives and solutions to real-world problems.

[0005] At the same time, the rapid development of large models also brings a series of technical challenges, especially in the management of model checkpoints. Checkpoint is a key mechanism in the machine learning process, used to save the state of the model for subsequent training or model recovery. In large-scale distributed training environments, efficient storage and fast recovery of checkpoints become particularly important. Distributed training is a method of parallel processing by dividing and distributing the training tasks of deep learning models to multiple computing devices (which can be referred to as training nodes, such as GPUs (Graphics Processing Units), which play a core role in distributed training by significantly accelerating the training process of deep learning models through parallel computing). This technology can effectively cope with the computing challenges brought by large-scale data sets and complex models, significantly improving training efficiency.

[0006] The related technical solutions simply use shared memory to accelerate checkpoint file saving, and only target single saving scenarios for a certain iteration. Moreover, the solutions often directly store the checkpoint file in the shared storage device after coupling it with the training framework during the training process. At this time, researchers need to fully understand the training framework mechanism to save the checkpoint file, which lacks universality. SUMMARY

[0007] The application provides a training task checkpoint file saving method, comprising:

[0008] In the synchronous training saving process of the large language model, the checkpoint file corresponding to the training task of the current iteration of the large language model is deconstructed to obtain a plurality of to-be-saved data, and the to-be-saved data is saved to the shared memory; and

[0009] In the asynchronous service storage process of the large language model, the plurality of to-be-saved data corresponding to the training task of the current iteration is recovered from the shared memory into a checkpoint file and stored in the shared storage device.

[0010] According to the training task checkpoint file saving method provided by the application, the checkpoint file corresponding to the training task of the current iteration of the large language model is deconstructed to obtain a plurality of to-be-saved data, and the to-be-saved data is saved to the shared memory, comprising:

[0011] A shared memory processing engine for processing the shared memory is constructed, so that the shared memory processing engine deconstructs the checkpoint file corresponding to the training task of the current iteration to obtain a plurality of to-be-saved data; and

[0012] A plurality of shared memory objects are constructed, so that each shared memory object saves one to-be-saved data in the plurality of to-be-saved data in the shared memory.

[0013] According to the training task checkpoint file saving method provided by the application, a plurality of shared memory objects are constructed, so that each shared memory object saves one to-be-saved data in the plurality of to-be-saved data in the shared memory, comprising:

[0014] The shared memory object calculates the space capacity occupied by one to-be-saved data in the plurality of to-be-saved data, and determines the space capacity;

[0015] The shared memory object divides a memory space corresponding to the space capacity in the shared memory, and stores the to-be-saved data in the divided memory space; and

[0016] The shared memory object configures an identifier for the to-be-saved data, and accordingly,

[0017] An identifier space is divided in the shared memory, identifiers corresponding to the plurality of to-be-saved data are merged into an identifier dictionary, and the identifier dictionary is saved in the identifier space.

[0018] According to the training task checkpoint file saving method provided by the application, the plurality of to-be-saved data is saved in the shared memory in order, and accordingly, a plurality of shared memory objects are constructed, comprising:

[0019] determining the current training node code and the current to-be-saved data sequence number when the current to-be-saved data needs to be saved in the shared memory;

[0020] constructing a shared memory object based on the current training node code and the current to-be-saved data sequence number, and configuring an object name for the constructed shared memory object, the object name containing the current training node code and the current to-be-saved data sequence number; and

[0021] Correspondingly, the current to-be-saved data corresponding identifier includes the current training node code and the current to-be-saved data sequence number; the dictionary name of the identifier dictionary to which the current to-be-saved data corresponding identifier belongs includes the current training node code; and the name of the memory space corresponding to the current to-be-saved data contains the current training node code and the current to-be-saved data sequence number.

[0022] According to the training task checkpoint file saving method provided in the present application, the plurality of to-be-saved data corresponding to the current iteration training task is recovered into a checkpoint file from the shared memory and stored into a shared storage device, comprising:

[0023] After monitoring that the plurality of to-be-saved data corresponding to the current iteration training task is successfully saved in the shared memory, the current training node code and the current to-be-saved data sequence number are obtained;

[0024] Based on the current training node code and the to-be-saved data sequence number, an identifier dictionary is screened out in an identifier space, and identifiers corresponding to each to-be-saved data are screened out in the identifier dictionary;

[0025] Based on the identifiers corresponding to each to-be-saved data, the to-be-saved data is obtained from the corresponding memory space; and

[0026] The plurality of to-be-saved data is recovered into a checkpoint file and stored into a shared storage device.

[0027] According to the training task checkpoint file saving method provided in the present application, the method further comprises:

[0028] Obtaining the duration of the to-be-saved data entering the saving process in the current iteration process;

[0029] When the duration exceeds the first duration, prompting the sequence number of the successfully saved to-be-saved data and the sequence number of the to-be-saved data currently being saved; and

[0030] When the duration exceeds the second duration, prompting that the current iteration process fails;

[0031] The second duration is longer than the first duration;

[0032] The first time length is half of the longest complete time length of successful saving of the data to be saved in all iteration processes before the current iteration process;

[0033] The second time length is an average time length of the complete time length of successful saving of the data to be saved in all iteration processes before the current iteration process.

[0034] According to the training task checkpoint file saving method provided in the application, the method further comprises:

[0035] Based on the number of shares of the data to be saved, the number of saving times is obtained; and

[0036] When it is determined that the continuous time length exceeds the first time length, if the serial number of the data to be saved being currently saved is less than the number of saving times, a new shared memory object is created based on the number of saving times.

[0037] According to the training task checkpoint file saving method provided in the application, the method further comprises:

[0038] When a piece of data to be saved is successfully saved, a saving record is generated; the saving record contains a record number, an iteration process number and a total number of saving; and

[0039] When a new piece of data to be saved is successfully saved, if the iteration process number in the generated saving record is the same as the iteration process number in the existing saving record, the record number and the total number of saving in the existing saving record are updated.

[0040] According to the training task checkpoint file saving method provided in the application, the method further comprises:

[0041] Before performing the synchronous training saving process of the current iteration process, it is detected whether the corresponding multiple pieces of data to be saved of the last iteration process are restored into the checkpoint file and successfully stored to the shared storage device, if the checkpoint file is successfully stored to the shared storage device, the synchronous training saving process of the current iteration process is performed.

[0042] According to the training task checkpoint file saving method provided in the application, the method further comprises:

[0043] A loop detection logic is configured, and a loop exit logic is configured, if it is judged in the loop logic that the checkpoint file is not successfully stored to the shared storage device, the loop is blocked; and

[0044] The loop exit logic is executed, if it is judged that the checkpoint file is not successfully stored to the shared storage device within a preset time, the whole training task is terminated.

[0045] According to the training task checkpoint file saving method provided in the application, the checkpoint file is stored into the shared storage device, comprising:

[0046] In the synchronization training saving process, for the shared memory object created in the current iteration process, the shared memory object is continuously used in the next iteration process.

[0047] According to the training task checkpoint file saving method provided in the application, the method further comprises:

[0048] When the current iteration process number reaches the preset number, and it is detected that the training task is updated, the shared memory object used in the current iteration process is deleted.

[0049] According to the training task checkpoint file saving method provided in the application, the checkpoint file corresponding to the training task of the current iteration of the large language model is deconstructed to obtain a plurality of to-be-saved data, including:

[0050] The checkpoint file corresponding to the training task of the current iteration of the large language model is deconstructed and analyzed to obtain the weight of the large language model, the state of the optimizer, and the activation value; and

[0051] The plurality of to-be-saved data is determined according to the weight of the large language model, the state of the optimizer, and the activation value.

[0052] According to the training task checkpoint file saving method provided in the application, the checkpoint file corresponding to the training task of the current iteration of the large language model is deconstructed and analyzed to obtain the weight of the large language model, the state of the optimizer, and the activation value, including:

[0053] The checkpoint file corresponding to the training task of the current iteration of the large language model is traversed to obtain a tensor of the checkpoint file, and the tensor is used to store the weight of the large language model, the state of the optimizer, and the activation value;

[0054] The plurality of to-be-saved data is determined according to the weight of the large language model, the state of the optimizer, and the activation value, including:

[0055] The attribute of the tensor is obtained, and the attribute of the tensor is taken as the to-be-saved data.

[0056] According to the training task checkpoint file saving method provided in the application, when the current iteration is the first training before the iteration training of the non-existing iteration process, the first time length is a configured standard time length.

[0057] According to the training task checkpoint file saving method provided in the application, the training task checkpoint file saving method further comprises:

[0058] When the current iteration process number reaches the preset number, the shared memory object used is saved.

[0059] According to the training task checkpoint file saving method provided in the application, after the step of deleting the shared memory object used in the current iteration process, the method further comprises the following steps of:

[0060] Starting to rebuild a new shared memory object.

[0061] The application further provides a training task checkpoint file saving device, comprising:

[0062] The synchronization processing module is configured to, in the synchronization training saving process of the large language model, deconstruct the checkpoint file corresponding to the training task of the current iteration of the large language model to obtain a plurality of to-be-saved data, and save the plurality of to-be-saved data to the shared memory.

[0063] The asynchronous processing module is configured to, in the asynchronous service storage process of the large language model, restore the plurality of to-be-saved data corresponding to the training task of the current iteration from the shared memory into the checkpoint file and store the checkpoint file in the shared storage device.

[0064] The application further provides an electronic device comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor implements any of the above training task checkpoint file saving methods when executing the program.

[0065] The application further provides a non-transitory computer-readable storage medium having computer-readable instructions stored thereon, wherein the computer-readable instructions are executed by a processor to implement any of the above training task checkpoint file saving methods.

[0066] The application further provides a computer-readable instruction product comprising computer-readable instructions, wherein the computer-readable instructions are executed by a processor to implement any of the above training task checkpoint file saving methods. BRIEF DESCRIPTION OF DRAWINGS

[0067] In order to more clearly illustrate the technical solutions of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0068] FIG. 1 is a flowchart of a training task checkpoint file saving method according to one or more embodiments of the application.

[0069] FIG. 2 is a schematic diagram of creating a shared memory object according to one or more embodiments of the application.

[0070] FIG. 3 is a schematic diagram of saving a plurality of to-be-saved data and an identifier according to one or more embodiments of the application.

[0071] FIG. 4 is a specific flow diagram of a training task checkpoint file saving method provided by one or more embodiments of the present application.

[0072] FIG. 5 is a structural diagram of a training task checkpoint file saving apparatus provided by one or more embodiments of the present application.

[0073] FIG. 6 is a structural diagram of an electronic device provided by one or more embodiments of the present application. DETAILED DESCRIPTION

[0074] For the purposes of the present application, the technical solutions and advantages thereof will be more clearly apparent from the following description of the technical solutions of the present application, which will be described in conjunction with the accompanying drawings of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0075] The training task checkpoint file saving method, apparatus, device and medium of the present application will be described below in conjunction with FIGS. 1-6.

[0076] FIG. 1 shows a flow diagram of a training task checkpoint file saving method provided by the present application, referring to FIG. 1, the method includes the following steps:

[0077] Step 11, in the synchronization training saving process of the large language model, the checkpoint file corresponding to the training task of the current iteration of the large language model is deconstructed to obtain a plurality of to-be-saved data, and the plurality of to-be-saved data is saved to the shared memory.

[0078] Step 12, in the asynchronous service storage process of the large language model, the plurality of to-be-saved data corresponding to the training task of the current iteration is recovered from the shared memory into a checkpoint file and stored in the shared storage device.

[0079] For the above steps 11-12, it should be noted that in the evolution of artificial intelligence, the rise of large models (such as large language models) marks a huge leap in computing power and data size. These models, known for their large number of parameters, have rapidly grown from early million-level parameters to current billion- or even hundred-billion-level parameters. This scale of expansion has enabled large models to exhibit unprecedented capabilities in complex tasks such as language understanding, image and video analysis, etc. They not only push the boundaries of technology, but also provide new perspectives and solutions to real-world problems.

[0080] Meanwhile, the rapid development of large models also brings a series of technical challenges, especially in the management of model checkpoints. Checkpoint is a key mechanism in the machine learning process, used to save the state of the model for subsequent training or model recovery. In a large-scale distributed training environment, efficient storage and fast recovery of checkpoints become particularly important. Distributed training is a method of parallel processing by dividing and distributing the training tasks of deep learning models to multiple computing devices (which can be referred to as training nodes, such as GPUs, which play a core role in distributed training, significantly accelerating the training process of deep learning models through parallel computing).

[0081] The method of the present application is suitable for the scenario where the model checkpoint needs to be split and saved, that is, the checkpoint file needs to be deconstructed into multiple data to be saved, and these data to be saved can be saved. Deconstructing the checkpoint file means analyzing and analyzing the checkpoint file generated in the deep learning training process to obtain the weight, optimizer state, activation value and other data of the model. Specifically, since the main element in the checkpoint file is a tensor (i.e. tensor), which is a multi-dimensional array, in deep learning, tensors are used to store model weights, optimizer states, activation values, and other attribute data. Therefore, the checkpoint file is first traversed to obtain the tensor therein, and then the tensor attribute is obtained. The obtained tensor attribute is used as the data to be saved, which is saved in the subsequent processing.

[0082] In the present application, the checkpoint file is deconstructed into multiple data to be saved and saved to shared memory. Shared memory is an efficient inter-process communication (IPC) mechanism that allows multiple processes to directly access the same physical memory area, enabling fast data exchange and sharing, and significantly improving training efficiency.

[0083] In the present application, the multiple data to be saved also need to be restored into a checkpoint file and stored in a shared storage device. The shared storage device is a persistent storage device (such as a hard disk).

[0084] The checkpoint file is stored in the shared storage device, which is regarded as shared storage landing. The shared storage landing refers to the process of writing data from memory or other temporary storage media to persistent storage devices in a distributed system, and refers to the process of writing data to persistent storage devices (such as hard disks) after sharing between multiple computing devices. This operation ensures the persistence and consistency of data, maintains the integrity of data even after system failure or restart, and facilitates subsequent use during the training process.

[0085] As can be seen from the above, each training node in the distributed training can create a mechanism for synchronously saving the training process and a mechanism for asynchronously serving the storage process when saving the to-be-saved data and storing the checkpoint file during the execution of its own training task. In the synchronous training saving process, the checkpoint file corresponding to the training task of the current iteration is deconstructed into multiple to-be-saved data, and the multiple to-be-saved data are saved to the shared memory. In the asynchronous service storage process, the multiple to-be-saved data corresponding to the current iteration process are recovered from the shared memory into a checkpoint file and stored in the shared storage device.

[0086] The training task checkpoint file saving method provided by the present application is suitable for the model checkpoint direct whole saving scene of the prior art. By deconstructing the checkpoint file into multiple to-be-saved data for saving in the shared memory, and extracting the multiple to-be-saved data from the shared memory to merge into a checkpoint file for storage in the shared storage device, the checkpoint file is avoided from being directly stored in the shared storage device after being coupled with the training framework during the training process, and the checkpoint file is also avoided from being saved only after completely understanding the training framework mechanism, thereby having universality.

[0087] In the further method of the training task checkpoint file saving method, the process of deconstructing the checkpoint file corresponding to the training task of the current iteration of the large language model to obtain multiple to-be-saved data and saving the to-be-saved data to the shared memory is explained as follows:

[0088] A shared memory processing engine for processing the shared memory is constructed, so that the checkpoint file corresponding to the training task of the current iteration of the shared memory processing engine is deconstructed into multiple to-be-saved data.

[0089] A plurality of shared memory objects are constructed, so that each shared memory object saves one to-be-saved data in the shared memory from the multiple to-be-saved data.

[0090] To this end, it should be noted that the training node (such as a GPU) needs to create a shared memory processing engine (shm_engine) to operate and maintain the shared memory of the data to be saved. The shared memory processing engine (shm_engine) is a software module or library for creating, managing and operating shared memory areas. In a distributed system, it allows different processes to exchange data efficiently through shared memory.

[0091] In this application, the shared memory processing engine is used to deconstruct the checkpoint file in the current iteration process of the training task into multiple data to be saved.

[0092] Since the checkpoint file is deconstructed into multiple data to be saved, in order to distinguish and save the data to be saved, multiple shared memory objects (shm objects) are constructed, each responsible for the saving operation of one data to be saved, and each shared memory object does not interfere with each other. Each shared memory object saves one data to be saved in the shared memory.

[0093] As shown in FIG. 2, the training node 1 creates a shared memory processing engine (shm_engine), and then creates a shared memory (shm) object 1, a shared memory (shm) object 2 and a shared memory (shm) object 3, which respectively save the 3 data to be saved deconstructed from the checkpoint file. Each data to be saved is saved once.

[0094] The further method of the present application can separate the saving of the data to be saved by creating a shared memory processing engine (shm_engine) and a shared memory (shm) object, making the saving method more flexible, avoiding mutual interference, and speeding up the saving efficiency.

[0095] In the further method of the above training task checkpoint file saving method, the process of constructing multiple shared memory objects and making each shared memory object save one data to be saved in the shared memory is mainly explained as follows:

[0096] The shared memory object calculates the space capacity of one data to be saved in the multiple data to be saved, and determines the space capacity.

[0097] The shared memory object divides a memory space corresponding to the space capacity in the shared memory, and stores the data to be saved in the divided memory space.

[0098] The shared memory object configures an identifier for the data to be saved, and accordingly,

[0099] An identifier space is divided in the shared memory, the identifiers corresponding to the multiple data to be saved are merged into an identifier dictionary, and the identifier dictionary is saved in the identifier space.

[0100] It should be noted that, in order to distinguish and save the to-be-saved data, a corresponding memory space needs to be divided for each to-be-saved data. Therefore, the shared memory object calculates the occupied space capacity of one to-be-saved data in the plurality of to-be-saved data, determines the space capacity, and then divides a memory space corresponding to the space capacity in the shared memory, and stores the to-be-saved data in the divided memory space.

[0101] The memory space is divided in order to store the to-be-saved data separately, and in order to be able to search the corresponding to-be-saved data from the memory space subsequently, an identifier corresponding to the to-be-saved data needs to be established. Therefore, the shared memory object configures an identifier for the to-be-saved data, and the identifier is used as metadata of the to-be-saved data, and the corresponding to-be-saved data can be searched based on the metadata.

[0102] Correspondingly, an identifier space is divided in the shared memory, identifiers corresponding to the plurality of to-be-saved data are combined into an identifier dictionary, and the identifier dictionary is saved in the identifier space.

[0103] Referring to FIG. 3, the shared memory processing engine (shm_engine) saves n to-be-saved data n times. Each to-be-saved data and identifier is regarded as a saving combination. Referring to the shared memory (shm) to-be-saved data 1 and the shared memory (shm) identifier (metadata) 1, …, the shared memory (shm) to-be-saved data n and the shared memory (shm) identifier n in FIG. 3, each saving combination corresponds to the first saving of the current iteration, …, the n-th saving of the current iteration.

[0104] The further method of the present application saves the to-be-saved data by dividing the memory space and configuring the identifier of each to-be-saved data, which is beneficial to subsequent query and accurate to-be-saved data acquisition in the memory, guarantees the integrity of the data, and is beneficial to the progress of the iterative training.

[0105] In the further method of the above training task checkpoint file saving method, the processing process of constructing a plurality of shared memory objects is mainly explained and described as follows:

[0106] When it is determined that the current to-be-saved data needs to be saved in the shared memory, the code of the current training node and the serial number of the current to-be-saved data are obtained.

[0107] Based on the code of the current training node and the serial number of the current to-be-saved data, a shared memory object is constructed, and an object name is configured for the constructed shared memory object, the object name containing the code of the current training node and the serial number of the current to-be-saved data.

[0108] Correspondingly, the identifier corresponding to the current to-be-saved data includes the code of the current training node and the serial number of the current to-be-saved data; the dictionary name of the identifier dictionary of the identifier corresponding to the current to-be-saved data includes the code of the current training node; and the name of the memory space corresponding to the current to-be-saved data contains the code of the current training node and the serial number of the current to-be-saved data.

[0109] To this end, it should be noted that, in order to accurately and correctly perform and use the saving of different training nodes and different to-be-saved data, when it is monitored that the current to-be-saved data needs to be saved in the shared memory, the code of the training node to which the to-be-saved data belongs and the serial number of the current to-be-saved data are obtained at this time.

[0110] Based on the code of the training node to which the to-be-saved data belongs and the serial number of the current to-be-saved data, a name is generated, which can be used as the object name of the constructed shared memory object.

[0111] In distributed training, each graphics processing unit (GPU) can indeed be regarded as an independent identifier (rank), especially during the process of using a data parallel model. In distributed training, the rank is a unique identifier used to distinguish different processes or devices. Each graphics processing unit (GPU) participating in training is assigned a unique identifier (rank) value. For this purpose, the identifier (rank) value can be used as the code of the training node.

[0112] For example, shm_rank_0 represents the name of the shared memory processing engine (shm_engine) held by rank_0, and the name of the shared memory object can be shm_1_rank_0, where rank_0 is used as the code of the training node, 1 in shm_1 is used as the serial number of the to-be-saved data, and shm is used as the abbreviation of the object.

[0113] Correspondingly, the identifier corresponding to the current to-be-saved data includes the code of the current training node and the serial number of the current to-be-saved data. shm_1_rank_0_meta. meta is used as the abbreviation of metadata or identifier.

[0114] The dictionary name of the identifier dictionary of the identifier corresponding to the current to-be-saved data includes the code of the current training node. For example, the dictionary contains shm_1_rank_0_meta, shm_2_rank_0_meta, shm_3_rank_0_meta, shm_4_rank_0_meta, and so on.

[0115] The name of the memory space corresponding to the current to-be-saved data contains the code of the current training node and the serial number of the current to-be-saved data. For example, space_1_rank_0. Space is short for space.

[0116] The further method of the present application can establish the connection of the content on each environment by configuring the name of the required content of each processing link based on the code of the current training node and the serial number of the current to-be-saved data, facilitate accurate extraction of data from the memory, and improve the correct use of data.

[0117] In the further method of the above training task checkpoint file saving method, the process of recovering the plurality of to-be-saved data corresponding to the current iteration training task from the shared memory into a checkpoint file and storing it into the shared storage device is mainly explained and described, and the specific process is as follows.

[0118] After monitoring that the plurality of to-be-saved data corresponding to the current iteration training task are successfully saved in the shared memory, the code of the current training node and the serial number of the current to-be-saved data are obtained.

[0119] Based on the code of the current training node and the serial number of the to-be-saved data, an identification dictionary is filtered out in an identification space, and the identification corresponding to each to-be-saved data is filtered out in the identification dictionary.

[0120] Based on the identification corresponding to each to-be-saved data, the to-be-saved data is obtained from the corresponding memory space.

[0121] The plurality of to-be-saved data are recovered into a checkpoint file and stored into the shared storage device.

[0122] For this, it needs to be explained that the trigger condition of the mechanism of the asynchronous service storage process is that the plurality of to-be-saved data corresponding to the checkpoint file in the current iteration process are successfully saved in the shared memory. Therefore, it is necessary to continuously monitor whether the to-be-saved data in the synchronization state has been completely saved into the shared memory. If it has not been completely saved into the shared memory, it continues to be monitored. If it has been completely saved into the shared memory, the code of the current training node and the serial number of the current to-be-saved data are obtained.

[0123] Then, based on the code of the current training node and the serial number of the to-be-saved data, an identification dictionary is filtered out in an identification space, and the identification corresponding to each to-be-saved data is filtered out in the identification dictionary. Then, based on the identification corresponding to each to-be-saved data, the to-be-saved data is obtained from the corresponding memory space, and the plurality of to-be-saved data are recovered into a checkpoint file and stored into the shared storage device.

[0124] The further method of the application, through the mechanism of the asynchronous service storage process triggered, based on the encoding of the training node and the serial number of the data to be saved, step by step screening and obtaining the corresponding data to be saved from the memory space, realizing the information association of each link in the synchronous training saving process, can quickly lock the memory space and quickly extract the data to be saved.

[0125] In the further method of the above training task checkpoint file saving method, mainly the condition of saving the data to be saved in the shared memory is monitored and explained, as follows:

[0126] Obtain the duration of the data to be saved in the current iteration process entering the saving process;

[0127] When the duration exceeds the first duration, prompt the serial number of the successfully saved data to be saved and the serial number of the currently saved data to be saved;

[0128] When the duration exceeds the second duration, prompt the current iteration process failure;

[0129] Wherein, the second duration is longer than the first duration;

[0130] The first duration is half of the longest complete duration of the successfully saved data to be saved in all iteration processes before the current iteration process;

[0131] The second duration is the average duration of the complete duration of the successfully saved data to be saved in all iteration processes before the current iteration process.

[0132] For this, it should be noted that the trigger condition of the mechanism of the asynchronous service storage process is that the multiple data to be saved corresponding to the checkpoint file in the current iteration process are successfully saved in the shared memory. The saving condition of the data to be saved in the shared memory needs to be monitored continuously. However, there are individual times of saving process that takes a long time or training failure that causes the information to be unsuitable for saving and stagnation. In order to know the saving condition in the first time and make timely response strategy, the saving condition can be monitored at a certain time point and the specific saving condition can be prompted.

[0133] At this time, the duration of the saving process of the to-be-saved data in the current iteration process is obtained. The duration is calculated from the beginning of the iteration. First, a first duration is set, which is half of the longest complete duration of the to-be-saved data successfully saved in all iteration processes before the current iteration process. If it is the first training process, the first duration can be configured as a standard duration. When the second training process, it can be regarded as the first iteration process, so the first duration is half of the complete duration of the to-be-saved data successfully saved in the first training process. The complete duration is the duration from the beginning of the iteration to the end of the monitoring of the to-be-saved data. When the n th iteration process, the longest complete duration can be obtained from the multiple complete durations obtained from the previous multiple iteration processes, and then half of the longest complete duration is taken as the first duration.

[0134] Correspondingly, the second duration is the average duration of the complete duration of the to-be-saved data successfully saved in all iteration processes before the current iteration process. That is, the average duration can be obtained from the multiple complete durations obtained from the previous multiple iteration processes.

[0135] When it is determined that the duration exceeds the first duration, the serial number of the to-be-saved data successfully saved is prompted, and the serial number of the to-be-saved data currently being saved is prompted. At this time, the specific saving situation is clear, and the R&D personnel are prompted.

[0136] When it is determined that the duration exceeds the second duration, it is prompted that the current iteration process fails. Since the average duration may be shorter than the complete duration of some special iteration processes. Therefore, an increment can be added to the average duration. Or directly take the longest complete duration from the multiple complete durations obtained from the previous multiple iteration processes as the second duration.

[0137] The further method of the present application reasonably limits the saving situation of the to-be-saved data in the monitoring iteration process by configuring two duration values, so as to timely understand the saving situation or training failure situation, and facilitate timely response measures.

[0138] In the further method of the above training task checkpoint file saving method, when the checkpoint file in the current iteration process is deconstructed into multiple to-be-saved data, the number of to-be-saved data can be obtained, and the number of saving times can also be obtained.

[0139] The trigger condition of the mechanism of the asynchronous service storage process is that the multiple to-be-saved data corresponding to the checkpoint file in the current iteration process are successfully saved in the shared memory. It is necessary to continuously monitor the saving situation of the to-be-saved data in the shared memory.

[0140] When it is determined that the duration exceeds the first duration, and the sequence number of the currently saving to-be-saved data is less than the saving times, a new shared memory object is created based on the saving times. At this time, it can be known that the to-be-saved data is saved to which one, and if the saving of the last to-be-saved data has not been reached, the shared memory object required by the subsequent to-be-saved data saving process can be established in advance, so as to reduce the saving action when reaching the saving, enter the saving process in time, and improve the saving efficiency.

[0141] In the further method of the above training task checkpoint file saving method, the saving process needs to be recorded in time, and record information is obtained, so as to understand the saving situation in the iteration process based on the record information. Specifically as follows:

[0142] When a to-be-saved data is successfully saved, a saving record is generated; the saving record contains a record number, an iteration process number and a total saving number. For example, record 1, iteration 1000, saving number 1. That is, in the 1000th iteration process, the record of the first saving is just completed.

[0143] When a new to-be-saved data is successfully saved, the iteration process number in the generated saving record is the same as that in the existing saving record, and the record number and the total saving number in the existing saving record are updated by value. For example, the original saving record is record 2, iteration 1000, saving number 2. The iteration process number of the new saving record is 1000, at this time, the new saving record obtained is record 3, iteration 1000, saving number 3.

[0144] The further method of the present application records the saving process in time, obtains record information, and understands the saving situation in the iteration process based on the record information, realizes transparent disclosure of the saving situation, and facilitates the research and development personnel to view.

[0145] In the further method of the above training task checkpoint file saving method, before the synchronous training saving process of the current iteration process is executed, it is detected whether the corresponding plurality of to-be-saved data of the last iteration process is restored into a checkpoint file and successfully stored to a shared storage device, if the checkpoint file is successfully stored to the shared storage device, the synchronous training saving process of the current iteration process is executed.

[0146] The further method of the present application enters the next iteration process after the checkpoint file of each iteration process is successfully stored to the shared storage device, guarantees the separate processing of different iteration processes, and avoids the mutual interference between the iteration processes.

[0147] In the further method of the above training task checkpoint file saving method, a loop detection logic is configured, and a loop exit logic is configured, and in the loop detection logic, if it is judged that the checkpoint file is not successfully stored in the shared storage device, the loop is blocked.

[0148] If it is judged that the checkpoint file is not successfully stored in the shared storage device within the preset time, the whole training task is terminated.

[0149] In the further method of the above training task checkpoint file saving method, a loop detection logic is configured, and a loop exit logic is configured, and in the loop detection logic, if it is judged that the checkpoint file is not successfully stored in the shared storage device, the loop is blocked.

[0150] In the further method of the above training task checkpoint file saving method, in the synchronous training saving process, the shared memory object created in the current iteration process is continuously used in the next iteration process.

[0151] The further method of the present application can reuse the shared memory object, which can reduce the creation process in each iteration process and reduce the complexity of process execution.

[0152] In the further method of the above training task checkpoint file saving method, when the iteration number reaches the preset number, that is, when the current iteration process number reaches the preset number, it indicates that the training node has completed the processing of the current training task, at this time, the shared memory object used is temporarily saved. In order to reprocess the current training task again in the future. However, when it is detected that the training task is updated, it indicates that the training node needs to process a new training task, and the number of data to be saved will also change, which will also cause the number of objects to change, so the shared memory object used in the current iteration process is deleted first, and a new shared memory object is started to be reconstructed. This way can ensure that different training tasks are processed separately, avoiding interference and affecting the accurate execution of the training task.

[0153] Referring to FIG. 4, which is a whole execution flowchart of the above content, referring to FIG. 4, it can be seen that the specific action flow of the training node in the synchronous training saving process and the asynchronous service storage process.

[0154] The present application is suitable for the model checkpoint direct whole saving scene compared with the prior art, which avoids directly storing the checkpoint file in the shared storage device after coupling with the training framework in the training process, and also avoids completely understanding the training framework mechanism to save the checkpoint file, and has universality.

[0155] The training task checkpoint file saving device provided in the present application is described below, and the training task checkpoint file saving device described below can be correspondingly referred to the training task checkpoint file saving method described above.

[0156] FIG. 5 shows a structural schematic diagram of a training task checkpoint file saving device provided in the present application, referring to FIG. 5, the device includes a synchronous processing module 51 and an asynchronous processing module 52, wherein:

[0157] The synchronous processing module 51 is configured to, in a synchronous training saving process of the large language model, deconstruct a checkpoint file corresponding to a training task of a current iteration of the large language model to obtain a plurality of to-be-saved data, and save the to-be-saved data to a shared memory;

[0158] The asynchronous processing module 52 is configured to, in an asynchronous service storage process of the large language model, recover the plurality of to-be-saved data corresponding to the training task of the current iteration from the shared memory into the checkpoint file and store the checkpoint file into a shared storage device.

[0159] The present application provides a training task checkpoint file saving device, which saves the checkpoint file by deconstructing it into a plurality of to-be-saved data in a shared memory, and extracts the plurality of to-be-saved data from the shared memory to merge them into a checkpoint file stored in a shared storage device, thereby avoiding directly storing the checkpoint file in the shared storage device after coupling it with a training framework in the training process, and also avoiding the need to fully understand the training framework mechanism to save the checkpoint file, and having universality.

[0160] FIG. 6 shows a structural schematic diagram of an electronic device, as shown in FIG. 6, the electronic device can include a processor 61, a communications interface 62, a memory 63 and a communications bus 64, wherein the processor 61, the communications interface 62 and the memory 63 complete mutual communication through the communications bus 64. The processor 61 can call logical instructions in the memory 63 to execute a training task checkpoint file saving method, which includes: in a synchronous training saving process of a large language model, deconstructing a checkpoint file corresponding to a training task of a current iteration of the large language model to obtain a plurality of to-be-saved data, and saving the to-be-saved data to a shared memory; in an asynchronous service storage process of the large language model, recovering the plurality of to-be-saved data corresponding to the training task of the current iteration from the shared memory into the checkpoint file and storing the checkpoint file into a shared storage device.

[0161] In addition, the logical instructions in the memory 63 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0162] In another aspect, the present application also provides a computer readable instruction product, which includes computer readable instructions, the computer readable instructions can be stored on a non-transitory computer readable storage medium, and the computer readable instructions can be executed by a processor to enable a computer to execute the training task checkpoint file saving method provided by the above-mentioned methods, the method comprising: in the synchronous training saving process of the large language model, deconstructing the checkpoint file corresponding to the training task of the current iteration of the large language model to obtain a plurality of to-be-saved data, and saving the to-be-saved data to a shared memory; in the asynchronous service storage process of the large language model, restoring the plurality of to-be-saved data corresponding to the training task of the current iteration from the shared memory into a checkpoint file and storing it in a shared storage device.

[0163] In another aspect, the present application also provides a computer readable instruction product, which includes computer readable instructions, the computer readable instructions can be stored on a non-transitory computer readable storage medium, and the computer readable instructions can be executed by a processor to enable a computer to execute the training task checkpoint file saving method provided by the above-mentioned methods, the method comprising: in the synchronous training saving process of the large language model, deconstructing the checkpoint file corresponding to the training task of the current iteration of the large language model to obtain a plurality of to-be-saved data, and saving the to-be-saved data to a shared memory; in the asynchronous service storage process of the large language model, restoring the plurality of to-be-saved data corresponding to the training task of the current iteration from the shared memory into a checkpoint file and storing it in a shared storage device.

[0164] The device embodiments described above are merely illustrative, wherein the units illustrated as separate components can or can not be physically separated, and the components illustrated as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0165] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of software products, and the computer software products can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and include a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods of the embodiments or some parts of the embodiments.

[0166] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A training task checkpoint file saving method characterized by, A training node applied to distributed training, comprising: In the saving process of the synchronous training of the large language model, the checkpoint file corresponding to the training task of the current iteration of the large language model is deconstructed to obtain a plurality of to-be-saved data, and the to-be-saved data is saved to the shared memory; and In the asynchronous service storage process of the large language model, the plurality of to-be-saved data corresponding to the training task of the current iteration is recovered from the shared memory into the checkpoint file and stored in the shared storage device.

2. The training task checkpoint file saving method according to claim 1, wherein, The checkpoint file corresponding to the training task of the current iteration of the large language model is deconstructed to obtain a plurality of to-be-saved data, and the to-be-saved data is saved to the shared memory, comprising: A shared memory processing engine for processing shared memory is constructed, so that the shared memory processing engine deconstructs the checkpoint file corresponding to the training task of the current iteration to obtain a plurality of to-be-saved data; and A plurality of shared memory objects are constructed, so that each shared memory object saves one of the plurality of to-be-saved data in the shared memory.

3. The training task checkpoint file saving method according to claim 2, wherein, The plurality of shared memory objects are constructed, so that each shared memory object saves one of the plurality of to-be-saved data in the shared memory, comprising: The shared memory object calculates the space capacity occupied by one of the plurality of to-be-saved data to determine the space capacity; The shared memory object divides a memory space corresponding to the space capacity in the shared memory, and stores the to-be-saved data in the divided memory space; and The shared memory object configures an identifier for the to-be-saved data, and accordingly, An identifier space is divided in the shared memory, the identifiers corresponding to the plurality of to-be-saved data are merged into an identifier dictionary, and the identifier dictionary is saved in the identifier space.

4. The training task checkpoint file saving method according to claim 3, wherein, The plurality of to-be-saved data are saved in the shared memory in order, and accordingly, the plurality of shared memory objects are constructed, comprising: When it is determined that the current to-be-saved data needs to be saved in the shared memory, the code of the current training node and the serial number of the current to-be-saved data are obtained; Based on the code of the current training node and the serial number of the current to-be-saved data, a shared memory object is constructed, and an object name is configured for the constructed shared memory object, the object name containing the code of the current training node and the serial number of the current to-be-saved data; and Accordingly, the identifier corresponding to the current to-be-saved data includes the code of the current training node and the serial number of the current to-be-saved data; the dictionary name of the identifier dictionary to which the identifier corresponding to the current to-be-saved data belongs includes the code of the current training node; and the name of the memory space corresponding to the current to-be-saved data contains the code of the current training node and the serial number of the current to-be-saved data.

5. The training task checkpoint file saving method according to claim 4, wherein, The plurality of to-be-saved data corresponding to the training task of the current iteration are recovered from the shared memory into the checkpoint file and stored in the shared storage device, comprising: After monitoring that the plurality of to-be-saved data corresponding to the training task of the current iteration are successfully saved in the shared memory, the code of the current training node and the serial number of the current to-be-saved data are obtained; Filtering an identification dictionary in the identification space based on the encoding of the current training node and the sequence numbers of the to-be-saved data, and filtering the identification corresponding to each to-be-saved data in the identification dictionary; Obtaining the to-be-saved data from the corresponding memory space based on the identification corresponding to each to-be-saved data; And Restoring the plurality of to-be-saved data into the checkpoint file and storing into the shared storage device.

6. The training task checkpoint file saving method according to claim 5, wherein, The method further comprises: Obtaining the duration of the to-be-saved data entering the saving process in the current iteration process; When the duration exceeds the first duration, prompting the sequence number of the successfully saved to-be-saved data and the sequence number of the to-be-saved data currently being saved; and When the duration exceeds the second duration, prompting the current iteration process failure; Wherein, the second duration is longer than the first duration; The first duration is half of the longest complete duration of the to-be-saved data successfully saved in all iteration processes before the current iteration process; The second duration is the average duration of the complete duration of the to-be-saved data successfully saved in all iteration processes before the current iteration process.

7. The training task checkpoint file saving method according to claim 6, wherein, The method further comprises: Based on the number of shares of the to-be-saved data, obtaining the number of saves; and In response to determining that the duration exceeds the first duration, prompting that the sequence number of the to-be-saved data currently being saved is less than the number of saves, and creating a new shared memory object based on the number of saves.

8. The training task checkpoint file saving method of claim 6, wherein, The method further comprises: When a to-be-saved data is successfully saved, generating a saving record; the saving record contains a record number, an iteration process number, and a total number of saves; and When a new to-be-saved data is successfully saved, if the iteration process number in the generated saving record is the same as the iteration process number in the existing saving record, the record number and the total number of saves in the existing saving record are updated.

9. The training task checkpoint file saving method of claim 1, wherein, The method further comprises: Before executing the synchronous training saving process of the current iteration process, detecting whether the corresponding plurality of to-be-saved data of the last iteration process is restored into the checkpoint file and successfully stored into the shared storage device, if the checkpoint file is successfully stored into the shared storage device, then executing the synchronous training saving process of the current iteration process.

10. The training task checkpoint file saving method of claim 9, wherein, The method further comprises: Configuring a loop detection logic and a loop exit logic, in the loop logic, if the checkpoint file is not successfully stored into the shared storage device, the loop is blocked; and Executing the loop exit logic, if it is determined that the checkpoint file is not successfully stored into the shared storage device within a preset time, the entire training task is terminated.

11. The training task checkpoint file saving method of claim 4, wherein, The method further comprises: In the synchronous training saving process, the shared memory object created in the current iteration process is continued to be used in the next iteration process.

12. The training task checkpoint file saving method of claim 11, wherein, The method further comprises: When the current iteration process number reaches a preset number, and it is detected that the training task is sent for update, the shared memory object used in the current iteration process is deleted.

13. The training task checkpoint file saving method of claim 1, wherein, The checkpoint file corresponding to the training task of the current iteration of the large language model is deconstructed to obtain a plurality of to-be-saved data, comprising: The checkpoint file corresponding to the training task of the current iteration of the large language model is deconstructed and analyzed to obtain the weights of the large language model, the state of the optimizer, and the activation value; and The plurality of to-be-preserved data is determined according to the weights of the large language model, the state of the optimizer, and the activation value.

14. The training task checkpoint file saving method according to claim 13, wherein, The checkpoint file corresponding to the training task of the current iteration of the large language model is deconstructed and analyzed to obtain the weights of the large language model, the state of the optimizer, and the activation value, including: The checkpoint file corresponding to the training task of the current iteration of the large language model is traversed to obtain a tensor of the checkpoint file, and the tensor is used to store the weights of the large language model, the state of the optimizer, and the activation value. The plurality of to-be-preserved data is determined according to the weights of the large language model, the state of the optimizer, and the activation value, including: The attributes of the tensor are obtained, and the attributes of the tensor are used as the to-be-preserved data.

15. The training task checkpoint file saving method of claim 6, wherein, When the current iteration is the first training without an existing iteration process, the first time length is a standard time length configured.

16. The training task checkpoint file saving method of claim 12, wherein, The method further includes: When the current iteration process number reaches a preset number, the shared memory object used is saved.

17. The training task checkpoint file saving method of claim 16, wherein, After the step of deleting the shared memory object used in the current iteration process, the method further includes: A new shared memory object is started to be reconstructed.

18. A training task checkpoint file saving apparatus characterized by comprising: Including: A synchronization processing module is configured to, in a synchronization training saving process of a large language model, deconstruct a checkpoint file corresponding to a training task of a current iteration of the large language model to obtain a plurality of to-be-preserved data, and save the to-be-preserved data to a shared memory. And An asynchronous processing module is configured to, in an asynchronous service storage process of the large language model, restore the plurality of to-be-preserved data corresponding to the training task of the current iteration from the shared memory to the checkpoint file and store the checkpoint file to a shared storage device.

19. An electronic device comprising a memory, a processor, and computer readable instructions stored on the memory and executable on the processor, wherein, The processor executes the program to implement the training task checkpoint file saving method of any one of claims 1-17.

20. A non-transitory computer-readable storage medium having stored thereon computer-readable instructions, wherein, The computer-readable instructions are executed by the processor to implement the training task checkpoint file saving method of any one of claims 1-17.

Images