Training task checkpoint file storage method, device, equipment and medium

Abstract

The present invention provides a training task checkpoint file saving method, device, equipment and medium, the method comprising: in the synchronous training saving process of a large language model, deconstructing the checkpoint file corresponding to the training task of the current iteration of the large language model to obtain multiple data to be saved, and saving the data to be saved in a shared memory; in the asynchronous service storage process, restoring the multiple data to be saved corresponding to the training task of the current iteration from the shared memory into a checkpoint file and storing it in a shared storage device. The present invention deconstructs the checkpoint file into multiple data to be saved and saves them in the shared memory, and extracts multiple data to be saved from the shared memory and merges them into a checkpoint file and stores them in the shared storage device, thereby avoiding storing the checkpoint file in the shared storage device after coupling it with the training framework during the training process, and also avoiding fully understanding the training framework mechanism before saving the checkpoint file, and has universal applicability.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and medium for saving a training task checkpoint file. Background Art

[0002] In the evolution of artificial intelligence, the rise of large models marks a huge leap in computing power and data scale. These models, known for their large number of parameters, have rapidly grown from millions of parameters in the early days to billions or even hundreds of billions of parameters today. This scale expansion has enabled large models to demonstrate unprecedented capabilities in complex tasks such as language understanding, image and video analysis. They not only push the boundaries of technology, but also provide new perspectives and solutions to solve real-world problems.

[0003] At the same time, the rapid development of large models has also brought a series of technical challenges, especially in the management of model checkpoints. Checkpoints are a key mechanism in the machine learning process, used to save the state of the model for subsequent training or model recovery. In a large-scale distributed training environment, efficient storage and fast recovery of checkpoints become particularly important. Distributed training is a method of decomposing and distributing the training tasks of deep learning models to multiple computing devices (which can be called training nodes, such as GPUs (graphics processing units), which play a core role in distributed training and significantly accelerate the training process of deep learning models through parallel computing) for parallel processing. This technology can effectively cope with the computing challenges brought by large-scale data sets and complex models, and significantly improve training efficiency.

[0004] The relevant technical solution simply uses shared memory to accelerate the saving of checkpoint files, and only saves them once for a certain iteration. In addition, the solution often directly couples the checkpoint files with the training framework during the training process and stores them in a shared storage device. At this time, R&D personnel need to fully understand the training framework mechanism to save the checkpoint files, which is not universal. Summary of the invention

[0005] In view of the problems existing in the prior art, the present invention provides a method, device, equipment and medium for saving a training task checkpoint file.

[0006] The present invention provides a method for saving a training task checkpoint file, comprising:

[0007] During the synchronous training and saving process of the large language model, the checkpoint file corresponding to the training task of the current iteration of the large language model is deconstructed to obtain a plurality of data to be saved, and the data to be saved is saved in the shared memory;

[0008] In the asynchronous service storage process of the large language model, the multiple to-be-saved data corresponding to the training task of the current iteration are restored from the shared memory into the checkpoint file and stored in the shared storage device.

[0009] According to a training task checkpoint file saving method provided by the present invention, the checkpoint file corresponding to the training task of the current iteration of the large language model is deconstructed to obtain multiple data to be saved, and the data to be saved is saved in a shared memory, including:

[0010] Construct a shared memory processing engine for processing shared memory, so that the shared memory processing engine deconstructs a plurality of data to be saved from a checkpoint file corresponding to a training task of a current iteration;

[0011] A plurality of shared memory objects are constructed, so that each shared memory object stores one of the plurality of data to be stored in the shared memory.

[0012] According to a training task checkpoint file saving method provided by the present invention, the construction of multiple shared memory objects, so that each shared memory object saves one of the multiple data to be saved in the shared memory, includes:

[0013] The shared memory object is enabled to calculate the space capacity occupied by one of the multiple data to be saved, and determine the space capacity;

[0014] The shared memory object is caused to divide a memory space in the shared memory that matches the space capacity, and the data to be saved is stored in the divided memory space;

[0015] Make the shared memory object treat the saved data configuration flag as follows, accordingly,

[0016] An identification space is divided in the shared memory, identifications corresponding to a plurality of data to be saved are merged into an identification dictionary, and the identification dictionary is saved in the identification space.

[0017] According to a training task checkpoint file saving method provided by the present invention, multiple data to be saved are saved in a shared memory in sequence. Accordingly, the construction of multiple shared memory objects includes:

[0018] When it is determined that the current data to be saved needs to be saved in the shared memory, the code of the current training node and the sequence number of the current data to be saved are obtained;

[0019] Based on the code of the current training node and the sequence number of the current data to be saved, a shared memory object is constructed, and an object name is configured for the constructed shared memory object, where the object name includes the code of the current training node and the sequence number of the current data to be saved;

[0020] Correspondingly, the identifier corresponding to the current data to be saved includes the code of the current training node and the serial number of the current data to be saved; the dictionary name of the identifier dictionary to which the identifier corresponding to the current data to be saved belongs includes the code of the current training node; the name of the memory space corresponding to the current data to be saved includes the code of the current training node and the serial number of the current data to be saved.

[0021] According to a training task checkpoint file saving method provided by the present invention, the multiple to-be-saved data corresponding to the training task of the current iteration are restored from the shared memory into the checkpoint file and stored in the shared storage device, including:

[0022] After monitoring that multiple data to be saved corresponding to the training task of the current iteration are successfully saved in the shared memory, obtain the code of the current training node and the sequence number of the current data to be saved;

[0023] Based on the code of the current training node and the serial number of the data to be saved, an identification dictionary is selected in the identification space, and the identification corresponding to each data to be saved is selected in the identification dictionary;

[0024] Based on the identifier corresponding to each piece of data to be saved, obtain the data to be saved from the corresponding memory space;

[0025] The plurality of data to be saved are restored into the checkpoint file and stored in the shared storage device.

[0026] According to a method for saving a training task checkpoint file provided by the present invention, the method further includes:

[0027] Get the duration of the saving process of the data to be saved in the current iteration;

[0028] When it is determined that the duration exceeds the first duration, the sequence number of the successfully saved data to be saved and the sequence number of the currently saved data to be saved are prompted;

[0029] When it is determined that the duration exceeds the second duration, prompting that the current iteration process fails;

[0030] Wherein, the second duration is longer than the first duration;

[0031] The first duration is half of the longest complete duration of successful storage of the data to be stored in all iterations before the current iteration;

[0032] The second duration is the average duration of the complete duration of successful storage of the data to be stored in all iteration processes before the current iteration process.

[0033] According to a method for saving a training task checkpoint file provided by the present invention, the method further includes:

[0034] Based on the number of copies of the data to be saved, the number of save times is obtained;

[0035] When it is determined that the duration exceeds the first duration, and it is prompted that the sequence number of the data to be saved currently being saved is less than the number of save times, a new shared memory object is created based on the number of save times.

[0036] According to a method for saving a training task checkpoint file provided by the present invention, the method further includes:

[0037] When a piece of data to be saved is successfully saved, a save record is generated; the save record contains the record number, the iteration process number and the total number of saves;

[0038] When the new data to be saved is successfully saved, and the iteration process number in the generated save record is the same as the iteration process number in the existing save record, the record number and the total number of saves in the existing save record are incremented and updated.

[0039] According to a method for saving a training task checkpoint file provided by the present invention, the method further includes:

[0040] Before executing the synchronous training and saving process of the current iteration process, detect whether the corresponding multiple data to be saved of the previous iteration process are restored to the checkpoint file and successfully stored in the shared storage device. If the checkpoint file is successfully stored in the shared storage device, execute the synchronous training and saving process of the current iteration process.

[0041] According to a method for saving a training task checkpoint file provided by the present invention, the method further includes:

[0042] Configure a loop detection logic and configure loop exit logic. If the checkpoint file is not successfully stored in the shared storage device in the loop logic, the loop will be blocked.

[0043] Execute the loop break logic. If the checkpoint file is not successfully stored in the shared storage device within the preset time, the entire training task is terminated.

[0044] According to a training task checkpoint file saving method provided by the present invention, the checkpoint file is stored in a shared storage device, comprising:

[0045] During the synchronous training and saving process, the shared memory objects created during the current iteration will continue to be used during the next iteration.

[0046] According to a method for saving a training task checkpoint file provided by the present invention, the method further includes:

[0047] When the current iteration number reaches the preset number and the training task sends an update, the shared memory objects used in the current iteration are deleted.

[0048] The present invention also provides a training task checkpoint file storage device, comprising:

[0049] A synchronization processing module, used for deconstructing the checkpoint file corresponding to the training task of the current iteration of the large language model during the synchronous training and saving process of the large language model, obtaining multiple data to be saved, and saving the multiple data to be saved into the shared memory;

[0050] The asynchronous processing module is used to restore the multiple to-be-saved data corresponding to the training task of the current iteration from the shared memory into the checkpoint file and store it in the shared storage device during the asynchronous service storage process of the large language model.

[0051] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements any of the above-mentioned training task checkpoint file saving methods.

[0052] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned training task checkpoint file saving methods.

[0053] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned training task checkpoint file saving methods.

[0054] The present invention provides a training task checkpoint file saving method, device, equipment and medium, which decomposes the checkpoint file into multiple data to be saved and saves them in a shared memory, and extracts multiple data to be saved from the shared memory and merges them into a checkpoint file and stores them in a shared storage device. This avoids directly storing the checkpoint file in a shared storage device after coupling it with a training framework during the training process, and also avoids having to fully understand the training framework mechanism before saving the checkpoint file. The invention has universal applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 It is a flowchart of the training task checkpoint file saving method provided by the present invention.

[0057] Figure 2 It is a schematic diagram of creating a shared memory object provided by the present invention.

[0058] Figure 3 It is a schematic diagram of storing multiple data and identifiers to be stored provided by the present invention.

[0059] Figure 4 It is a specific flow chart of the training task checkpoint file saving method provided by the present invention.

[0060] Figure 5 It is a structural schematic diagram of the training task checkpoint file storage device provided by the present invention.

[0061] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] Combine the following Figure 1-Figure 6 The present invention describes a training task checkpoint file storage method, device, equipment and medium.

[0064] Figure 1 A flow chart of a method for saving a training task checkpoint file provided by the present invention is shown in FIG. Figure 1 , the method comprises the following steps:

[0065] Step 11: During the synchronous training and saving process of the large language model, the checkpoint file corresponding to the training task of the current iteration of the large language model is deconstructed to obtain multiple data to be saved, and the multiple data to be saved are saved in the shared memory.

[0066] Step 12: During the asynchronous service storage process of the large language model, multiple data to be saved corresponding to the training task of the current iteration are restored from the shared memory into a checkpoint file and stored in the shared storage device.

[0067] Regarding steps 11 to 13 above, it should be noted that in the evolution of artificial intelligence, the rise of large models (such as large language models) marks a huge leap in computing power and data scale. These models, known for their large number of parameters, have rapidly grown from millions of parameters in the early days to billions or even hundreds of billions of parameters today. This scale expansion has enabled large models to demonstrate unprecedented capabilities in complex tasks such as language understanding, image and video analysis. They not only push the boundaries of technology, but also provide new perspectives and solutions for solving real-world problems.

[0068] At the same time, the rapid development of large models has also brought a series of technical challenges, especially in the management of model checkpoints. Checkpoints are a key mechanism in the machine learning process, used to save the state of the model for subsequent training or model recovery. In a large-scale distributed training environment, efficient storage and fast recovery of checkpoints become particularly important. Distributed training is a method of decomposing and distributing the training tasks of deep learning models to multiple computing devices (which can be called training nodes, such as graphics processing units (GPUs), which play a core role in distributed training and significantly accelerate the training process of deep learning models through parallel computing) for parallel processing. This technology can effectively cope with the computing challenges brought by large-scale data sets and complex models, and significantly improve training efficiency.

[0069] The method of the present invention is suitable for scenarios where model checkpoints need to be segmented and saved, that is, the checkpoint file needs to be deconstructed into multiple data to be saved, and these data to be saved can be saved. Deconstructing the checkpoint file refers to parsing and analyzing the checkpoint file generated during the deep learning training process to obtain the model weights, optimizer status, activation values ​​and other data. Specifically: Since the main element in the checkpoint file is a tensor (i.e., tensor), a tensor is a multidimensional array. In deep learning, tensors are used to store model weights, optimizer status, activation values ​​and other attribute data. So first traverse the checkpoint file to obtain the tensors therein, and then obtain the tensor attributes. The obtained tensor attributes are used as the data to be saved and saved in the subsequent processing process.

[0070] In the present invention, the checkpoint file is decomposed into multiple data to be saved and then saved in shared memory. Shared memory is an efficient inter-process communication (IPC) mechanism that allows multiple processes to directly access the same physical memory area, thereby achieving rapid data exchange and sharing, and significantly improving training efficiency.

[0071] In the present invention, it is also necessary to restore a plurality of data to be saved into a checkpoint file and store it in a shared storage device, which is a persistent storage device (such as a hard disk).

[0072] Checkpoint files are stored in shared storage devices, which is considered shared storage writes. Shared storage writes to disk. In a distributed system, data is written from memory or other temporary storage media to persistent storage devices. It refers to the process in which data is shared between multiple computing devices and finally written to persistent storage devices (such as hard disks). This operation ensures data persistence and consistency, and maintains data integrity even after system failure or restart, which is beneficial for subsequent use during training.

[0073] From the above, it can be seen that when each training node in distributed training executes its own training task, when saving the data to be saved and storing the checkpoint file, a mechanism for the synchronous training saving process and a mechanism for the asynchronous service storage process can be created. In the synchronous training saving process, the checkpoint file corresponding to the training task of the current iteration deconstructs multiple data to be saved, and saves the multiple data to be saved to the shared memory. In the asynchronous service storage process, the multiple data to be saved corresponding to the current iteration process are restored from the shared memory into a checkpoint file and stored in the shared storage device.

[0074] Compared with the prior art, the training task checkpoint file saving method provided by the present invention is suitable for the scenario of directly saving the model checkpoint as a whole. It decomposes the checkpoint file into multiple data to be saved and saves them in the shared memory, and extracts multiple data to be saved from the shared memory and merges them into the checkpoint file and stores them in the shared storage device. It avoids directly storing the checkpoint file in the shared storage device after coupling it with the training framework during the training process, and also avoids fully understanding the training framework mechanism before saving the checkpoint file. It has universal applicability.

[0075] In the further method of the above training task checkpoint file saving method, the checkpoint file corresponding to the current iteration of the large language model training task is mainly deconstructed to obtain multiple data to be saved, and the processing process of saving the data to be saved in the shared memory is explained as follows:

[0076] Construct a shared memory processing engine for processing shared memory, so that the checkpoint file corresponding to the training task of the current iteration of the shared memory processing engine is deconstructed into multiple data to be saved;

[0077] A plurality of shared memory objects are constructed, so that each shared memory object stores one of the plurality of data to be stored in the shared memory.

[0078] In this regard, it should be noted that the training node (such as GPU) needs to create a shared memory processing engine (shm_engine) to operate and maintain the shared memory of the data to be saved. The shared memory processing engine (shm_engine) is a software module or library used to create, manage and operate shared memory areas. In a distributed system, it allows different processes to exchange data efficiently through shared memory.

[0079] In the present invention, a shared memory processing engine is used to deconstruct a checkpoint file in a current iteration of a training task into a plurality of data to be saved.

[0080] Since the checkpoint file deconstructs multiple data to be saved, in order to distinguish and save the data to be saved, multiple shared memory objects (shm objects) are constructed, each of which is responsible for the saving operation of one data to be saved, and the shared memory objects do not interfere with each other. Each shared memory object saves one data to be saved among the multiple data to be saved in the shared memory.

[0081] like Figure 2 As shown, training node 1 creates a shared memory processing engine (shm_engine), and then creates shared memory (shm) object 1, shared memory (shm) object 2, and shared memory (shm) object 3, and saves the three data to be saved deconstructed from the checkpoint file respectively. The saving of each data to be saved is called one save.

[0082] A further method of the present invention can save the data to be saved separately by creating a shared memory processing engine (shm_engine) and a shared memory (shm) object, so that the saving method is more flexible, mutual interference is avoided, and the saving efficiency is accelerated.

[0083] In the further method of the above-mentioned training task checkpoint file saving method, the main process of constructing multiple shared memory objects so that each shared memory object saves one of the multiple data to be saved in the shared memory is explained as follows:

[0084] The shared memory object is enabled to calculate the space capacity occupied by one of the multiple data to be saved, and determine the space capacity;

[0085] The shared memory object is allocated a memory space in the shared memory that matches the space capacity, and the data to be saved is stored in the allocated memory space;

[0086] Make the shared memory object treat the saved data configuration flag as follows, accordingly,

[0087] An identification space is divided in the shared memory, identifications corresponding to a plurality of data to be saved are merged into an identification dictionary, and the identification dictionary is saved in the identification space.

[0088] In this regard, it should be noted that, in the present invention, in order to distinguish and save the data to be saved, it is necessary to allocate a corresponding memory space for each data to be saved. Therefore, the shared memory object is made to calculate the space capacity occupied by one of the multiple data to be saved to determine the space capacity. Then, the shared memory object is made to allocate a memory space in the shared memory that matches the space capacity, and the data to be saved is stored in the allocated memory space.

[0089] The purpose of dividing the memory space is to store the data to be saved separately. In order to search the corresponding data to be saved from the memory space later, it is necessary to establish an identifier corresponding to the data to be saved. Therefore, the shared memory object is used to configure the data to be saved. The identifier here serves as the metadata of the data to be saved. The corresponding data to be saved can be searched based on the metadata.

[0090] Accordingly, an identification space is divided in the shared memory, identifications corresponding to a plurality of data to be saved are merged into an identification dictionary, and the identification dictionary is saved in the identification space.

[0091] See also Figure 3 , the shared memory processing engine (shm_engine) saves n data to be saved n times. Each data to be saved and the identifier are regarded as a saved combination. See Figure 3 The shared memory (shm) to be saved data 1 and the shared memory (shm) identifier (metadata) 1, ..., the shared memory (shm) to be saved data n and the shared memory (shm) identifier n, each save combination corresponds to the first save of this iteration, ..., the nth save of this iteration.

[0092] A further method of the present invention, by dividing the memory space to save the data to be saved and configuring the identifier of each data to be saved, is conducive to the subsequent query and acquisition of accurate data to be saved in the memory, ensures the integrity of the data, and is conducive to the progress of iterative training.

[0093] In the further method of the above training task checkpoint file saving method, the processing process of constructing multiple shared memory objects is mainly explained as follows:

[0094] When it is determined that the current data to be saved needs to be saved in the shared memory, the code of the current training node and the sequence number of the current data to be saved are obtained;

[0095] Based on the code of the current training node and the sequence number of the current data to be saved, a shared memory object is constructed, and an object name is configured for the constructed shared memory object, where the object name includes the code of the current training node and the sequence number of the current data to be saved;

[0096] Correspondingly, the identifier corresponding to the current data to be saved includes the code of the current training node and the serial number of the current data to be saved; the dictionary name of the identifier dictionary of the identifier corresponding to the current data to be saved includes the code of the current training node; the name of the memory space corresponding to the current data to be saved includes the code of the current training node and the serial number of the current data to be saved.

[0097] In this regard, it should be noted that in order to accurately save and use different training nodes and different data to be saved, when it is monitored that the current data to be saved needs to be saved in shared memory, the code of the training node to which the data to be saved belongs and the serial number of the current data to be saved are obtained.

[0098] Based on the code of the training node to which the data to be saved belongs and the sequence number of the current data to be saved, a name is generated, which can be used as the object name of the constructed shared memory object.

[0099] In distributed training, each GPU can indeed be considered as a separate rank, especially when using data parallel models. In distributed training, a rank is a unique identifier that distinguishes different processes or devices. Each GPU participating in training is assigned a unique rank value. For this purpose, the rank value can be used as an encoding for the training node.

[0100] For example, shm_rank_0 represents the name of the shared memory processing engine (shm_engine) held by rank_0, so the name of the shared memory object can be shm_1_rank_0, where rank_0 is the code of the training node, 1 in shm_1 is the sequence number of the data to be saved, and shm is the abbreviation of the object.

[0101] Accordingly, the identifier corresponding to the current data to be saved includes the code of the current training node and the sequence number of the current data to be saved. shm_1_rank_0_meta. meta is the abbreviation of metadata or identifier.

[0102] The dictionary name of the identification dictionary corresponding to the current data to be saved includes the code of the current training node. For example, the dictionary includes shm_1_rank_0_meta, shm_2_rank_0_meta, shm_3_rank_0_meta, shm_4_rank_0_meta, and so on.

[0103] The name of the memory space corresponding to the current data to be saved contains the code of the current training node and the sequence number of the current data to be saved, such as space_1_rank_0. Space is the abbreviation of space.

[0104] A further method of the present invention can establish a connection between the contents in each environment by configuring the names of the contents required for each processing link based on the code of the current training node and the serial number of the current data to be saved, thereby facilitating the subsequent accurate extraction of data from the memory and improving the correct use of the data.

[0105] In a further method of the above-mentioned training task checkpoint file saving method, the processing of restoring the multiple to-be-saved data corresponding to the current iteration of the training task from the shared memory into a checkpoint file and storing it in the shared storage device is mainly explained as follows;

[0106] After monitoring that multiple data to be saved corresponding to the training task of the current iteration are successfully saved in the shared memory, obtain the code of the current training node and the sequence number of the current data to be saved;

[0107] Based on the code of the current training node and the serial number of the data to be saved, an identification dictionary is selected in the identification space, and the identification corresponding to each data to be saved is selected in the identification dictionary;

[0108] Based on the identifier corresponding to each piece of data to be saved, obtain the data to be saved from the corresponding memory space;

[0109] Restore multiple data to be saved into checkpoint files and store them in a shared storage device.

[0110] In this regard, it should be noted that the triggering condition of the mechanism of the asynchronous service storage process is that the multiple data to be saved corresponding to the checkpoint file in the current iteration process are successfully saved in the shared memory. Therefore, it is necessary to continuously monitor whether the data to be saved in the synchronization state has been completely saved to the shared memory. If it has not been completely saved to the shared memory, continue to monitor. If it has been completely saved to the shared memory, obtain the code of the current training node and the sequence number of the current data to be saved in order.

[0111] Then, based on the code of the current training node and the serial number of the data to be saved, the identification dictionary is filtered out in the identification space, and the identification corresponding to each data to be saved is filtered out in the identification dictionary. Then, based on the identification corresponding to each data to be saved, the data to be saved is obtained from the corresponding memory space, and multiple data to be saved are restored into checkpoint files and stored in the shared storage device.

[0112] A further method of the present invention, after being triggered by the mechanism of the asynchronous service storage process, screens and obtains the corresponding data to be saved from the memory space step by step based on the encoding of the training node and the serial number of the data to be saved, thereby realizing information association of each link in the synchronous training and saving process, and being able to quickly lock the memory space and quickly extract the data to be saved.

[0113] In the further method of the above training task checkpoint file saving method, the main purpose is to monitor the status of the data to be saved in the shared memory for explanation, as follows:

[0114] Get the duration of the saving process of the data to be saved in the current iteration;

[0115] When it is determined that the duration exceeds the first duration, the sequence number of the successfully saved data to be saved and the sequence number of the currently saved data to be saved are prompted;

[0116] When it is determined that the duration exceeds the second duration, it is prompted that the current iteration process fails;

[0117] wherein the second duration is longer than the first duration;

[0118] The first duration is half of the longest complete duration of successful storage of the data to be stored in all iterations before the current iteration;

[0119] The second duration is the average duration of the complete duration of successful saving of the data to be saved in all iteration processes before the current iteration process.

[0120] In this regard, it should be noted that the triggering condition of the asynchronous service storage process mechanism is that the multiple data to be saved corresponding to the checkpoint file in the current iteration process are successfully saved in the shared memory. It is necessary to continuously monitor the storage status of the data to be saved in the shared memory. However, there are individual times when the saving process takes a long time or there are training failures that cause the information to be unsuitable for saving and stagnate. In order to be able to know the saving status at the first time and make a response strategy in time, the saving status can be monitored at a certain point in time and the specific saving status can be prompted.

[0121] At this time, the duration of the data to be saved entering the saving process in the current iteration process will be obtained. This duration is calculated from the beginning of the iteration. First, the first duration will be set. The first duration is half of the longest complete duration in which the data to be saved is successfully saved in all iterations before the current iteration process. If it is the first training process, a standard duration can be configured for the first duration at this time. When it is the second training process, it can be regarded as the first iteration process. Then the first duration is half of the complete duration in which the data to be saved is successfully saved in the first training process. The complete duration here is the duration from the start of the iteration to the monitoring to the end of the saving of the data to be saved. When it is the nth iteration process, the longest complete duration can be obtained from the multiple complete durations obtained from the previous multiple iterations, and then half of the longest complete duration is taken as the first duration.

[0122] Accordingly, the second duration is the average duration of the complete durations of successfully saving the data to be saved in all iterations before the current iteration, that is, the average duration can be obtained from multiple complete durations obtained in multiple previous iterations.

[0123] When it is determined that the duration exceeds the first duration, the sequence number of the successfully saved data to be saved and the sequence number of the currently saved data to be saved are prompted. At this time, the specific saving situation is clarified and prompted to the R&D personnel.

[0124] When it is determined that the duration exceeds the second duration, it is indicated that the current iteration process has failed. Since the average duration may be shorter than the complete duration of some special iteration processes, an increment can be added based on the average duration. Alternatively, the longest complete duration can be directly obtained from the multiple complete durations obtained in the previous multiple iteration processes as the second duration.

[0125] A further method of the present invention reasonably limits the preservation status of the data to be preserved during the monitoring iteration process by configuring two duration values, so as to timely understand the preservation status or training failure status and take response measures in time.

[0126] In a further method of the above-mentioned training task checkpoint file saving method, when a plurality of data to be saved are deconstructed from the checkpoint file in the current iteration process, the number of copies of the data to be saved can be known, and thus the number of save times can also be known.

[0127] The triggering condition of the asynchronous service storage process mechanism is that the multiple data to be saved corresponding to the checkpoint file in the current iteration process are successfully saved in the shared memory. It is necessary to continuously monitor the storage status of the data to be saved in the shared memory.

[0128] When it is determined that the duration exceeds the first duration, it is prompted that the sequence number of the data to be saved currently being saved is less than the number of save times, and a new shared memory object is created based on the number of save times. At this time, it can be known which data to be saved is saved. If the last data to be saved has not yet been saved, the shared memory objects required for the subsequent saving process of the data to be saved can be established in advance, so as to reduce the saving action when the saving is reached, and to enter the saving process in time, thereby improving the saving efficiency.

[0129] In a further method of the above training task checkpoint file saving method, the saving process needs to be recorded in time and the record information is obtained so as to understand the saving status in the iteration process based on the record information. The details are as follows:

[0130] When a piece of data to be saved is successfully saved, a save record is generated; the save record contains the record number, iteration process number and total number of saves. For example, record 1, iteration 1000, and save quantity 1. This means the record that has just been saved for the first time during the 1000th iteration.

[0131] When the new data to be saved is successfully saved, the iteration process number in the generated save record is the same as the iteration process number in the existing save record, then the record number and the total number of saves in the existing save record are incremented and updated. For example, the original save record is record 2, iteration 1000, and the number of saves is 2. The iteration process number of the new save record is 1000, at this time, the new save record obtained is record 3, iteration 1000, and the number of saves is 3.

[0132] A further method of the present invention requires timely recording of the preservation process and obtaining record information so as to understand the preservation status during the iteration process based on the record information, realize transparent disclosure of the preservation status, and facilitate R&D personnel to view it.

[0133] In a further method of the above-mentioned training task checkpoint file saving method, before executing the synchronous training saving process of the current iteration process, it is detected whether the corresponding multiple data to be saved of the previous iteration process are restored to the checkpoint file and successfully stored in the shared storage device. If the checkpoint file is successfully stored in the shared storage device, the synchronous training saving process of the current iteration process is executed.

[0134] A further method of the present invention enters the next iteration process after the checkpoint file of each iteration process is successfully stored in a shared storage device, thereby ensuring separate processing of different iteration processes and avoiding mutual interference between iteration processes.

[0135] In a further method of the above-mentioned training task checkpoint file saving method, a loop detection logic is configured, and a loop exit logic is configured. If it is determined in the loop logic that the checkpoint file is not successfully stored in the shared storage device, the loop is blocked.

[0136] If the checkpoint file is not successfully stored in the shared storage device within the preset time, the entire training task is terminated.

[0137] A further method of the present invention can automatically monitor the storage status of the checkpoint file by configuring the loop detection logic and configuring the loop exit logic, which is simple and fast.

[0138] In a further method of the above-mentioned training task checkpoint file saving method, during the synchronous training saving process, the shared memory object created during the current iteration process continues to be used during the next iteration process.

[0139] A further method of the present invention can share the reuse of memory objects, which can reduce the creation process in each iteration process and reduce the complexity of process execution.

[0140] In a further method of the above-mentioned training task checkpoint file saving method, when the number of iterations reaches a preset number, that is, when the current iteration process number reaches a preset number, it indicates that the training node has completed processing the current training task. At this time, the shared memory object used is temporarily saved. So that the current training task can be reprocessed later. However, when it is detected that the training task sends an update, it indicates that the training node needs to process a new training task, and the number of times the data needs to be saved will also change, which will also cause the number of objects to change. Therefore, the shared memory object used in the current iteration process is deleted first, and the subsequent reconstruction of the new shared memory object is started. This method can ensure that different training tasks are processed separately, avoid negotiations, and affect the precise execution of the training tasks.

[0141] See also Figure 4 This is the overall execution flow chart of the above content, see Figure 4 , we can see the specific action flow of the training node in the synchronous training saving process and the asynchronous service storage process.

[0142] Compared with the prior art, the present invention is suitable for the scenario of directly saving the model checkpoint as a whole. The checkpoint file is decomposed into multiple data to be saved and saved in the shared memory, and the multiple data to be saved are extracted from the shared memory and merged into the checkpoint file and stored in the shared storage device. This avoids directly storing the checkpoint file in the shared storage device after coupling it with the training framework during the training process, and also avoids fully understanding the training framework mechanism before saving the checkpoint file. The present invention has universal applicability.

[0143] The training task checkpoint file saving device provided by the present invention is described below. The training task checkpoint file saving device described below and the training task checkpoint file saving method described above can be referenced to each other.

[0144] Figure 5 A schematic diagram of the structure of a training task checkpoint file storage device provided by the present invention is shown. Figure 5 The device comprises a synchronous processing module 51 and an asynchronous processing module 52, wherein:

[0145] The synchronization processing module 51 is used to deconstruct the checkpoint file corresponding to the training task of the current iteration of the large language model during the synchronous training and saving process of the large language model, obtain multiple data to be saved, and save the data to be saved into the shared memory;

[0146] The asynchronous processing module 52 is used to restore the multiple to-be-saved data corresponding to the training task of the current iteration from the shared memory into a checkpoint file and store it in the shared storage device during the asynchronous service storage process of the large language model.

[0147] The present invention provides a training task checkpoint file storage device, which decomposes the checkpoint file into multiple data to be saved and stores them in a shared memory, and extracts multiple data to be saved from the shared memory and merges them into a checkpoint file and stores them in a shared storage device. This avoids directly storing the checkpoint file in the shared storage device after coupling it with a training framework during the training process, and also avoids having to fully understand the training framework mechanism before saving the checkpoint file. The device has universal applicability.

[0148] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor 61, a communication interface 62, a memory 63 and a communication bus 64, wherein the processor 61, the communication interface 62 and the memory 63 communicate with each other through the communication bus 64. The processor 61 may call the logic instructions in the memory 63 to execute the training task checkpoint file saving method, which includes: in the synchronous training and saving process of the large language model, deconstructing the checkpoint file corresponding to the training task of the current iteration of the large language model to obtain a plurality of data to be saved, and saving the data to be saved in the shared memory; in the asynchronous service storage process of the large language model, restoring the plurality of data to be saved corresponding to the training task of the current iteration from the shared memory into the checkpoint file and storing it in the shared storage device.

[0149] In addition, the logic instructions in the above-mentioned memory 63 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0150] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the training task checkpoint file saving method provided by the above methods, the method comprising: in the synchronous training and saving process of the large language model, deconstructing the checkpoint file corresponding to the training task of the current iteration of the large language model to obtain multiple data to be saved, and saving the data to be saved in a shared memory; in the asynchronous service storage process of the large language model, restoring the multiple data to be saved corresponding to the training task of the current iteration from the shared memory into a checkpoint file and storing it in a shared storage device.

[0151] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the training task checkpoint file saving method provided by the above methods, the method comprising: in the synchronous training and saving process of the large language model, deconstructing the checkpoint file corresponding to the training task of the current iteration of the large language model to obtain a plurality of data to be saved, and saving the data to be saved in a shared memory; in the asynchronous service storage process of the large language model, restoring the plurality of data to be saved corresponding to the training task of the current iteration from the shared memory into a checkpoint file and storing it in a shared storage device.

[0152] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0153] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0154] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for saving a training task checkpoint file, characterized in that: Training nodes used in distributed training include: During the synchronous training and saving process of the large language model, the checkpoint file corresponding to the training task of the current iteration of the large language model is deconstructed to obtain a plurality of data to be saved, and the data to be saved is saved in the shared memory; In the asynchronous service storage process of the large language model, a plurality of to-be-saved data corresponding to the training task of the current iteration is restored from the shared memory into the checkpoint file and stored in the shared storage device; The step of deconstructing the checkpoint file corresponding to the training task of the current iteration of the large language model to obtain a plurality of data to be saved, and saving the data to be saved into the shared memory includes: Construct a shared memory processing engine for processing shared memory, so that the shared memory processing engine deconstructs a plurality of data to be saved from a checkpoint file corresponding to a training task of a current iteration; Constructing multiple shared memory objects, so that each shared memory object stores one of the multiple data to be saved in the shared memory; The step of restoring the plurality of to-be-saved data corresponding to the training task of the current iteration from the shared memory into the checkpoint file and storing the files in the shared storage device comprises: After monitoring that multiple data to be saved corresponding to the training task of the current iteration are successfully saved in the shared memory, obtain the code of the current training node and the sequence number of the current data to be saved; Based on the code of the current training node and the serial number of the data to be saved, an identification dictionary is selected in the identification space, and the identification corresponding to each data to be saved is selected in the identification dictionary; wherein the identification space is in the shared memory, the identification dictionary is formed by merging the identifications corresponding to the multiple data to be saved, and the identification dictionary is stored in the identification space; Based on the identifier corresponding to each piece of data to be saved, obtain the data to be saved from the corresponding memory space; The plurality of data to be saved are restored into the checkpoint file and stored in the shared storage device.

2. The training task checkpoint file saving method according to claim 1, characterized in that: The step of constructing a plurality of shared memory objects so that each shared memory object stores one of the plurality of to-be-stored data in the shared memory includes: The shared memory object is enabled to calculate the space capacity occupied by one of the multiple data to be saved, and determine the space capacity; The shared memory object is made to divide a memory space in the shared memory that matches the space capacity, and the data to be saved is stored in the divided memory space.

3. The training task checkpoint file saving method according to claim 2, characterized in that: The multiple data to be saved are saved in the shared memory in sequence. Accordingly, the multiple shared memory objects are constructed, including: When it is determined that the current data to be saved needs to be saved in the shared memory, the code of the current training node and the sequence number of the current data to be saved are obtained; Based on the code of the current training node and the sequence number of the current data to be saved, a shared memory object is constructed, and an object name is configured for the constructed shared memory object, where the object name includes the code of the current training node and the sequence number of the current data to be saved; Correspondingly, the identifier corresponding to the current data to be saved includes the code of the current training node and the serial number of the current data to be saved; the dictionary name of the identifier dictionary to which the identifier corresponding to the current data to be saved belongs includes the code of the current training node; the name of the memory space corresponding to the current data to be saved includes the code of the current training node and the serial number of the current data to be saved.

4. The training task checkpoint file saving method according to claim 1, characterized in that: The method further comprises: Get the duration of the saving process of the data to be saved in the current iteration; When it is determined that the duration exceeds the first duration, the sequence number of the successfully saved data to be saved and the sequence number of the currently saved data to be saved are prompted; When it is determined that the duration exceeds the second duration, prompting that the current iteration process fails; Wherein, the second duration is longer than the first duration; The first duration is half of the longest complete duration of successful storage of the data to be stored in all iterations before the current iteration; The second duration is the average duration of the complete duration of successful storage of the data to be stored in all iteration processes before the current iteration process.

5. The training task checkpoint file saving method according to claim 4, characterized in that: The method further comprises: Based on the number of copies of the data to be saved, the number of save times is obtained; When it is determined that the duration exceeds the first duration, and it is prompted that the sequence number of the data to be saved currently being saved is less than the number of save times, a new shared memory object is created based on the number of save times.

6. The training task checkpoint file saving method according to claim 4, characterized in that: The method further comprises: When a piece of data to be saved is successfully saved, a save record is generated; the save record contains the record number, the iteration process number and the total number of saves; When the new data to be saved is successfully saved, and the iteration process number in the generated save record is the same as the iteration process number in the existing save record, the record number and the total number of saves in the existing save record are incremented and updated.

7. The training task checkpoint file saving method according to claim 1, characterized in that: The method further comprises: Before executing the synchronous training and saving process of the current iteration process, detect whether the corresponding multiple data to be saved of the previous iteration process are restored to the checkpoint file and successfully stored in the shared storage device. If the checkpoint file is successfully stored in the shared storage device, execute the synchronous training and saving process of the current iteration process.

8. The training task checkpoint file saving method according to claim 7, characterized in that: The method further comprises: Configure a loop detection logic and configure loop exit logic. If the checkpoint file is not successfully stored in the shared storage device in the loop logic, the loop will be blocked. Execute the loop break logic. If the checkpoint file is not successfully stored in the shared storage device within the preset time, the entire training task is terminated.

9. The training task checkpoint file saving method according to claim 2, characterized in that: The method further comprises: During the synchronous training and saving process, the shared memory objects created during the current iteration will continue to be used during the next iteration.

10. The training task checkpoint file saving method according to claim 9, characterized in that: The method further comprises: When the current iteration number reaches the preset number and the training task sends an update, the shared memory objects used in the current iteration are deleted.

11. A training task checkpoint file storage device, characterized in that: include: A synchronization processing module, used for deconstructing the checkpoint file corresponding to the training task of the current iteration of the large language model during the synchronous training and saving process of the large language model, obtaining multiple data to be saved, and saving the data to be saved into the shared memory; An asynchronous processing module, used for restoring the multiple to-be-saved data corresponding to the training task of the current iteration from the shared memory into the checkpoint file and storing it in the shared storage device during the asynchronous service storage process of the large language model; The step of deconstructing the checkpoint file corresponding to the training task of the current iteration of the large language model to obtain a plurality of data to be saved, and saving the data to be saved into the shared memory includes: Construct a shared memory processing engine for processing shared memory, so that the shared memory processing engine deconstructs a plurality of data to be saved from a checkpoint file corresponding to a training task of a current iteration; Constructing multiple shared memory objects, so that each shared memory object stores one of the multiple data to be saved in the shared memory; The step of restoring the plurality of to-be-saved data corresponding to the training task of the current iteration from the shared memory into the checkpoint file and storing the files in the shared storage device comprises: After monitoring that multiple data to be saved corresponding to the training task of the current iteration are successfully saved in the shared memory, obtain the code of the current training node and the sequence number of the current data to be saved; Based on the code of the current training node and the serial number of the data to be saved, an identification dictionary is selected in the identification space, and the identification corresponding to each data to be saved is selected in the identification dictionary; wherein the identification space is in the shared memory, the identification dictionary is formed by merging the identifications corresponding to the multiple data to be saved, and the identification dictionary is stored in the identification space; Based on the identifier corresponding to each piece of data to be saved, obtain the data to be saved from the corresponding memory space; The plurality of data to be saved are restored into the checkpoint file and stored in the shared storage device.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the training task checkpoint file saving method as described in any one of claims 1-10 is implemented.

13. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for saving a training task checkpoint file as described in any one of claims 1 to 10 is implemented.

Images