Model check point file storage method and electronic equipment

Abstract

The invention provides a model check point file storage method and electronic equipment, and after the electronic equipment obtains training process information associated with a target model training task, the training process information indicates at least one training process used for executing the target model training task and a check point memory address allocated for a check point file. The check point file comprises target model state data updated in the target model training task execution process, and target abnormal events generated by the training process can be intercepted based on the training process information, so that the target abnormal events can be intercepted based on the check point memory address during the target abnormal event interception period. And storing the updated check point file in the memory of the electronic equipment to the storage equipment.

Description

Technical Field

[0001] The present application mainly relates to the field of computer application technology, and in particular to a model checkpoint file storage method and electronic device. Background Art

[0002] During the long training process of artificial intelligence (AI) models, training is often interrupted due to failures in electronic devices (such as various software and hardware, system failures, or framework failures). The model parameters in the memory of the electronic device are lost, and the AI ​​model training task has to be started from scratch, which greatly wastes training time and cost.

[0003] In order to reduce the risk of model parameter loss, a model checkpoint mechanism is usually used to periodically save model states such as model parameters, optimizer status, training rounds, etc. during the training process as checkpoint files, and store them on storage devices such as disks, so that training can be resumed from the latest saved checkpoint after training is interrupted, avoiding the waste of training time caused by starting training from scratch.

[0004] However, when an electronic device fails, the corresponding training process of the AI ​​model will be directly terminated, resulting in the inability of the training process to save the model checkpoint file obtained in this cycle. It is still necessary to re-execute the training process of this cycle after the fault is resolved, affecting the efficiency of model training. Summary of the invention

[0005] In view of the above problems, this application provides the following technical solutions:

[0006] The first aspect of the present application provides a method for saving a model checkpoint file, the method comprising:

[0007] Acquire training process information associated with a target model training task; wherein the training process information indicates at least one training process used to execute the target model training task, and a checkpoint memory address allocated to a checkpoint file, wherein the checkpoint file contains target model state data updated during the execution of the target model training task;

[0008] Based on the training process information, intercepting a target abnormal event generated by the training process;

[0009] While the target abnormal event is intercepted, the updated checkpoint file in the memory of the electronic device is saved to a storage device based on the checkpoint memory address.

[0010] In a possible implementation, intercepting a target abnormal event generated by the training process based on the training process information includes:

[0011] Based on the training process information, determining an update status of a corresponding checkpoint file in a memory of the electronic device and an abnormal event to be processed generated by the training process;

[0012] If the update status is the update completion status, the unprocessed abnormal event generated by the training process is determined as a target abnormal event for interception.

[0013] In a possible implementation, the method further includes:

[0014] Creating a checkpoint user agent and a checkpoint kernel agent; the checkpoint user agent runs in the user space of the electronic device, and the checkpoint kernel agent runs in the kernel space of the electronic device;

[0015] The acquiring of training process information associated with the target model training task, and intercepting a target abnormal event generated by the training process based on the training process information, includes:

[0016] The checkpoint user agent obtains training progress information associated with the target model training task;

[0017] The checkpoint user agent loads the training process information into the kernel space;

[0018] The checkpoint kernel agent intercepts a target abnormal event generated by the training process based on the training process information.

[0019] In a possible implementation, during the period when the target abnormal event is intercepted, based on the checkpoint memory address, saving the checkpoint file in the memory of the electronic device to a storage device includes:

[0020] The checkpoint kernel agent triggers the generation of a process pre-exit event for the training process in response to interception of the target abnormal event;

[0021] The checkpoint user agent monitors the process pre-exit event, determines a checkpoint memory address associated with the process pre-exit event,

[0022] The checkpoint user agent reads the checkpoint file that has been updated in the memory of the electronic device based on the determined checkpoint memory address;

[0023] The read checkpoint file is written into a storage device.

[0024] In a possible implementation, the checkpoint user agent obtains training progress information associated with the target model training task, including:

[0025] The checkpoint user agent determines a training process to be created for a target model training task, and a processor type to run the training process;

[0026] Configuring a training process identifier corresponding to the training process;

[0027] Based on a process communication mechanism corresponding to the processor type, converting a virtual memory address allocated for the checkpoint file into a physical memory address;

[0028] wherein the physical memory address is determined as the checkpoint memory address of the checkpoint file;

[0029] The checkpoint memory address and the training process identifier constitute the training process information associated with the target model training task.

[0030] In a possible implementation, the checkpoint user agent monitors the process pre-exit event, and determines the checkpoint memory address associated with the process pre-exit event, including:

[0031] The checkpoint user agent monitors the process pre-exit event, and determines a first training process identifier and event information associated with the process pre-exit event; the first training process identifier is the training process identifier of the training process that generates the target abnormal event, and the event information indicates the abnormal event type of the target abnormal event;

[0032] Based on the event information, determining that the target abnormal event is an abnormal process exit event;

[0033] Obtain a checkpoint memory address associated with the first training process identifier.

[0034] In a possible implementation, if the training process used to execute the target model training task includes a main process and multiple sub-processes, obtaining the checkpoint memory address associated with the first training process identifier includes any of the following:

[0035] If the abnormal training process having the first training process identifier is the main process, determining the second training process identifiers of each sub-process associated with the main process, and obtaining the checkpoint memory addresses corresponding to the first training process identifier and the second training process identifier;

[0036] If the abnormal training process with the first training process identifier is any of the sub-processes, determine the second training process identifier of the main process corresponding to the sub-process, and the third training process identifiers of other sub-processes associated with the main process, and obtain the checkpoint memory addresses corresponding to the first training process identifier, the second training process identifier and the third training process identifier.

[0037] In a possible implementation, the method further includes at least one of the following:

[0038] The checkpoint user agent creates a first mapping table associated with the target model training task, and loads the first mapping table into the kernel space; the first mapping table is used to record the training process identifier and the checkpoint memory address;

[0039] The checkpoint user agent creates a second mapping table associated with the target model training task, and loads the second mapping table into the kernel space; the second mapping table is used to record the process pre-exit event triggered by the checkpoint kernel agent, and the first training process identifier and event information associated with the process pre-exit event;

[0040] The checkpoint user agent successfully writes the read checkpoint file into the storage device, and updates the checkpoint save state value recorded in the first mapping table in the kernel space;

[0041] The checkpoint user agent determines, based on the event information associated with the process pre-exit event, that the target abnormal event is an abnormal operation event for the corresponding training process, and performs fault repair on the abnormal operation event based on the fault handling strategy recorded in the fault knowledge base;

[0042] The checkpoint user agent monitors the target model training task creation event, creates a checkpoint configuration file for the corresponding training process, and determines the checkpoint memory address associated with the process pre-exit event based on the checkpoint configuration file when the process pre-exit event is monitored; the checkpoint configuration file is used to record the training process identifier, checkpoint memory address and processor type corresponding to the training process;

[0043] The checkpoint kernel agent determines, based on the checkpoint save status value recorded in the first mapping table in the kernel space, that the updated checkpoint file in the memory of the electronic device has been saved to the storage device, releases the interception of the target abnormal event, and performs a process cleanup operation on the training process that generates the target abnormal event.

[0044] In a possible implementation, the creating of a checkpoint user agent and a checkpoint kernel agent includes:

[0045] Obtaining a checkpoint user agent program file and a checkpoint kernel agent program file, wherein the checkpoint user agent program file and the checkpoint kernel agent program file are obtained based on an extended Berkeley packet filter architecture;

[0046] Executing the checkpoint user agent program file to run the corresponding checkpoint user agent in the user space of the electronic device;

[0047] The checkpoint user agent loads the checkpoint kernel agent program file into the kernel space of the electronic device;

[0048] The kernel space is based on the checkpoint kernel agent program file, mounts the corresponding checkpoint kernel agent to a designated kernel hook and then runs it; the designated kernel hook is used to track and detect unprocessed abnormal events generated during the execution of the target model training task.

[0049] A second aspect of the present application provides an electronic device, the electronic device comprising at least one memory and at least one processor, wherein:

[0050] The memory is used to store computer programs;

[0051] The processor is used to execute the computer program to implement the following steps:

[0052] Acquire training process information associated with a target model training task; wherein the training process information indicates at least one training process used to execute the target model training task, and a checkpoint memory address allocated to a checkpoint file, wherein the checkpoint file contains target model state data updated during the execution of the target model training task;

[0053] Based on the training process information, intercepting a target abnormal event generated by the training process;

[0054] While the target abnormal event is intercepted, the updated checkpoint file in the memory of the electronic device is saved to a storage device based on the checkpoint memory address. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0056] Figure 1 This is a flow chart of the method for saving a model checkpoint file proposed in Example 1 of the present application;

[0057] Figure 2 This is a flow chart of the method for saving a model checkpoint file proposed in Embodiment 2 of the present application;

[0058] Figure 3 A schematic diagram of a system architecture applicable to the model checkpoint file saving method proposed in an embodiment of the present application;

[0059] Figure 4 This is a flow chart of the model checkpoint file saving method proposed in Embodiment 3 of the present application;

[0060] Figure 5 This is a flow chart of the method for saving a model checkpoint file proposed in Embodiment 4 of the present application;

[0061] Figure 6 A schematic diagram of a process cleanup operation in the model checkpoint file saving method proposed in Embodiment 5 of the present application;

[0062] Figure 7 A schematic diagram of another system architecture applicable to the model checkpoint file saving method proposed in the embodiment of the present application;

[0063] Figure 8 A schematic diagram of a process cleanup operation in the model checkpoint file saving method proposed in Embodiment 6 of the present application;

[0064] Fig. 9 A schematic diagram of the structure of a model checkpoint file storage device provided in an embodiment of the present application;

[0065] Fig.10 The present invention is a schematic diagram of the hardware structure of an electronic device suitable for the model checkpoint file saving method proposed in this application. DETAILED DESCRIPTION

[0066] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation mode of the present application are only used to explain the specific embodiments of the present application and are not intended to limit the present application. The embodiments of the present application are described below in conjunction with the drawings. It is known to those of ordinary skill in the art that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0067] The term "first", "second" etc. that the full text of the application relates to is used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable in appropriate circumstances, and this is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the application. In addition, the terms "including" and "having" and any of their deformations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0068] In view of the description in the background technology section, a model checkpoint saving mechanism is used to periodically save the model state of the artificial intelligence (AI) model training, so that when the electronic device performing the model training task fails, a checkpoint saving semaphore is sent to the training process. The training process will execute the checkpoint saving function, and the checkpoint file composed of model state data such as model parameters, optimizer status, training rounds, etc. updated in the training process of this cycle, which is cached in the memory of the electronic device, is written to a storage device such as a disk for persistent storage to achieve breakpoint resumption of training. Among them, since a cycle time is too long, the time to resume model training will also be long, which will also block the training process. The cycle time can be shortened, and even the model state data updated in each round of training can be saved in real time to shorten the time for resuming training and avoid blocking training problems.

[0069] However, when the internal exception handling mechanism of the electronic device system is triggered due to an electronic device failure, the corresponding training process will be directly terminated, resulting in the training process being unable to perform any cleanup operations in the user state (user space), and being unable to receive the checkpoint save semaphore, and even more unable to save the checkpoint file by executing the checkpoint save function. The checkpoint file cached in the electronic device memory will be directly cleared, and the current cycle training process will need to be re-executed after the fault is repaired.

[0070] Based on this, in order to improve the timeliness and reliability of model checkpoint file preservation and reduce model training overhead, the embodiment of the present application provides a model checkpoint file preservation method and electronic device, which can timely and reliably save the checkpoint file updated in this round of training when various faults occur during the model training process, so that after the fault is resolved, the saved checkpoint file can be used to quickly restore to the model state completed in this round of training, and continue to perform the next round of model training, avoiding the overhead caused by repeatedly executing the same round of model training process. The model checkpoint file preservation method of the embodiment of the present application is described in detail below in conjunction with the accompanying drawings.

[0071] Reference Figure 1 , is a flow chart of the model checkpoint file saving method proposed in the first embodiment of the present application. The model checkpoint file saving method can be applied to electronic devices, which are training devices for performing model training tasks, and can be one or more terminal devices or servers with data processing capabilities. The present application does not limit the deployment structure of the training device for performing model training tasks, which can be determined according to the circumstances. Figure 1 As shown, the model checkpoint file saving method proposed in this embodiment may include but is not limited to the following steps:

[0072] Step S11, obtaining training process information associated with the target model training task;

[0073] In an embodiment of the present application, the target module may be a machine learning model created or retrieved according to the requirements of the application scenario, and may be an artificial intelligence (AI) model suitable for scenarios such as natural language processing, recommendation systems, image / video processing, autonomous driving, medical diagnosis, financial analysis, education, content creation, games, smart homes, smart cities, or industrial automation. The present application does not impose any restrictions on the network structure of the target model, which may be determined depending on the circumstances.

[0074] Based on this, for target model training tasks with different structures, the corresponding training environment architecture and strategy can be deployed in combination with factors such as the number and complexity of target model parameters, as well as performance requirements such as model training efficiency, so as to achieve the target model training task through a single training process or a distributed training process (i.e., a distributed architecture consisting of multiple training processes). This application does not elaborate on the creation and implementation process of the target model training task.

[0075] It can be seen that the training process information in step S11 can indicate at least one training process for executing the target model training task, and a checkpoint memory address allocated to a checkpoint file, which contains the target model state data updated during the execution of the target model training task, such as various model parameters, optimizer status, and training rounds, etc. This application does not limit the content of the checkpoint file and the implementation method of its memory address allocation.

[0076] It should be understood that the target model is usually iteratively trained for multiple rounds to obtain a module that meets the requirements of the application scenario. At least one model state data is usually generated in each round of training of the target model. In order to avoid a long breakpoint recovery time for the target model and affect the model training efficiency, a checkpoint file save process can be performed for each round of training of the target model. This process can be acquired and processed by the corresponding training process. This application does not elaborate on the acquisition and processing process of each target model state data contained in the checkpoint file.

[0077] Step S12, intercepting target abnormal events generated by the training process based on the training process information;

[0078] In the process of executing the target model training task through at least one created training process, an abnormal event of interrupting the target model training task may occur due to one or more failures, such as various hardware failures of the electronic device (such as errors in the processor running the training process, execution of illegal instructions, memory allocation failure, etc.), framework failures (such as OOM (Out of Memory), Runtime error, etc.), system failures (such as system timeout, file system corruption, etc.), program logic errors (such as assertion failure), etc.

[0079] In actual applications, the system kernel of an electronic device can detect various abnormal events generated by various processes running in the user space. The present application can identify the target abnormal events generated by the training process used to execute the target model training task based on the training process information associated with the target model training task. The present application will intercept the target abnormal event in time to avoid executing the exception handling mechanism for the target abnormal event, directly terminating the corresponding training process, and causing the failure to save the checkpoint file completed in this round of training update.

[0080] It can be seen that the above-mentioned target abnormal event of the present application is an abnormal event generated when the training process obtains the checkpoint file updated by the current round of training, due to various fault problems listed above, resulting in abnormal exit or timeout exit. The implementation process of step S12 can be implemented based on the pre-configured information in the system kernel that meets the checkpoint file storage requirements, such as pre-configuring a hook function for executing event interception operations, and triggering the call of the hook function to implement the judgment conditions for intercepting the target abnormal event, etc. The identification of the target abnormal event of the present application and its interception method are not limited.

[0081] Step S13, while the target abnormal event is intercepted, based on the checkpoint memory address, the updated checkpoint file in the electronic device memory is saved to the storage device.

[0082] Since the present application timely intercepts the target exception event generated by the training process, before the target exception event is handled through the exception handling mechanism, the checkpoint file obtained by the training process that generated the target exception event still exists in the memory of the electronic device. According to the checkpoint memory address allocated for this round of training, the checkpoint file in the memory can be successfully saved to a storage device, such as a disk in the electronic device or a connected external storage device, etc., to achieve persistent storage of the checkpoint file. The present application does not limit the type of storage device and its storage method for checkpoint files.

[0083] Combined with the above description of the target abnormal event, when the target abnormal event occurs, the training process has completed the current round of training update of the target model state, and there is no need to continue to update the target model state during the current round of training, so that the checkpoint file cached in the electronic device memory contains the complete target model state data required for the current round of training update. In this way, based on the checkpoint file saved by the storage device, it is possible to quickly restore to the target model state that has completed the current round of training, without re-executing the current round of training on the target model, and the next round of training on the target model can be directly executed, which reliably reduces the overhead caused by repeating at least one round of training of the target model, and is conducive to improving the efficiency and reliability of model training.

[0084] Reference Figure 2 , is a flow chart of the model checkpoint file saving method proposed in the second embodiment of the present application. This embodiment can describe an optional implementation method of determining and intercepting the target abnormal event proposed above, such as Figure 2 As shown, the method proposed in the embodiment of the present application may include:

[0085] Step S21, creating a checkpoint user agent and a checkpoint kernel agent, running the checkpoint user agent in the user space of the electronic device, and running the checkpoint kernel agent in the kernel space of the electronic device;

[0086] During the execution of the target model training task, in order to reduce the risk of checkpoint saving failure, improve system security and stability, etc., the embodiment of the present application proposes to create a third-party checkpoint agent module to implement the model checkpoint file saving method. For the operating modes (such as user state and kernel state) / operating spaces (corresponding to user space and kernel space) required for the electronic device to run the different steps of the method, a checkpoint user agent running in the user space of the electronic device and a checkpoint kernel agent running in the kernel space of the electronic device will be created. This application does not limit the creation and implementation methods of these two agents.

[0087] It should be noted that during the implementation of step S21, the configuration of the proxy module (i.e., the corresponding checkpoint user agent or checkpoint kernel agent) running in the corresponding operating space can be implemented according to the content of the method steps executed in different operating spaces of the electronic device, thereby ensuring that the two proxy modules (i.e., the checkpoint user agent and the checkpoint kernel agent) created by the electronic device interact with each other and reliably execute the model checkpoint file saving method proposed in this application. This application does not limit the configuration content of each proxy module.

[0088] Step S22, the checkpoint user agent obtains training process information associated with the target model training task;

[0089] In an embodiment of the present application, in response to a target model training task request submitted by a user, at least one training process for executing the target model training task can be created in the user space, and an identifier for identifying different training processes can be configured. The target model training task can be independently executed as required by the training process, or part of the target model training task can be executed, etc. Memory space (which can be represented by a corresponding memory address) is allocated for each target model state data obtained when executing the corresponding training task to cache the acquired target model state data.

[0090] It should be understood that the total memory space allocated to all training processes that execute the target model training task is greater than or equal to the amount of data of all target model state data required to be stored in one round of training process to ensure the integrity of the target model state data cached in the memory. This application does not limit the implementation method of memory space allocation.

[0091] During the creation of the training process of the above-mentioned target model training task, the checkpoint user agent running in the user space listens to the event, and obtains the configuration content such as the identifier of at least one training process used to execute the target model training task as indicated above, and determines the checkpoint memory address used to indicate the total memory space allocated from the electronic device memory for caching the complete target model state data updated in this round of training, to form the training process information associated with the execution of the target training task in this round and store it for subsequent query to implement subsequent steps. This application does not limit the content contained in the training process information and its acquisition method, and can be flexibly configured or adjusted according to actual processing needs.

[0092] In practical applications, if the size of the complete checkpoint files updated for different rounds of training of the target model is different, the size of the memory space allocated to them may be different, and the corresponding checkpoint memory address determined by the checkpoint user agent may be different. It is necessary to ensure that it indicates the storage location in memory of all the target model state data contained in the complete checkpoint file updated for this round of training, so that when executing the checkpoint save process, the checkpoint user agent can reliably read all the target model state data completed for this training update from the memory and write it to the storage device for persistent storage. This application does not limit the method for determining the checkpoint memory address.

[0093] Step S23, the checkpoint user agent loads the training process information into the kernel space;

[0094] Since various events generated by processes running in user space are usually handled by the system kernel, the checkpoint agent needs to share the training process information with the checkpoint kernel agent to ensure that the checkpoint kernel agent can promptly intercept the target abnormal events generated by at least one training process (the process used to execute the target model training task). Therefore, after obtaining the training process information associated with the target model training task, the checkpoint user agent can load the training process information into the kernel space through the communication method between the user space and the kernel space, and the kernel space stores the training process information associated with the current round of training of the target model. This application does not limit the storage method of the training process information loaded into the kernel space.

[0095] In one possible implementation, the present application may first create a kernel mapping table in the kernel space for storing training process information. During each round of training of the target model training task, the checkpoint user agent may load the training process information acquired this time into the kernel mapping table, so that the checkpoint kernel agent running in the kernel space can determine whether the training process generates the target abnormal event in a timely and accurate manner based on the training process information recorded in the kernel mapping table.

[0096] Step S24, the checkpoint kernel agent determines the update status of the corresponding checkpoint file in the memory of the electronic device and the abnormal event to be processed generated by the training process based on the training process information loaded in the kernel space;

[0097] Following the above analysis, in the process of intercepting the target abnormal events generated by the training process based on the training process information loaded by the kernel space, the checkpoint kernel agent can timely and accurately determine the update status of the current round of training of the target model (here it means the current round of training progress of the target model) by tracking the system calls related to the checkpoint memory address read and write operations based on the training process information.

[0098] Exemplarily, the checkpoint kernel agent tracks the checkpoint read and write system calls based on the training process information loaded in the kernel space this time. If a write request to the checkpoint memory address is detected, that is, a request to write at least one target model state data updated in this round of training to the checkpoint memory address, it means that the target model state of this round of training has not been updated, and the checkpoint file under the checkpoint memory address contains incomplete target model state data, and the update status of the checkpoint file at this time is determined to be an incomplete update state, which can be recorded as Updating. If a write completion request to the checkpoint memory address is detected, it means that the update of the target model state in this round of training is completed, and the checkpoint file under the checkpoint memory address in the memory contains complete target model state data updated in this round of training, and the update status of the checkpoint file at this time is determined to be an update completion state, which can be recorded as Finished.

[0099] In actual applications, the present application can determine the update status of the checkpoint file in a timely manner in the manner described above, but not limited to, and store it in the kernel space, such as writing it to the corresponding field of the mapping table in the kernel space for subsequent query. Optionally, in combination with the above description of the different update states of the checkpoint file, the training process can be in the process of executing various read and write requests on the checkpoint memory address. After executing the read and write requests, the checkpoint user agent determines the update status of the corresponding checkpoint file, and then loads it into the system kernel for the checkpoint kernel agent to consult. The present application does not limit the acquisition and storage methods of the update status of the checkpoint file.

[0100] Step S25, the checkpoint kernel agent determines that the update status is the update completion status, and determines the to-be-processed abnormal event generated by the training process as a target abnormal event for interception;

[0101] Step S26, the checkpoint user agent monitors that the target abnormal event is intercepted, and based on the checkpoint memory address, saves the checkpoint file in the electronic device memory to the storage device.

[0102] When a failure occurs during the execution of a target model training task by an electronic device, in the target model breakpoint resumption training application based on the checkpoint saving mechanism, it is necessary to use the complete target model state data saved in the latest round of training updates to quickly restore to the target model state of the latest round of training updates. If the target model state data in the saved checkpoint file is incomplete, it is impossible to restore to the target model state when the target model training task was interrupted.

[0103] Therefore, in order to avoid persistent storage of incomplete target model state data, when executing checkpoint file saving processing, the checkpoint kernel agent running in the kernel space of the present application determines that the training process generates an unprocessed exception event. It is necessary to determine that the checkpoint update status of the corresponding checkpoint file at this time is the update completion status, that is, to determine that the current round of training update of the target model state data is completed, so that the unprocessed exception event generated by the training process can be determined as the target exception event, and the target exception event can be intercepted in time. After waiting for the completion of the checkpoint file saving of the current round of training update (here refers to the persistent storage realized by using the storage device), the interception of the target exception event can be restored, so as to process it through the exception handling mechanism and improve the reliability and accuracy of the complete checkpoint file saving.

[0104] Among them, for the preservation of the checkpoint file updated after this round of training, the present application will be implemented by the checkpoint user agent running in the user space. The checkpoint user agent may monitor that the target abnormal event generated by the training process is intercepted by the checkpoint kernel agent. During the period when the target abnormal event is intercepted, the checkpoint memory address associated with the training process that generated the target abnormal event is determined from the stored training process information, or after reading the corresponding checkpoint memory address from the kernel space when the target abnormal event is intercepted, the complete checkpoint file currently cached in the memory of the electronic device is read in time according to the checkpoint memory address, that is, the complete target model status data updated in this round of training, and send it to the storage device for persistent storage. Compared with notifying the training process to execute the checkpoint file save processing operation, the present application is changed to be executed by the checkpoint user agent, which effectively and reliably reduces the risk of checkpoint file save failure.

[0105] To summarize, in an embodiment of the present application, a checkpoint user agent is created to run in the user space of an electronic device, and a checkpoint kernel agent is created to run in the kernel space of an electronic device. When at least one training process for executing a target model training task is created in the user space, the checkpoint user agent obtains the training process information associated with the target model training task, which indicates the at least one training process and the checkpoint memory address allocated to the checkpoint file. In this way, after loading it into the kernel space, the checkpoint kernel agent can determine the pending exception events generated by the at least one training process based on this, and determine whether the checkpoint file in the memory contains the complete target model status data updated in this round of training, that is, determine whether the update status of the checkpoint file is the update completion status. If so, it means that the pending exception event generated by the training process is the target exception event, and the checkpoint kernel agent will intercept the target exception event to avoid unnecessary interception of pending exception events generated when the checkpoint file is in an incomplete update state, thereby affecting the exception handling efficiency.

[0106] Once the checkpoint user agent detects that the target abnormal event has been intercepted, it can save the complete checkpoint file in the electronic device's memory to the storage device in a timely manner during the interception period based on the corresponding checkpoint memory address, thereby realizing persistent storage of the complete target model state data updated in this round of training, avoiding accidental interruption of the training process and inability to execute the checkpoint save processing operation, improving the timeliness and reliability of checkpoint saving, and being able to directly continue to the next round of training without repeating the current round of training when resuming training. The real-time saving of the target model training task status in this application minimizes the number of checkpoint file saves, avoids taking up too much effective training time, and at the same time reduces storage space occupancy, thereby improving model training efficiency and reliability.

[0107] In some embodiments, for the creation of the above-mentioned checkpoint user agent and checkpoint kernel agent, the present application can be implemented using the extended Berkeley Packet Filter (eBPF) framework. Among them, eBPF is a virtual machine technology in the Linux kernel, which allows users to run customized program code in the kernel space to achieve higher-level network filtering, monitoring and security functions, etc. Therefore, the present application can generate a checkpoint user agent program file and a checkpoint kernel agent program file based on the eBPF architecture. After the electronic device obtains these two program files (checkpoint user agent program file and checkpoint kernel agent program file), it can execute the checkpoint user agent program file to run the corresponding checkpoint user agent in the user space of the electronic device.

[0108] After that, the checkpoint user agent can load the checkpoint kernel agent program file into the kernel space of the electronic device. Based on the checkpoint kernel agent program file, the kernel space mounts the corresponding checkpoint kernel agent to the specified kernel hook and runs it to track and intercept the target abnormal events. It can be seen that the specified kernel hook is used to track and detect the pending abnormal events generated during the execution of the target model training task, and based on the training process information of the current round of training target model loaded in the kernel space, identify whether there are target abnormal events.

[0109] Combined with the above analysis, the embodiment of the present application can adopt frameworks such as BCC (BPF Compiler Collection, an open source project, which is a collection of BPF compilers) or Libbpf + BPF CO-RE (Compile Once – Run Everywhere, BPF portability) to develop an eBPF program for the target checkpoint file saving method steps executed in the kernel space, so that the eBPF program can be loaded from the user space to the kernel space through the interface provided by eBPF, and triggered by the target abnormal event (such as the event generated when the process is created, exited, etc.). Among them, BCC is a development tool chain for eBPF programs, which combines Python user space code with eBPF code written in C. This application does not describe its processing process in detail.

[0110] In the practical application of this application, refer to Figure 3 As shown in the flowchart, the task manager receives a target model training task creation request submitted by a user and creates at least one training process for it, such as Figure 3The distributed training process shown, i.e., the architecture formed by the main process and multiple sub-processes, is implemented in the user space by the checkpoint user agent to execute the model checkpoint file saving method steps, such as loading the eBPF program into the kernel space, compiling it into machine code, i.e., verifying the security of the checkpoint kernel agent, and the contents described in the above steps S21 and S22. In this way, the checkpoint kernel agent running in the kernel space can search for the training process identifier of the training process, match the process associated with the system call, i.e., determine the unprocessed abnormal event generated by at least one training process used to execute the target model training task, match the associated process identifier with the training process identifier loaded in the kernel space, and combine the checkpoint memory address associated with the training process identifier to determine that the training process has obtained the complete update completion state of the target state data at this time, and timely intercept the target abnormal event. The checkpoint user agent monitors this interception and saves the corresponding checkpoint file in time.

[0111] Reference Figure 4 , which is a signaling flow diagram of the model checkpoint file saving method proposed in Example 3 of the present application. This embodiment can be another optional implementation method for determining and intercepting target abnormal events when creating a checkpoint user agent and a checkpoint kernel agent, such as Figure 4 As shown, the method may include:

[0112] Step S41, the checkpoint user agent determines the training process created for the target model training task and the type of processor running the training process;

[0113] Step S42, the checkpoint user agent configures the training process identifier corresponding to the training process;

[0114] The task manager responds to the target model training task creation request submitted by the user and creates at least one training process for executing the target model training task, such as a single training process or multiple distributed training processes, which can be divided into a main process (which can also be a lancher process) and at least one child process (which can be a corresponding worker process). This application does not describe in detail the creation and implementation process of the training process.

[0115] When the checkpoint user agent listens to the creation of a training process, it can determine which training processes are created by the user space to execute the target model training task, and configure corresponding training process identifiers, such as training process IDs, for each determined training process to identify the corresponding training process. This application does not limit the content of the training process identifier and its configuration implementation method, which can be determined based on the corresponding processing logic contained in the checkpoint user agent program file.

[0116] For the created training process, it is run in the processor of the electronic device to execute at least part of the target model training task assigned to each. The processor may include but is not limited to one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP) or other special processors, etc. This application does not limit the processor running the training process. Because different types of processors run training processes, the checkpoint user agent obtains the checkpoint memory address in different ways, this application can first determine the type of processor running at least one training process.

[0117] In actual applications, since electronic devices may implement multiple model training tasks, in order to distinguish them, corresponding task service identifiers, such as task service names, can be configured. In the subsequent processing, the task service identifier is first used to determine the existence of the target model training task, and then the various training tasks used to execute the target model training task are further queried to implement the configuration of the training process identifier. For the above-listed contents such as task service name, training process identifier, and processor type, a checkpoint configuration file can be created for storage for subsequent query.

[0118] Step S43, the checkpoint user agent converts the virtual memory address allocated for the checkpoint file into a physical memory address based on the process communication mechanism corresponding to the processor type; the physical memory address is determined as the checkpoint memory address of the checkpoint file;

[0119] In an embodiment of the present application, in order to improve the processing accuracy and efficiency of the checkpoint files updated in the memory for the current round of training of the target model, after the checkpoint user agent obtains the virtual memory address assigned to the checkpoint file, it can convert it into a physical memory address and determine it as the checkpoint memory address of the checkpoint file, which is used to reliably indicate the cache location of the checkpoint file in the memory. The present application does not limit the conversion processing method of the virtual memory address.

[0120] In a possible implementation, if the processor running the training process is a GPU, the GPU memory virtual address of the training process checkpoint metadata and resource data can be obtained through the IPC (Inter-Process Communication, which can be achieved through shared memory and memory mapped files, and through shared memory areas to achieve data sharing between processes) process communication mechanism, thereby using the training process identifier and memory virtual address of the corresponding training process running on the GPU to obtain the physical memory address of the checkpoint file, and store it as the checkpoint memory address in the checkpoint configuration file created by the checkpoint user agent, so that the training process memory data can be operated accordingly in the future. Among them, the checkpoint resource data variables may include one or more variables such as model parameters, optimizers, schedulers, model weights, and training gradients, and the checkpoint metadata variables may include one or more variables such as Epoch variables and checkpoint file storage paths. After these variables are taken, they constitute the checkpoint file as the target model state data.

[0121] In a possible implementation, if the processor running the training process is a CPU, a training process tracking session can be created through the Ptrace process communication mechanism, and the current checkpoint user agent process is attached to the training task process. Based on the checkpoint metadata variable and the resource data variable virtual address, the actual physical memory address is calculated and recorded as the checkpoint memory address in the checkpoint configuration file. Among them, the Ptrace process communication mechanism is a system call used for training process tracking and control in the Linux system. It allows a process (called tracer) to monitor and control the execution of another process (called tracee). This application does not elaborate on the working principle.

[0122] It should be noted that in the process of converting the virtual memory address allocated to the checkpoint file into a physical memory address, the processor types suitable for implementation using the IPC process communication mechanism include but are not limited to GPUs. Similarly, the processor types suitable for implementation using the Ptrace process communication mechanism include but are not limited to CPUs. Of course, for other types of processors, other process communication mechanisms can also be used to implement virtual memory address conversion processing. This application does not limit the type of processor running the training process and the process communication mechanism used to implement memory virtual address conversion, which can be determined based on actual conditions.

[0123] Step S44, the checkpoint user agent loads the training process identifier and the checkpoint memory address into the kernel space;

[0124] In the embodiment of the present application, the training process identifier and the checkpoint memory address obtained in the above manner constitute the training process information and are loaded into the kernel space. If a kernel mapping table for recording the training process information is created in the kernel space in advance, the checkpoint user agent can load the training process identifier and the checkpoint memory address into the kernel mapping table for storage, so that the checkpoint kernel agent can perform subsequent steps accordingly. It should be noted that other contents can also be recorded in the kernel mapping table according to actual needs, such as the update status of the checkpoint file, the save status of the checkpoint file, etc.

[0125] Step S45, the checkpoint kernel agent triggers the generation of a process pre-exit event for the training process in response to the interception of the target abnormal event;

[0126] Step S46: the checkpoint user agent monitors the process pre-exit event, determines the checkpoint memory address associated with the process pre-exit event,

[0127] Step S47, the checkpoint user agent reads the corresponding updated checkpoint file in the memory of the electronic device based on the determined checkpoint memory address;

[0128] Step S48: The checkpoint user agent writes the read checkpoint file into the storage device.

[0129] In order to avoid the inability to read the checkpoint file to perform the save processing operation due to the clearing of memory resources after the training process exits, this application adds a process pre-exit (Process_preexit) system call. When it is determined according to the method described above that the training process generates a target abnormal event, the process pre-exit event is triggered to notify the checkpoint user agent that the training process is about to exit, or the process pre-exit event is passed to the checkpoint user agent to save the checkpoint file in the memory to the storage device in time.

[0130] In some embodiments, the checkpoint user agent running in the user space can monitor events generated in the kernel space in real time. In this way, when a process pre-exit event is monitored, based on the pre-defined content of the process pre-exit event, it is known that the current training process has generated a target exception event and will be cleaned up based on the exception handling mechanism. Before the corresponding checkpoint file in the memory is cleared, the checkpoint user agent needs to save the checkpoint file to the storage device in time. Compared with the preservation of the checkpoint file by the training process where the target exception event occurs, the present application uses the checkpoint user agent to perform checkpoint file reading and writing operations in accordance with the method described above, which can reliably realize the persistent storage of the checkpoint file updated in this round of training, so that it can quickly restore to the recorded target model state later, thereby improving the efficiency and reliability of model training.

[0131] In a possible implementation, the process pre-exit event generated by the kernel space can also be recorded in a mapping table storage method, that is, when the process pre-exit event is triggered, the process pre-exit event and its related information are stored in the kernel mapping table, so as to realize the interaction between the checkpoint user agent and the checkpoint kernel agent through the kernel mapping table, so that the checkpoint user agent can directly monitor the kernel mapping table, timely determine the newly generated process pre-exit event, and save the checkpoint file in time. For various contents loaded into or generated by the kernel space, the storage of the contents in the kernel space can be realized by using, but not limited to, a mapping table storage created for each type of content, or a mapping table and other storage methods.

[0132] Reference Figure 5 , is a signaling flow diagram of the model checkpoint file saving method proposed in the fourth embodiment of the present application. This embodiment can provide a detailed description of the checkpoint file saving process of the checkpoint user agent in the model checkpoint file saving method proposed above, such as Figure 5 As shown, the method may include:

[0133] Step S51, the checkpoint user agent monitors a process pre-exit event, and determines a first training process identifier and event information associated with the process pre-exit event;

[0134] In an embodiment of the present application, the first training process identifier is the training process identifier of the training process that generates the target abnormal event, such as the first training process ID. The event information can indicate the abnormal event type of the target abnormal event, such as the description information of the target abnormal event, etc. The present application does not limit the method of obtaining it.

[0135] Step S52, the checkpoint user agent determines that the target abnormal event is an abnormal process exit event based on the event information;

[0136] According to the above description of event information, by analyzing the content of event information, it is determined whether the target abnormal event is an abnormal process exit event or an abnormal operation event for the training process. The subsequent processing methods for different types of target abnormal events may be different.

[0137] In a possible implementation, if the target abnormal event is the above-mentioned abnormal operation event, the checkpoint user agent can perform fault repair on the abnormal operation event based on the fault handling strategy recorded in the fault knowledge base to try to solve the fault of the electronic device. If it cannot be solved, the method described below can be used to save the checkpoint file and trigger the abnormal handling mechanism to terminate the training process that generates the abnormal operation event. Of course, in the case where the target abnormal event is an abnormal operation event, it can also be directly processed according to the subsequent processing strategy for such a fault that cannot be solved.

[0138] Step S53, the checkpoint user agent obtains the checkpoint memory address associated with the first training process identifier;

[0139] Combined with the above analysis, since the target model training task may be a stand-alone training task or a distributed training task, a single training process can be created for a stand-alone training task; for a distributed training task, multiple training processes including a main process and a sub-process need to be created for execution. It can be seen that for distributed training tasks, the training process that generates the target abnormal event may be the main process or the sub-process. For training processes of different levels, the implementation method of determining the checkpoint memory address of the target model state data updated for the current round of training of the target model is different. Therefore, the present application can determine the subsequent corresponding processing operations by identifying whether the training process with the first training process identifier belongs to the main process or the sub-process.

[0140] Based on this, if the checkpoint kernel agent determines that the abnormal training process with the first training process identifier is the main process of the target model training task, it can determine the second training process identifier of each sub-process associated with the main process, and obtain the checkpoint memory address corresponding to the first training process identifier and the second training process identifier, thereby determining the checkpoint memory address corresponding to all training processes that execute the target model training task, ensuring that the checkpoint memory address stores the complete target model status data updated in this round of training, laying the foundation for the reliable preservation of subsequent checkpoint files.

[0141] If the checkpoint kernel agent determines that the abnormal training process with the first training process identifier is a subprocess (such as any subprocess of a distributed training task, such as Figure 3 The process i shown, i=1, 2, 3, etc. are integers), determines the second training process identifier of the main process corresponding to the sub-process, and the third training process identifier of other sub-processes associated with the main process, and obtains the checkpoint memory addresses corresponding to the first training process identifier, the second training process identifier, and the third training process identifier. It can be seen that in the case where a certain sub-process generates a target abnormal event, the distributed architecture between multiple training processes can be used to determine the main process ID (i.e., the second training process identifier, which can be expressed as PID, etc.) corresponding to the sub-process, so as to further query the training process IDs of other sub-processes associated with the main process based on the main process ID (i.e., the third training process identifier, which can be expressed as PPID, etc.), thereby determining the training process identifiers of all training processes that execute the target model training task, and ensuring the integrity and reliability of the checkpoint memory addresses determined accordingly.

[0142] It should be understood that, in the case where distributed training includes a main process and a child process, in the above execution process, the first training process identifier and the second training process identifier are determined, and the checkpoint memory address corresponding to the first training process identifier and the second training process identifier is obtained. In addition, for a single-machine training task, the single training process created for it can be used as both a main process and a child process. In this case, the checkpoint memory address corresponding to the first training process identifier can be directly obtained, and the physical memory address of the complete target model state data updated in this round of training can be obtained.

[0143] Step S54, during the period when the target abnormal event is intercepted, the checkpoint user agent reads the corresponding updated checkpoint file in the electronic device memory based on the checkpoint memory address and writes it into the storage device;

[0144] Step S55, the checkpoint user agent determines that the checkpoint file is successfully written into the storage device, and updates the corresponding checkpoint save state value to the kernel space;

[0145] Step S56: The checkpoint kernel agent releases the interception of the target abnormal event based on the updated checkpoint save state value and the first training process identifier, and performs a cleanup operation on the training process that generates the target abnormal event.

[0146] Following the above analysis, after the checkpoint user agent determines the checkpoint memory address of the complete checkpoint file that needs to be saved, it can read all the target model state data updated in this round of training and write it to the storage device, thereby achieving complete and reliable preservation of the current training state data of the target model. When resuming training, it can be used to directly restore to the current training state of the target model without repeating the current round of training of the target model.

[0147] After completing the saving of the checkpoint file, the checkpoint guarantee status corresponding to each training process executing the target model training task can be updated, and the updated checkpoint save state value can be determined according to the meaning represented by the pre-configured different checkpoint save state values. Exemplarily, the checkpoint save state value is configured to 1, indicating that the saving of the checkpoint file has been completed, and the checkpoint kernel agent can be notified to release the interception of the target abnormal event and execute the normal process exit system call, that is, to perform a cleanup operation on the corresponding training process. If the checkpoint save state value is 0 or other values, it means that the checkpoint file has not been saved yet, and the interception of the target abnormal event is still maintained.

[0148] For the implementation process of step S56, in a possible implementation, refer to Figure 6As shown in the flowchart, according to the method described above, it is determined that the checkpoint file has been successfully saved and the target abnormal event can be processed. Therefore, the checkpoint kernel agent can release the interception of the target abnormal event and use the determined first training process identifier to perform a cleanup operation on the training process that generates the target abnormal event, such as Figure 6 The cleaning operation process shown can trigger the exception handling mechanism within the system. The system processing module that executes the mechanism calls the exit function exit() to terminate the corresponding training process. Specifically, the training process can be suspended and the cached data in the memory can be cleared to cause the training process to exit, but the method is not limited to this process cleaning operation implementation method.

[0149] According to the model checkpoint file saving method described in the above embodiments, in the implementation scenario of using a kernel mapping table to store training process information, various states and events, the kernel mapping table can be created and loaded into the kernel space during the checkpoint user agent initialization phase. If the kernel mapping table to be created includes a first mapping table and a second mapping table, the first mapping table is used to record the training process identifier and the checkpoint memory address, and can also record the update status of the checkpoint file and the checkpoint save status value as needed. The second mapping table is used to record the process pre-exit event triggered by the checkpoint kernel agent, as well as the first training process identifier and event information associated with the process pre-exit event.

[0150] So, refer to Figure 7 In the schematic diagram of the system architecture shown, the checkpoint user agent creates a first mapping table associated with the target model training task, loads the first mapping table into the kernel space, and creates a second mapping table associated with the target model training task, loads the second mapping table into the kernel space, and implements data storage according to the configuration requirements for the storage contents of the two mapping tables. According to the above description of the model checkpoint file saving method, the second mapping table in the kernel space can be used to implement event interaction between the checkpoint user agent and the checkpoint kernel agent, such as the interaction of the above process pre-exit event, to improve the timeliness of checkpoint file saving. The implementation process is not described in detail in this embodiment.

[0151] Among them, for the training process information stored in the first mapping table, such as Figure 7 As shown, the checkpoint user agent may monitor the target model training task creation event, create a checkpoint configuration file for the corresponding training process, and obtain the training process identifier and checkpoint memory address ( Figure 7Only taking the scenario where the CPU runs the training process as an example to illustrate the process of obtaining the checkpoint memory address) writes the checkpoint configuration file into the checkpoint configuration file. Of course, the checkpoint configuration file can also store other content such as the task service identifier of the target model training task, the processor type, etc., which can be obtained according to actual needs. This application does not limit the content of the checkpoint configuration file and its creation implementation method.

[0152] In one possible implementation, the present application can create a checkpoint kernel agent based on eBPF and load it into the kernel space to run during the checkpoint user agent initialization phase, so as to achieve real-time capture of pending exception events generated by the training process during the execution of the target model training task, so as to intercept the target exception event in time when it is determined that it belongs to the target exception event, and notify the checkpoint user agent to save the checkpoint file to reduce the risk of checkpoint loss.

[0153] Take the development of eBPF program based on BCC architecture and the implementation of checkpoint kernel agent as an example to illustrate. Figure 7 As shown, the bpf() interface can be called to load the compiled eBPF bytecode program (i.e., the checkpoint kernel agent program file) into the kernel space, perform security verification on the bytecode program in the kernel space, use JIT (Just In Time) technology to compile the eBPF bytecode program into local eBPF machine code (i.e., the checkpoint kernel agent that can run in the kernel space), and mount the eBPF machine code to the specified kernel hook, thereby implementing tracking of process exit system call events or other abnormal events (such as NCCL AllReduce operation network timeout, network error, etc.), so as to capture the pending abnormal events in real time when the training process fails, and intercept the target abnormal events.

[0154] Among them, in the tracking and processing flow of the training process that executes the target model training task by the checkpoint kernel agent, based on the training process identifier and the checkpoint memory address recorded in the first mapping table, the read and write system calls to the checkpoint memory address are tracked and checked. After determining the update status of the checkpoint file, it can be updated to the first mapping table record. In this way, the checkpoint kernel agent can monitor the process exit system call or other training process abnormal events in real time. Once the training process exits abnormally or there is a software or hardware failure, the checkpoint kernel agent captures the relevant pending abnormal events in real time and Figure 8 The process shown in the figure realizes the timely and reliable preservation of checkpoint files.

[0155] like Figure 8As shown, each unprocessed abnormal event generated by the training process is detected, and it is determined whether there is a corresponding training process identifier in the first mapping table. If not, the process cleanup operation can be directly performed on the corresponding training process; if so, the update status of the checkpoint file corresponding to the training process identifier in the first mapping table can be continued to determine whether it is an update completion status. If it is not an update completion status, the target model status data of this round of training has not been updated, and the checkpoint file content is incomplete or incorrect, then it is directly skipped and subsequent process cleanup operations are performed; if it is an update completion status, it means that the target model status data of this round of training has been fully updated, and the checkpoint file content is complete and correct, which can trigger a process pre-exit event, and write the corresponding training process identifier, as well as event information such as system call or operation exceptions into the second mapping table.

[0156] At this time, the checkpoint user agent that monitors the second mapping table will promptly monitor the process pre-exit event, obtain the training process identifier and event information of the training process that generated the target abnormal event, and implement the processing operation when the training process exits abnormally. The implementation process can refer to the description of the corresponding part of the above embodiment, and the embodiment of the present application will not be repeated here. It should be noted that after the checkpoint kernel agent triggers the process pre-exit event, it starts to monitor the changes in the checkpoint save state value in the first mapping table. If it is determined that the checkpoint save state value is the target value, such as 1, it means that the checkpoint user agent can write the checkpoint file to the storage device at this time. After the process ends, the system call for the training process exit can be executed, memory resources can be released, and the kernel data structure related to the training process can be destroyed. The training process exit operation is completed, that is, the process cleanup operation for the training process is executed.

[0157] Among them, the preservation of the checkpoint file can be realized in the form of snapshot. Based on the checkpoint memory address, the checkpoint snapshot of the checkpoint file is obtained and sent to the storage device, but it is not limited to this storage form. It is also possible to read each target model state data and write it into a file of a specified format, and then send the file containing the complete target model state data to the storage device for persistent storage, etc., which can be flexibly configured according to actual conditions.

[0158] A method for saving a model checkpoint file provided in an embodiment of the present application is introduced above. A device for executing the above-mentioned method for saving a model checkpoint file will be introduced below.

[0159] Reference Fig. 9 , is a schematic diagram of the structure of a model checkpoint file storage device provided in an embodiment of the present application. Fig. 9 As shown, the model checkpoint file storage device may include:

[0160] A training process information acquisition module 91 is used to acquire training process information associated with a target model training task; wherein the training process information indicates at least one training process used to execute the target model training task, and a checkpoint memory address allocated to a checkpoint file, wherein the checkpoint file contains target model state data updated during the execution of the target model training task;

[0161] An interception module 92, configured to intercept a target abnormal event generated by the training process based on the training process information;

[0162] The saving processing module 93 is used to save the updated checkpoint file in the memory of the electronic device to a storage device based on the checkpoint memory address during the period when the target abnormal event is intercepted.

[0163] In some embodiments, the interception module 92 may include:

[0164] A first determining unit, configured to determine, based on the training process information, an update status of a corresponding checkpoint file in a memory of the electronic device and whether the training process generates an abnormal event to be processed;

[0165] The interception unit is used to determine the unprocessed abnormal event generated by the training process as a target abnormal event for interception if the update status is an update completion status.

[0166] In some embodiments, the above apparatus may further include:

[0167] An agent creation module, used to create a checkpoint user agent and a checkpoint kernel agent; the checkpoint user agent runs in the user space of the electronic device, and the checkpoint kernel agent runs in the kernel space of the electronic device;

[0168] Based on this, the above-mentioned training process information acquisition module 91 is specifically used by the checkpoint user agent to obtain the training process information associated with the target model training task, and load the training process information into the kernel space; the interception module 92 is specifically used by the checkpoint kernel agent to intercept the target abnormal event generated by the training process based on the training process information.

[0169] In a possible implementation, the storage processing module 93 may include:

[0170] A process pre-exit event triggering unit, configured for the checkpoint kernel agent to trigger the generation of a process pre-exit event for the training process in response to interception of the target abnormal event;

[0171] a checkpoint memory address determining unit, configured for the checkpoint user agent to monitor the process pre-exit event and determine the checkpoint memory address associated with the process pre-exit event,

[0172] A checkpoint file reading unit, configured for the checkpoint user agent to read the corresponding updated checkpoint file in the memory of the electronic device based on the determined checkpoint memory address;

[0173] The checkpoint file writing unit is used to write the read checkpoint file into the storage device.

[0174] In a possible implementation, the training process information acquisition module 91 may include:

[0175] A second determination unit, configured for the checkpoint user agent to determine a training process to be created for a target model training task, and a processor type for running the training process;

[0176] A training process identifier configuration unit, used to configure a training process identifier corresponding to the training process;

[0177] a memory address translation unit, configured to translate a virtual memory address allocated to the checkpoint file into a physical memory address based on a process communication mechanism corresponding to the processor type;

[0178] wherein the physical memory address is determined as the checkpoint memory address of the checkpoint file;

[0179] The checkpoint memory address and the training process identifier constitute the training process information associated with the target model training task.

[0180] In a possible implementation, the checkpoint memory address determination unit may include:

[0181] A third determining unit is used for the checkpoint user agent to monitor the process pre-exit event and determine a first training process identifier and event information associated with the process pre-exit event; the first training process identifier is the training process identifier of the training process that generates the target abnormal event, and the event information indicates the abnormal event type of the target abnormal event;

[0182] A fourth determining unit, configured to determine, based on the event information, that the target abnormal event is an abnormal process exit event;

[0183] A checkpoint memory address acquisition unit is used to acquire a checkpoint memory address associated with the first training process identifier.

[0184] In the above embodiment, if the training process for executing the target model training task includes a main process and multiple sub-processes, the checkpoint memory address acquisition unit may include any of the following sub-units:

[0185] A first acquisition subunit is used to determine the second training process identifiers of each sub-process associated with the main process if the abnormal training process with the first training process identifier is the main process, and to obtain the checkpoint memory address corresponding to the first training process identifier and the second training process identifier;

[0186] The second acquisition sub-unit is used to determine the second training process identifier of the main process corresponding to the sub-process and the third training process identifiers of other sub-processes associated with the main process if the abnormal training process with the first training process identifier is any of the sub-processes, and obtain the checkpoint memory addresses corresponding to the first training process identifier, the second training process identifier and the third training process identifier.

[0187] Based on the above analysis, the above device may further include at least one of the following modules:

[0188] A first mapping table processing module, used for the checkpoint user agent to create a first mapping table associated with the target model training task, and load the first mapping table into the kernel space; the first mapping table is used to record the training process identifier and the checkpoint memory address;

[0189] A second mapping table processing module is used for the checkpoint user agent to create a second mapping table associated with the target model training task, and load the second mapping table into the kernel space; the second mapping table is used to record the process pre-exit event triggered by the checkpoint kernel agent, and the first training process identifier and event information associated with the process pre-exit event;

[0190] A checkpoint save state update module, used for the checkpoint user agent to successfully write the read checkpoint file into the storage device, and update the checkpoint save state value recorded in the first mapping table in the kernel space;

[0191] A fault repair module, used for the checkpoint user agent to determine that the target abnormal event is an abnormal operation event for the corresponding training process based on the event information associated with the process pre-exit event, and to perform fault repair on the abnormal operation event based on the fault handling strategy recorded in the fault knowledge base;

[0192] A checkpoint configuration file creation module is used for the checkpoint user agent to monitor the target model training task creation event and create a checkpoint configuration file for the corresponding training process, so as to determine the checkpoint memory address associated with the process pre-exit event based on the checkpoint configuration file when the process pre-exit event is monitored; the checkpoint configuration file is used to record the training process identifier, checkpoint memory address and processor type corresponding to the training process;

[0193] A process cleanup module is used for the checkpoint kernel agent to determine, based on the checkpoint save status value recorded in the first mapping table in the kernel space, that the updated checkpoint file in the memory of the electronic device has been saved to the storage device, release the interception of the target abnormal event, and perform a process cleanup operation on the training process that generated the target abnormal event.

[0194] In some embodiments, the above-mentioned proxy creation module may include:

[0195] An agent file acquisition unit, used for acquiring a checkpoint user agent file and a checkpoint kernel agent file, wherein the checkpoint user agent file and the checkpoint kernel agent file are obtained based on an extended Berkeley packet filter architecture;

[0196] A first execution unit, configured to execute the checkpoint user agent program file to run a corresponding checkpoint user agent in a user space of the electronic device;

[0197] A checkpoint kernel agent program file loading processing unit, used for the checkpoint user agent to load the checkpoint kernel agent program file into the kernel space of the electronic device;

[0198] A checkpoint kernel agent processing unit is used for the kernel space to mount the corresponding checkpoint kernel agent to a designated kernel hook and then run it based on the checkpoint kernel agent program file; the designated kernel hook is used to track and detect unprocessed abnormal events generated during the execution of the target model training task.

[0199] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the model checkpoint file saving methods provided in the embodiments of the present application.

[0200] Wherein, when loading and executing computer-readable instructions on a computer, all or part of the process or function described in the embodiment of the present application is generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer-readable instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer-readable instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center to meet the data / instruction transmission requirements during the execution of the model checkpoint file preservation method proposed in the embodiment of the present application.

[0201] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any model checkpoint file saving method provided in an embodiment of the present application.

[0202] The computer-readable storage medium may be any available medium that can be stored by a computer or a data storage device such as a training device or a data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)).

[0203] Reference Fig.10 , is a hardware structure diagram of an electronic device applicable to the model checkpoint file saving method proposed in this application, the electronic device may include but is not limited to mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptops, netbooks, robots or business terminals, etc., and a server may also be configured to implement the model checkpoint file saving method proposed in this application. Fig.10 As shown, the electronic device may include but is not limited to: at least one memory 101 and at least one processor 102, wherein:

[0204] At least one memory 101 and at least one processor 102 can communicate via a bus. The bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 Only one bidirectional line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0205] The memory 101 can be used to store a computer program that implements the method for saving the intent recognition model checkpoint file proposed in the embodiment of the present application; the processor 102 can load and execute the computer program stored in the memory 101 to implement the various steps of the method for saving the model checkpoint file proposed in the embodiment of the present application. The implementation process can refer to the description of the corresponding part of the method embodiment above.

[0206] In the embodiment of the present application, the memory 101 may include the storage media listed above. The processor 102 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or a field-programmable gate array (FPGA).

[0207] In combination with the description of the above embodiment, the electronic device may also include a storage device such as a disk, which is used to implement persistent storage of the checkpoint file. Optionally, the above storage device may also be an external device. In this case, the electronic device may also be used to implement a communication element for communication connection between the electronic device and the storage device. The communication element may include a communication element corresponding to wireless communication methods such as wifi, Bluetooth, 5G / 6G and other mobile communications. Of course, in order to realize data transmission between components inside the electronic device, the communication element may also include one or more interfaces that support wired communication methods, such as a general-purpose input / output (GPIO) interface, a USB interface, a universal asynchronous receiver / transmitter (UART) interface, etc. This application does not limit the composition structure of the communication element to realize this function and its corresponding communication transmission mechanism, which may be determined according to the circumstances.

[0208] It should be understood that Fig.10 The structure of the electronic device shown does not constitute a limitation on the electronic device in the embodiment of the present application. In practical applications, the electronic device may include Fig.10 More or fewer components as shown, or a combination of certain components, such as a microphone (which can collect user input of a target model training task creation request, etc.), a speaker, a display, various sensors, antennas, a power module, a radio frequency component, an external port, and other input / output components, etc., can be determined based on the processing function requirements, and this application does not give detailed examples one by one.

[0209] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0210] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0211] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. In this specification, each embodiment is described in a progressive or parallel manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referred to each other. For the device and electronic device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

Claims

1. A method for saving a model checkpoint file, the method comprising: Acquire training process information associated with a target model training task; wherein the training process information indicates at least one training process used to execute the target model training task, and a checkpoint memory address allocated to a checkpoint file, wherein the checkpoint file contains target model state data updated during the execution of the target model training task; Based on the training process information, intercepting a target abnormal event generated by the training process; While the target abnormal event is intercepted, the updated checkpoint file in the memory of the electronic device is saved to a storage device based on the checkpoint memory address.

2. The method according to claim 1, wherein intercepting a target abnormal event generated by the training process based on the training process information comprises: Based on the training process information, determining an update status of a corresponding checkpoint file in a memory of the electronic device and an abnormal event to be processed generated by the training process; If the update status is the update completion status, the unprocessed abnormal event generated by the training process is determined as a target abnormal event for interception.

3. The method according to claim 1 or 2, further comprising: Create a Checkpoint User Agent and a Checkpoint Kernel Agent; The checkpoint user agent runs in the user space of the electronic device, and the checkpoint kernel agent runs in the kernel space of the electronic device; The acquiring of training process information associated with the target model training task, and intercepting a target abnormal event generated by the training process based on the training process information, includes: The checkpoint user agent obtains training progress information associated with the target model training task; The checkpoint user agent loads the training process information into the kernel space; The checkpoint kernel agent intercepts a target abnormal event generated by the training process based on the training process information.

4. The method according to claim 3, wherein during the interception of the target abnormal event, based on the checkpoint memory address, the checkpoint file in the memory of the electronic device is saved to a storage device, comprising: The checkpoint kernel agent triggers the generation of a process pre-exit event for the training process in response to interception of the target abnormal event; The checkpoint user agent monitors the process pre-exit event, determines a checkpoint memory address associated with the process pre-exit event, The checkpoint user agent reads the checkpoint file that has been updated in the memory of the electronic device based on the determined checkpoint memory address; The read checkpoint file is written into a storage device.

5. According to the method of claim 3, the checkpoint user agent obtains training progress information associated with the target model training task, including: The checkpoint user agent determines a training process to be created for a target model training task, and a processor type to run the training process; Configuring a training process identifier corresponding to the training process; Based on a process communication mechanism corresponding to the processor type, converting a virtual memory address allocated for the checkpoint file into a physical memory address; wherein the physical memory address is determined as the checkpoint memory address of the checkpoint file; The checkpoint memory address and the training process identifier constitute the training process information associated with the target model training task.

6. The method according to claim 4, wherein the checkpoint user agent monitors the process pre-exit event and determines the checkpoint memory address associated with the process pre-exit event, comprising: The checkpoint user agent monitors the process pre-exit event and determines a first training process identifier and event information associated with the process pre-exit event; The first training process identifier is a training process identifier of a training process that generates the target abnormal event, and the event information indicates an abnormal event type of the target abnormal event; Based on the event information, determining that the target abnormal event is an abnormal process exit event; Obtain a checkpoint memory address associated with the first training process identifier.

7. According to the method of claim 6, if the training process used to execute the target model training task includes a main process and multiple sub-processes, obtaining the checkpoint memory address associated with the first training process identifier includes any of the following: If the abnormal training process having the first training process identifier is the main process, determining the second training process identifiers of each sub-process associated with the main process, and obtaining the checkpoint memory addresses corresponding to the first training process identifier and the second training process identifier; If the abnormal training process with the first training process identifier is any of the sub-processes, determine the second training process identifier of the main process corresponding to the sub-process, and the third training process identifiers of other sub-processes associated with the main process, and obtain the checkpoint memory addresses corresponding to the first training process identifier, the second training process identifier and the third training process identifier.

8. The method according to claim 4, further comprising at least one of the following: The checkpoint user agent creates a first mapping table associated with the target model training task, and loads the first mapping table into the kernel space; the first mapping table is used to record the training process identifier and the checkpoint memory address; The checkpoint user agent creates a second mapping table associated with the target model training task, and loads the second mapping table into the kernel space; the second mapping table is used to record the process pre-exit event triggered by the checkpoint kernel agent, and the first training process identifier and event information associated with the process pre-exit event; The checkpoint user agent successfully writes the read checkpoint file into the storage device, and updates the checkpoint save state value recorded in the first mapping table in the kernel space; The checkpoint user agent determines, based on the event information associated with the process pre-exit event, that the target abnormal event is an abnormal operation event for the corresponding training process, and performs fault repair on the abnormal operation event based on the fault handling strategy recorded in the fault knowledge base; The checkpoint user agent monitors the target model training task creation event, creates a checkpoint configuration file for the corresponding training process, and determines the checkpoint memory address associated with the process pre-exit event based on the checkpoint configuration file when the process pre-exit event is monitored; the checkpoint configuration file is used to record the training process identifier, checkpoint memory address and processor type corresponding to the training process; The checkpoint kernel agent determines, based on the checkpoint save status value recorded in the first mapping table in the kernel space, that the updated checkpoint file in the memory of the electronic device has been saved to the storage device, releases the interception of the target abnormal event, and performs a process cleanup operation on the training process that generates the target abnormal event.

9. The method according to claim 3, wherein the creating a checkpoint user agent and a checkpoint kernel agent comprises: Obtaining a checkpoint user agent program file and a checkpoint kernel agent program file, wherein the checkpoint user agent program file and the checkpoint kernel agent program file are obtained based on an extended Berkeley packet filter architecture; Executing the checkpoint user agent program file to run the corresponding checkpoint user agent in the user space of the electronic device; The checkpoint user agent loads the checkpoint kernel agent program file into the kernel space of the electronic device; The kernel space is based on the checkpoint kernel agent program file, mounts the corresponding checkpoint kernel agent to a designated kernel hook and then runs it; the designated kernel hook is used to track and detect unprocessed abnormal events generated during the execution of the target model training task.

10. An electronic device, comprising at least one memory and at least one processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to implement the following steps: Acquire training process information associated with a target model training task; wherein the training process information indicates at least one training process used to execute the target model training task, and a checkpoint memory address allocated to a checkpoint file, wherein the checkpoint file contains target model state data updated during the execution of the target model training task; Based on the training process information, intercepting a target abnormal event generated by the training process; While the target abnormal event is intercepted, the updated checkpoint file in the memory of the electronic device is saved to a storage device based on the checkpoint memory address.

Images