Checkpoint-based application dump method, checkpoint-based application restart method, device and storage medium

By integrating and storing the status description information of target specific devices for HPC applications on the computing node, the problem that traditional checkpoints and recovery solutions cannot recover HPC applications is solved, and normal recovery of applications and efficient utilization of resources are achieved.

WO2024239866A1PCT designated stage expired Publication Date: 2024-11-28HANGZHOU ALICLOUD FEITIAN INFORMATION TECH CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2024/088676
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-24
Filing Date
2024-04-18
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

In a cloud computing environment, traditional checkpoints and recovery solutions cannot effectively support the elastic use of high-performance computing (HPC) applications, resulting in the problem that HPC applications cannot be recovered.

Method used

Ensure that the application can restore to the correct device state when it is restored by integrating status description information for targeted devices on the compute node and adding it to the checkpoint file to support dumping and recovery of device state.

Benefits of technology

It realizes normal recovery of HPC applications in a stateful specific device situation, supports applications to use computing resources resiliently, improves resource utilization and cluster throughput, and saves costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024088676_28112024_PF_FP_ABST
    Figure CN2024088676_28112024_PF_FP_ABST
Patent Text Reader

Abstract

Provided in the embodiments of the present application are a checkepoint-based application dump method, a checkpoint-based application restart method, a device and a storage medium. The present application innovatively proposes integrating state description information under a target application for a target specific device assembled at a computing node, and adding the state description information into a checkpoint file created for the target application; on this basis, during the process of restarting the target application on the basis of the checkpoint, the state description information can be read from the checkpoint file and, on the basis of the state description information, the device state of the target specific device under the target application is restarted to a specified checkpoint, which can provide a correct device state for the target application, thus ensuring the normal restart of the target application. Therefore, by reforming dump processes and restart processes, the embodiments of the present application can ensure that target applications can still be normally restarted when using stateful specific devices.
Need to check novelty before this filing date? Find Prior Art

Description

Checkpoint-based application dump and recovery method, device, and storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 24, 2023, with application number 202310608335.0 and invention name “A checkpoint-based application dump and recovery method, device and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of cloud computing technology, and in particular to a checkpoint-based application dump and recovery method, device, and storage medium. Background Art

[0003] Public clouds are designed for multi-tenant environments, providing flexible computing power to meet customers' sudden demands. Many HPC (High Performance Computing) applications are typically heavy-duty, placing significant pressure on computing resources. Many applications also require multi-node parallel computing. Therefore, supporting flexible use of computing resources for HPC applications in cloud environments is a key means of reducing customers' total cost of ownership.

[0004] Currently, the structure of computing nodes in cloud environments is becoming increasingly diverse, and the underlying logic supporting application operations on computing nodes is also becoming increasingly diverse. This results in the HPC application often being unable to recover after simply copying the memory data of the HPC application according to the traditional checkpoint and restart (CR) solution, making it impossible to support the elastic use of computing resources by HPC applications.

[0005] Summary of the Invention

[0006] Multiple aspects of the present application provide a checkpoint-based application dumping and recovery method, device, and storage medium to better support application dumping and recovery.

[0007] An embodiment of the present application provides a checkpoint-based application dump method, which is applicable to a computing node, on which a target specific device is installed. The method includes: in response to a checkpoint creation instruction for a target application, obtaining state description information of the target specific device under the target application, wherein the state description information is used to support restoring the device state of the target specific device under the target application to a current checkpoint; adding the state description information to a checkpoint file constructed for the target application; and dumping the checkpoint file to restore the device state of the target specific device based on the state description information when restoring the target application to the current checkpoint.

[0008] An embodiment of the present application also provides a checkpoint-based application recovery method, which is applicable to a computing node, on which a target specific device is installed. The method includes: in response to a recovery instruction to restore a target application to a specified checkpoint, obtaining a checkpoint file corresponding to the target application at the specified checkpoint; reading status description information of the target specific device under the target application from the checkpoint file; and restoring the device status of the target specific device under the target application to the specified checkpoint based on the status description information, so as to restore the target application to the specified checkpoint.

[0009] An embodiment of the present application also provides a computing node, including a memory, a processor and a communication component; the memory is used to store one or more computer instructions; the processor is coupled with the memory and the communication component, and is used to execute the one or more computer instructions to execute the aforementioned checkpoint-based application dump method or the aforementioned checkpoint-based application recovery method.

[0010] An embodiment of the present application also provides a computer-readable storage medium storing computer instructions, which, when executed by one or more processors, causes the one or more processors to execute the aforementioned checkpoint-based application dump method or the aforementioned checkpoint-based application recovery method. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0012] FIG1 is a flow chart of a checkpoint-based application dump method provided by an exemplary embodiment of the present application;

[0013] FIG2 is a logical diagram of a checkpoint-based application dump method provided by an exemplary embodiment of the present application;

[0014] FIG3 is a flow chart of a checkpoint-based application recovery method provided by an exemplary embodiment of the present application;

[0015] FIG4 is a logical diagram of a checkpoint-based application recovery method provided by an exemplary embodiment of the present application;

[0016] FIG5 is a schematic diagram of the structure of a computing node provided by another exemplary embodiment of the present application. DETAILED DESCRIPTION

[0017] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0018] Currently, the problem of HPC applications being unable to recover often occurs, making it impossible to support the flexible use of computing resources by HPC applications. To this end, in some embodiments of the present application: in the process of dumping the target application based on the checkpoint, it is pioneered to integrate the state description information under the target application for the target specific device installed on the computing node, and add the state description information to the checkpoint file constructed for the target application. In this way, in addition to containing the content required for traditional application recovery, the checkpoint file also adds state description information for supporting the device state of the target specific device under the target application to the specified checkpoint. On this basis, in the process of restoring the target application based on the checkpoint, the state description information can be read from the checkpoint file, and the device state of the target specific device under the target application can be restored to the specified checkpoint based on the state description information. This can provide the target application with the correct device state, thereby ensuring the normal recovery of the target application. Therefore, in the embodiments of the present application, by modifying the dump process and the recovery process, it can be ensured that the target application can still be restored normally when using a stateful specific device.

[0019] The following describes in detail the technical solutions provided by various embodiments of the present application in conjunction with the accompanying drawings.

[0020] Figure 1 is a flow chart of a checkpoint-based application dump method provided by an exemplary embodiment of the present application, and Figure 2 is a logic diagram of a checkpoint-based application dump method provided by an exemplary embodiment of the present application. The method can be executed by a data processing device, which can be implemented as software, hardware, or a combination of software and hardware, and can be integrated into a computing node. Referring to Figure 1, the method may include:

[0021] Step 100: In response to a checkpoint creation instruction for a target application, obtain state description information of a target specific device under a target application, where the state description information is used to support restoring the device state of the target specific device under the target application to a current checkpoint. Step 101: Add the state description information to a checkpoint file constructed for the target application. Step 102: Dump the checkpoint file to restore the device state of the target specific device based on the state description information when restoring the target application to the current checkpoint.

[0022] The application dump method provided in this embodiment is applicable to scenarios requiring application dump and recovery. For example, scenarios supporting elastic resource usage in cloud environments, or scenarios involving automatic fault tolerance based on checkpoints, are examples. This embodiment does not limit the application scenarios.

[0023] Checkpoint and Restart (CR) is a backward recovery technology. By setting checkpoints during the normal operation of an application and saving the corresponding checkpoint files, it is possible to roll back the application to the checkpoint and re-execute the application from the checkpoint, without having to execute the application from the source. A checkpoint can be understood as a snapshot of an application at a certain moment. Currently, the principle of checkpoint and recovery technology is to construct the application's memory data into a checkpoint file and store the checkpoint file persistently, so that when needed, the memory data of the application can be restored on the original computing node or a new computing node based on the checkpoint file, thereby restoring the application to the checkpoint. The memory data basically stores information that is originally maintained in memory, such as the status of the process itself and the operating system environment, so we will not give too many examples here.

[0024] However, during the research process, the inventors found that after simply copying the application's memory data according to the traditional checkpoint and restart (CR) solution, the application often cannot be recovered. Therefore, this embodiment provides an improved solution to improve the problem of application non-recovery.

[0025] Referring to Figure 2 , in this embodiment, a computing node may be equipped with specific devices. The number of specific devices on a computing node may be one or more, and the types of specific devices may be one or more, although this embodiment does not limit this. Applications hosted on the computing node may use these specific devices. The computing node in this embodiment may be a cloud server or server cluster, etc., although this embodiment does not limit this.

[0026] In this embodiment, the specific device is generally a stateful device, which may include but is not limited to a stateful IO device or a stateful heterogeneous acceleration device. Among them, a stateful device refers to a type of device that uses the device state as a guide for behavior, that is, the device state affects the behavior of this type of device. If the application needs to call a target specific device, the computing node needs to calculate the device state of the target specific device under the target process and return it to the target application so that the target application can perform operations on the target specific device based on the calculated device state. Therefore, enabling the target application to obtain the correct device state of the target specific device is a necessary condition for ensuring the normal operation of the target application. However, the above-mentioned calculation process for the device state is transparent and imperceptible to the memory data of the target application, that is, the state value in the specific device is not included in the memory data. This results in the inability to copy the memory data according to the traditional checkpoint and recovery scheme and to achieve the dump of the device state. It is understandable that it is precisely because the device state cannot be dumped correctly that the problem of application failure to recover often occurs under the traditional checkpoint and recovery scheme.

[0027] To this end, this embodiment innovatively proposes integrating the state description information of a specific device installed on a computing node under a target application to support dumping and restoring the device state.

[0028] For ease of description, this embodiment uses a target specific device installed on a compute node as an example to illustrate the technical solution. It should be understood that the target specific device can be any specific device installed on the compute node that is used by the target application. From the perspective of the target application, however, there can be multiple specific devices used on the compute node. The technical solution provided in this embodiment can also be used to address other specific devices used by the target application on the compute node besides the target specific device.

[0029] 1 , in step 100 , in response to a checkpoint creation instruction for a target application, state description information of a target specific device under a target application may be obtained.

[0030] In this embodiment, the state description information is used to support restoring the device state of the target specific device within the target application to the current checkpoint. That is, during the subsequent application recovery process, the device state of the target specific device within the target application can be restored to the device state that would have been presented to the target application at the current checkpoint based on the state description information. To this end, the state description information in this embodiment may include various information required to calculate the device state for the specific device.

[0031] As mentioned above, the computing node needs to calculate the device state under the target process for the target specific device and return it to the target application so that the target application can perform operations on the target specific device based on the calculated device state. Therefore, the state value recorded in the specific device and the device state ultimately returned to the target application may be inconsistent. Among them, it is a relatively common phenomenon that the state value recorded in the specific device and the device state ultimately returned to the target application may be inconsistent. There are many reasons for this phenomenon. For example, the current device state of a specific device needs to refer to its previous device state (which can be called context information) to be confirmed. In the process of determining the current device state, in addition to reading the state value from the specific device, it is also necessary to combine its previous device state to calculate the current device state and return it to the application. Obviously, the current device state returned to the application is inconsistent with the state value read from the specific device.

[0032] To this end, this embodiment proposes that, in addition to the status value originally recorded in the specific device, the status description information also includes other information required for the device status calculation process to support correct restoration of the device status during the recovery process.

[0033] It should be understood that for different types of specific devices, the information items included in the state description information may not be exactly the same, which is related to the internal operating principles of the specific device, etc. In this embodiment, the information items included in the state description information are not limited. Several exemplary information items may include the identifier of the device register in the specific device, the attributes of the device register, the state value of the device register, the state value of the device memory, the state value of the device driver software, context information used to support state transitions, and / or mapping relationships used to support state mapping. Among them, the attributes of the device register may include, but are not limited to, read-only, write-only, simulation required, or mapping required. The state value of the device register refers to the state data recorded in the device register, which can be used as the basis for calculating the device state of the specific device. It is worth noting that in this embodiment, the information items included in the state description information can be set as needed and are not limited to this. It can be seen that the state description information in this embodiment not only includes the various state values ​​recorded in the target specific device, but also includes auxiliary information such as context information and mapping relationships to support the correct recovery of the device state.

[0034] In this embodiment, various implementation methods can be used to obtain the state description information of the target specific device in the target application. In one implementation method, the target device type corresponding to the target specific device can be determined; a data collection interface adapted for the target device type can be invoked to collect the state description information of the target specific device in the target application. Different device types are adapted to different data collection interfaces, and the data collection interfaces define the information items and collection logic required for the supported device types.

[0035] In this implementation, it is taken into account that the information items required to calculate the device status for different types of specific devices may not be exactly the same, and different types of specific devices may also have different storage locations and storage methods for related information items. For this reason, in this implementation, a variety of data acquisition interfaces can be developed in advance, and different device types can be adapted to different data acquisition interfaces. In addition, the information items and acquisition logic required to be collected under the device types it supports can be pre-defined in the data acquisition interface. Among them, the information items and acquisition logic in the data acquisition interface can be defined with reference to the product manual of the specific device, other processing logic related to the specific device in the operating system, etc., and are not specifically limited here. Based on this, in this implementation, by calling the appropriate data acquisition interface, the status description information of the target specific device under the target application can be obtained comprehensively and accurately.

[0036] Of course, other implementation methods can also be used in this embodiment to obtain the state description information of the target specific device under the target application. The main concept here is to pioneer the collection and integration of various information items used to support the restoration of the device state of the target specific device under the target application to the current checkpoint. The implementation methods used for collection and integration can be flexible and diverse, and this embodiment does not limit this.

[0037] 1 , in step 101 , the state description information obtained in step 100 may be added to a checkpoint file constructed for a target application.

[0038] The checkpoint file can be constructed using conventional checkpoint and recovery techniques. This embodiment does not limit the content contained in conventional checkpoint files, nor does it provide further explanation. This aspect can be implemented using methods known in the relevant art. Thus, step 101 can be understood as adding the state description information pioneered by this embodiment to the conventional checkpoint file.

[0039] It should be understood that steps 100 and 101 illustrate the logic for obtaining state description information and adding it to the checkpoint file using the target specific device as an example. In actual applications, state description information for other specific devices on the compute node under the target application will also be obtained and added to the checkpoint file. Thus, from the compute node's perspective, the checkpoint file generated for the target application will contain both the traditional checkpoint file content and the state description information for each specific device used by the target application prior to the time corresponding to the current checkpoint.

[0040] On this basis, referring to FIG. 1 , in step 102 , the checkpoint file may be dumped so as to restore the device state of the target specific device based on the state description information when restoring the target application to the current checkpoint.

[0041] In actual applications, the checkpoint file generated for the target application at the current checkpoint in step 101 can be dumped into a persistent storage resource. The type of persistent storage resource is not limited here, and can be a cloud disk or a cloud database.

[0042] At this point, the response to this checkpoint creation instruction is complete. In this embodiment, subsequent checkpoint creation instructions can be issued on the compute node for the target application to create more checkpoints for the target application, and the corresponding checkpoint files can also be persistently stored. In this way, when the target application process is subsequently restored, the desired checkpoint can be specified as needed, and the checkpoint file corresponding to the specified checkpoint will be used as the basis for recovery.

[0043] It can be understood that since the checkpoint file in this embodiment contains the aforementioned status description information, when performing application recovery based on the checkpoint file, in addition to obtaining traditional recovery results, it is also possible to restore the device status of a specific device under the target application to ensure the normal operation of the target application after recovery.

[0044] In this embodiment, for heavy-load applications such as HPC applications that require multiple processes to run in parallel and that may be distributed on different computing nodes, a timeout period can be set during the process of dumping the target application process to avoid the problem of premature exit of computing nodes in multiple computing nodes that carry the target application. In addition, if the relevant data on other computing nodes of a specific target device under the target application is needed in the process of calculating the device status of the target application, these data will also be carried in the checkpoint file as information items in the status description information. This further ensures that in the process of recovering such applications, the consistency of the device status on multiple computing nodes can be achieved to avoid errors. Based on this, in actual applications, it is possible to support the transfer and storage of stateful HPC loads (that is, HPC applications involving device status), thereby achieving barrier-free HPC application migration, and thus better supporting cluster expansion and contraction or allowing multi-node parallel jobs with higher priority submitted later to be executed in advance, thereby achieving the purpose of improving resource utilization.

[0045] In summary, in this embodiment, in the process of dumping the target application based on the checkpoint, it is pioneered to integrate the state description information under the target application for the target specific device installed on the computing node, and add the state description information to the checkpoint file constructed for the target application. In this way, in addition to containing the content required for traditional application recovery, the checkpoint file also adds state description information for supporting the restoration of the device state of the target specific device under the target application to the specified checkpoint. On this basis, in the process of restoring the target application based on the checkpoint, the state description information can be read from the checkpoint file, and the device state of the target specific device under the target application can be restored to the specified checkpoint based on the state description information. This can provide the target application with the correct device state, thereby ensuring the normal recovery of the target application. Therefore, in the embodiment of the present application, by modifying the dump process and the recovery process, it can be ensured that the target application can still be restored normally when using a stateful specific device.

[0046] In the above or below embodiments, various implementations may be used to support the operation of obtaining the state description information of the target specific device under the target application in step 100 .

[0047] In one implementation, before the time corresponding to the current checkpoint, if a target process in a target application initiates a state access operation for a target-specific device, then the state description data corresponding to the state access operation is obtained. The state description data corresponding to the state access operation is used to calculate the device state to be returned to the target process. The state description data of the target-specific device under the target process is updated with the state description data corresponding to the state access operation. The target process can be any process in the target application running on a computing node that has used the target-specific device before the time corresponding to the current checkpoint.

[0048] In this implementation, the target device's device state is constantly changing due to the ongoing operation of the target application. Prior to operating on the target device, the target application typically needs to initiate a state access operation on the target device to obtain the device's real-time state. Therefore, the state description information corresponding to each state access operation is obtained. It should be understood that, consistent with the definition of state description information, the state description data can be used to support restoring the target device's device state under the target process to the current checkpoint.

[0049] In this way, based on this implementation, the state description information may include state description data of the target specific device under at least one process in the target application.

[0050] In actual applications, the target process in the target application may initiate multiple state access operations for the target specific device before the time corresponding to the current checkpoint. In this embodiment, it is not necessary to retain the state description data corresponding to all state access operations. Instead, it is sufficient to retain the state description data corresponding to the last state access operation initiated by the target process for the target specific device before the time corresponding to the current checkpoint. The retained state description data is sufficient to support restoring the device state of the target specific device under the target process to the current checkpoint.

[0051] To this end, this implementation proposes updating the state description data for the target specific device under the target process with the state description data corresponding to the state access operation. That is, before the current checkpoint, the aforementioned update operation is performed each time a state access operation on the target specific device occurs by the target process. This ensures that the state description information for the target specific device under the target application only retains the most recent state description data for the target specific device under at least one process in the target application. This effectively reduces the data cost of application dumps.

[0052] Following the data acquisition interface mentioned in the above embodiment, in this implementation, when a state access operation occurs, the corresponding state description data can be obtained for the state access operation by calling an appropriate data acquisition interface.

[0053] In this way, before the moment corresponding to the current checkpoint, the status description information of the target specific device under the target application can be continuously integrated to support obtaining the status description information of the target specific device under the target application in response to the checkpoint creation instruction for the target application in step 100.

[0054] In one embodiment, a designated memory area can be allocated for the target application, and the state description information of the target specific device under the target application can be stored in the designated memory area. Of course, similarly, the state description information of other specific devices under the target application can also be stored in the designated memory area. Based on this, in the above implementation scheme: the state description data of the target specific device under the target process can be updated to the state description data corresponding to the state access operation in the designated memory area. That is, every time a state access operation for the target specific device occurs in the target application, an update operation for the designated memory area can be triggered, and the update operation is used to update the corresponding state description data stored in the designated memory area. In this way, the designated memory area can be continuously updated during the operation of the target application to maintain the state description information of each specific device under the target application in the designated memory area.

[0055] Based on the designated memory area, the data in the designated memory area can be added to the checkpoint file constructed for the target application in step 101. In other words, in addition to dumping the target application's traditional memory data according to conventional methods, it is proposed to also dump the memory data in the designated memory area additionally allocated for the target application.

[0056] Of course, in addition to the above exemplary implementation scheme, other implementation schemes can also be used in this embodiment to integrate the state description information of the target specific device under the target application. For example, after receiving a checkpoint creation instruction, the last state access operation initiated by each process in the target application on the target specific device is traced back, and the corresponding state description data is obtained for each traced state access operation. This embodiment does not limit the method for maintaining state description information.

[0057] In summary, in this embodiment, various implementation schemes can be used to maintain the status description information of the target specific device under the target application in the computing node, so as to support the comprehensive and accurate acquisition of the status description information of the target specific device under the target application when a checkpoint creation instruction occurs in step 100.

[0058] In the above or below embodiments, a pre-installed daemon process can also be started on the computing node before the target application is run. This daemon process can then execute the aforementioned steps 100-101. For example, in step 100, the daemon process can read the state description information of the target specific device under the target application from the designated memory area allocated for the target application. In step 101, the daemon process can add the read state description information to a checkpoint file constructed for the target application. Finally, in step 102, the daemon process can dump the checkpoint file.

[0059] Furthermore, while the target application is running on the compute node, the daemon process can also maintain relevant state description information for the target application. Specifically, the daemon process can call appropriate data collection interfaces to obtain state description data corresponding to relevant state access operations and update the data into a designated memory area.

[0060] There are two possible design concepts here:

[0061] In one design concept, a compute node can return the device status of a specific target device to each target application using a traditional device status response method. In traditional response methods, the compute node's operating system or the device controller of a specific device might perform the device status calculation. However, the daemon process in this embodiment only retrieves the status description data and does not participate in the device status response.

[0062] In another design concept, in the computing node, each state access operation initiated by each process in the target application for a specific target device can be intercepted, and the daemon process in this embodiment can be used to perform the operation on its behalf. Under this design concept, a preset daemon process can be used to calculate the device state to be returned to the target process based on the acquired state description data; the daemon process can be used to return the calculated device state to the target process as a response to the state access operation. Among them, the calculation logic for the device state can be pre-defined in the daemon process. In actual application, the calculation logic in the traditional device state response method can be defined with reference to it. In addition, modern operating systems can provide user space tracking of kernel space. For example, for the Linux operating system, the PRELOAD and ptrace mechanisms can be used to intercept user process access to devices on various paths. Therefore, various feasible solutions can be used here to implement the interception of state access operations.

[0063] The following explains the two aforementioned design concepts: In the latter design concept, device state calculation is essentially independent of the compute node's operating system and other dependent environments. Instead, the daemon process independently calculates device state based on the acquired state description information. This independent device state calculation method offers significant advantages during application recovery. This is because during application recovery, the compute node may have been replaced, and the original dependent environment may not be fully restored. However, the daemon process can independently calculate the relevant device state based on the state description information contained in the checkpoint file, thus achieving correct device state recovery.

[0064] As can be seen from the above description, both of the above design concepts can be used in the compute node before performing an application dump to achieve the application dump effect required in this embodiment. The latter design concept, on the one hand, can better avoid the omission of state access operations during the application dump process, thereby better ensuring the integrity of the state description information maintained for the target application; on the other hand, the processing logic defined within the daemon process can be directly reused on the compute node where the application is restored, achieving a coordinated application dump and restore process.

[0065] In summary, in this embodiment, the application dump logic in this embodiment can be implemented by starting a preset daemon process on the computing node. Moreover, the daemon process can separate the calculation work of the device status from the dependent environment in the computing node, thereby avoiding application recovery errors caused by the inability to fully restore the dependent environment.

[0066] Figure 3 is a flow chart of a checkpoint-based application recovery method provided by an exemplary embodiment of the present application. Figure 4 is a logic diagram of a checkpoint-based application recovery method provided by an exemplary embodiment of the present application. This method can be executed by a data processing device, which can be implemented as software, hardware, or a combination of software and hardware. The data processing device can be integrated into a computing node. Referring to Figure 3, the method may include:

[0067] Step 300: In response to a restore instruction to restore a target application to a specified checkpoint, obtain a checkpoint file corresponding to the target application at the specified checkpoint;

[0068] Step 301: Read the status description information of the target specific device under the target application from the checkpoint file;

[0069] Step 302: Restore the device state of the target specific device under the target application to a specified checkpoint according to the state description information, so as to restore the target application to the specified checkpoint.

[0070] Referring to Figure 2, the computing node in this embodiment may be equipped with multiple specific devices. The number of specific devices on the computing node may be one or more, and the type of specific devices may be one or more, which is not limited in this embodiment. The applications carried on the computing node may use these specific devices. For ease of description, in this embodiment, the target specific device assembled on the computing node will be used as an example to illustrate the technical solution. It should be understood that the target specific device may be any specific device assembled on the computing node and used by the target application. From the perspective of the target application, the specific devices used on the computing node may be multiple. For other specific devices used by the target application on the computing node except the target specific device, the technical solution provided in this embodiment may also be used to restore the device status.

[0071] In this embodiment, the specific device is generally a stateful device, which may include but is not limited to a stateful IO device or a stateful heterogeneous acceleration device, etc. The definition of a stateful device will not be repeated here.

[0072] Among them, in addition to the content required for application recovery designed in the traditional checkpoint and recovery scheme, the checkpoint file in this embodiment also includes the status description information of at least one specific device on the computing node under the target application, and the status description information of the target specific device under the target application can be used to support the recovery of the device status of the target specific device under the target application to a specified checkpoint.

[0073] This embodiment does not limit the checkpoint file generation method. In this embodiment, the checkpoint-based application dump method shown in FIG1 can be used to generate the checkpoint file required in this embodiment. For relevant technical details, please refer to the previous embodiment and will not be repeated here. However, the checkpoint file generation method in this embodiment is not limited to this.

[0074] In addition, the checkpoint file in this embodiment can be stored in a persistent storage resource. In step 300, the checkpoint file corresponding to the target application at the specified checkpoint can be read from the persistent storage resource. In this embodiment, the target application can have multiple checkpoints, and the restore instruction in step 300 can specify the specific checkpoint to be restored to.

[0075] Referring to Figure 3, in step 301, the state description information of the target specific device in the target application can be read from the checkpoint file. In one embodiment, the state description information includes state description data of the target specific device in at least one process in the target application. Consistent with the definition of the state description information, the state description data is used to support restoring the device state of the target specific device in the target process to a specified checkpoint.

[0076] Based on this, in step 302, the device state of the target specific device under the target application can be restored to the specified checkpoint according to the state description information read from the checkpoint file, so as to restore the target application to the specified checkpoint.

[0077] The state description information in this embodiment may include various information required to calculate the device state for a specific device. It should be understood that for different types of specific devices, the information items included in the state description information may not be exactly the same, depending on the internal operating principles of the specific device. In this embodiment, the information items included in the state description information are not limited. Several exemplary information items may include the identifier of a device register in the specific device, the attributes of the device register, the state value of the device register, the state value of the device memory, the state value of the device driver software, context information used to support state transitions, and / or mapping relationships used to support state mapping. The attributes of the device register may include, but are not limited to, read-only, write-only, simulation required, or mapping required. The state value of the device register refers to the state data recorded in the device register, which can be used as the basis for calculating the device state of the specific device. It is worth noting that in this embodiment, the information items included in the state description information can be set as needed and are not limited to this.

[0078] Therefore, during the device status recovery process in step 302, there may be various situations. This embodiment provides the following recovery solutions for several exemplary situations:

[0079] In an exemplary case: if the state description data of the target specific device under the target process in the target application indicates that the attribute of the first device register needs to be simulated, then based on the context information associated with the first device register in the state description data, the state transition process of the first device register is simulated to restore the device state of the first device register under the target process to a specified checkpoint; wherein the first device register is any device register included in the target specific device, and the target process is any process running on the computing node in the target application.

[0080] In another exemplary case: if the attribute of the first device register indicated in the state description data is that mapping is required, then based on the mapping relationship associated with the first device register in the state description data, the state value provided by the first device register is mapped to restore the device state of the first device register under the target process to the specified checkpoint.

[0081] For example, if one of the status values ​​recorded in the first device register is a DMA access address, but due to the change of the computing node, the DMA access address recorded in the first device register is usually the DMA access address assigned by the operating system, and the DMA access address may correspond to a different memory address in the hardware environment of a different computing node. Here, based on the address mapping relationship carried in the status description data, the DMA access address recorded in the first register can be mapped to the correct address, and the correct address can be returned to the target process, which can ensure that the target address subsequently initiates a DMA operation instruction to the target specific device according to the correct address.

[0082] That is, according to the properties of different device registers in the target specific device, appropriate recovery logic can be used for each device register to restore the device state, so as to ensure that the device state of each device register is correctly restored.

[0083] In this embodiment, a preset daemon process may be run on the computing node, and the daemon process may be used to implement the above-mentioned application recovery solution.

[0084] Based on this, in step 302, the recovery logic under various attributes can be pre-defined in the daemon process. In this way, the daemon process can read the information items required for calculation from the status description information carried by the checkpoint file according to the appropriate recovery logic, thereby correctly restoring the device status of each device register.

[0085] The computing node in this embodiment can be the original computing node where the target application is located before dumping, or can be a new computing node. In this embodiment, it is only necessary to ensure that the computing node is equipped with the specific device required by the target application.

[0086] In one implementation, during the recovery process of the target application, the state access operations initiated by each process in the target application to the target specific device may be monitored, and after the state access operations are monitored, steps 301 and 302 are executed.

[0087] In this implementation, monitored state access operations can be intercepted;

[0088] Use the preset daemon process to read the state description data corresponding to the intercepted state access operation from the checkpoint file, and calculate the device state to be returned to the target process, where the target process is the process that initiated the state access operation;

[0089] The daemon process is used to return the calculated device state to the target process in response to the state access operation.

[0090] Thus, in this implementation, when a state access operation occurs, the traditional device state response logic is no longer executed. Instead, the daemon process performs the device state calculation on its behalf. The daemon process can complete the device state calculation independently of the dependency environment on the original compute node. It simply reads the correct state description data from the checkpoint file and runs the appropriate internal predefined recovery logic to calculate the correct device state. This way, even if the compute node where the application is restored no longer has the complete dependency environment of the original compute node, the daemon process can still correctly restore the device state of the target specific device under the target application on the compute node where the application is restored, thereby ensuring the normal recovery of the target application.

[0091] In this implementation, upon monitoring the first or earliest N status access operations initiated by the target process for a specific target device, the daemon process can return the device status to the target process according to the aforementioned proxy operation logic. Subsequent status access operations initiated by the target process for the specific target device can be handled according to the traditional device status response method. Of course, it is also possible to consistently maintain the daemon process returning the device status for each status access operation initiated by the target process according to the aforementioned proxy operation logic, while completely disabling the traditional device status response method. This is not limited here.

[0092] In summary, in this embodiment, it is pioneered to carry the state description information of the target specific device under the target application in the checkpoint file constructed for the target application. In this way, in addition to containing the content required for traditional application recovery, the checkpoint file also adds state description information for supporting the restoration of the device state of the target specific device under the target application to the specified checkpoint. On this basis, in the process of restoring the target application based on the checkpoint, the state description information can be read from the checkpoint file, and the device state of the target specific device under the target application can be restored to the specified checkpoint based on the state description information. This can provide the target application with the correct device state, thereby ensuring the normal recovery of the target application. Therefore, in the embodiment of the present application, by modifying the dump process and the recovery process, it can be ensured that the target application can still be restored normally when using a specific device with a state.

[0093] Furthermore, the application dump and restore method provided in this embodiment can be applied to HPC cluster systems, enabling the dump and restore of stateful multi-process parallel jobs. This can improve HPC cluster system utilization, increase cluster throughput, and save costs. By transferring HPC jobs to storage and then restoring them after scheduling, job migration can be centralized, freeing up more available nodes for subsequent jobs or releasing idle nodes. This also allows cluster jobs with higher priority to be submitted later to be preempted without notice. In the past, preemption of parallel jobs often involved simply killing the executing job in the job system, allowing resources for the higher-priority job to be scheduled and executed. This solution, through checkpointing and restoring parallel jobs, enables multi-machine job preemption. Furthermore, it facilitates cluster resource maintenance. When cluster node operation and maintenance actions need to be performed, system administrators can temporarily transfer jobs on the nodes to storage, then perform resource maintenance operations and wait for node maintenance to complete before restoring them. This, combined with the scheduling algorithms and strategies within the HPC cluster system, can benefit the HPC cluster system in terms of operation and maintenance, troubleshooting, and other aspects.

[0094] In an embodiment of the present application, in the process of dumping the target application based on the checkpoint, it is pioneered to integrate the state description information under the target application for the target specific device installed on the computing node, and add the state description information to the checkpoint file constructed for the target application. In this way, in addition to containing the content required for traditional application recovery, the checkpoint file also adds state description information for supporting the restoration of the device state of the target specific device under the target application to the specified checkpoint. On this basis, in the process of restoring the target application based on the checkpoint, the state description information can be read from the checkpoint file, and the device state of the target specific device under the target application can be restored to the specified checkpoint based on the state description information. This can provide the target application with the correct device state, thereby ensuring the normal recovery of the target application. Therefore, in an embodiment of the present application, by modifying the dump process and the recovery process, it can be ensured that the target application can still be restored normally when using a stateful specific device.

[0095] It should be noted that some of the processes described in the above embodiments and accompanying drawings include multiple operations that appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order in which they appear in this document or may be executed in parallel. The sequence numbers of the operations, such as 801 and 802, are merely used to distinguish between different operations and do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel.

[0096] FIG5 is a schematic diagram of a computing node structure provided by another exemplary embodiment of the present application. As shown in FIG5 , the computing device includes: a memory 50 , a processor 51 , and a target specific device 52 .

[0097] The processor 51 is coupled to the memory 50 and is used to execute the computer program in the memory 50, so as to: obtain state description information of the target specific device 52 under the target application in response to a checkpoint creation instruction for the target application, the state description information being used to support restoring the device state of the target specific device 52 under the target application to the current checkpoint; add the state description information to a checkpoint file constructed for the target application; and dump the checkpoint file to restore the device state of the target specific device 52 based on the state description information when restoring the target application to the current checkpoint.

[0098] In one embodiment, the state description information includes state description data of the target specific device 52 in at least one process of the target application. Before responding to the checkpoint creation instruction for the target application, the processor 51 may further be configured to:

[0099] Before the moment corresponding to the current checkpoint, if it is monitored that the target process in the target application initiates a state access operation for the target specific device 52, the state description data corresponding to the state access operation is obtained, and the state description data corresponding to the state access operation is used to calculate the device state to be returned to the target process; the state description data of the target specific device 52 under the target process is updated to the state description data corresponding to the state access operation; wherein, the target process is any process running on the computing node in the target application.

[0100] In one embodiment, when the processor 51 obtains the status description data corresponding to the status access operation, it can be specifically used to: determine the target device type corresponding to the target specific device 52; call the data acquisition interface adapted to the target device type to collect the status description information corresponding to the status access operation; wherein, different device types are adapted to different data acquisition interfaces, and the data acquisition interface defines the information items and acquisition logic that need to be collected under the device types it supports.

[0101] In one embodiment, the processor 51 can also be used to: intercept status access operations; use a preset daemon process to calculate the device status to be returned to the target process based on the acquired status description data; and use the daemon process to return the calculated device status to the target process as a response to the status access operation.

[0102] In one embodiment, the state description information includes one or more of an identifier of a device register in the target specific device 52, attributes of the device register, a state value of the device register, a state value of the device memory, a state value of the device driver software, context information for supporting state transitions, and a mapping relationship for supporting state mapping.

[0103] In one embodiment, the status description information is stored in a designated memory area allocated for the target application. When the processor 51 updates the status description data of the target specific device 52 under the target process to the status description data corresponding to the status access operation, it can be specifically used to: update the status description data of the target specific device 52 under the target process in the designated memory area to the status description data corresponding to the status access operation.

[0104] In one embodiment, the processor 51 can also be used to: in response to a recovery instruction to restore the target application to the current checkpoint, obtain the checkpoint file corresponding to the target application at the current checkpoint; read the status description information from the checkpoint file; based on the status description information, restore the device status of the target specific device 52 under the target application to the current checkpoint to restore the target application to the current checkpoint.

[0105] In one embodiment, the target specific device 52 is a stateful device such as a stateful network device or a stateful heterogeneous acceleration device.

[0106] In other possible designs, a checkpoint-based application recovery solution can also be provided based on the computing node shown in FIG5 . In these possible designs, the processor 51 can be configured to execute a computer program in the memory 50 to: in response to a recovery instruction to restore the target application to a specified checkpoint, obtain a checkpoint file corresponding to the target application at the specified checkpoint; read state description information of the target specific device 52 under the target application from the checkpoint file; and restore the device state of the target specific device 52 under the target application to the specified checkpoint based on the state description information, thereby restoring the target application to the specified checkpoint.

[0107] In one embodiment, the state description information includes state description data of the target specific device 52 under at least one process in the target application. When the processor 51 restores the device state of the target specific device 52 under the target application to a specified checkpoint based on the state description information, it can be specifically used to: if the state description data of the target specific device 52 under the target process in the target application indicates that the attribute of the first device register needs to be simulated, then based on the context information associated with the first device register in the state description data, simulate the state conversion process of the first device register to restore the device state of the first device register under the target process to the specified checkpoint; wherein, the first device register is any device register included in the target specific device 52, and the target process is any process running on the computing node in the target application.

[0108] In one embodiment, the processor 51 may further be configured to: if the attribute of the first device register indicated in the state description data is that mapping is required, then based on the mapping relationship associated with the first device register in the state description data, map the state value provided by the first device register to restore the device state of the first device register under the target process to a specified checkpoint.

[0109] Furthermore, as shown in Figure 5 , the computing node also includes other components, such as a communication component 53 and a power supply component 54. Figure 5 only schematically illustrates some of the components, and does not imply that the computing node only includes the components shown in Figure 5 . Furthermore, the computing node shown in Figure 5 can be a cloud server or a logical node composed of multiple cloud servers, without limitation here.

[0110] It is worth noting that the technical details of the above-mentioned computing node embodiments can be referred to the relevant descriptions in the aforementioned method embodiments. In order to save space, they will not be repeated here, but this should not cause any loss of the protection scope of this application.

[0111] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement the steps that can be executed by a computing device in the above method embodiment.

[0112] The memory in Figure 5 is used to store computer programs and can be configured to store various other data to support operations on the computing platform. Examples of such data include instructions for any application or method operating on the computing platform, contact data, phone book data, messages, pictures, videos, etc. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0113] The communication component in Figure 5 above is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0114] The power supply assembly in Figure 5 provides power to various components of the device in which the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply assembly is located.

[0115] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0116] The present application is described with reference to the flow chart and / or block diagram of the method, device (system), and computer program product according to the embodiment of the present application. It should be understood that each flow process and / or box in the flow chart and / or block diagram and the combination of the flow process and / or box in the flow chart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processing machine or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one flow chart flow or multiple flows and / or one box or multiple boxes of the block diagram.

[0117] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0118] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0119] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0120] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0121] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included in the protection scope of the present application.

Claims

1. A checkpoint-based application dump method, applicable to a computing node, wherein the computing node is equipped with a target specific device, the method comprising: In response to a checkpoint creation instruction for a target application, obtaining state description information of the target specific device under the target application, wherein the state description information is used to support restoring a device state of the target specific device under the target application to a current checkpoint; adding the state description information to a checkpoint file constructed for the target application; The checkpoint file is dumped to restore the device state of the target specific device based on the state description information when the target application is restored to the current checkpoint.

2. The method according to claim 1, wherein the state description information includes state description data of the target specific device under at least one process in the target application, and before responding to the checkpoint creation instruction for the target application, the method further comprises: Before the time corresponding to the current checkpoint, if it is monitored that the target process in the target application initiates a state access operation for the target specific device, then the state description data corresponding to the state access operation is obtained, and the state description data corresponding to the state access operation is used to calculate the device state to be returned to the target process; Updating the state description data of the target specific device under the target process to the state description data corresponding to the state access operation; The target process is any process in the target application running on the computing node.

3. According to the method of claim 2, the step of obtaining the state description data corresponding to the state access operation comprises: Determine a target device type corresponding to the target specific device; Calling a data collection interface adapted to the target device type to collect status description information corresponding to the status access operation; Different device types are adapted to different data collection interfaces, and the data collection interface defines the information items and collection logic that need to be collected under the device types it supports.

4. The method according to claim 2, further comprising: intercepting the state access operation; Utilizing a preset daemon process to calculate the device status to be returned to the target process based on the acquired status description data; The calculated device state is returned to the target process by the daemon process as a response to the state access operation.

5. According to the method described in any one of claims 1 to 4, the state description information includes one or more of an identifier of a device register in the target specific device, an attribute of the device register, a state value of the device register, a state value of a device memory, a state value of a device driver software, context information for supporting state transitions, and a mapping relationship for supporting state mapping.

6. The method according to claim 2, wherein the state description information is stored in a designated memory area allocated for the target application, and the state description data of the target specific device under the target process is updated to the state description data corresponding to the state access operation, comprising: In the designated memory area, the state description data of the target specific device under the target process is updated to the state description data corresponding to the state access operation.

7. The method according to claim 1, further comprising: In response to a restore instruction to restore the target application to the current checkpoint, obtaining a checkpoint file corresponding to the target application at the current checkpoint; Read the status description information from the checkpoint file; Based on the state description information, the device state of the target specific device under the target application is restored to the current checkpoint, so as to restore the target application to the current checkpoint.

8. According to the method of claim 1, the target specific device is a stateful network device or a stateful heterogeneous acceleration device.

9. A checkpoint-based application recovery method, applicable to a computing node, wherein the computing node is equipped with a target specific device, the method comprising: In response to a restore instruction to restore a target application to a specified checkpoint, obtaining a checkpoint file corresponding to the target application at the specified checkpoint; Reading the state description information of the target specific device under the target application from the checkpoint file; According to the state description information, the device state of the target specific device under the target application is restored to the specified checkpoint, so as to restore the target application to the specified checkpoint.

10. The method according to claim 9, wherein the state description information includes state description data of the target specific device under at least one process in the target application, and according to the state description information, restoring the device state of the target specific device under the target application to the specified checkpoint comprises: If the target specific device indicates in the state description data under the target process in the target application that the attribute of the first device register needs to be simulated, then based on the context information associated with the first device register in the state description data, simulating the state conversion process of the first device register, so as to restore the device state of the first device register under the target process to the specified checkpoint; The first device register is any device register included in the target specific device, and the target process is any process in the target application running on the computing node.

11. The method according to claim 10, further comprising: If the attribute of the first device register indicated in the state description data is that mapping is required, the state value provided by the first device register is mapped based on the mapping relationship associated with the first device register in the state description data to restore the device state of the first device register under the target process to the specified checkpoint.

12. A computing node comprising a memory, a processor, and a communication component; The memory is used to store one or more computer instructions; The processor is coupled to the memory and the communication component, and is used to execute the one or more computer instructions to execute the checkpoint-based application dump method described in any one of claims 1-8 or the checkpoint-based application recovery method described in any one of claims 9-11.

13. A computer-readable storage medium storing computer instructions, which, when executed by one or more processors, causes the one or more processors to execute the checkpoint-based application dump method described in any one of claims 1 to 8 or the checkpoint-based application recovery method described in any one of claims 9 to 11.

Citation Information

Patent Citations

  • Cluster fault-tolerance system, apparatus and method

    CN101369241A

  • Method for restoring application software exception in Android system

    CN106959909A

  • File system data access method and file system

    CN110554834A

  • Event interaction type process control method, device and system and related equipment

    CN114296889A

  • Application dump and recovery method and device based on check points and storage medium

    CN116610493A