Process state storage and recovery method, system and equipment and storage medium

Abstract

The embodiment of the invention discloses a process state storage and recovery method and system, electronic equipment and a storage medium, relates to the technical field of computers, and facilitates complete storage and accurate recovery of process states including remote direct memory access resources. The method comprises the following steps: in a process storage stage, closing remote direct memory access resources of a target process so as to convert the target process into a degradation process, and storing running state information of the degradation process; based on the stored operation state information of the degradation process, recovering the degradation process; opening a remote direct memory access resource, so that the recovered degradation process is converted into a target process containing the remote direct memory access resource; and unfreezing the target process so that the target process continues to be executed from the storage point.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and in particular, to a process state saving and recovery method and system, an electronic device, and a storage medium. BACKGROUND

[0002] With the development of high-performance computing and distributed cluster technology, the running period of computing tasks is continuously extended, and fault recovery and task migration become key requirements for ensuring system stability and resource utilization. Task saving and recovery technology provides core technical support for the above-mentioned fault recovery and task migration scenarios by completely saving process running state information and restoring execution when needed.

[0003] In related technologies of task saving and recovery, a passive process state saving scheme has attracted attention because it does not require application intervention. The CRIU (checkpoint-restore in userspace) tool is a typical representative, which can freeze a process and save its running state information as an image file, and then restore and execute it on the same or similar hardware platform. However, for a process that depends on remote direct memory access (RDMA) resources, since RDMA data transmission bypasses the kernel and is directly performed in the user space, the kernel cannot perceive the resource state, resulting in that the CRIU tool cannot completely save and accurately restore RDMA-related resources and running state information. SUMMARY

[0004] Therefore, the embodiments of the present application provide a process state saving and recovery method, system, and electronic device, which facilitate complete saving and accurate recovery of process state information including remote direct memory access resources.

[0005] In a first aspect, the embodiments of the present application provide a process state saving and recovery method, which includes: freezing a target process, closing remote direct memory access resources of the target process, so that the target process is converted into a degraded process; saving running state information of the degraded process; restoring the degraded process based on the saved running state information of the degraded process, wherein the degraded process does not include remote direct memory access resources; opening the remote direct memory access resources, so that the restored degraded process is converted into a target process including remote direct memory access resources; unfreezing the target process, so that the target process continues to execute from a save point.

[0006] According to a specific implementation manner of the embodiments of the present application, the closing of the remote direct memory access resources of the target process includes: starting an operation for closing a remote direct memory access resource through a remote direct memory access virtual layer; the remote direct memory access virtual layer comprises a resource pool for storing resource information records, the resource information records comprising all remote direct memory access resources of the target process and a dependency relationship between the remote direct memory access resources; According to the all remote direct memory access resources of the target process and the dependency relationship, remote direct memory access resource closing interfaces are sequentially called to release the remote direct memory access resources.

[0007] According to a specific implementation manner of the embodiment of the application, the remote direct memory access virtual layer is a dynamic link library; the method further comprises: an environment variable is pre-configured to make an application preferentially link the dynamic link library, so as to intercept interface calling of the application to a remote direct memory access core library before the remote direct memory access resource closing interfaces are sequentially called.

[0008] According to a specific implementation manner of the embodiment of the application, before the target process is frozen, the method further comprises: based on the remote direct memory access virtual layer, a creation operation, a modification operation and a deletion operation of the target process to a remote direct memory access resource are intercepted; if the operation is the creation operation, the created remote direct memory access resource, a corresponding virtual resource and a creation operation configuration parameter are recorded, and an initial mapping relationship of the virtual resource to the remote direct memory access resource is established, to form a resource information record; the creation operation configuration parameter is a configuration attribute required by a remote direct memory access resource core library to create a physical resource, and the remote direct memory access resource is a physical resource; if the operation is the modification operation, only the last information of the modification operation is recorded and is updated to the resource information record; if the operation is the deletion operation, corresponding physical resources, virtual resources, initial mapping relationships and creation operation configuration parameters are deleted from the resource information record; a dependency relationship between the remote direct memory access resources is recorded, the dependency relationship is integrated into the resource information record, and the finally formed resource information record is stored into the resource pool.

[0009] According to a specific implementation manner of the embodiment of the application, before the target process is frozen, the method further comprises: when an application calls a remote direct memory access resource interface to initiate a resource access request, the virtual resource transmitted by the application in the access request is received; the access request is intercepted, and based on an initial mapping relationship recorded in the resource pool, a physical resource corresponding to the virtual resource is searched; the searched physical resource is taken as a return parameter and is transmitted to a same-name interface of a remote direct memory access core library, so that the remote direct memory access core library executes an actual remote direct memory access operation through the physical resource.

[0010] According to a specific implementation manner of the embodiment of the present application, the resuming the degradation process based on the saved running state information of the degradation process comprises: parsing the running state information file of the degradation process, extracting the memory data and the register state carried in the running state information file; based on the memory data, reconstructing the address space of the degradation process, and based on the register state, resuming the execution context of the degradation process for resuming the degradation process.

[0011] According to a specific implementation manner of the embodiment of the present application, the opening the remote direct memory access resource comprises: based on the remote direct memory access virtual layer, starting an operation for reopening the remote direct memory access resource, reading the creation operation configuration parameters of each remote direct memory access resource and the dependency relationship between each remote direct memory access resource from the resource pool; sequentially calling the remote direct memory access resource creation interface according to the dependency relationship, and based on the corresponding creation operation configuration parameters, reconstructing each remote direct memory access resource; according to the last information of the modification operation of each remote direct memory access resource in the resource information record, adjusting the attribute of each reconstructed remote direct memory access resource to the state before being closed.

[0012] According to a specific implementation manner of the embodiment of the present application, the closing operation for closing the remote direct memory access resource and the opening operation for reopening the remote direct memory access resource are triggered and executed in the following manners: injecting the code for executing the operation into the address space of the target process, and triggering the execution at a preset time; or starting an independent thread in the remote direct memory access virtual layer, and executing the closing or opening operation by receiving external instructions.

[0013] According to a specific implementation manner of the embodiment of the present application, after the remote direct memory access resource is opened, before the target process is unfrozen, the method further comprises: based on the remote direct memory access virtual layer, establishing the mapping relationship between the reconstructed remote direct memory access resource and the virtual resource, so as to update the initial mapping relationship in the resource pool; wherein the reconstructed remote direct memory access resource is a physical resource generated by a remote direct memory access core library through a resource creation interface according to the creation operation parameters; and the virtual resource is visible identification information used by an application when accessing a remote direct memory access resource before the target process is converted into a degradation process.

[0014] According to a specific implementation manner of the embodiment of the present application, the unfreezing the target process to enable the target process to continue execution from the save point comprises: releasing the frozen state of the target process; and when receiving an access request initiated by an application through the virtual resource, directing the access request to a reconstructed remote direct memory access resource corresponding to the virtual resource based on the mapping relationship, to continue the execution logic of the target process before the save point.

[0015] In a second aspect, the embodiment of the present application further provides a process state saving and restoring system, comprising: a hardware layer comprising a processor, a memory and a remote direct memory access network card; and a software layer comprising a target process running on the processor, a remote direct memory access virtual layer and a remote direct memory access core library, wherein the remote direct memory access core library is configured to provide an operation interface of a remote direct memory access resource; the remote direct memory access virtual layer is configured to: during freezing of the target process by the processor, close a remote direct memory access resource used by the target process through the remote direct memory access network card, convert the target process into a degraded process, and save running state information of the degraded process to the memory; based on the running state information saved in the memory, control the processor to restore the degraded process; open the remote direct memory access resource by calling the operation interface provided by the remote direct memory access core library, convert the restored degraded process into a target process comprising a remote direct memory access resource; and control the processor to unfreeze the target process, to enable the target process to continue execution from a save point.

[0016] In a third aspect, the embodiment of the present application further provides an electronic device, comprising: a housing, a processor, a memory, a circuit board and a power supply circuit, wherein the circuit board is arranged inside a space enclosed by the housing, and the processor and the memory are arranged on the circuit board; the power supply circuit is configured to supply power to each circuit or device of the electronic device; the memory is configured to store executable program code; and the processor is configured to run a program corresponding to the executable program code by reading the executable program code stored in the memory, to execute the method of any one of the first aspect.

[0017] In a fourth aspect, the embodiment of the present application further provides a computer readable storage medium, which stores one or more programs, and the one or more programs are executable by one or more processors to implement the method of any one of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or in the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description only represent some embodiments of the present application, and all other drawings obtained by those of ordinary skill in the art without creative effort based on these drawings belong to the protection scope of the present application.

[0019] Figure 1 The schematic flow chart of the process state saving and recovery method provided by an embodiment of the present application is shown in the figure. Figure 2 The schematic block diagram of the software and hardware system structure corresponding to the process state saving and recovery method provided by an embodiment of the present application is shown in the figure. Figure 3 The RDMA resource management schematic diagram provided by an embodiment of the present application is shown in the figure. Figure 4 The schematic diagram of constructing the mapping relationship between the virtual resource and the physical resource in an embodiment of the present application is shown in the figure. Figure 5 The schematic diagram of the isolation and conversion principle between the RDMA virtual resource and the physical resource in an embodiment of the present application is shown in the figure. Figure 6 The schematic flow chart of the process state saving process provided by an embodiment of the present application is shown in the figure. Figure 7 The schematic flow chart of the process state recovery process provided by an embodiment of the present application is shown in the figure. Figure 8 The schematic diagram of creating the RDMA resource information record in the resource pool before the process saving in an embodiment of the present application is shown in the figure. Figure 9 The schematic diagram of closing the RDMA resource in the process saving stage in an embodiment of the present application is shown in the figure. Figure 10 The schematic diagram of rebuilding the RDMA resource in the process recovery stage of the RDMA resource in an embodiment of the present application is shown in the figure. Figure 11 The schematic diagram of the architecture of an embodiment of the process state saving and recovery system provided by the present application is shown in the figure. Figure 12 The schematic diagram of the electronic device provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0020] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0021] It should be clear that the described embodiments are only some embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort belong to the protection scope of the present application.

[0022] For those skilled in the art to better understand the technical concepts, embodiments and beneficial effects of the present application, the following will be described in detail through specific embodiments.

[0023] Process saving technology is mainly applied in two scenarios of fault recovery and task migration. In the fault recovery scenario, the state of one process or multiple processes of a computing task is saved at regular intervals to form a backup. If a hardware failure occurs, the process execution is restored on other nodes using the latest backup data, without starting the task from scratch, which greatly improves the computing efficiency. In the task migration scenario, the state of the specified process is saved, and the saved data is migrated to other nodes to restore the process running, in order to achieve load balancing or reduce energy consumption.

[0024] In high-performance computing and distributed cluster systems, as the running period of computing tasks is prolonged, fault recovery and task migration become key requirements to ensure system stability and resource utilization. Among them, the passive process state saving scheme is suitable for most applications because it does not require the application itself to implement saving and recovery logic, and it can perform task migration and recovery in real time without rolling back the task.

[0025] Currently, the mainstream passive process state saving tool, such as the CRIU (checkpoint-restore in userspace) tool, has the following problems when handling processes that rely on remote direct memory access (RDMA) resources in a cluster system: Although the native CRIU tool can freeze the process and save its running state information, since RDMA communication bypasses the kernel and directly transmits memory data, CRIU saves the process state by freezing the process, reading the process address space, querying the device file state and register values, etc. However, its core principle is not to interfere with or modify the user state of the process. Since the RDMA connection and resource management are completely implemented in the user state, CRIU cannot modify the user state RDMA protocol like interfering with kernel resources, such as adding a pause connection mechanism, resulting in the inability to directly save and restore the RDMA process state. To solve this problem, related solutions require modifying the RDMA protocol and network card firmware, which destroys the transparency of CRIU, and because developers cannot modify the network card firmware, the feasibility is low.

[0026] The present application provides a process state saving and restoring method suitable for cluster systems with RDMA communication. Process saving refers to completely saving the process state and data of a running process in a computing environment, such as a physical machine, virtual machine or container. The saved information can be used to restore the process on another node device and continue running from the saved point.

[0027] In view of the problem that the native CRIU tool cannot handle the saving and restoring of a process containing an RDMA resource, refer to Figure 1 The embodiment of the present application provides a process state saving and restoring method, which comprises the following steps: S110, freezing a target process, and closing remote direct memory access (RDMA) resources of the target process, so that the target process is converted into a degenerate process, S115, saving running state information of the degenerate process.

[0028] S120, restoring the degenerate process based on the saved running state information of the degenerate process.

[0029] S130, opening the remote direct memory access resources, so that the restored degenerate process is converted into a target process containing remote direct memory access resources.

[0030] S140, unfreezing the target process, so that the target process continues to execute from a save point.

[0031] Refer to Figure 2 In the embodiment, the freezing of the target process can be realized by using a CRIU tool, which can pause all computing and communication operations of the target process, so that the target process is in a frozen state.

[0032] Since the RDMA resources, such as a queue pair and a protection domain, that is, objects related to the RDMA device in the address space of the process, adopt a direct memory access mode bypassing the kernel, the state of the RDMA resources cannot be perceived by the kernel and the CRIU, so that the CRIU cannot directly identify and record the information of the resources. Therefore, refer to Figure 2 In the embodiment, an RDMA virtual layer is inserted between an application program and an RDMA core library, and the virtual layer is configured to convert the target process into a degenerate process not containing the RDMA resources by closing the RDMA resources of the target process when receiving an application program call, so as not to affect the normal work of the CRIU tool, so that the CRIU tool can normally save the state of a process only including non-RDMA resources, and the process state of the non-RDMA resources includes memory data such as variable values, heap and stack information, and register states such as a program counter and a stack pointer.

[0033] Specifically, the saving of the running state information of the degenerate process comprises the following steps: collecting memory data and register states of the degenerate process; and recording and associating the memory data, the register states and the resource information to generate a running state information file of the degenerate process.

[0034] Afterwards, the degradation process is resumed based on the saved running state information of the degradation process. Specifically, the running state information file of the degradation process is parsed to extract the memory data and the register state; the address space of the degradation process is reconstructed based on the memory data, and the execution context of the degradation process, including the instruction execution position, the stack pointer direction, etc., is recovered based on the register state, so as to restore the degradation process state to the state before freezing.

[0035] After the process without RDMA resources is resumed by using the CRIU tool, the process is in the user mode and can normally reopen the RDMA resources, and the opened RDMA resources are completely consistent with the state before being closed, so as to restore the degradation process without RDMA resources to the target process including the RDMA resources again, convert the restored degradation process to the target process including the RDMA resources again, and unfreeze the target process, so that the target process can continue to execute from the save point, thereby realizing complete saving and restoring of the target process state including the RDMA resources.

[0036] The application programming interface (API) in the RDMA core library is implemented in the RDMA virtual layer and can be compiled into a dynamic link library. When the application program is executed, the dynamic link library corresponding to the RDMA virtual layer is specified in advance through an environment variable, for example, LD_PRELOAD, so that the application program preferentially links the API in the RDMA virtual layer. The virtual layer also performs at least two other tasks in the running process, including resource management and resource virtualization. The resource management mainly processes the RDMA resources, for example, intercepts and records the increase, deletion and modification information of the resources during the running, and updates the resource list and the mapping relationship in the resource pool in real time, and then performs explicit linking after the resource processing is completed, refer to Figure 6 and Figure 7 When the RDMA resources need to be closed or reopened, the same function in the RDMA core library can be called to achieve this.

[0037] Therefore, in some embodiments, the remote direct memory access virtual layer is a dynamic link library; and the method further includes: pre-configuring an environment variable to make the application program preferentially link the dynamic link library, so as to intercept the interface call of the application program to the remote direct memory access core library before sequentially calling each remote direct memory access resource closing interface. In this way, the RDMA virtual layer can effectively capture and record the RDMA resource information, so as to ensure the subsequent accurate recovery of the process state of the RDMA resources.

[0038] Of course, according to the description of the tasks performed by the virtual layer, the intercepted application program interface call to the remote direct memory access core library is mainly to manage the RDMA resources, so the intercepted application program interface call to the remote direct memory access core library is not limited to before calling the remote direct memory access resource closing interface, that is, not limited to before closing the RDMA resource. After closing, the virtual layer will still perform the task of resource management to update the resource information record of the resource pool.

[0039] When closing the RDMA resource, it is necessary to clearly identify the RDMA resource to be closed and the way or order to close, otherwise it is easy to have the problem of missing closing the resource or incorrect release order. In some embodiments, the closing the remote direct memory access resource of the target process comprises: starting an operation for closing the remote direct memory access resource by the remote direct memory access virtual layer; the remote direct memory access virtual layer comprises a resource pool for storing resource information records, the resource information records comprising: all remote direct memory access resources of the target process, and the dependency relationship between each remote direct memory access resource; according to all remote direct memory access resources of the target process and the dependency relationship, each remote direct memory access resource closing interface is called in turn to release each remote direct memory access resource.

[0040] In this embodiment, by adding an RDMA virtual layer between the application program and the RDMA core library, and configuring a resource pool in the virtual layer, the resource pool is used to centrally record each RDMA resource and the mutual dependency relationship, and the RDMA closing operation is performed according to the resource information record in the resource pool, which can cover all RDMA resources and avoid omission; and based on the reverse order closing sequence of the dependency relationship, the rationality of the resource release order can be ensured, and resource leakage or system crash caused by incorrect order can be prevented.

[0041] Due to the working mechanism of RDMA, in the prior art, the communication of RDMA bypasses the kernel, and the kernel cannot perceive and control the communication state, so CRIU cannot effectively save and restore the RDMA state with the help of the kernel. Referring to Figure 2 and Figure 3In the embodiment, specifically, a virtual layer is added between the application program and the RDMA kernel library, and before the target process is frozen, the method further includes: based on the remote direct memory access virtual layer, intercepting the target process's creation operation, modification operation and deletion operation on the remote direct memory access resource; if it is a creation operation, recording the created remote direct memory access resource, the corresponding virtual resource and the creation operation configuration parameter, and establishing an initial mapping relationship from the virtual resource to the remote direct memory access resource, to form a resource information record; the creation operation configuration parameter is a configuration attribute required for the remote direct memory access resource core library to create a physical resource, and the remote direct memory access resource is a physical resource; If it is a modification operation, only the last information of the modification operation is recorded, and the resource information record is updated; if it is a deletion operation, the corresponding physical resource, virtual resource, initial mapping relationship and creation operation configuration parameter are deleted from the resource information record; the dependency relationship between each remote direct memory access resource is recorded, the dependency relationship is integrated into the resource information record, and the finally formed resource information record is stored in the resource pool, so that the RDMA resource information is completely saved, and the RDMA resource process state can be accurately saved and restored based on the resource information record in the resource pool when needed.

[0042] Before the target process is frozen, the method further includes: when the application program calls a remote direct memory access resource interface to initiate a resource access request, receiving the virtual resource transmitted by the application program in the access request; intercepting the access request, and based on the initial mapping relationship recorded in the resource pool, finding the physical resource corresponding to the virtual resource; the found physical resource is transmitted as a return parameter to the same interface of the remote direct memory access core library, so that the remote direct memory access core library executes the actual remote direct memory access operation through the physical resource.

[0043] Referring to Figure 3 , the processing of the RDMA resource in the resource management includes: the API of the RDMA resource creation, modification or deletion operation, the dependency parameters of the API of the above operation, including the parameters required for creating the resource and the parameters for modifying the resource, and the resource information record of the created resource is maintained in the resource pool, and the RDMA resource is closed or recreated to recover the RDMA resource according to the resource information record in the resource pool.

[0044] Referring to Figure 3 step 1 in the embodiment, for the resource creation API, the parameters required for creating the RDMA resource and the created RDMA resource are recorded. For example, if a resource res2 is created by calling the resource creation API, the created res2 and the configuration parameter dep2 required for creating the resource res2 are recorded in the resource pool.

[0045] For the resource modification API, record the modification information, if the information is modified multiple times, record the last modification information. For example, call the resource modification API to modify a certain attribute of resource res2, after multiple modifications, only record the last modification information dep5, as shown in step 2. Figure 3 Step 2.

[0046] Referring to Figure 3 As shown in step 3 of

[0047] The resource management in the RDMA virtual layer updates the resource state in the resource pool in real time at runtime. In this embodiment, through the resource information record in the resource pool, it can be determined which RDMA resources to close in the process saving stage, which RDMA resources to open in the process recovery stage, and in what order or manner to open. In this way, in the process saving stage, all RDMA resources in the resource pool are closed in sequence according to the dependency relationship. In the process recovery stage, the RDMA resources are reopened using the recorded parameters required to open the resources, the resource attributes are modified using the recorded parameters to modify the resources, and the resources are restored to the state before being closed.

[0048] Next, the aforementioned resource virtualization is introduced. It can be understood that for memory resource access, the application side sees virtual resources, and the RDMA core library side is physical resources, i.e., the aforementioned remote direct memory access resources are physical resources, and the reopened RDMA resources also belong to physical resources, and their addresses or identifiers may change due to changes in the system environment, such as process migration, resource reallocation, etc., and the application program still relies on the initial mapping relationship between the original virtual resources and the remote direct memory access resources for access, which may cause access failure and fail to achieve transparent recovery of the process. Therefore, in some embodiments, resource virtualization is needed to achieve isolation and conversion of RDMA resources, and in the process saving and recovery process, the RDMA resources are rebuilt and the mapping relationship is updated, so that even if the physical resources change, the virtual resources remain unchanged. In this way, after the process is recovered, the application program can still continue to run transparently using virtual resources, without the need to perceive changes in the underlying physical resources, and to ensure that the application can continue to run transparently using virtual resources after the process is recovered.

[0049] In the above process of reconstruction and recovery of RDMA resources, resource virtualization plays an important role. Specifically, resource virtualization refers to the construction of mapping relationship between virtual resources and physical resources in the RDMA virtual layer, so that the application program only perceives virtual resources, and the RDMA core library directly operates physical resources. In the present embodiment, since the reconstructed RDMA physical resource can be different from that before shutdown, such as the original RDMA physical resource p_res0 (note: the use of underlines in this text is to make the example consistent with the conventional symbol representation in the computer field), it can become p_res2 after reconstruction. In order to make the application program continue to run transparently using virtual resources without perceiving the change of underlying hardware resources, an isolation and conversion mechanism realized by resource virtualization is needed to decouple the direct association between virtual resources and physical resources.

[0050] Referring to Figure 4 In the resource management process, when the resource creation API is called, the RDMA core library creates a physical resource res0, and the RDMA virtual layer creates a corresponding virtual resource v_res0 for each reconstructed physical resource res0, and establishes the mapping relationship between the two.

[0051] In some embodiments, before the freezing of the target process, the method further comprises: based on the remote direct memory access virtual layer, intercepting the creation operation, modification operation and deletion operation of the target process on the remote direct memory access resource; if it is a creation operation, recording the created remote direct memory access resource, the corresponding virtual resource and the creation operation configuration parameter, and establishing the initial mapping relationship of the virtual resource to the remote direct memory access resource to form a resource information record; the creation operation configuration parameter is the configuration attribute required for the remote direct memory access resource core library to create a physical resource, and the remote direct memory access resource is a physical resource; if it is a modification operation, only the last information of the modification operation is recorded, and the resource information record is updated; if it is a deletion operation, the corresponding physical resource, virtual resource, initial mapping relationship and creation operation configuration parameter are deleted from the resource information record; the dependency relationship between each remote direct memory access resource is recorded, the dependency relationship is integrated into the resource information record, and the finally formed resource information record is stored in the resource pool.

[0052] Referring to Figure 5When the application initiates an access through the virtual resource v_res0, the RDMA virtual layer finds the corresponding physical resource p_res0 through the virtual-to-physical V2P mapping relationship, and directs the access request to p_res0, and passes p_res0 as a parameter to the same function of the RDMA core library to realize the specific communication function; the physical resource result p_res1 returned by the RDMA core library is then converted into a virtual resource v_res1 through a physical-to-virtual P2V mapping and returned to the application. Even if the physical resource changes after reconstruction, such as p_res0 changes to p_res2, the application can still access the physical resource through the original virtual resource v_res0 according to the mapping relationship to realize transparent recovery execution.

[0053] Therefore, in some embodiments, before the freezing of the target process, the method further comprises: when an application calls a remote direct memory access resource interface to initiate a resource access request, receiving the virtual resource transmitted by the application in the access request; intercepting the access request, finding the physical resource corresponding to the virtual resource based on the initial mapping relationship recorded in the resource pool; and passing the found physical resource as a return parameter to the same interface of the remote direct memory access core library, so that the remote direct memory access core library executes the actual remote direct memory access operation through the physical resource.

[0054] In some embodiments, the opening of the remote direct memory access resource comprises: based on the remote direct memory access virtual layer, starting an operation for reopening the remote direct memory access resource, reading the creation operation configuration parameters of each remote direct memory access resource and the dependency relationship between each remote direct memory access resource from the resource pool; sequentially calling the remote direct memory access resource creation interface according to the dependency relationship, and reconstructing each remote direct memory access resource based on the corresponding creation operation configuration parameters; and adjusting the properties of each reconstructed remote direct memory access resource to the state before shutdown according to the last information of the modification operation of each remote direct memory access resource in the resource information record, so as to realize the saving of the RDMA resource and the accurate recovery of the process.

[0055] Referring to Figure 7 , the opening of the RDMA resource can be realized by the open_res function. This function reads the resource creation parameters recorded in the resource pool by the RDMA virtual layer running before, such as the initialization parameters of the queue pair, the configuration information of the protection domain, and the dependency relationship between resources, reconstructs the resources by calling the RDMA resource creation interface in the dependency order, adjusts the resource state according to the last recorded modification information, ensures that the reconstructed RDMA resource is completely consistent with the state before shutdown, and then updates the mapping between the virtual resource and the physical resource, so that the restored degraded process is reconverted into the target process containing the RDMA resource.

[0056] Specifically, the closing operation for closing the remote direct memory access resource and the opening operation for reopening the remote direct memory access resource are triggered to be executed in the following manners: in a first triggering manner, codes for executing the operations are injected into the address space of the target process, and execution is triggered at a preset time; and in a second triggering manner, codes for executing the closing operation close_res function and the opening operation open_res function are injected into the address space of the target process through a parasitic code mechanism of the CRIU itself. When the closing operation is triggered after the CRIU freezes the process, and when the opening operation is triggered after the degenerate process is restored, the codes are called by the execution logic of the process itself to implement the resource operations. For example, after the CRIU freezes the target process, the injected close_res function is informed to execute through a signal mechanism, so as to ensure that the resource closing operation is synchronized with the process freezing state. This code injection manner can ensure that the operation is closely synchronized with the process state, and is suitable for scenarios with high real-time requirements.

[0057] In a second triggering manner, an independent thread is started in the remote direct memory access virtual layer, and the closing or opening operation is called to be executed by receiving an external instruction. Specifically, an independent thread can be started in the RDMA virtual layer, and the thread continuously listens to a control signal from the CRIU tool. When the closing operation needs to be executed, the thread receives a closing resource instruction and calls the close_res function; and when the opening operation needs to be executed, the thread receives an opening resource instruction and calls the open_res function. This implementation manner avoids environmental interference that may be caused by code injection, does not need to modify the address space of the process, and is suitable for scenarios with high compatibility requirements.

[0058] It can be understood that if the mapping between the virtual resource and the physical resource is not updated in time after the RDMA resource is rebuilt, the application program will fail to find the physical resource through the original virtual resource after being restored, which will cause access failure. For example Figure 8 In the RDMA resource is not closed, the RDMA resource (i.e., the physical resource) is qp0; refer to Figure 10 In the recovery phase, the physical resource changes to qp1, and at this time, it is necessary to rebuild the mapping relationship between the original virtual resource and the changed RDMA resource. Therefore, refer to Figure 7 、 Figure 8 and Figure 10In some embodiments, after the opening of the remote direct memory access resource, before the thawing of the target process, the method further comprises: based on the remote direct memory access virtual layer, establishing a mapping relationship between the reconstructed remote direct memory access resource and a virtual resource to update the initial mapping relationship in the resource pool; wherein the reconstructed remote direct memory access resource is a physical resource generated by a resource creation interface according to the creation operation parameter by a remote direct memory access core library; the virtual resource is visible identification information used by an application when accessing a remote direct memory access resource before the target process is converted into a degraded process.

[0059] After opening the RDMA resource, before thawing the target process, the RDMA virtual layer performs a final confirmation operation of the mapping relationship: the reconstructed and attribute-adjusted physical resource, such as p_res2 (not shown in the figure), is stably mapped with the virtual resource, such as v_res0, used by the application when accessing the resource before the process is closed, that is, it is updated to v_res0→p_res2.

[0060] Specifically, the virtual resource v_res0 is the virtual resource determined by the application based on the virtual resource through the RDMA virtual layer before the process closes the RDMA resource, that is, v_res0 is mapped to p_res0; after the reconstruction of the physical resource, the virtual layer updates the mapping relationship so that v_res0 points to p_res2 after reconstruction instead of p_res0. During this process, the visible virtual resource v_res0 of the application does not change at all, and only the physical resource it points to is updated from p_res0 to p_res2.

[0061] In some embodiments, the thawing of the target process to continue the execution of the target process from the save point comprises: releasing the frozen state of the target process; when receiving an access request initiated by the application through the virtual resource, directing the access request to the reconstructed remote direct memory access resource corresponding to the virtual resource based on the mapping relationship to continue the execution logic of the target process before the save point.

[0062] The CRIU releases the frozen limit of the process, the previously re-established mapping relationship v_res0→p_res2 directs the request to the corresponding reconstructed physical resource p_res2; after the p_res2 function operation of the RDMA core library returns the result to the application through the physical-to-virtual P2V mapping conversion to the virtual resource result v_res1, the process continues to execute from the save point. Since the state of the reconstructed RDMA resource is consistent with that before the shutdown, and the memory data, register state, etc. of the process have been completely restored, the process can seamlessly continue the execution logic before the freezing.

[0063] The process state saving and recovery method provided by the embodiments of the present application can ensure that the state saved by CRIU contains all necessary basic information, and avoid state omission caused by the fact that the RDMA resource is not perceivable, by actively closing the RDMA resource of the target process in the process saving stage, so that the process is converted into a "degraded process" that can be completely processed by CRIU. In the recovery stage, the degraded process is recovered based on the saved basic information first, so that the basic running environment of the process is consistent with that before freezing. Then, the previously closed RDMA resource is reopened, so that the resource state is completely matched with the resource state of the process before being closed. Finally, the process is unfrozen. Since the basic state and the RDMA resource of the process are accurately recovered at this time, the process can be seamlessly continued from the saving point. Thus, the problem that the native CRIU cannot process the saving and recovery of the RDMA resource is solved, and complete saving and accurate recovery of the process containing the RDMA resource are realized.

[0064] It should be noted that the process state saving and recovery method provided by each of the embodiments described above in the present application can be fixed in a manufactured physical hardware device in the form of software or a program, and when the software or program is run, the method flow described above can be reproduced.

[0065] Referring to Figure 11 The embodiments of the present application also provide a process state saving and recovery system, which comprises: A hardware layer 210, which comprises a processor, a memory and a remote direct memory access network card; A software layer 220, which comprises a target process running on the processor, a remote direct memory access virtual layer and a remote direct memory access core library, wherein the remote direct memory access core library is configured to provide an operation interface of the remote direct memory access resource; The remote direct memory access virtual layer is configured to perform: During freezing of the target process by the processor, the remote direct memory access resource used by the target process through the remote direct memory access network card is closed, so that the target process is converted into a degraded process, and the running state information of the degraded process is saved to the memory; Based on the running state information saved in the memory, the processor is controlled to recover the degraded process; By calling the operation interface provided by the remote direct memory access core library, the remote direct memory access resource is opened, so that the recovered degraded process is converted into a target process containing the remote direct memory access resource; The processor is controlled to unfreeze the target process, so that the target process continues to execute from the saving point.

[0066] The system provided by the embodiment can be used to execute the technical solutions of each embodiment of the foregoing method, and has similar implementation principles and technical effects to Embodiment 1, which will not be described in detail again, and can be mutually referred to.

[0067] To help more clearly understand the technical solutions of one embodiment of the process state saving and recovery in the present application and the technical effects thereof, the following will be described in detail in combination with Figure 6 to Figure 10 The following will be described in detail in combination with the actual application scenario of a high-performance computing cluster. The present embodiment takes a target process that depends on a remote direct memory access resource for communication in a high-performance computing cluster as an example. For the convenience of description, it is assumed that the target process is marked as process P. Wherein, v_qp is a virtual resource visible to an application program, qp0 and qp1 are physical resources actually created in an RDMA core library, p_qp is a pointer to the physical resource in the RDMA virtual layer, ibv_create_qp, ibv_post_send, ibv_destroy_qp and ibv_modify_qp are interface functions of the RDMA core library, close_res is a function for closing an RDMA resource in the RDMA virtual layer, and open_res is a function for opening an RDMA resource in the RDMA virtual layer.

[0068] When the process P normally runs on the cluster node A, it needs to perform high-speed data communication with other nodes through the RDMA resource. When the application program calls the ibv_create_qp function to create a queue pair resource, the node A pre-configures an environment variable to preferentially link the RDMA virtual layer, and actually calls the ibv_create_qp function of the virtual layer. The function first explicitly links and calls the same named function of the RDMA core library, and creates the physical resource qp0 in the core library. After returning, the virtual layer creates a data structure, records the dependency parameter dep and the virtual resource v_qp of qp0, and the pointer p_qp pointing to qp0, at this time p_qp→qp0, and stores the structure into a resource pool, as shown in the following table. Figure 8

[0069] When the process P communicates with other nodes of the cluster, the application program calls the ibv_post_send function to send data, and the input parameter is v_qp. The RDMA virtual layer finds the corresponding qp0 through v_qp, and then calls the ibv_post_send function of the core library with qp0 as the parameter, to complete the actual data transmission, and this process does not affect the normal execution of the application.

[0070] When the cluster node A has a hardware failure and needs to perform failure recovery on the process P, or needs to migrate the process P to the node B due to load balancing, the state of the process P needs to be saved first. At this time, according to the process state saving and recovery scheme provided by the present application, the process P is saved according to the following steps. Figure 6 ​The flowchart shows that the CRIU tool initiates a process saving operation, first freezes the target process P to suspend its execution. At this time, the RDMA virtual layer starts the close_res function, and according to the dependency relationship recorded in the resource pool, closes the RDMA resources one by one. After closing, the target process is saved after degradation, which does not contain RDMA resources.

[0071] Referring to Figure 9 , for v_qp, the function calls the ibv_destroy_qp function of the core library to close the corresponding physical resource qp0. After all the RDMA resources are closed, the process P is degraded to a degraded process that does not contain RDMA resources, and the CRIU saves the memory data and register state of the degraded process, and associates the v_qp, dep and other information in the resource pool to generate a running state information file.

[0072] On node B (the target node of fault recovery or the migration node), the process P is restored based on the saved running state information file. Referring to Figure 8 , S1, preparation before recovery, deploy the RDMA virtual layer and configure the environment variables to ensure that it is loaded first; obtain the running state information file of process P, and initialize the resource pool to store the v_qp, dep and other information parsed from the file.

[0073] Referring to Figure 7 , S2, rebuild the degraded process and RDMA resources, specifically: the CRIU parses the running state information file, rebuilds the address space and execution context of the process P, and restores the degraded process to the basic running state information before freezing. Subsequently, the RDMA virtual layer starts the open_res function to rebuild the RDMA resources according to the dependency relationship. The dep calls the ibv_create_qp function of the core library to create the physical resource qp1, and then calls the ibv_modify_qp function to adjust the state of qp1 to the state before qp0 is closed, and updates p_qp to point to qp1, i.e. p_qp→qp1, as shown in Figure 10 .

[0074] S3, establish a mapping relationship, specifically: before unfreezing the process P, the RDMA virtual layer re-establishes the mapping relationship between v_qp and qp1 to ensure that the application program still accesses the RDMA physical resource through v_qp.

[0075] After unfreezing the process P, it continues to execute from the save point. When the application program calls the ibv_post_send function to send data, it still passes in v_qp as a parameter. The RDMA virtual layer finds qp1 through v_qp, and calls the core library function with qp1 as a parameter to complete data transmission. Since the state of the restored qp1 is the same as that of the closed qp0, the communication between the process P and other nodes of the cluster is seamlessly continued.

[0076] In the embodiments of the present application, before the CRIU saves the process state, the RDMA resource is closed, and the process is degraded to a non-RDMA process. After the degraded non-RDMA process is restored, the restored RDMA resource is opened, and the process is restored to the state before the RDMA resource is closed. In this way, the process state containing the RDMA resource can be completely saved and restored. Thus, the CRIU-based RDMA process saving and restoration is realized, and the passive process state saving scheme is applied in a cluster system. In the cluster fault recovery and migration scenarios, the fault recovery can be realized in a low-cost and high-performance manner, the system availability is improved, and the task migration is realized, and the system performance is improved.

[0077] Further, through the technical scheme of unchanged virtual resource and dynamically reconstructed and mapped physical resource, after the physical resource qp0 of the process P on the node A fails, the qp1 is reconstructed on the node B and the v_qp is maintained unchanged, so that the application program can be transparently recovered and executed without modification. This further makes the passive saving and recovery scheme based on CRIU applicable in a high-performance computing cluster relying on RDMA, and significantly improves the fault tolerance capability and resource scheduling flexibility of the cluster.

[0078] The embodiments of the present application also provide an electronic device, such as Figure 12 The electronic device provided by the embodiments of the present application can include a housing 51, a processor 52, a memory 53, a circuit board 54, and a power supply circuit 55, wherein the circuit board 54 is arranged inside a space enclosed by the housing 51, and the processor 52 and the memory 53 are arranged on the circuit board 54; the power supply circuit 55 is configured to supply power to each circuit or device of the electronic device; the memory 53 is configured to store executable program codes; and the processor 52 is configured to run programs corresponding to the executable program codes by reading the executable program codes stored in the memory 53, and execute the process state saving and restoration method provided by any of the preceding embodiments.

[0079] The specific execution process of the processor 52 on the above steps and the steps further executed by the processor 52 by running the executable program codes can refer to the description of the preceding embodiments, and will not be described here again.

[0080] The embodiments of the present application also provide a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the process state saving and restoration method described in any of the preceding embodiments.

[0081] To sum up, the process state saving and recovery method and system provided by the embodiments of the present application actively close the RDMA resources of the target process in the process saving stage, so that the process is converted into a degenerate process that can be completely processed by CRIU, and it is ensured that the state saved by CRIU contains all necessary basic information, avoiding the omission of state caused by the unperceivable RDMA resources; secondly, in the recovery stage, the degenerate process is recovered based on the saved basic information first, so as to ensure that the basic running environment of the process is consistent with that before freezing; then the previously closed RDMA resources are reopened, so that the resource state is completely matched with the resource state of the process before being closed, and finally the process is unfrozen. Since the basic state and the RDMA resources of the process have been accurately recovered at this time, the process can be seamlessly continued from the save point. Thus, the problem that the native CRIU cannot process the saving and recovery of the RDMA resources is solved, and the complete saving and accurate recovery of the process containing the RDMA resources are realized.

[0082] It should be noted that in this paper, the schemes described in each embodiment have different focuses, but there is some kind of mutual relationship between each embodiment. When understanding the schemes of the present application, each embodiment can be referred to each other. In addition, relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or measurement control unit including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or measurement control unit. Without more limitations, the element defined by the statement "including a" does not exclude the existence of other identical elements in the process, method, article or measurement control unit including the element.

[0083] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any changes or replacements within the technical scope disclosed by the present application can be easily thought by those skilled in the art, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for saving and restoring process state, characterized in that, include: Freeze the target process and disable the remote direct memory access resources of the target process so that the target process is converted into a degenerate process; Save the running status information of the degradation process; Based on the running status information of the degraded process, the degraded process is restored, wherein the degraded process does not contain remote direct memory access resources; Open the remote direct memory access resource so that the restored degenerate process is converted into a target process that includes the remote direct memory access resource; Unfreeze the target process so that it can continue execution from the save point.

2. The method according to claim 1, characterized in that, The closure of the remote direct memory access resources of the target process includes: An operation to disable remote direct memory access resources is initiated through a remote direct memory access virtual layer; the remote direct memory access virtual layer includes a resource pool for storing resource information records, the resource information records including: all remote direct memory access resources of the target process, and the dependencies between the various remote direct memory access resources; Based on all remote direct memory access resources of the target process and their dependencies, the closing interfaces of each remote direct memory access resource are called sequentially to release each remote direct memory access resource.

3. The method according to claim 2, characterized in that, The remote direct memory access virtual layer is a dynamic link library; The method further includes: pre-configuring environment variables to make the application link the dynamic link library preferentially, so as to intercept the application's interface calls to the remote direct memory access core library before sequentially calling the closing interfaces of each remote direct memory access resource.

4. The method according to claim 2, characterized in that, Prior to freezing the target process, the method further includes: Based on the aforementioned Remote Direct Memory Access virtual layer, the target process's creation, modification, and deletion operations on remote direct memory access resources are intercepted; If it is a creation operation, the created remote direct memory access resource, the corresponding virtual resource, and the creation operation configuration parameters are recorded, and an initial mapping relationship between the virtual resource and the remote direct memory access resource is established to form a resource information record; the creation operation configuration parameters are the configuration attributes required by the remote direct memory access resource core library to create physical resources, and the remote direct memory access resource is a physical resource; If it is a modification operation, only the last information of the modification operation is recorded and updated to the resource information record; If it is a deletion operation, then the corresponding physical resources, virtual resources, initial mapping relationships and creation operation configuration parameters are deleted from the resource information record; Record the dependencies between each remote direct memory access resource, integrate the dependencies into the resource information record, and store the final resource information record into the resource pool.

5. The method according to claim 4, characterized in that, Prior to freezing the target process, the method further includes: When an application calls the Remote Direct Memory Access Resource Interface to initiate a resource access request, the virtual resource passed in by the application in the access request is received; The access request is intercepted, and the physical resource corresponding to the virtual resource is found based on the initial mapping relationship recorded in the resource pool. The found physical resource is passed as a return parameter to the remote direct memory access core library's interface of the same name, so that the remote direct memory access core library can perform the actual remote direct memory access operation through the physical resource.

6. The method according to claim 1, characterized in that, The step of restoring the degradation process based on the saved runtime status information of the degradation process includes: Parse the runtime status information file of the degenerate process and extract the memory data and register status carried in the runtime status information file; Based on the memory data, the address space of the degraded process is reconstructed, and based on the register state, the execution context of the degraded process is restored for the purpose of restoring the degraded process.

7. The method according to claim 4, characterized in that, Opening the remote direct memory access resource includes: Based on the remote direct memory access virtual layer, an operation to reopen remote direct memory access resources is initiated, and the creation operation configuration parameters of each remote direct memory access resource and the dependency relationships between each remote direct memory access resource are read from the resource pool. According to the aforementioned dependency relationship, the remote direct memory access resource creation interface is called sequentially, and each remote direct memory access resource is rebuilt based on the corresponding creation operation configuration parameters. Based on the last information of the modification operation of each remote direct memory access resource in the resource information record, adjust the attributes of each reconstructed remote direct memory access resource to the state before it was closed.

8. The method according to claim 7, characterized in that, The closing operation for disabling remote direct memory access resources and the opening operation for reopening remote direct memory access resources are triggered in the following ways: The code that performs the closing or opening operation is injected into the address space of the target process and executed at a preset time. or, A separate thread is started in the remote direct memory access virtual layer to execute the close or open operation by receiving external instructions.

9. The method according to claim 7, characterized in that, After opening the remote direct memory access resource and before unfreezing the target process, the method further includes: Based on the Remote Direct Memory Access (RDA) virtual layer, a mapping relationship is established between the reconstructed RDA resources and virtual resources to update the initial mapping relationship in the resource pool; wherein, the reconstructed RDA resources are physical resources generated by the RDA core library through the resource creation interface according to the creation operation parameters; the virtual resources are visible identification information used by the application when accessing RDA resources before the target process is converted into a degenerate process.

10. The method according to claim 9, characterized in that, Unfreezing the target process so that it can continue execution from the save point includes: Release the frozen state of the target process; When an access request initiated by an application through the virtual resource is received, the access request is directed to the remote direct memory access resource reconstructed corresponding to the virtual resource based on the mapping relationship, so as to continue the execution logic of the target process before the save point.

11. A process state saving and restoring system, characterized in that, include: The hardware layer includes a processor, a memory, and a remote direct memory access network interface card. The software layer includes a target process running on the processor, a remote direct memory access virtual layer, and a remote direct memory access core library, wherein the remote direct memory access core library is used to provide an operation interface for remote direct memory access resources. The remote direct memory access virtual layer is configured to perform: During the period when the processor freezes the target process, the remote direct memory access resources used by the target process through the remote direct memory access network card are closed, the target process is converted into a degenerate process, and the running status information of the degenerate process is saved to the memory; Based on the running state information stored in the memory, the processor is controlled to resume the degradation process; By calling the operation interface provided by the Remote Direct Memory Access (RDM) core library, the RDM resource is opened, thereby converting the restored degraded process into a target process containing the RDM resource. The processor is controlled to unfreeze the target process so that the target process can continue execution from the save point.

12. An electronic device, characterized in that, The electronic device includes: a housing, a processor, a memory, a circuit board, and a power supply circuit, wherein the circuit board is disposed inside the space enclosed by the housing, and the processor and the memory are disposed on the circuit board; the power supply circuit is used to supply power to various circuits or devices of the electronic device; the memory is used to store executable program code; the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, for executing the method of any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the method of any one of claims 1 to 10.

Images