Methods and electronic devices for recovering computational tasks

By using the address mapping relationship of task snapshot files during the computing task recovery process, data blocks are restored to the same virtual address, solving the problem of low efficiency in computing task recovery and achieving non-intrusive fast recovery and reduced latency.

CN123114667APending Publication Date: 2026-11-17LENOVO SOFTWARE
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611317002.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-27
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing technologies have low recovery efficiency for computing tasks, and cold restarting processes requires executing a full set of initialization procedures, resulting in long processing times.

Method used

By utilizing the address mapping relationship in the task snapshot file, data blocks in the application cache data can be restored to the same virtual address in the video memory of the target computing device, achieving fast recovery without code modification and address remapping.

Benefits of technology

It enables non-intrusive and rapid recovery of computing tasks, reduces recovery latency, and improves recovery efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123114667A_ABST
    Figure CN123114667A_ABST
Patent Text Reader

Abstract

This application discloses a method and electronic device for recovering a computing task. The method includes: receiving a task snapshot file sent by a storage device in response to a recovery instruction for a computing task; the task snapshot file includes at least: application cache data and an address mapping relationship; the application cache data is business data stored in the video memory of a target application on a first computing device when executing a computing task; the address mapping relationship represents the correspondence between each data block in the application cache data and its virtual address and data block size; according to the address mapping relationship, transferring each data block in the application cache data to the corresponding virtual address in the video memory of a second computing device, so that the computing task of the target application can be resumed on the second computing device; the virtual address corresponding to the data block in the video memory of the second computing device is consistent with the virtual address corresponding to the data block in the address mapping relationship.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method for recovering a computing task and an electronic device. Background Technology

[0002] With technological advancements, computational tasks such as artificial intelligence model inference are being deployed on a large scale. Current computational tasks rely on computing devices, such as graphics processing units (GPUs). In various scenarios, rapid recovery of task data is required for these computational tasks.

[0003] Currently, the recovery method of process cold restart is mostly adopted to achieve rapid recovery of GPU applications. However, the process cold restart method requires executing a complete initialization process from scratch, such as interface runtime environment initialization, graph capture, full loading of model weights, inference core cache initialization and other time-consuming operations.

[0004] Therefore, the current method of rapidly recovering task data suffers from low recovery efficiency. Summary of the Invention

[0005] In view of this, this application provides a method and electronic device for recovering computing tasks, as follows:

[0006] A method for recovering a computational task, comprising:

[0007] In response to a recovery command for the computing task, the system receives a task snapshot file sent by the storage device.

[0008] The task snapshot file includes at least: application cache data and address mapping relationships; the application cache data is the business data stored in the video memory of the target application on the first computing device when executing the computing task; the address mapping relationship represents the correspondence between each data block in the application cache data and its virtual address and data block size;

[0009] According to the address mapping relationship, each data block in the application cache data is transferred to the corresponding virtual address in the video memory of the second computing device, so that the computing task of the target application can resume running on the second computing device;

[0010] Wherein, the virtual address corresponding to the data block in the video memory of the second computing device is consistent with the virtual address corresponding to the data block in the address mapping relationship.

[0011] Optionally, in the above method, the virtual addresses in the address mapping relationship are addresses allocated in a continuous allocation manner, which at least includes: connecting the starting address of the newly allocated virtual address to the ending address of the already allocated virtual address;

[0012] The process of transferring each data block in the application cache data to the corresponding virtual address in the video memory of the second computing device includes:

[0013] Multiple data blocks with consecutive virtual addresses in the application cache data are merged and transmitted to the corresponding virtual addresses in the video memory of the second computing device.

[0014] Optionally, the above method involves merging and transmitting multiple consecutive data blocks with virtual addresses in the application cache data to the corresponding virtual addresses in the video memory of the second computing device, including:

[0015] In the case where there is one computing device in the computing node where the second computing device is located, multiple data blocks with consecutive virtual addresses in the application cache data are merged and transmitted to the corresponding virtual addresses in the video memory of the second computing device;

[0016] In the case where there are multiple computing devices in the computing node where the second computing device is located, the application cache data is divided into multiple cache data sets; each cache data set corresponds to one computing device in the computing node.

[0017] Multiple data blocks with consecutive virtual addresses in each of the cached data sets are merged and transmitted to the third computing device in the computing node, so that the third computing device sends the cached data sets to the corresponding virtual addresses in the video memory of the second computing device through a high-speed interconnect channel.

[0018] The high-speed interconnect channel is the data transmission channel between the computing devices.

[0019] Optionally, the above method may involve merging and transmitting multiple data blocks with consecutive virtual addresses in the cached data set to a third computing device in the computing node, including:

[0020] If the size of the cached data set exceeds the target threshold, the cached data set will be divided into multiple data subsets.

[0021] In a parallel manner, multiple data blocks with consecutive virtual addresses in each of the data subsets are merged and transmitted to a third computing device in the computing node.

[0022] Optionally, the task snapshot file further includes runtime state data for maintaining the computing task's operating environment.

[0023] The method further includes:

[0024] Using the target tools provided by the driver of the second computing device, the runtime state data in the task snapshot file is transferred to the video memory of the second computing device.

[0025] Optionally, the task snapshot file can be obtained through the following method:

[0026] In response to a snapshot instruction for the computing task, the application cache data in the video memory of the first computing device is saved according to the address mapping relationship to obtain a first snapshot file;

[0027] Release the application cache data in the video memory of the first computing device;

[0028] The runtime state data in the video memory of the first computing device is saved to obtain a second snapshot file;

[0029] The first snapshot file and the second snapshot file together form the task snapshot file.

[0030] Optionally, in the above method, the physical address corresponding to the data block in the video memory of the second computing device is different from the physical address corresponding to the data block in the video memory of the first computing device.

[0031] The method further includes:

[0032] Update the address binding relationship of the data block according to the virtual address and physical address of the data block in the video memory of the second computing device;

[0033] The address binding relationship represents the correspondence between the virtual address and the physical address corresponding to the data block; the address binding relationship is used to access the data block.

[0034] Optionally, the data blocks in the application cache data are stored in the video memory of the first computing device in the following manner:

[0035] Intercept data storage requests for the video memory of the first computing device; the data storage request contains at least the data block size of the first data block to be stored;

[0036] According to the contiguous allocation method and the size of the first data block, a first virtual address and a first physical address are allocated for the first data block in the video memory of the first computing device; the starting address of the first virtual address is connected to the ending address of the allocated virtual address;

[0037] Establish an address mapping relationship between the first data block, the first virtual address, and the data block size of the first data block;

[0038] Establish an address binding relationship between the first virtual address and the first physical address.

[0039] Optionally, the continuous allocation method described above includes:

[0040] Check if there is a free address segment in the allocated address segment; the allocated address segment is an address segment consisting of virtual addresses allocated from the video memory of the first computing device; the free address segment is an address segment consisting of virtual addresses corresponding to data blocks released in response to a video memory release request;

[0041] If there is no free address segment in the allocated address segment, a first virtual address is allocated according to the data block size of the first data block; the starting address of the first virtual address is connected to the ending address of the allocated address segment;

[0042] If a free address segment exists within the allocated address segment, a first virtual address matching the size of the first data block is allocated from the free address segment; the starting address of the first virtual address is the starting address of the free address segment.

[0043] Optionally, in the above method, allocating a first virtual address from the free address segment that matches the size of the first data block includes:

[0044] If there exists a free address segment whose address length is greater than or equal to the data block size of the first data block, then select a target address segment from the free address segments whose address length is greater than or equal to the data block size of the first data block and whose address length is the smallest.

[0045] From the target address range, allocate a first virtual address that matches the size of the first data block;

[0046] If the address length of each of the free address segments is less than the data block size of the first data block, a first virtual address is allocated from the multiple free address segments according to the address length of the free address segments and the data block size of the first data block.

[0047] An electronic device, comprising:

[0048] A storage device is used to store task snapshot files; wherein, the task snapshot files include at least: application cache data and address mapping relationships; the application cache data is business data stored in the video memory of the target application on the first computing device when executing the computing task; the address mapping relationship represents the correspondence between each data block in the application cache data and its virtual address and data block size;

[0049] The processor is configured to, in response to a recovery instruction for the computing task, receive a task snapshot file sent by the storage device; and, according to the address mapping relationship, transfer each data block in the application cache data to the corresponding virtual address in the video memory of the second computing device, so that the computing task of the target application can resume running on the second computing device.

[0050] Wherein, the virtual address corresponding to the data block in the video memory of the second computing device is consistent with the virtual address corresponding to the data block in the address mapping relationship.

[0051] As can be seen from the above technical solutions, the computing task recovery method and electronic device disclosed in this application utilize the address mapping relationship in the task snapshot file to restore the data blocks in the application cache data to the same virtual address in the video memory of the target computing device according to the address mapping relationship. Thus, for the target application, the access address of the data block does not change. Therefore, for the upper-layer target application, no code modification or address remapping operation is required to resume execution from the breakpoint of the computing task, realizing a fast recovery that is completely non-intrusive to the application layer, thereby reducing the recovery latency of the computing task and further improving the recovery efficiency of the computing task. Attached Figure Description

[0052] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 A flowchart illustrating a method for recovering a computational task, as provided in an embodiment of this application;

[0054] Figure 2 This is a schematic diagram of the virtual address of a data block in an embodiment of this application;

[0055] Figure 3 This is an example diagram illustrating high-speed parallel transmission implemented via NVLink in the embodiments of this application;

[0056] Figure 4 Another flowchart illustrating a method for recovering a computational task provided in an embodiment of this application;

[0057] Figure 5 A partial flowchart illustrating a method for recovering a computing task provided in an embodiment of this application;

[0058] Figure 6 Another part of the flowchart of a method for recovering a computing task provided in an embodiment of this application;

[0059] Figure 7 This is another part of a flowchart illustrating a method for recovering a computational task provided in an embodiment of this application;

[0060] Figure 8 A schematic diagram of a computing task recovery device provided in an embodiment of this application;

[0061] Figure 9 Another structural schematic diagram of a computing task recovery device provided in an embodiment of this application;

[0062] Figure 10 Another schematic diagram of a computing task recovery device provided in an embodiment of this application;

[0063] Figure 11 Another structural schematic diagram of a computing task recovery device provided in an embodiment of this application;

[0064] Figure 12 Another schematic diagram of a computing task recovery device provided in an embodiment of this application;

[0065] Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0066] Figure 14 This is an example diagram of the snapshot process in the embodiments of this application;

[0067] Figure 15 This is an example diagram of the recovery process in the embodiments of this application;

[0068] Figure 16 This is a schematic diagram showing the continuous allocation of virtual addresses corresponding to the three memory requests in this embodiment. Detailed Implementation

[0069] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0070] refer to Figure 1This is a flowchart illustrating a method for recovering computing tasks according to an embodiment of this application. This method is applicable to electronic devices, such as servers and laptops, that connect to multiple computing devices to perform computing tasks such as artificial intelligence model inference. The electronic device also houses a central processing unit (CPU), which acts as the control core, executing the recovery control flow of the target application's computing tasks and sending control commands to the computing devices via a driver. The electronic device is connected to multiple computing devices, which can be accelerated computing hardware such as GPUs or neural processing units (NPUs) to execute the target application's computing tasks. The electronic device is also connected to a storage device that stores task snapshot files. The technical solution in this embodiment primarily aims to improve the recovery efficiency of computing tasks within computing devices.

[0071] Specifically, the method in this embodiment may include the following steps:

[0072] Step 101: In response to a recovery command for a computing task, receive a task snapshot file sent by the storage device.

[0073] The computational tasks refer to accelerated computational tasks running on GPUs, including but not limited to: artificial intelligence model inference tasks, deep learning model training tasks, and scientific computing tasks. For example, a computational task could be an inference service for a large language model (LLM), which loads model weight data and key-value cache (KV cache) data generated during inference into the GPU's video memory.

[0074] It should be noted that a recovery command refers to an instruction triggered by the user or the scheduling system in the electronic device to instruct the resumption of computing tasks. For example, when an electronic device detects a GPU failure and needs to perform fault tolerance, smooth service migration, or elastic load scaling, the scheduling system automatically generates a recovery command.

[0075] The task snapshot file is a pre-saved data file used to restore the computational task. The task snapshot file can be stored on a storage device. For example, the storage device can be a hard disk drive (HDD), a solid-state drive (SSD), or a non-volatile memory (NSM). In other implementations, the task snapshot file can also be stored in a remote storage system or a distributed file system. When the CPU receives a restore command, the electronic device can read the task snapshot file from the storage device into the host memory.

[0076] Specifically, the task snapshot file includes at least: application cache data and address mapping relationships. The application cache data is the business data stored in the GPU memory of the target application on the first computing device when executing the computing task. In other words, the application cache data is the business payload data generated or managed by the upper-layer target application and stored in the GPU memory.

[0077] For example, in large-scale AI model inference scenarios, application cached data can include: model weight data (such as the parameter matrix of a neural network) and KV cache data generated during inference. The KV cache includes key and value tensor data cached in the self-attention mechanism to accelerate inference. It should be noted that application cached data constitutes the largest portion of the data during the computation task recovery process, and its recovery efficiency directly determines the overall computation task recovery latency.

[0078] Address mapping represents the correspondence between each data block in the application cache and its virtual address and size. For example, each address mapping records the virtual address of each data block in the GPU memory and the size of that data block. Through this address mapping, data blocks can be accurately placed at the correct virtual address during recovery.

[0079] Step 102: Based on the address mapping relationship, transfer each data block in the application cache data to the corresponding virtual address in the video memory of the second computing device, so that the computing task of the target application can resume running on the second computing device.

[0080] In this context, the virtual address corresponding to the data block in the video memory of the second computing device is consistent with the virtual address corresponding to the data block in the address mapping relationship. The first computing device refers to the computing device where the computing task originally ran, such as GPU1; the second computing device refers to the computing device where the computing task will run after it is restored, such as GPU2.

[0081] It should be noted that a virtual address refers to an address in the virtual address space of the video memory of a computing device. Data blocks are also allocated physical addresses. There is an address binding relationship between physical addresses and virtual addresses. The address binding relationship can be recorded in the memory management unit (MMU). The computing device can use the MMU to translate virtual addresses into physical addresses, thereby accessing the actual physical location of the video memory and realizing data block access.

[0082] Specifically, the CPU in the electronic device can parse the address mapping relationship in the task snapshot file to obtain the virtual address and data block size of each data block at the time of the snapshot. Based on this, the CPU can call the GPU driver interface of the second computing device, request the corresponding virtual address space in the video memory of the second computing device according to the virtual address, and allocate the physical address. The CPU sends a page table update command to the GPU's MMU through the driver interface to establish the binding relationship between the virtual address and the physical address. Subsequently, the CPU uses the high-speed serial computer interconnect express (PCIe) standard or direct memory access (DMA) between the host and the GPU to transfer each data block from the host memory to the location of the physical address in the video memory of the second computing device.

[0083] It's important to note that, to ensure the computational tasks function correctly after recovery, the virtual address corresponding to the data block in the second computing device's video memory is consistent with the virtual address of the data block in the address mapping relationship. In other words, the location of the data block in the restored GPU virtual address space is exactly the same as its location in the first computing device's GPU virtual address space at the time of the snapshot. This allows the target application's code logic to directly access the restored data without any modification, achieving a seamless and rapid recovery process.

[0084] Specifically, after the application cache data transfer is complete and the virtual address mapping is finished, the CPU of the electronic device resumes the execution of the target application on the second computing device. Since the application cache data has been restored to the same virtual address location as at the time of the snapshot, the target application's computational tasks can resume execution from the breakpoint at the time of the snapshot without re-executing the initialization process. At this point, the computational tasks resume running on the second computing device.

[0085] For example, taking a large language model inference service as an example, the first computing device (such as GPU1) occupies 42GB of video memory, of which runtime state data only occupies about 2GB, and application cache data (model weights + KV cache) occupies about 40GB. When the electronic device restores the application cache data, it allocates the same virtual address in the video memory of the second computing device (such as GPU2) as in the video memory of GPU1. As a result, the GPU application in the electronic device can continue execution on GPU2 according to the breakpoint position on GPU1 without code modification, address remapping, or other operations, thereby speeding up the recovery of computing tasks.

[0086] As can be seen from the above technical solutions, in the computing task recovery method provided by the embodiments of this application, the address mapping relationship in the task snapshot file is used to restore the data blocks in the application cache data to the same virtual address in the video memory of the target computing device according to the address mapping relationship. Thus, for the target application, the access address of the data block has not changed. Therefore, for the target application at the upper layer, no code modification or address remapping operation is required to resume execution from the breakpoint of the computing task, realizing a fast recovery that is completely non-intrusive to the application layer, thereby reducing the recovery latency of the computing task and further improving the recovery efficiency of the computing task.

[0087] In one implementation, the virtual addresses in the address mapping relationship are allocated in a contiguous manner, which includes at least the following: the starting address of the newly allocated virtual address is connected to the ending address of the already allocated virtual address.

[0088] In other words, during the normal operation of a computing task, whenever a target application requests video memory (such as GPU memory), the electronic device can use an interception mechanism to prevent arbitrary allocation of virtual addresses according to the default memory management method. Instead, it sets the starting address of each newly requested virtual address to the address following the last address of the previously allocated virtual address. Thus, although multiple requests for video memory by the upper-layer target application may be non-contiguous in physical address space, in virtual address space, the data blocks corresponding to these multiple requests are aggregated into a continuous virtual address range. Therefore, the continuous allocation of virtual addresses is beneficial for merging the transmission of multiple data blocks into a single transmission during the recovery phase, thereby reducing the number of transmissions and improving transmission efficiency.

[0089] Based on this, in step 102, when transferring each data block in the application cache data to the corresponding virtual address in the video memory of the second computing device, the specific steps are to merge multiple consecutive data blocks in the application cache data to the corresponding virtual address in the video memory of the second computing device.

[0090] Because the address mapping records the virtual address and size of each data block, and these virtual addresses are allocated contiguously, during recovery, the electronic device can identify multiple data blocks with consecutive virtual addresses as a single continuous address range. All data blocks corresponding to this address range are then transferred as a whole to the video memory of the second computing device in one go. In other words, data blocks that would otherwise require multiple transfers are merged into a single transfer, reducing the number of transfer operations and overhead.

[0091] In one implementation, the CPU in the electronic device can traverse the address mapping relationship and check whether the virtual addresses of adjacent data blocks are contiguous, in ascending order of virtual addresses. If the starting address of the virtual address of data block A plus the size of data block A (the address after the last address of data block A) equals the starting address of the virtual address of data block B, then data blocks A and B are determined to be contiguous in terms of virtual address and can be merged for transmission. This process continues, merging multiple data blocks with contiguous virtual addresses into a single large data block.

[0092] For example, suppose the address mapping records three data blocks: data block 1, data block 2, and data block 3. Data block 1 has a virtual address of 0x1000 (starting address) and a size of 0x1000 bytes; data block 2 has a virtual address of 0x2000 and a size of 0x2000 bytes; and data block 3 has a virtual address of 0x5000 and a size of 0x1000 bytes. Data block 1's tail address is 0x1FFF (0x1000 + 0x1000 - 1), and data block 2's starting address is 0x2000; these are contiguous and can be combined for transmission. Data block 2's tail address is 0x3FFF (0x2000 + 0x2000 - 1), and data block 3's starting address is 0x5000; these are not contiguous (0x4000~0x4FFF are free space) and need to be transmitted separately. Figure 2 As shown in the diagram. Therefore, data block 1 and data block 2 are combined into one transmission, while data block 3 is transmitted separately.

[0093] Specifically, after merging multiple consecutive data blocks with virtual addresses, the electronic device can call the driver of the second computing device to request the same virtual address space in the video memory based on the merged consecutive virtual address range, and then transfer the merged whole data block to the corresponding virtual address location in the video memory of the second computing device via PCIe or DMA.

[0094] For example, in a large language model inference service, the GPU application loads multiple tensors from the model weight file sequentially during the initialization phase, with each load corresponding to a GPU memory allocation. Using traditional allocation methods, the virtual addresses corresponding to these multiple memory allocations may be scattered across different locations in the address space, requiring each tensor to be transferred separately during recovery. However, by using the continuous allocation method in this embodiment, the virtual addresses of these multiple allocations are arranged consecutively in the address space. During recovery, multiple tensors can be merged into a single large block transfer, significantly improving transmission efficiency.

[0095] As can be seen, in this embodiment, virtual addresses are allocated to the data blocks of application cache data in the video memory of the computing device in a manner of continuous virtual address allocation. Therefore, during recovery, multiple data blocks with consecutive virtual addresses are merged and transmitted, which can reduce the number of transmissions and thus reduce the power consumption overhead caused by transmission operations, thereby improving the recovery efficiency of application cache data.

[0096] In one specific implementation, when merging and transmitting multiple consecutive data blocks with virtual addresses in the application cache data in step 102, the number of computing devices deployed in the computing node where the second computing device is located can be determined first. If there is only one computing device in the computing node where the second computing device is located, the multiple consecutive data blocks with virtual addresses in the application cache data are directly merged and transmitted to the corresponding virtual address in the video memory of the second computing device. If there are multiple computing devices in the computing node where the second computing device is located, the application cache data can be divided into multiple cache data sets, and each cache data set corresponds to one computing device in the computing node. Then, the multiple consecutive data blocks with virtual addresses in each cache data set are merged and transmitted to the third computing device in the computing node (i.e., all computing devices in the computing node, including the second computing device), so that the third computing device sends the cache data set to the corresponding virtual address in the video memory of the second computing device through the high-speed interconnect channel.

[0097] It should be noted that after receiving the cached data set, the second computing device in the computing node does not need to transmit it back to itself. That is, each third computing device in the computing node, other than the second computing device, sends its received cached data set to the corresponding virtual address in the video memory of the second computing device through the high-speed interconnect channel.

[0098] High-speed interconnect channels serve as data transmission channels between computing devices, such as NVLink and AMD's Infinity Fabric. The transmission rate of high-speed interconnect channels is significantly higher than that of the PCIe bus.

[0099] It should be noted that a compute node can be understood as a physical server or a virtual machine instance. A compute node can have one or more compute devices such as GPUs deployed on it. In this embodiment, the number of available compute devices on a compute node (excluding compute devices that have failed) can be determined by querying the hardware topology information or driver interface of the compute node.

[0100] If a computing node has only one available computing device (which can be the second computing device itself), it is not possible to use multiple computing devices for parallel transmission. In this case, the merged data block is directly transferred from the host memory to the video memory of the second computing device via PCIe.

[0101] When a computing node contains multiple computing devices, parallel transmission across these devices can be utilized to increase bandwidth. Specifically, electronic devices can divide application cache data into multiple cache data sets. Each cache data set can be of the same size or differentiated based on the transmission capabilities of each computing device. The number of cache data sets equals the number of available computing devices in the computing node, such as N, with each cache data set corresponding to one computing device. However, each cache data set is not directly transmitted to the second computing device; instead, it is first transmitted to its corresponding third computing device, and then forwarded to the second computing device by other third computing devices (not the second computing device) via a high-speed interconnect channel.

[0102] It should be noted that the transmission of the aforementioned multiple cached data sets to each computing device occurs simultaneously. That is, N cached data sets are transmitted in parallel from the host memory to each computing device via the PCIe channels of N computing devices. Simultaneously, N-1 third computing devices (not the target recovery devices) forward the cached data sets to the second computing devices via high-speed interconnect channels, either at the same time or after receiving them. In this way, the total transmission bandwidth between the electronic devices and the second computing devices is increased from the bandwidth of a single PCIe channel to a combined bandwidth of "the sum of the bandwidths of N PCIe channels (limited by the total bandwidth on the host side) + high-speed interconnect bandwidth," significantly improving transmission efficiency.

[0103] For example, suppose a compute node has four GPUs: GPU0, GPU1, GPU2, and GPU3, and the second compute device is the target recovery device, such as GPU2. The total size of the application cache data is 40GB. The CPU divides the application cache data into four cache data sets, each approximately 10GB, corresponding to GPU0, GPU1, GPU2, and GPU3 respectively. These four cache data sets are sent to GPU0, GPU1, GPU2, and GPU3 respectively via PCIe. Then, GPU0, GPU1, and GPU3 each send their received cache data sets to GPU2 via NVLink. Figure 3 As shown in the diagram. Since the transmission rate of NVLink is significantly higher than that of PCIe, this embodiment uses the high-transmission-rate NVLink to share the amount of data that needs to be transmitted on PCIe, thereby improving data recovery efficiency.

[0104] As can be seen, in this embodiment, when the computing node contains multiple computing devices, a multi-path parallel transmission method of data sharding, parallel transmission of multiple computing devices, and high-speed interconnection aggregation can be adopted, which can break through the bandwidth limitation of a single PCIe and thus significantly improve the transmission rate of application cache data.

[0105] Furthermore, in step 102, when merging and transmitting multiple data blocks with consecutive virtual addresses in the cached data set to the third computing device in the computing node, the cached data set can be further split and transmitted in parallel. Specifically, if the size of the cached data set exceeds the target threshold, the cached data set is divided into multiple data subsets. If the split data subsets are still larger than the target threshold, the splitting continues until the size of each data subset does not exceed the target threshold. Then, multiple data blocks with consecutive virtual addresses in each data subset are merged and transmitted to the third computing device in the computing node in parallel.

[0106] Parallel processing refers to establishing an independent transmission channel or stream for each data subset and transmitting it synchronously. Specifically, the CPU of an electronic device can adopt a multi-stream mechanism of Compute Unified Device Architecture (CUDA) to create a CUDA stream for each data subset. Data transmission operations are executed in parallel on the same computing device through multiple streams to achieve concurrent execution of transmission tasks and maximize the utilization of hardware transmission bandwidth.

[0107] The target threshold is used to determine whether a single cached data set is too large. If the cached data set is too large, it can be further split to increase parallelism. The target threshold can be dynamically determined based on factors such as the computing power of the computing device, the bandwidth of the high-speed interconnect channel, and the PCIe bus bandwidth, or it can be preset based on empirical values. For example, assuming the cached data set size is 12GB and the target threshold is 5GB, the CPU will divide the cached data set into three equal subsets, each 4GB, none exceeding 5GB. As another example, assuming the cached data set size is 12GB and the target threshold is 5GB, the CPU will first split the cached data set into two 6GB subsets, and then further split each 6GB subset into two 3GB subsets, ultimately resulting in four 3GB subsets. As yet another example, assuming the cached data set size is 10GB and the target threshold is 5GB, the CPU will split the cached data set into two 5GB subsets, resulting in two 5GB subsets.

[0108] Based on this, in this embodiment, on the basis of data block merging and transmission, large data blocks are further split into multiple data subsets and transmitted in parallel through multiple streams. When the size of the cached data set exceeds the target threshold, the hardware transmission bandwidth resources are fully utilized through splitting and multi-stream concurrency. This can give full play to the parallel transmission capabilities of the computing device hardware, keep the PCIe bus and high-speed interconnect channel at high utilization, and further improve the efficiency of cached data recovery.

[0109] based on Figure 1 In one specific implementation of the technical solution illustrated, the task snapshot file may further include runtime state data used to maintain the computing task's runtime environment. Taking a GPU as an example, the runtime state data is GPU runtime environment context data, which may include, but is not limited to: CUDA runtime context, GPU hardware resource binding handles, driver layer kernel state, CUDA graph and its execution state, etc. The runtime state data is characterized by its relatively small data volume (typically below GB), but it is crucial for restoring the GPU's executable environment.

[0110] Based on this, in this embodiment, before step 102, the following processing can also be performed, such as... Figure 4 As shown:

[0111] Step 103: Using the target tool provided by the driver of the second computing device, transfer the runtime state data in the task snapshot file to the video memory of the second computing device.

[0112] Taking a GPU as an example, the target tool can be a tool natively provided by the GPU driver for saving and restoring GPU runtime state data. For example, the target tool could be a checkpointing tool (such as cuda-checkpoint) provided by NVIDIA CUDA. This tool is embedded in the GPU driver layer and can efficiently save and restore the GPU's control-state data and hardware context, outperforming traditional API interception solutions.

[0113] Specifically, the CPU of the electronic device can invoke the target tool to transfer runtime state data from the host memory to the video memory of the second computing device via the PCIe bus. The restoration of runtime state data takes precedence over the restoration of application cache data. That is, runtime state data is restored first to put the second computing device into an executable state, and then application cache data is restored. In this way, the runtime environment of the second computing device is already ready during the application cache data restoration process, and application execution can resume immediately once the application cache data restoration is complete.

[0114] In one specific implementation, the details of restoring runtime state data may include at least one of the following:

[0115] Based on the context in the runtime state data, reconstruct the underlying runtime environment of the second computing device in the video memory of the second computing device;

[0116] Based on the execution flow state and synchronization event state in the runtime state data, restore the control state of the second computing device in the video memory of the second computing device.

[0117] Based on the hardware resource binding information in the runtime state data, restore the hardware context of the second computing device in the video memory of the second computing device.

[0118] Based on the computation graph, kernel functions, and their execution status in the runtime state data, the runtime foundation required for the execution of the computation task breakpoint is restored in the video memory of the second computing device.

[0119] For example, when using the NVIDIA CUDA-checkpoint tool to restore runtime state data, because this tool is embedded in the driver layer and does not require application programming interface (API) interception and replay, the time to restore a GPU context is typically in the milliseconds to seconds. Meanwhile, application cache data uses the multi-path parallel transmission mechanism of this application, which is far more efficient than directly using the transmission method provided by CUDA-checkpoint. The combination of these two methods reduces the overall recovery latency by more than 50% compared to using CUDA-checkpoint alone.

[0120] As can be seen, this embodiment splits the data to be recovered from the computing task into two parts: runtime state data and application cache data. A layered and differentiated recovery approach is adopted. For example, for runtime state data, the underlying operating environment of the computing device is quickly reconstructed using the driver layer capabilities; for application cache data, the efficient transmission mechanism described in the aforementioned embodiments is used for recovery. The two are decoupled (e.g., runtime state data is recovered first, followed by application cache data), and each uses the optimal recovery method, thereby maximizing the overall recovery efficiency of the computing task.

[0121] Based on the above implementation scheme, the task snapshot file in this embodiment can be obtained in the following ways, such as... Figure 5 As shown:

[0122] Step 501: In response to a snapshot instruction for a computing task, the application cache data in the video memory of the first computing device is saved according to the address mapping relationship to obtain a first snapshot file.

[0123] Snapshot commands are instructions triggered by users or scheduling systems to instruct the saving of computing tasks for restoration on other computing devices. Specifically, snapshot commands can be triggered in various scenarios, such as proactive backups, preventative snapshots before GPU failures, state saving before elastic load scaling, and preparations before service migration.

[0124] It's important to note that during the snapshot phase, the application cache data is saved first. Specifically, the electronic device's CPU traverses all data blocks based on the maintained address mapping relationship, sequentially reading the content of each data block via DMA and saving it to the storage device. Simultaneously, the virtual address and data block size corresponding to the data block, i.e., the address mapping relationship, need to be saved to ensure that the address mapping relationship is available during restoration.

[0125] In one implementation, in this embodiment, when saving application cache data, multiple data blocks with consecutive virtual addresses can be merged and written to the storage device to improve write efficiency.

[0126] Step 502: Release the application cache data in the video memory of the first computing device.

[0127] After the application cache data is successfully saved to the storage device, the CPU of the electronic device can release the application cache data in the video memory of the first computing device through the driver. The purpose of releasing the data is to reduce the video memory usage, so that the runtime state data that needs to be saved next can be efficiently processed by the driver-level tools.

[0128] It should be noted that since the application cache data has been persistently saved to the storage device, releasing the application cache data in the video memory will not cause data loss.

[0129] Step 503: Save the runtime state data in the video memory of the first computing device to obtain the second snapshot file.

[0130] The first snapshot file and the second snapshot file together form the task snapshot file. The CPU of the electronic device stores them together as the task snapshot file for subsequent recovery of computing tasks. During recovery, the second snapshot file (runtime state data) is recovered first, followed by the first snapshot file (application cache data and address mapping relationships).

[0131] It should be noted that after the application cache data is released, only runtime state data remains in the video memory of the first computing device. At this point, the CPU of the electronic device uses the target tool provided by the driver (such as NVIDIA cuda-checkpoint) to save the runtime state data, obtaining a second snapshot file. Since only runtime state data remains in the video memory, the amount of data is small, and the target tool provided by the driver can complete the saving in a very short time.

[0132] For example, when the first computing device (GPU1) has 42GB of video memory (40GB of application cache data + 2GB of runtime state data), the CPU of the electronic device first saves 40GB of application cache data (which can also be optimized by merging transmission and multi-stream concurrency), then releases the video memory, and finally saves 2GB of runtime state data through cuda-checkpoint. The overall snapshot time is reduced by about 40% compared to the traditional solution.

[0133] As can be seen, this embodiment divides the snapshot process into two sequential stages: application cache data saving and runtime state data saving. First, the application cache data is saved and released from the video memory. Then, the remaining runtime state data in the video memory is saved. This layered snapshot strategy allows each part of the snapshot to be taken in the optimal way: application cache data is managed through a lightweight interception mechanism, and runtime state data is efficiently saved through driver-level tools, which can improve the overall snapshot efficiency.

[0134] In one implementation, the physical address corresponding to the data block in the video memory of the second computing device is different from the physical address corresponding to the data block in the video memory of the first computing device.

[0135] Specifically, in this embodiment, when allocating a virtual address for a data block in the video memory of the second computing device, a physical address is also allocated. The virtual address of the data block remains consistent with the address mapping in the snapshot to ensure that the target application's code can access the data normally without modification. However, the physical address is the actual physical storage location in the video memory of the second computing device. The electronic device can call the driver interface of the second computing device to allocate a physical address for the data block in the video memory of the second computing device. Unlike the virtual address, the physical address is managed by the MMU of the computing device, and its allocation location depends on the current free state of the video memory. That is, the physical address can be allocated according to the actual physical address space of the video memory of the second computing device. Because the first computing device and the second computing device are different physical devices, their video memory free distribution and allocation strategies are different. The physical address allocated from the video memory of the second computing device when the same data block is restored may be different from its physical address in the video memory of the first computing device.

[0136] Based on this, in this embodiment, after allocating virtual and physical addresses to the data block in the video memory of the second computing device, the address binding relationship corresponding to the data block can be updated according to the virtual and physical addresses corresponding to the data block in the video memory of the second computing device.

[0137] The address binding relationship represents the correspondence between the virtual address and the physical address of the data block, and is used to access the data block.

[0138] Specifically, the electronic device can call the driver interface of the second computing device to establish an address binding relationship between the virtual address and physical address of a data block by issuing a page table update command to the MMU of the second computing device. The address binding relationship is stored in the page table of the second computing device and maintained by the MMU of the second computing device. When the target application accesses a certain virtual address, the second computing device queries the page table through the MMU to translate the virtual address into the corresponding physical address, thereby accessing the actual video memory content.

[0139] For example, in the video memory of the first computing device (GPU1), the virtual address of data block A is 0x1000, and the physical address is 0x5000. In the video memory of the second computing device (GPU2), the virtual address remains 0x1000 upon recovery, but the physical address is 0x8000. The second computing device (GPU2) records the mapping of virtual address 0x1000 to physical address 0x8000 in its MMU page table. When the target application accesses virtual address 0x1000 on the second computing device, the second computing device automatically translates it to physical address 0x8000, without the target application being aware of this.

[0140] As can be seen, this embodiment decouples the management of virtual addresses and physical addresses: virtual addresses are used for addressing of upper-layer target applications, while physical addresses are allocated on demand by the computing device driver. During recovery, only the consistency of virtual addresses needs to be ensured; physical addresses can be reallocated and bound, thereby achieving both application-layer seamlessness and flexible allocation of physical resources.

[0141] based on Figure 1 In one specific implementation of the technical solution shown, the data blocks in the application cache data are stored in the video memory of the first computing device in the following manner: Figure 6 As shown:

[0142] Step 601: Intercept data storage requests for the video memory of the first computing device.

[0143] The data storage request must include at least the size of the first data block to be stored. A data storage request is a memory allocation request initiated by the target application during runtime via the CUDA API or other programming interfaces, such as a call to the cudaMalloc function.

[0144] Specifically, in this embodiment, the electronic device can intercept data storage requests through a lightweight interception mechanism. For example, the interception management module can be preloaded into the target application's process using the LD_PRELOAD environment variable, thereby intercepting all CUDA interface calls related to video memory allocation, video memory release, and video memory read / write.

[0145] It should be noted that the interception mechanism in this embodiment is lightweight, that is, it only intercepts interfaces related to video memory management and does not intercept other computing interfaces (such as kernel boot interfaces) in order to minimize the impact on the normal performance of the target application.

[0146] Step 602: According to the contiguous allocation method and the data block size of the first data block, allocate a first virtual address and a first physical address for the first data block in the video memory of the first computing device.

[0147] The first virtual address is connected to the last address of the allocated virtual address.

[0148] Specifically, the electronic device can call the driver interface of the first computing device to allocate a first virtual address and a first physical address for the first data block in the video memory of the first computing device according to the contiguous allocation method. The first virtual address is the address in the virtual address space of the video memory of the first computing device, and the first physical address is the actual physical storage location of the video memory of the first computing device.

[0149] In this contiguous allocation method, the starting address of a newly allocated virtual address is contiguous with the ending address of an already allocated virtual address. Specifically, the electronic device can maintain a virtual address allocation pointer, recording the ending address of currently allocated virtual addresses. When a new data storage request is received, the starting address of the newly allocated virtual address is set to the value of this pointer, and then the pointer is moved forward by the length of the data block. In this way, all allocated virtual addresses form a contiguous address range in the address space.

[0150] It should be noted that in the above contiguous allocation method, although the virtual addresses are contiguous, the physical addresses do not have to be contiguous. That is, the physical address allocated to each data block can be any currently free physical page, and does not need to be in the same order as the virtual addresses.

[0151] Step 603: Establish the address mapping relationship between the first data block, the first virtual address, and the data block size of the first data block.

[0152] The address mapping records the identifier (or content) of each data block, its corresponding virtual address, and its size. This address mapping can be stored in the electronic device's memory or in persistent storage. During snapshots, the address mapping is saved to the storage device along with the application cache data.

[0153] Step 604: Establish the address binding relationship between the first virtual address and the first physical address.

[0154] In this system, the electronic device can call the driver interface of the first computing device to establish an address binding relationship between a first virtual address and a first physical address by issuing a page table binding command to the MMU of the first computing device. The address binding relationship represents the mapping between the first virtual address and the first physical address, is stored in the page table of the first computing device, and is managed by the MMU of the first computing device. When a target application accesses a certain virtual address, the first computing device looks up the corresponding physical address through the page table to complete the address translation.

[0155] It should be noted that the address allocation and address binding operations described above are performed by the driver running on the CPU of the electronic device. The CPU sends control commands to the GPU via the bus, and the GPU's MMU configures the page table entries at the hardware level according to the commands, thereby realizing the mapping from virtual addresses to physical addresses.

[0156] In this embodiment, the address mapping relationship and the address binding relationship are correspondences at different levels: the address mapping relationship is the application-level address mapping management, used to record the correspondence between data blocks and virtual addresses, serving snapshots and recovery, while the address binding relationship is the hardware-level address binding, used for address translation, serving normal data access.

[0157] For example, the target application requests video memory three times in sequence: 1GB the first time, 2GB the second time, and 1.5GB the third time. In this embodiment, the first allocation is 0x1000~0x40000000 (1GB), the second allocation is 0x40000000~0xC0000000 (2GB), and the third allocation is 0xC0000000~0x180000000 (1.5GB). These three allocations are contiguous in the virtual address space, and the 4.5GB of data can be merged and transmitted at once during recovery.

[0158] As can be seen, through the above continuous allocation method, the virtual addresses corresponding to multiple memory requests are arranged continuously in the address space, which provides the prerequisite for merging and transmitting during subsequent snapshots.

[0159] In one specific implementation, the virtual address space of video memory may contain free address segments caused by video memory release. Therefore, in this embodiment, after intercepting a data storage request, the free address segments need to be considered when allocating virtual addresses according to the contiguous allocation method. Based on this, the address allocation process according to the contiguous allocation method is as follows: Figure 7 As shown:

[0160] Step 701: Check if there is a free address segment in the allocated address segments; if there is no free address segment in the allocated address segments, proceed to step 702; if there is a free address segment in the allocated address segments, proceed to step 703.

[0161] The allocated address segment consists of the virtual addresses allocated from the video memory of the first computing device; the free address segment consists of the virtual addresses corresponding to the data blocks released in response to the video memory release request. It should be noted that when a virtual address is released, the bound physical address is also released. Furthermore, if multiple adjacent free address segments exist within the allocated address segment, these adjacent free address segments are merged into a single free address segment. For example, assuming the allocated virtual address range is 0x1000~0x5000, and data blocks within the range 0x2000~0x3000 are released, then 0x2000~0x3000 becomes the free address segment.

[0162] Step 702: Allocate a first virtual address according to the data block size of the first data block.

[0163] The first virtual address is connected to the end of the allocated address segment. In other words, if there is no free address segment to reuse, a new first virtual address is allocated at the end of the allocated address segment according to the continuous allocation method.

[0164] Step 703: Allocate a first virtual address from the free address segment that matches the size of the first data block.

[0165] The first virtual address is the starting address of the free address segment.

[0166] As can be seen, in this embodiment, when there is a free address range, the virtual address in the free address range is reused first, so as to make full use of the allocated but released address space and avoid the infinite expansion of the virtual address space.

[0167] In one implementation, this embodiment may employ memory allocation algorithms such as first-fit or best-fit when allocating virtual addresses from free address segments. For example, this embodiment may use the best-fit algorithm to select the free address segment whose address length is closest to the size of the first data block from all free address segments for allocation. Alternatively, this embodiment may use the first-fit algorithm to allocate the first virtual address for the first data block starting from the first address segment among all free address segments.

[0168] It should be noted that when allocating virtual addresses from free address ranges, the continuity of virtual addresses should be maintained as much as possible. That is, if a free address range happens to meet the allocation requirements, the starting address of that free address range should be directly allocated as the starting address of the first virtual address.

[0169] As can be seen, this embodiment introduces a reuse scheme for free address segments on the basis of continuous virtual address allocation. While ensuring the continuity of virtual addresses, it effectively utilizes the free address segments generated by the release of video memory, thus avoiding the waste and unlimited growth of virtual address space.

[0170] Specifically, in step 703, when allocating a first virtual address from the free address segment that matches the size of the first data block, this can be achieved in the following way:

[0171] First, the system filters for free address segments whose address length is greater than or equal to the size of the first data block. If such a segment exists, a target address segment with the smallest address length that is greater than or equal to the size of the first data block is selected from the free address segments. In other words, the target address segment with the smallest address length is chosen from multiple free address segments with address lengths greater than or equal to the size of the first data block. Then, a first virtual address matching the size of the first data block is allocated from the target address segment. For example, starting from the beginning of the target address segment, a first virtual address with an address length equal to the size of the first data block is allocated. The remaining address space in the target address segment continues as a new free address segment for subsequent allocation. Therefore, this embodiment uses the best-fit algorithm to allocate a first virtual address to the first data block from free address segments. This avoids dividing large free address segments into smaller ones, reducing address fragmentation.

[0172] When the address lengths of all free address segments are smaller than the size of the first data block, a single free address segment cannot meet the allocation requirements. Therefore, in this embodiment, the first virtual address can be allocated from multiple free address segments according to the address length of the free address segments and the size of the first data block. In other words, the first data block can be divided into multiple sub-blocks, and multiple free address segments can be selected. Each free address segment is provided to its corresponding sub-block, and these free address segments form the first virtual address. The starting address of the first free address segment among the multiple free address segments is the starting address of the first virtual address. This allows for the combination and reuse of small free address segments and reduces address fragmentation.

[0173] In another implementation, when the address lengths of all free address segments are smaller than the data block size of the first data block, the fragmentation of the free address segments is high and they cannot be combined into contiguous address segments. In this case, the reuse of free address segments can be abandoned, and new virtual addresses can be allocated directly at the end of the allocated address segments.

[0174] It should be noted that in this embodiment, allocating the first virtual address within a free address segment maximizes address space utilization while ensuring virtual address continuity. However, in actual implementations, if severe fragmentation leads to decreased allocation efficiency, additional allocation can be made directly without considering reusing the free address segment. In specific scenarios, the appropriate method can be determined based on the actual situation, and this application does not impose any limitations on this.

[0175] As can be seen, in this embodiment, the best-fit algorithm is used to filter target address segments in the free address segments. When no single free address segment meets the requirements, multiple free address segments are flexibly combined for allocation or the allocation is appended to the end of the address segment. Under the premise of ensuring continuous allocation of virtual addresses, fragmentation in the address space is effectively managed, thereby improving the utilization rate of the address space.

[0176] refer to Figure 8 This diagram illustrates the implementation structure of a computing task recovery device according to an embodiment of this application. This device can be deployed in an electronic device, such as a server or laptop, that connects multiple computing devices to perform computing tasks such as artificial intelligence model inference. The technical solution in this embodiment primarily aims to improve the recovery efficiency of computing tasks within computing devices.

[0177] Specifically, the apparatus in this embodiment may include the following units:

[0178] The snapshot receiving unit 801 is configured to receive a task snapshot file sent by the storage device in response to a recovery command for the computing task;

[0179] The task snapshot file includes at least: application cache data and address mapping relationships; the application cache data is the business data stored in the video memory of the target application on the first computing device when executing the computing task; the address mapping relationship represents the correspondence between each data block in the application cache data and its virtual address and data block size;

[0180] The cache recovery unit 802 is used to transfer each data block in the application cache data to the corresponding virtual address in the video memory of the second computing device according to the address mapping relationship, so that the computing task of the target application can resume running on the second computing device.

[0181] Wherein, the virtual address corresponding to the data block in the video memory of the second computing device is consistent with the virtual address corresponding to the data block in the address mapping relationship.

[0182] As can be seen from the above technical solution, in the computing task recovery device provided in this application embodiment, the address mapping relationship in the task snapshot file is used to restore the data block in the application cache data to the same virtual address in the video memory of the target computing device according to the address mapping relationship. Thus, for the target application, the access address of the data block has not changed. Therefore, for the upper-layer target application, no code modification or address remapping operation is required to resume execution from the breakpoint of the computing task, realizing a fast recovery that is completely non-intrusive to the application layer, thereby reducing the recovery latency of the computing task and further improving the recovery efficiency of the computing task.

[0183] In one implementation, the virtual addresses in the address mapping relationship are addresses allocated in a continuous allocation manner, which at least includes: the starting address of the newly allocated virtual address is connected to the ending address of the already allocated virtual address;

[0184] Specifically, the cache recovery unit 802 is used to: merge multiple data blocks with consecutive virtual addresses in the application cache data and transmit them to the corresponding virtual addresses in the video memory of the second computing device. For example, if there is only one computing device in the computing node where the second computing device is located, the multiple data blocks with consecutive virtual addresses in the application cache data are merged and transmitted to the corresponding virtual addresses in the video memory of the second computing device; if there are multiple computing devices in the computing node where the second computing device is located, the application cache data is divided into multiple cache data sets; each cache data set corresponds to one computing device in the computing node; and the multiple data blocks with consecutive virtual addresses in each cache data set are merged and transmitted to a third computing device in the computing node, so that the third computing device sends the cache data set to the corresponding virtual addresses in the video memory of the second computing device through a high-speed interconnect channel; wherein, the high-speed interconnect channel is a data transmission channel between the computing devices.

[0185] Specifically, when the cache recovery unit 802 merges and transmits multiple data blocks with consecutive virtual addresses in the cache data set to the third computing device in the computing node, it is specifically used to: divide the cache data set into multiple data subsets when the data size of the cache data set exceeds a target threshold; and merge and transmit multiple data blocks with consecutive virtual addresses in each data subset to the third computing device in the computing node in a parallel manner.

[0186] In one implementation, the task snapshot file further includes: runtime state data for maintaining the computing task's operating environment;

[0187] The apparatus in this embodiment may further include the following units, such as... Figure 9 As shown:

[0188] The runtime recovery unit 803 is used to transfer the runtime state data in the task snapshot file to the video memory of the second computing device using a target tool provided by the driver of the second computing device.

[0189] Additionally, the apparatus in this embodiment may further include a snapshot acquisition unit 804 for obtaining task snapshot files, such as... Figure 10As shown, the snapshot acquisition unit 804 is specifically used for: responding to a snapshot instruction for the computing task, saving the application cache data in the video memory of the first computing device according to the address mapping relationship to obtain a first snapshot file; releasing the application cache data in the video memory of the first computing device; and saving the runtime state data in the video memory of the first computing device to obtain a second snapshot file; wherein the first snapshot file and the second snapshot file constitute a task snapshot file.

[0190] In one implementation, the physical address corresponding to the data block in the video memory of the second computing device is different from the physical address corresponding to the data block in the video memory of the first computing device;

[0191] The apparatus in this embodiment may further include the following units, such as... Figure 11 As shown:

[0192] The binding update unit 805 is used to update the address binding relationship corresponding to the data block according to the virtual address and physical address corresponding to the data block in the video memory of the second computing device; wherein, the address binding relationship represents the correspondence between the virtual address and physical address corresponding to the data block; the address binding relationship is used to access the data block.

[0193] In one implementation, the apparatus in this embodiment may further include a data storage unit 806 for storing data blocks from the application cache data into the video memory of the first computing device, such as... Figure 12 As shown, the data storage unit 806 is specifically used for: intercepting data storage requests for the video memory of the first computing device; the data storage request at least includes the data block size of the first data block to be stored; allocating a first virtual address and a first physical address for the first data block in the video memory of the first computing device according to the contiguous allocation method and the data block size of the first data block; connecting the starting address of the first virtual address to the ending address of the allocated virtual address; establishing an address mapping relationship between the first data block and the first virtual address and the data block size of the first data block; and establishing an address binding relationship between the first virtual address and the first physical address.

[0194] The contiguous allocation method used by data storage unit 806 specifically includes:

[0195] Check if there is a free address segment in the allocated address segment; the allocated address segment is an address segment consisting of virtual addresses allocated from the video memory of the first computing device; the free address segment is an address segment consisting of virtual addresses corresponding to data blocks released in response to a video memory release request;

[0196] If there is no free address segment in the allocated address segment, a first virtual address is allocated according to the data block size of the first data block; the starting address of the first virtual address is connected to the ending address of the allocated address segment;

[0197] If a free address segment exists within the allocated address segments, a first virtual address matching the block size of the first data block is allocated from the free address segments; the starting address of the first virtual address is the starting address of the free address segment. For example, if the address length of a free address segment is greater than or equal to the block size of the first data block, a target address segment with the smallest address length that is greater than or equal to the block size of the first data block is selected from the free address segments; a first virtual address matching the block size of the first data block is allocated from the target address segment; if the address lengths of all free address segments are less than the block size of the first data block, a first virtual address is allocated from multiple free address segments according to the address lengths of the free address segments and the block size of the first data block.

[0198] It should be noted that the specific implementation of each unit in this embodiment can be referred to the corresponding technical content above, and will not be described in detail here.

[0199] refer to Figure 13 This is a schematic diagram of an electronic device provided in an embodiment of this application. This electronic device can be one that connects multiple computing devices to perform computational tasks such as artificial intelligence model inference, such as a server or laptop. The technical solution in this embodiment is mainly used to improve the recovery efficiency of computational tasks within the computing devices.

[0200] Specifically, the electronic device in this embodiment may include the following structure:

[0201] Storage device 1301 is used to store task snapshot files; wherein, the task snapshot files include at least: application cache data and address mapping relationships; the application cache data is business data stored in the video memory of the target application on the first computing device when executing the computing task; the address mapping relationship represents the correspondence between each data block in the application cache data and its virtual address and data block size;

[0202] Processor 1302 is configured to, in response to a recovery instruction for the computing task, receive a task snapshot file sent by the storage device; and, according to the address mapping relationship, transfer each data block in the application cache data to the corresponding virtual address in the video memory of the second computing device, so that the computing task of the target application resumes operation on the second computing device.

[0203] Wherein, the virtual address corresponding to the data block in the video memory of the second computing device is consistent with the virtual address corresponding to the data block in the address mapping relationship.

[0204] As can be seen from the above technical solution, in the electronic device provided by the embodiments of this application, the address mapping relationship in the task snapshot file is used to restore the data block in the application cache data to the same virtual address in the video memory of the target computing device according to the address mapping relationship. Thus, for the target application, the access address of the data block does not change. Therefore, for the target application at the upper layer, no code modification or address remapping operation is required to resume execution from the breakpoint of the computing task. This achieves fast recovery without any intrusion on the application layer, thereby reducing the recovery latency of the computing task and further improving the recovery efficiency of the computing task.

[0205] Taking a GPU application as an example, the technical solution of this application is illustrated below:

[0206] First, this application proposes an efficient method for rapid recovery of GPU applications. Through core innovative designs such as a two-stage recovery architecture, decoupling of virtual and physical memory addresses, continuous aggregation of discrete memory, adaptive multi-path parallel transmission, and multi-stream concurrent transmission, the method comprehensively optimizes the GPU checkpoint and recovery process. Under the premise of full compatibility with upper-layer business and non-intrusiveness to applications, it significantly improves the recovery efficiency and stability of GPU services.

[0207] The core points of this application are as follows:

[0208] 1. Two-Phase Layered Checkpoint and Recovery Architecture. This application splits the recovery content into two parts: GPU runtime state data and business cache data (i.e., application cache data mentioned above), and adopts a layered differentiated recovery strategy. For GPU runtime state data (such as runtime context), this application reuses an efficient driver-integrated recovery mechanism, avoiding the cumbersome API interception, serialization storage, and API replay processes of traditional interception solutions, and quickly rebuilds the underlying GPU runtime environment. This allows the recovered business (i.e., the computing tasks of the GPU application) to resume execution from the checkpoint breakpoint, significantly shortening the basic recovery time. For business cache data, which occupies the largest amount of GPU memory, this application adopts a lightweight GPU memory allocation interception mechanism, uniformly managing GPU memory content and addresses in user space, independently ensuring the address consistency and transmission efficiency of cache data, and achieving decoupling optimization between efficient runtime state recovery and high-speed data transmission.

[0209] 2. A method of decoupling virtual and physical addresses is adopted. After intercepting a video memory request (i.e., a data storage request), a fixed virtual address is allocated for it, and then a physical address is allocated. The physical address is then bound to the virtual address, and the virtual address is returned to the GPU application. Simultaneously, the virtual address and data block size allocated each time are recorded. During each recovery, video memory is allocated strictly according to the recorded virtual address and data block size.

[0210] 3. When allocating virtual addresses, try to allocate multiple requests in a contiguous address space. During snapshot / restore, multiple memory copies can be merged into a single memory copy, greatly improving transmission efficiency.

[0211] 4. Adaptive Multi-Path Parallel Transmission Mechanism Based on GPU Topology: This application automatically detects the number of GPUs and the hardware interconnect topology in the computing node. When the number of GPUs in the recovery node is greater than 1 and a high-speed interconnect environment such as NVLink is available, the data transmission method will be automatically split into multi-channel parallel transmission. That is, the data will be split into the same number of parts as the number of GPUs. One part will be copied directly from the host memory to the GPU via PCIe, while the other parts will be transmitted to different GPUs, and then transmitted to the recovery GPU via NVLink. Through concurrent multi-path transmission, the PCIe bandwidth limitation is overcome, and the transmission efficiency is improved.

[0212] 5. During the GPU checkpointing and recovery phases, no business computing tasks are executed in the system, and there are no data read / write synchronization conflicts or resource contention issues. Each transmission can be further divided into fine-grained blocks, and an asynchronous parallel transmission queue can be built using multiple streams to fully utilize hardware transmission bandwidth resources, maximize parallelism, and further improve the transmission rate of GPU memory data snapshots and recovery.

[0213] like Figure 14 The diagram shown is an example of the snapshot process provided in this application. The specific process is as follows:

[0214] 1. Implement a low-level video memory access interception module in the electronic device to perform lightweight interception of all video memory allocation, release, and read / write related CUDA interfaces of the application-side APP. Specifically, this application can preload the interception management module into the process via LD_PRELOAD.

[0215] 2. After the CPU of the electronic device intercepts a memory allocation request for GPU1, it splits the memory allocation request into two parts: a virtual address allocation and a physical address allocation. The CPU then binds the results of these two allocations and returns the virtual address to the application (i.e., the GPU application). Specifically, the CPU calls the GPU driver interface to first allocate a virtual address (reserving the virtual address), then allocate a physical address, and finally bind the physical address to the virtual address.

[0216] 3. Each time the CPU requests a data block, it records the virtual address and the size of the data block, forming an address mapping relationship.

[0217] 4. The starting address of each virtual address requested by the CPU is specified as the byte following the last byte of the previously requested virtual address.

[0218] 5. During each snapshot, the CPU must not only save the contents of the allocated video memory, but also save the address mapping relationship between the video memory virtual address and the size of each data block.

[0219] 6. When taking a snapshot, the CPU first saves the application cache data and the corresponding virtual address. After saving, the application cache data is released. At this point, only the GPU's runtime state data remains in the video memory.

[0220] 7. Next, the electronic device takes a snapshot of the GPU's runtime state data. At this time, the checkpointing tool integrated into the driver (such as NVIDIA cuda-checkpoint) is used directly to take the snapshot.

[0221] 8. After the snapshot process is complete, two snapshot files can be obtained: one corresponds to the GPU runtime state data (i.e., the second snapshot file), and the other corresponds to the application cache data and address mapping relationship (i.e., the first snapshot file).

[0222] like Figure 15 The diagram shown is an example of the recovery process provided in this application. The specific process is as follows:

[0223] 1. During the recovery process, the CPU first recovers the GPU runtime state data and directly uses the checkpointing tool integrated in the driver (such as NVIDIA cuda-checkpoint) to transfer the runtime state data to the video memory of the second computing device (GPU2).

[0224] 2. CPU restores application cache data. First, check the GPU topology. If the number of GPUs on the node is N, and N is greater than 1 and the GPUs are connected via high-speed connections such as NVLink, then the application cache data is divided into N equal parts. One part is directly transferred from the host to GPU2 via PCIe, and the remaining N-1 parts are divided and transferred to the remaining N-1 other GPUs, such as GPU0, GPU1, and GPU3. Then, these GPUs use NVLink to aggregate the received data back to GPU2. It should be noted that the above transmissions occur simultaneously.

[0225] 3. Specific steps of data transfer: The CPU first reads the mapping table (i.e., address mapping) between application cache data and virtual addresses, calls the GPU2 driver interface, and requests a virtual address based on the virtual address and data block size recorded in the table, ensuring consistency with the virtual address in the mapping table. Then, it requests a physical address and binds the physical address and virtual address by issuing a page table update command to the GPU's MMU. Finally, the CPU copies the corresponding data to the corresponding video memory via DMA.

[0226] 4. When the transmitted data block exceeds 5GB (i.e., the target threshold), the data block is further divided into equal parts. If each split data block is still larger than 5GB, the splitting continues until the condition (less than or equal to 5GB) is met. A CUDA stream is created for each split data block and transmitted synchronously.

[0227] 5. Once the GPU runtime state data and application cache data have been restored, resume GPU application execution, and the entire recovery process is complete.

[0228] like Figure 16 The diagram illustrates the continuous allocation of virtual addresses corresponding to three memory allocation requests in this embodiment. The CPU of the electronic device allocates a virtual address starting at the byte following the last byte of the previously allocated virtual address. Specifically, the first allocation assigns virtual address segment A (from V1 to V2), the second allocation assigns a virtual address starting at the byte following V2, forming virtual address segment B (from V2+1 to V3), and the third allocation assigns a virtual address starting at the byte following V3, forming virtual address segment C (from V3+1 to V4). The three allocated virtual addresses are arranged consecutively in the address space, allowing data segments A, B, and C to be merged into a single transmission during snapshot / restore.

[0229] It is evident that the technical solution of this application has the following advantages:

[0230] 1. It adopts a two-stage snapshot / restore approach, eliminating the need for time-consuming operations such as API replay on runtime state data, thus greatly improving recovery efficiency.

[0231] 2. Through multi-level data transmission optimization such as virtual and physical memory decoupling, discrete memory aggregation, multi-path topology transmission, and multi-stream concurrent acceleration, the pain points of low PCIe bandwidth utilization and slow data recovery of large memory cache in traditional driver solutions are completely solved, greatly improving the snapshot and recovery efficiency of large model and large memory services.

[0232] 3. The overall technical solution is fully deployed at the underlying resource management layer, completely decoupled from the upper-layer inference framework, business code, and model structure. No application modifications are required. It features zero intrusion, strong versatility, and wide adaptability, and can seamlessly adapt to various GPU-accelerated computing services.

[0233] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0234] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0235] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0236] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for recovering a computational task, comprising: In response to a recovery command for the computing task, the system receives a task snapshot file sent by the storage device. The task snapshot file includes at least: application cache data and address mapping relationships; the application cache data is the business data stored in the video memory of the target application on the first computing device when executing the computing task; the address mapping relationship represents the correspondence between each data block in the application cache data and its virtual address and data block size; According to the address mapping relationship, each data block in the application cache data is transferred to the corresponding virtual address in the video memory of the second computing device, so that the computing task of the target application can resume running on the second computing device; Wherein, the virtual address corresponding to the data block in the video memory of the second computing device is consistent with the virtual address corresponding to the data block in the address mapping relationship.

2. The method according to claim 1, wherein the virtual address in the address mapping relationship is an address allocated according to a contiguous allocation method, the contiguous allocation method including at least: The starting address of the newly allocated virtual address is connected to the ending address of the already allocated virtual address; The process of transferring each data block in the application cache data to the corresponding virtual address in the video memory of the second computing device includes: Multiple data blocks with consecutive virtual addresses in the application cache data are merged and transmitted to the corresponding virtual addresses in the video memory of the second computing device.

3. The method according to claim 2, wherein multiple data blocks with consecutive virtual addresses in the application cache data are merged and transmitted to corresponding virtual addresses in the video memory of the second computing device, comprising: In the case where there is one computing device in the computing node where the second computing device is located, multiple data blocks with consecutive virtual addresses in the application cache data are merged and transmitted to the corresponding virtual addresses in the video memory of the second computing device; In the case where there are multiple computing devices on the computing node where the second computing device is located, the application cache data is divided into multiple cache data sets; Each of the cached data sets corresponds to a computing device in the computing node; Multiple data blocks with consecutive virtual addresses in each of the cached data sets are merged and transmitted to the third computing device in the computing node, so that the third computing device sends the cached data sets to the corresponding virtual addresses in the video memory of the second computing device through a high-speed interconnect channel. The high-speed interconnect channel is the data transmission channel between the computing devices.

4. The method according to claim 3, wherein multiple data blocks with consecutive virtual addresses in the cached data set are merged and transmitted to a third computing device in the computing node, comprising: If the size of the cached data set exceeds the target threshold, the cached data set will be divided into multiple data subsets. In a parallel manner, multiple data blocks with consecutive virtual addresses in each of the data subsets are merged and transmitted to a third computing device in the computing node.

5. The method according to claim 1 or 2, wherein the task snapshot file further comprises: Runtime state data used to maintain the operating environment of the computing task; The method further includes: Using the target tools provided by the driver of the second computing device, the runtime state data in the task snapshot file is transferred to the video memory of the second computing device.

6. The method according to claim 5, wherein the task snapshot file is obtained in the following manner: In response to a snapshot instruction for the computing task, the application cache data in the video memory of the first computing device is saved according to the address mapping relationship to obtain a first snapshot file; Release the application cache data in the video memory of the first computing device; The runtime state data in the video memory of the first computing device is saved to obtain a second snapshot file; The first snapshot file and the second snapshot file together form the task snapshot file.

7. The method according to claim 1, wherein the physical address corresponding to the data block in the video memory of the second computing device is different from the physical address corresponding to the data block in the video memory of the first computing device; in, The method further includes: Update the address binding relationship of the data block according to the virtual address and physical address of the data block in the video memory of the second computing device; The address binding relationship represents the correspondence between the virtual address and the physical address corresponding to the data block; the address binding relationship is used to access the data block.

8. The method according to claim 1, wherein the data blocks in the application cache data are stored in the video memory of the first computing device in the following manner: Intercept data storage requests for the video memory of the first computing device; the data storage request contains at least the data block size of the first data block to be stored; According to the contiguous allocation method and the size of the first data block, a first virtual address and a first physical address are allocated for the first data block in the video memory of the first computing device; the starting address of the first virtual address is connected to the ending address of the allocated virtual address; Establish an address mapping relationship between the first data block, the first virtual address, and the data block size of the first data block; Establish an address binding relationship between the first virtual address and the first physical address.

9. The method according to claim 8, wherein the continuous allocation method includes: Check if there is a free address segment in the allocated address range; The allocated address segment is an address segment consisting of virtual addresses allocated from the video memory of the first computing device; The free address segment is an address segment consisting of the virtual addresses corresponding to the data blocks released in response to the video memory release request; If there is no free address segment in the allocated address segment, allocate a first virtual address according to the data block size of the first data block; The starting address of the first virtual address is connected to the ending address of the allocated address segment; If a free address segment exists within the allocated address segment, a first virtual address matching the size of the first data block is allocated from the free address segment; the starting address of the first virtual address is the starting address of the free address segment.

10. An electronic device, comprising: A storage device is used to store task snapshot files; wherein, the task snapshot files include at least: application cache data and address mapping relationships; the application cache data is business data stored in the video memory of the target application on the first computing device when executing the computing task; the address mapping relationship represents the correspondence between each data block in the application cache data and its virtual address and data block size; The processor is configured to, in response to a recovery instruction for the computing task, receive a task snapshot file sent by the storage device; and, according to the address mapping relationship, transfer each data block in the application cache data to the corresponding virtual address in the video memory of the second computing device, so that the computing task of the target application can resume running on the second computing device. Wherein, the virtual address corresponding to the data block in the video memory of the second computing device is consistent with the virtual address corresponding to the data block in the address mapping relationship.