Heterogeneous computing power instance scheduling and state management method, device, medium and chip

Abstract

Embodiments of the present application relate to the field of cloud computing, and provide a heterogeneous computing power instance scheduling and state management method, device, medium and chip, comprising: receiving a state change request of a target heterogeneous computing power instance; if it is a stop request, triggering a state saving process corresponding to the target heterogeneous computing power instance to perform a snapshot operation on a root file system of the target heterogeneous computing power instance before releasing heterogeneous computing power resources, and recording snapshot metadata corresponding to the snapshot operation; if it is a start request, determining a recovery node based on the snapshot metadata, performing dynamic environment reconstruction and recovering the root file system, and after the root file system is recovered, allocating heterogeneous computing power resources to the target heterogeneous computing power instance and starting the target heterogeneous computing power instance. The present application can reduce the cost of idle computing power and the cost of environment reconstruction, realize on-demand start and stop, rapid recovery and environment consistency reconstruction, and reduce resource fragmentation, recovery failure probability and operation and maintenance scheduling overhead.

Description

Technical Field

[0001] This application relates to the field of cloud computing, and more specifically to a method, apparatus, medium, and chip for scheduling and managing heterogeneous computing power instances. Background Technology

[0002] With the continuous development of AI training, inference, and interactive development, cloud services for heterogeneous computing resources are gradually becoming an important infrastructure capability. These services typically rely on multiple runtime elements, including driver environments, compilers, deep learning frameworks, dependency libraries, and user code, resulting in a complex overall environment and high deployment and rebuild costs. Simultaneously, heterogeneous computing resources are expensive, and users generally prefer to release computing resources during non-computing periods to reduce costs and restore the previous runtime environment and business state when needed again.

[0003] In existing technologies, one type of solution relies on container orchestration platforms for unified management of heterogeneous computing resources. This type of solution typically deploys and runs computing tasks through resource declaration and scheduling mechanisms, quantifies and allocates accelerator card resources using device plugins, and externally stores user data through persistent volumes. However, because the instance management mechanism of container orchestration platforms is usually stateless, after an instance is stopped or deleted, it often only retains a portion of the data in the external data volume, making it difficult to persist the container system disk. This results in the loss of container dependencies, runtime configurations, and context information when the instance is destroyed. Users then need to re-pull the image and reinstall dependencies upon restarting, impacting efficiency and business continuity.

[0004] Another approach uses virtualization technology to suspend, snapshot, and restore computing instances. This approach can achieve system-level state preservation to a certain extent, but the virtualization layer introduces additional resource overhead and performance loss, and the startup and recovery process is usually slow, making it difficult to meet the application requirements of high-frequency start-stop, elastic scheduling, and efficient reuse of heterogeneous computing resources.

[0005] In addition, existing AI task scheduling systems are mostly geared towards batch processing scenarios, and typically aim to release resources as soon as the task is completed. They lack effective support for interactive, long-cycle, and pauseable / recoverable development environments, making it difficult to balance the need for on-demand release of computing power with the need to maintain continuous operation. Summary of the Invention

[0006] This application provides a method, apparatus, medium, and chip for scheduling and managing heterogeneous computing instances, enabling coordinated control of state preservation and resource release for heterogeneous computing instances. This achieves the technical effects of rapid recovery, maintaining environmental consistency, and flexible scheduling, further reducing the cost of idle computing power, minimizing environmental reconstruction overhead, and improving the continuity and stability of heterogeneous computing services. It avoids the problems of system disk state loss, difficulty in rebuilding the operating environment, long recovery time, and low utilization of computing resources in the prior art after container instances stop.

[0007] In a first aspect, embodiments of this application provide a method for scheduling and managing the state of heterogeneous computing power instances, the method comprising: Receive a state change request for a target heterogeneous computing power instance, the state change request including at least a stop request and a start request; If the state change request is a stop request, the state saving process corresponding to the target heterogeneous computing power instance is triggered to perform a snapshot operation on the root file system of the target heterogeneous computing power instance before releasing the heterogeneous computing power resources, and record the snapshot metadata corresponding to the snapshot operation. If the status change request is a startup request, the target node for restoring the target heterogeneous computing instance is determined based on the snapshot metadata. Perform dynamic environment reconstruction on the target node and restore the root file system of the target heterogeneous computing instance based on the snapshot metadata; After the root file system is restored, heterogeneous computing resources are allocated to the target heterogeneous computing instance and the target heterogeneous computing instance is started.

[0008] Secondly, embodiments of this application provide a heterogeneous computing instance scheduling and state management device, which has the function of implementing the heterogeneous computing instance scheduling and state management method corresponding to the first aspect described above. The function can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above function, and the modules can be software and / or hardware.

[0009] In one embodiment, the heterogeneous computing instance scheduling and state management device includes: The input / output module is configured to receive state change requests for a target heterogeneous computing power instance, the state change requests including at least a stop request and a start request; The processing module is configured to, if the state change request is a stop request, trigger a state saving process corresponding to the target heterogeneous computing instance, to perform a snapshot operation on the root file system of the target heterogeneous computing instance before releasing the heterogeneous computing resources, and record snapshot metadata corresponding to the snapshot operation; if the state change request is a start request, determine a target node for restoring the target heterogeneous computing instance based on the snapshot metadata; perform dynamic environment reconstruction on the target node, and restore the root file system of the target heterogeneous computing instance based on the snapshot metadata; after the root file system is restored, allocate heterogeneous computing resources to the target heterogeneous computing instance and start the target heterogeneous computing instance.

[0010] Thirdly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the heterogeneous computing instance scheduling and state management method as described in the first aspect.

[0011] Fourthly, embodiments of this application provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the heterogeneous computing instance scheduling and state management method described in the first aspect.

[0012] Fifthly, embodiments of this application provide a chip including a processor coupled to a transceiver for executing the technical solution provided in the first aspect of this application. In one possible design, the chip can also be a dedicated hardware structure for implementing the technical solution provided in the first aspect above; for example, processing involving neural network models can be implemented by a dedicated neural network processor or a graphics processor.

[0013] Compared to existing technologies, this application embodiment performs a snapshot operation on the root file system and records snapshot metadata when the target heterogeneous computing instance stops. When the target heterogeneous computing instance starts, a recovery node is determined based on the snapshot metadata, and the root file system is restored after dynamic environment reconstruction is performed on the recovery node. Because this application embodiment collaboratively saves and restores the root file system state and runtime context of the target heterogeneous computing instance, rather than simply persisting external data volumes or relying on the entire virtual machine to suspend for recovery as in existing technologies, this application embodiment can retain the instance's running state while releasing heterogeneous computing resources, and can quickly restore the instance's runtime environment after subsequent resource reallocation.

[0014] As can be seen, in this embodiment, the release process of heterogeneous computing resources is decoupled from the instance state maintenance process. Since the root file system snapshot and snapshot metadata jointly represent the environment state when the instance stops, and the recovery phase further combines dynamic environment reconstruction to rebuild the running configurations such as network storage mounting, memory parameters, and network ports, the recovered instance obtained in this embodiment can not only maintain the historical state at the file system level, but also maintain the configuration consistency at the runtime environment level. This enables environment consistency recovery and effectively avoids the problems of long rebuilding time, error-prone configuration, and inconsistent running state after recovery in the prior art.

[0015] Furthermore, compared to existing technologies where the system disk state is lost upon instance deletion, this application embodiment guides the selection of recovery nodes through snapshot metadata, enabling the system to perform subsequent recovery operations based on the saved state. This reduces the idle cost of heterogeneous computing resources and the cost of rebuilding the user environment. Since this application embodiment supports releasing heterogeneous computing resources when the instance is not running and restoring the original instance state when needed, the resource usage method obtained by this application embodiment can achieve on-demand start / stop and elastic reuse, thereby improving the utilization rate of heterogeneous computing resources. Attached Figure Description

[0016] The objectives, features, and advantages of the embodiments of this application will become readily understood by referring to the accompanying drawings and the detailed description of the embodiments. Wherein: Figure 1 This is a flowchart illustrating the heterogeneous computing instance scheduling and state management method in the embodiments of this application; Figure 2 This is an interactive diagram illustrating the heterogeneous computing power instance scheduling and state management method according to an embodiment of this application; Figure 3 This application provides a schematic diagram of the structure of a heterogeneous computing instance scheduling and state management device.

[0017] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0018] This application provides a method, apparatus, medium, and chip for scheduling and managing heterogeneous computing instances, applicable to heterogeneous computing service systems in cloud computing scenarios requiring state maintenance, resource release, and rapid recovery of heterogeneous computing instances. The heterogeneous computing service system may include a user service interface module, a persistent instance control module, a resource scheduling module, a root file system persistence and state management module, and a runtime interface module. These modules can be deployed integratedly or separately. The user service interface module is at least used to receive user requests for creating, starting, stopping, and deleting target heterogeneous computing instances. The persistent instance control module is used to respond to state change requests and coordinate state saving, state recovery, and instance lifecycle management. The resource scheduling module is used to determine recovery nodes based on instance specifications, node health status, resource fragmentation level, and topology matching. The root file system persistence and state management module is used to perform snapshots, storage, and recovery on the root file system of the target heterogeneous computing instance. The runtime interface module is used to allocate heterogeneous computing resources to containers and enable the runtime environment when the instance starts.

[0019] The solutions provided in this application involve technologies such as cloud computing, container orchestration, heterogeneous computing resource management, and artificial intelligence training and inference support, which are specifically illustrated through the following embodiments.

[0020] Cloud computing technology provides unified resource pooling, elastic scheduling, and service-oriented delivery capabilities for heterogeneous computing resources. Container orchestration technology can be used for unified management of instance deployment, scheduling, scaling, and lifecycle. Heterogeneous computing resource management technology can be used for the discovery, allocation, and isolation of acceleration resources such as TPUs, GPUs, and NPUs. Artificial intelligence-related technologies typically rely on multiple runtime elements, including drivers, compilation environments, deep learning frameworks, dependency libraries, and user code. Therefore, maintaining a complete runtime environment after an instance stops and quickly rebuilding the context when an instance resumes are critical issues affecting system availability and user experience.

[0021] In existing technologies, stateless instance management based on container orchestration platforms is commonly used to schedule heterogeneous computing resources. This approach relies on the fact that standard container orchestration frameworks already possess resource scheduling, service discovery, and device plugin capabilities, facilitating the rapid construction of a unified heterogeneous computing service platform. Therefore, instance operation and data preservation are typically achieved through resource declarations, device plugins, and persistent volumes. However, this method often only retains data in external data volumes, making it difficult to preserve the container system disk and its driver environment, dependency libraries, runtime configurations, and user context. This results in the loss of environment state after the instance stops, requiring the image to be pulled again and dependencies to be reinstalled during subsequent recovery. Another existing technology uses virtualization for whole-machine snapshots and recovery. This approach can achieve relatively complete state preservation, but due to the high I / O overhead, computational loss, and startup latency introduced by the virtualization layer, it is difficult to meet the application requirements of high-frequency start / stop, elastic reuse, and rapid recovery of heterogeneous computing instances. In addition, existing heterogeneous computing power scheduling is usually based solely on the number of resources, lacking a comprehensive understanding of device interconnection topology, node health status, and resource fragmentation. This makes it difficult to ensure consistent topology performance when instances are restarted or restored, and also makes it difficult to avoid problems such as scheduling failures of large-scale tasks and decreased resource utilization.

[0022] Compared to existing technologies, this application embodiment receives a state change request for a target heterogeneous computing instance, triggers a state preservation process upon a stop request, and performs a snapshot operation on the root file system of the target heterogeneous computing instance before releasing heterogeneous computing resources. Upon a start request, a recovery node is determined based on the snapshot metadata, and dynamic environment reconstruction and root file system recovery are performed on the recovery node. This achieves coordinated control of heterogeneous computing resource release and instance state preservation. Since this application embodiment does not merely persist external data volumes or rely on the entire virtual machine to suspend and recover as in existing technologies, but rather coordinates the preservation and recovery of the root file system state, snapshot metadata, and runtime context of the target heterogeneous computing instance, it can retain the instance's runtime environment while releasing heterogeneous computing resources and quickly restore the instance state after subsequent resource reallocation. Furthermore, by combining snapshot metadata to determine the target node during the recovery phase and performing dynamic environment reconstruction on the target node, the recovered instance can simultaneously obtain consistency in file system state and runtime environment configuration, thus effectively solving the problems of complex environment reconstruction, low recovery efficiency, unstable performance after recovery, and low resource utilization in existing technologies.

[0023] In some implementations, the aforementioned heterogeneous computing service system can be implemented by separating the control plane and the compute plane. The heterogeneous computing instance scheduling and state management method provided in this application can be implemented based on a heterogeneous computing service system. This heterogeneous computing service system may include control plane nodes and compute plane nodes. The control plane node may deploy a user service interface module, a persistent instance control module, and a resource scheduling module. The compute plane node may deploy a root file system persistence and state management module and a runtime interface module.

[0024] The control plane node can receive user-submitted heterogeneous computing instance state change requests. Upon receiving a stop request, it coordinates the compute plane nodes to perform a snapshot operation on the root file system of the target heterogeneous computing instance, recording snapshot metadata and releasing the corresponding heterogeneous computing resources after the state is saved. Upon receiving a start request, the control plane node can determine the target node based on snapshot metadata, candidate node health status, resource status, and topology matching, and send a recovery command to the target node. After receiving the recovery command, the target node can restore the root file system of the target heterogeneous computing instance based on the snapshot metadata and perform dynamic environment reconstruction based on the user's latest configuration to rebuild context information such as network disk mounting, shared memory, large page memory, and service port mapping. After completing the recovery, it allocates heterogeneous computing resources to the target heterogeneous computing instance and starts the instance.

[0025] It should be noted that the computing devices involved in the embodiments of this application can be servers and / or terminal devices. The server can be an independent physical server, a server cluster, a distributed system, or a cloud server providing cloud services, cloud computing, cloud storage, network services, middleware services, and artificial intelligence platform capabilities. The terminal device can be a user device used to initiate instance management requests, view instance status, or access instance services, such as a personal computer, workstation, mobile terminal, or other device with network communication capabilities. The embodiments of this application are not limited to a specific chip architecture and are applicable to various AI accelerator cards such as TPUs, GPUs, and NPUs that support PCIe interfaces. Whether it is a virtualization device supporting SR-IOV / MIG or a standard physical dedicated device, both can serve as the underlying computing resources of this system.

[0026] Reference Figure 1 , Figure 1 This is a flowchart illustrating a heterogeneous computing instance scheduling and state management method provided in an embodiment of this application. The method can be executed by a heterogeneous computing instance scheduling and state management device, which can be deployed in a cloud computing platform, container orchestration platform, or heterogeneous computing service platform. It is suitable for scenarios requiring state maintenance, resource release, and rapid recovery of heterogeneous computing instances.

[0027] Container orchestration engines are used to deploy, scale, and manage containerized applications. An orchestration engine is deployed in a collection of physical or virtual machines called a cluster. A cluster consists of a control plane and nodes. The control plane is responsible for managing the state and configuration of the entire cluster. The control plane includes at least an Application Programming Interface (API) server, a scheduler, and a key-value store. The API server is the front end of the control plane, providing the entry point for cluster operations and handling data validation and configuration. The scheduler allocates unassigned Pods to appropriate nodes. The key-value store stores all cluster data.

[0028] For example, the container orchestration engine can be Kubernetes. The container orchestration engine is typically deployed in a cluster consisting of multiple physical machines and / or virtual machines, the cluster including a control plane and multiple worker nodes. The control plane maintains the global state, resource information, and scheduling policies of the cluster, and manages various resource objects within the cluster in a unified manner. The control plane may include at least an API server, a scheduler, and a key-value storage system. The API server serves as the unified access point for the cluster, receiving resource creation, startup, shutdown, deletion, and configuration update requests submitted by users or upper-layer business systems, and performing authentication, verification, object persistence, and configuration distribution on the requests. The scheduler performs node filtering and node scoring for Pods to be run, thereby allocating Pods to target nodes that meet resource requirements. The key-value storage system persistently stores cluster resource objects, node states, scheduling-related data, and instance status information.

[0029] Furthermore, to adapt to the state maintenance and rapid recovery requirements of heterogeneous computing instances, the control plane can also deploy extended components corresponding to the heterogeneous computing instance scheduling and state management device. These extended components may include a heterogeneous computing resource and scheduling extension module, a persistent instance controller, and a state management component. Specifically, the heterogeneous computing resource and scheduling extension module can work in conjunction with the scheduler to perform resource pre-screening, fault label filtering, bin-based resource allocation, and topology matching scoring on candidate nodes during the scheduling phase; the persistent instance controller can coordinate the state saving process when an instance stops and the recovery process when an instance starts, based on user-submitted target state change requests; and the state management component is used to perform snapshots, recovery, and version management on the instance root file system.

[0030] Each node in the cluster is used to collaboratively run and manage containerized applications. Any node can be a physical server or a virtual machine. Each node can run a node agent, which communicates with the API server and manages the running status of Pods and containers on its node according to instructions issued by the control plane. Nodes can also deploy container runtimes, heterogeneous computing device plugins, and lightweight heterogeneous computing runtime interface modules. The device plugins report available heterogeneous computing resources to the container orchestration engine. The lightweight heterogeneous computing runtime interface module transparently transmits the target heterogeneous computing device to the container when the Pod starts, and performs device cleanup, state reset, and resource reclamation operations when the instance stops or the container is destroyed, thereby improving security and resource reuse capabilities in multi-tenant scenarios.

[0031] In some implementations, one or more Pods can run on a node. A Pod is the smallest deployable and schedulable unit in a container orchestration engine, and a Pod can contain one or more containers. For the heterogeneous computing instances involved in this application, in addition to the business containers, auxiliary containers co-located with the business containers can also be deployed in the Pod. The auxiliary containers can cooperate with lifecycle hooks to perform root file system snapshot operations during the instance shutdown phase and cooperate to complete snapshot restoration during the instance startup phase, thereby supporting the heterogeneous computing instances to maintain their running environment state after releasing heterogeneous computing resources and to achieve rapid recovery when heterogeneous computing resources are subsequently reallocated.

[0032] The following describes a method for scheduling and managing heterogeneous computing power instances using specific embodiments. This method includes at least steps 101-105: Step 101: Receive a state change request for the target heterogeneous computing instance. The state change request includes at least a stop request and a start request.

[0033] In this embodiment, the target heterogeneous computing power instance refers to an instance object in the platform used to carry user tasks and requiring state maintenance, resource release, and recovery startup management. The target heterogeneous computing power instance can be a stateful container instance, such as a TPU container instance for a heterogeneous computing power cloud service platform. Further, the target heterogeneous computing power instance can be represented and managed by a persistent instance object, which records the instance identifier, instance specifications, current state, target state, and snapshot information related to state recovery. The instance specifications may include the type and quantity of heterogeneous computing power devices and configuration requirements related to the operating environment.

[0034] The state change request refers to a control request used to instruct a change in the running state of the target heterogeneous computing instance. The state change request includes at least a stop request and a start request. Specifically, the stop request instructs the target heterogeneous computing instance to be switched from a running state to a stopped state, and a state saving operation is performed before releasing the heterogeneous computing resources; the start request instructs the target heterogeneous computing instance to be switched from a stopped state to a running state, and resource scheduling, environment reconstruction, and instance recovery operations are performed based on the saved state.

[0035] In this embodiment, to achieve unified management of heterogeneous computing container instances with state preservation capabilities, a custom resource definition, PersistentTPUContainer (PTC), can be introduced. Users can apply for and manage a TPU container instance with state preservation capabilities by creating the PTC object. The PTC object can be used to characterize the desired configuration and actual operating status of the target heterogeneous computing instance. The `spec` field can be used to describe the user's desired instance configuration and target status, which may include Running and Stopped. The `status` field can be used to describe the instance's current actual status, the most recent snapshot information, node information, and resource allocation information.

[0036] In some implementations, a PTC controller corresponding to the PTC object can be deployed in the platform. The PTC controller listens for creation, update, and deletion events of the PTC object and coordinates the execution of corresponding processing flows by various functional modules based on the PTC object's spec and status fields. For example, when a user changes desiredState from Running to Stopped, the PTC controller can trigger a state saving process for the target heterogeneous computing instance, and delete the actually running Pod or release the corresponding heterogeneous computing resources after the state saving is completed. When the user changes desiredState from Stopped to Running, the PTC controller can further trigger resource scheduling, dynamic environment reconstruction, and instance recovery processes. Therefore, the PTC controller can serve as a core control component in the platform's instance management service, enabling fine-grained lifecycle management of persistent TPU container instances.

[0037] In some implementations, users can submit the state change request through the RESTful API, command-line tool, or console page provided by the user service interface module. The state change request can be used to instruct the target heterogeneous computing instance to switch from a running state to a stopped state, or vice versa. In some scenarios, the target heterogeneous computing instance can be a stateful container instance represented and managed by a persistent instance object, such as a TPU container instance for a TPU computing power cloud service platform. After receiving the state change request, the controller can parse the instance identifier, instance specifications, current state, and target state of the target heterogeneous computing instance, and trigger subsequent state saving or state recovery processes based on the parsing results. The instance specifications may include the type and quantity of heterogeneous computing devices and configuration requirements related to the operating environment, and the target state may include states such as Running and Stopped.

[0038] Further optionally, before releasing the heterogeneous computing power resources, the processes occupying the heterogeneous computing power devices may be cleared; the state of the heterogeneous computing power devices may be reset; residual data on the heterogeneous computing power devices may be erased; a health check may be performed on the heterogeneous computing power devices, and the device availability status may be updated based on the health check results.

[0039] Before releasing the heterogeneous computing resources, a heterogeneous computing device cleanup process can be performed. This cleanup process can be completed collaboratively by a lightweight heterogeneous computing runtime interface module, device plugins, or a node-side resource management module. This is to clear the occupancy status and residual operating traces of the previous instance when the target heterogeneous computing instance stops and is ready to release its heterogeneous computing resources, so as to avoid subsequent instances being affected by the previous operating status when reusing the same heterogeneous computing device.

[0040] In some implementations, cleaning up processes occupying heterogeneous computing power devices may include identifying processes still occupying device files of the target heterogeneous computing power device and cleaning up related zombie processes, abnormal residual processes, or user-space processes that have not exited normally. For example, in scenarios targeting TPU devices, process handles still occupying the corresponding device file can be detected, and process termination operations can be performed to prevent the device from being unable to be reallocated due to unreleased handles. In this way, problems such as device busy, device handle leakage, or incomplete resource reclamation can be avoided when subsequent instances apply for heterogeneous computing power devices.

[0041] Furthermore, resetting the state of heterogeneous computing devices may include calling the underlying driver interface or device management interface to reset the register state, runtime context, task queue state, or error counter of the heterogeneous computing devices. In the TPU scenario, the chip's runtime state can be reset through the underlying driver, and abnormal device-side state information such as ECC error counts can be cleared. By performing device state reset, it is possible to prevent abnormal contexts, error states, or residual execution environments generated during the operation of the previous instance from affecting the normal startup and operation of the next instance.

[0042] Furthermore, erasing residual data on heterogeneous computing devices can include performing data zeroing or overwriting operations on the video memory, cache, or other onboard storage areas of the heterogeneous computing devices. In some implementations, hardware-level memory erasure instructions can be invoked, or zero-value data can be written to the corresponding storage area via DMA to clear model weights, inference intermediate results, training cache data, or other sensitive information left behind by the previous instance on the device side. This approach can improve isolation and security in multi-tenant environments and reduce the risk of information leakage caused by cross-instance data residue.

[0043] Furthermore, the step of performing health checks on heterogeneous computing devices and updating the device availability status based on the health check results may include performing a self-check process on the target heterogeneous computing device after completing process cleanup, device reset, and residual data erasure to determine whether the device is in a healthy state where it can be reassigned. The health check may include at least one of the following: device online status detection, basic communication detection, driver response detection, error count check, and simple calculation self-check. If the health check result indicates that the target heterogeneous computing device is in a normal state, the device can be marked as available, allowing it to be rejoined to the resource pool for subsequent scheduling. If the health check result indicates that the target heterogeneous computing device is abnormal, the device can be marked as faulty, unschedulable, or sub-healthy, and the node label, device label, or resource status table can be updated synchronously for filtering or demotion during subsequent API layer pre-scheduling and Kubernetes scheduling layer scoring.

[0044] For example, in an implementation scenario of a TPU computing power cloud service platform, after a user submits a stop request, the controller, after completing the root file system snapshot and snapshot metadata recording, can trigger the node-side runtime module to perform a cleanup process on the TPU devices currently allocated to the target heterogeneous computing power instance. First, the system can identify and clean up training processes still occupying the TPU device files. Then, it calls the TPU driver interface to reset the chip register state and clear the device error count. Next, it erases residual model parameters, intermediate tensor data, and cached data in the video memory. Finally, it uses a device self-test program to determine if the TPU device has returned to a usable state. If the self-test passes, the TPU device is re-marked as allocable; if the self-test fails, the TPU device is marked as faulty and reported to the scheduling system via a fault tag. In this way, when the next start request arrives, the API layer pre-scheduling and scheduling layer scoring can automatically avoid devices or nodes with fault risks.

[0045] It should be noted that although the above-mentioned heterogeneous computing power device purification process occurs before the release of heterogeneous computing power resources, its implementation is usually triggered by a stop request. Therefore, this process has a triggering relationship with the state change request in step 101 and an execution connection relationship with the state saving and resource release process in step 102. In other words, the stop request in step 101 not only instructs the instance to switch from the running state to the stopped state, but can also further trigger subsequent processing links such as state saving, device purification, and resource reclamation, thereby ensuring that heterogeneous computing power resources are in a reusable, schedulable, and secure state before being redistributed.

[0046] By introducing the aforementioned device cleanup mechanism, the integrity of heterogeneous computing resource release can be improved, preventing resources from being unable to be properly reclaimed due to process remnants, state remnants, or handle leaks. Furthermore, it can enhance data security and device isolation in multi-tenant environments, and update device availability status through health check results, providing more accurate basic status information for subsequent resource scheduling, thereby improving the reliability of lifecycle management and resource reuse capabilities of target heterogeneous computing instances.

[0047] By executing step 101, user requests for lifecycle control of heterogeneous computing instances can be uniformly integrated into the platform control chain, providing triggering conditions for subsequent execution of state saving, state recovery, and resource scheduling, thereby improving the automation level and control accuracy of instance management. Simultaneously, by explicitly transmitting the target state to the controller, a unified control entry point can be provided for fine-grained management of the instance lifecycle.

[0048] It is worth noting that the lightweight TPU container (i.e. heterogeneous computing instance) runtime interface module of this application can be implemented in a variety of integration methods. Its purpose is to ensure that TPU resources can be efficiently and correctly allocated and exposed to the application in the container, while minimizing runtime overhead.

[0049] As an optional implementation, the lightweight TPU container runtime interface module can be implemented using a combination of general-purpose CRIShim and OCI Hooks. Specifically, the system does not need to rely entirely on a separately customized lightweight container runtime, but can be implemented based on standard container runtime interfaces, such as containerd or CRI-O, to complete the basic creation, startup, and lifecycle management of the container. Furthermore, the hook mechanism defined in the OCI runtime specification can be used to execute TPU-related initialization logic in a preset phase before container startup. For example, prestart hooks can be used to complete TPU device file injection, driver library mounting, runtime environment variable configuration, device access permission settings, and initialization operations related to TPU device management before the container officially enters the running state. In this way, while maintaining compatibility with the standard CRI architecture, the TPU device enabling logic can be embedded into the container startup process in a pluggable and hook-based manner, thereby reducing the complexity of deep modification of the underlying container runtime.

[0050] For example, after the target heterogeneous computing instance is scheduled and bound to the target node, the node-side container runtime can first create the basic container runtime environment according to the standard CRI process. Subsequently, before the container officially starts, preprocessing logic related to the TPU device is executed through OCI Hooks to map the corresponding device files to the container namespace and mount the required driver files or runtime libraries, enabling the application inside the container to recognize and call the target TPU resources. In this way, the existing container runtime ecosystem can be reused, while also meeting the special initialization requirements of heterogeneous computing devices in container scenarios.

[0051] As an alternative implementation, the lightweight TPU container runtime interface module can also be implemented by direct integration with the TPU driver or TPU management software. Specifically, if the TPU manufacturer provides corresponding driver interfaces, management tools, runtime libraries, or device control commands, the device plugin or lightweight TPU container runtime interface module can directly call these tools or interfaces to complete operations such as TPU device configuration, driver loading, device state initialization, container enabling, and resource reclamation. In this implementation, the system can reduce intermediate adaptation layers, making the device configuration process closer to the underlying hardware control logic, thereby improving the efficiency of TPU resource allocation and device activation.

[0052] For example, during the startup of a target heterogeneous computing instance, the lightweight TPU container runtime interface module can directly call device management commands or driver interfaces provided by the TPU manufacturer to perform initialization configuration on the TPU devices allocated to the instance, and complete device status verification, access path preparation, and exposure within the container before container startup. Furthermore, during the container stopping or destruction phase, the corresponding driver interface can be called to complete device status reset, memory cleanup, and health checks to support the secure reuse of subsequent heterogeneous computing resources. This direct integration method can further improve the accuracy of device configuration and reduce the additional runtime overhead caused by multi-layer adaptation.

[0053] It should be noted that the two integration methods described above are not mutually exclusive. In some implementations, the standard CRI implementation, the OCI Hooks mechanism, and the driver interface provided by the TPU vendor can be combined. For example, basic container lifecycle control can be implemented through the standard CRI, TPU initialization logic can be inserted before startup through OCI Hooks, and further fine-tuning of the device configuration can be achieved by calling the underlying tools provided by the TPU vendor within the hooks. This combination approach can balance standardization compatibility, deployment flexibility, and underlying hardware control capabilities.

[0054] Therefore, the integration method of the lightweight TPU container runtime interface module can be flexibly selected based on the platform technology stack, TPU vendor capabilities, and runtime performance requirements. Whether using a combination of general CRI Shim and OCI Hooks, or direct integration with TPU drivers or management software, the core objective is to ensure that TPU resources can be efficiently and correctly allocated and exposed to applications within the container, while minimizing runtime processing overhead, thereby improving the startup efficiency, resource utilization, and operational stability of the target heterogeneous computing instance.

[0055] Step 102: If the state change request is a stop request, trigger the state saving process corresponding to the target heterogeneous computing power instance to perform a snapshot operation on the root file system of the target heterogeneous computing power instance before releasing the heterogeneous computing power resources, and record the snapshot metadata corresponding to the snapshot operation.

[0056] In this embodiment, the root file system of the target heterogeneous computing instance refers to the system-level file system that hosts the runtime environment of the target heterogeneous computing instance. The root file system includes at least operating system base files, driver files, compilers, deep learning frameworks, dependency libraries, user code, and other runtime environment configurations. By taking a snapshot of the root file system, the software environment state corresponding to the instance's shutdown can be preserved before the heterogeneous computing resources are released, so that it can be used again during subsequent recovery.

[0057] In practical applications, the root file system of the target heterogeneous computing instance can be a system-level file system mounted by the container instance at runtime. It can consist of a base image layer and a currently active writable layer, or be implemented by an underlying storage object that supports snapshot functionality. The root file system should at least include operating system base files, heterogeneous computing driver files, compilers, deep learning frameworks, dependency libraries, user code, configuration files, and environment state information written to the system disk during instance runtime. For cloud drive mounting, external data volumes, shared memory configurations, large page memory configurations, and network port mappings, these can be re-established through dynamic environment reconstruction during the subsequent recovery phase, and do not necessarily need to be included in the root file system snapshot.

[0058] It's important to note that the root file system differs from external data volumes or network storage mounts. The root file system is primarily used to store the instance's runtime environment itself, while external data volumes, network drive mounts, and other storage resources can be remounted during instance recovery, thus collectively achieving consistent environment restoration.

[0059] In step 102, after the controller recognizes that the state change request is a stop request, it can first update the state of the target heterogeneous computing instance to "stopping" and trigger the termination process of the associated container. For example, the persistent instance controller can delete the associated Pod through the container orchestration platform interface, or adjust the number of replicas of the associated workload to zero to trigger container termination.

[0060] Before the container exits, an atomic snapshot operation is performed on the root file system of the target heterogeneous computing instance through the PreStop lifecycle hook of the Sidecar container co-located with the business container to save the runtime environment state of the instance when it stops. The PreStop lifecycle hook is a preprocessing mechanism automatically triggered by the container orchestration platform before the container terminates, used to perform preset operations before the container officially exits. In this embodiment, the co-located auxiliary container can perform a root file system snapshot operation before the business container terminates through the PreStop lifecycle hook, thereby saving the runtime environment state of the target heterogeneous computing instance before releasing the heterogeneous computing resources. In some implementations, the atomic snapshot operation can be implemented based on an underlying storage system that supports snapshot functionality, such as a snapshot mechanism based on the ZFS file system to perform a snapshot of the currently active writable layer, and after the snapshot is completed, it can be renamed or promoted to correspond to the persistent layer used for subsequent recovery.

[0061] The snapshot metadata includes, but is not limited to, snapshot identifier, snapshot data location, topology information corresponding to the state saving, and other descriptive information used for subsequent recovery. The snapshot data location can be used to characterize the node, storage path, or storage medium location where the snapshot data is located, and the topology information can be used to characterize the interconnection relationship between heterogeneous computing devices when the state is saved.

[0062] After completing the snapshot operation and snapshot metadata recording, the heterogeneous computing resources occupied by the target heterogeneous computing instance can be released. Before or during the release of resources, a device cleanup operation can also be performed on the heterogeneous computing device to clean up occupying processes, reset the device state, and erase residual data.

[0063] By executing step 102, the system disk state, environment configuration, and user context of the instance can be preserved while releasing heterogeneous computing resources. This avoids the problem of complete loss of the runtime environment after instance deletion in existing technologies, thereby reducing environment reconstruction costs and improving resource reuse capabilities. Simultaneously, recording the state preservation results through snapshot metadata provides a state basis for subsequent node selection and instance recovery.

[0064] As an optional embodiment, in step 102, before releasing the heterogeneous computing resources, a snapshot operation is performed on the root file system of the target heterogeneous computing instance, including: in response to the stop request, the persistent TPU container controller triggers a container termination process, wherein the container termination process includes deleting the associated Pod through the Kubernetes interface or adjusting the number of associated workload replicas to zero; before the business container terminates, the PreStop lifecycle hook of the co-resident auxiliary container is triggered to perform the snapshot operation; after the snapshot operation is completed, a snapshot identifier is recorded, and the snapshot metadata is updated based on the snapshot identifier.

[0065] Specifically, after a user switches the target state of the target heterogeneous computing instance from running to stopped through the user service interface module, the persistent TPU container controller can listen to the state change of the corresponding persistent instance object, such as listening to the change of desiredState from Running to Stopped. Subsequently, the persistent TPU container controller can first update the current state of the instance to stopped, for example, by updating status.currentState to Stopping, to indicate that the state saving process has started. Afterwards, the persistent TPU container controller can trigger the container termination process. The container termination process may include deleting the Pod associated with the target heterogeneous computing instance through the Kubernetes interface, or adjusting the number of replicas of the associated workload to zero, thereby triggering the Kubernetes termination management process for the business container and co-resident auxiliary containers. In some implementations, to ensure that the snapshot operation can be completed before the business container is forcibly terminated, the container termination process can also rely on container termination grace time parameters, PreStop lifecycle hook execution time limit parameters, and underlying storage snapshot timeout parameters for coordinated control. In other words, after issuing the termination command, the controller does not immediately release the heterogeneous computing resources. Instead, it reserves a time window for performing a root file system snapshot operation. At this time, the co-resident auxiliary container can trigger a snapshot operation through the PreStop lifecycle hook before the business container terminates, saving the current root file system state of the target heterogeneous computing instance. After the snapshot operation is completed, the co-resident auxiliary container can generate and report the corresponding snapshot identifier, such as snapshotID. The persistent TPU container controller then updates the snapshot metadata based on this snapshot identifier and can further write the most recent snapshot field in the instance status object, such as status.lastSavedSnapshotID. Through the above processing, the stop request and the state saving process can be atomically linked, enabling the target heterogeneous computing instance to complete the solidification of a recoverable state before releasing the heterogeneous computing resources.

[0066] Further optionally, after triggering the PreStop lifecycle hook of the co-resident auxiliary container to execute the snapshot operation before the business container terminates, it can also be implemented as follows: identifying the currently active writable layer of the root file system corresponding to the business container; using the currently active writable layer to perform a solidification state operation to generate an atomic snapshot; performing a renaming operation on the atomic snapshot to achieve hierarchical elevation of the atomic snapshot and mapping the atomic snapshot to a persistent layer for subsequent recovery; writing the identification information corresponding to the persistent layer and the association between the identification information and the snapshot identifier corresponding to the atomic snapshot into the snapshot metadata, so that the root file system can be recovered based on the snapshot metadata during subsequent startup.

[0067] In the above steps, the co-resident auxiliary container can first identify the active writable layer corresponding to the current root file system of the business container, such as Active Writable Clone. The active writable layer can be understood as a writable file system layer that carries incremental changes during the runtime of the current instance on top of the base image layer. It stores software packages added by the user during this run, configuration files generated during the run, updated dependency environment, user code changes, and other state information written to the system disk.

[0068] After identifying the active writable layer, the co-resident auxiliary container can invoke the underlying storage mechanism that supports snapshot capabilities to perform a state-fixing operation on the active writable layer to generate an atomic snapshot. For example, in a ZFS-based implementation, a snapshot command can be executed on the corresponding dataset or writable clone volume to generate a read-only snapshot that corresponds one-to-one with the current file system state. Key parameters here may include the active writable layer identifier, snapshot name, snapshot creation time, instance identifier, and corresponding dataset path, etc.

[0069] Subsequently, the atomic snapshot can be renamed or promoted to a persistent layer for subsequent recovery. Technically, this persistent layer elevates the root file system state at the time of instance shutdown from a temporary running layer to a recovery base layer that can be directly referenced in subsequent startup phases. In some implementations, the layer identifier, dataset name, mount path, volume identifier, and their association with the snapshot ID corresponding to this persistent layer can be written into the snapshot metadata, enabling the system to directly locate the recovery source based on the snapshot metadata during subsequent startup phases.

[0070] For example, if a TPU container instance has already installed a specified version of the TPU driver, Python environment, and deep learning framework in the root file system before stopping, and has saved training scripts and configuration files in the user's working directory, then the aforementioned active writable layer snapshot and its upgraded persistent layer can completely preserve this runtime environment state. When the instance is subsequently restarted, the root file system can be quickly restored based on this persistent layer without reinstalling the aforementioned software environment. In this way, fine-grained preservation of the root file system state, traceable management of snapshot results, and rapid location of subsequent recovery processes can be achieved.

[0071] Further optionally, after triggering the PreStop lifecycle hook of the co-resident auxiliary container to execute the snapshot operation before the business container terminates, it can also be implemented as follows: if the co-resident auxiliary container fails to execute the snapshot operation, a non-zero exit code is returned; the persistent container controller captures the snapshot failure information corresponding to the non-zero exit code; the status of the target heterogeneous computing instance is updated to the snapshot failure status based on the snapshot failure information; the Pod corresponding to the target heterogeneous computing instance is retained; and an alarm message is triggered to notify the operation and maintenance to intervene.

[0072] In the above steps, if an exception occurs during the snapshot operation of the co-resident auxiliary container, such as insufficient underlying storage space, snapshot command execution failure, abnormal dataset status, inaccessible path, or snapshot creation timeout, the co-resident auxiliary container can return a non-zero exit code to explicitly indicate that the snapshot operation failed. Correspondingly, the persistent TPU container controller can capture the snapshot failure information corresponding to the non-zero exit code by listening to Pod termination results, reading shared state files, receiving Sidecar reports, or querying event logs. The snapshot failure information may include at least one of the following: failure time, failure reason, failure stage, failure object identifier, snapshot command return value, and associated instance identifier. After obtaining the snapshot failure information, the persistent TPU container controller can update the status of the target heterogeneous computing instance to a snapshot failure status, such as updating it to SnapshotFailed, so that subsequent control logic and operations personnel can clearly identify that this stop operation failed to successfully save the state.

[0073] Furthermore, in this abnormal scenario, the Pod corresponding to the target heterogeneous computing instance can be retained instead of being immediately deleted, thus preventing the complete loss of the runtime environment due to continued resource release when the snapshot fails. Simultaneously, an alarm can be triggered, reporting the snapshot failure event to the monitoring platform, the operation and maintenance alarm system, or the instance management page, prompting operations and maintenance personnel to intervene. For example, if a target heterogeneous computing instance fails to create a snapshot during shutdown due to insufficient remaining space in the ZFS storage pool, the persistent TPU container controller can mark the instance state as SnapshotFailed after the co-resident auxiliary container returns a non-zero exit code, preserving the current Pod from destruction and sending a storage space shortage alarm to the operations and maintenance system, allowing operations and maintenance personnel to expand the storage pool or clean up historical snapshots. This abnormal handling mechanism prevents the risk of environment loss caused by directly releasing resources in the event of state preservation failure and improves the reliability and maintainability of the state management process.

[0074] Further optionally, after triggering the container termination process in response to the stop request, the process further includes: if a node malfunctions and causes the PreStop lifecycle hook to not be executed, then triggering a background cleanup task when the node recovers or the Pod is evicted; checking whether there is a dangling snapshot corresponding to the target heterogeneous computing instance through the background cleanup task; and performing repair or cleanup operations on the dangling snapshot when the existence of the dangling snapshot is detected.

[0075] In the above steps, if a node crashes, the node network is interrupted, the node agent malfunctions, the container runtime malfunctions, or the Pod is abnormally evicted, the PreStop lifecycle hook may fail to execute normally, or the snapshot process may only be partially completed, resulting in dangling snapshots related to the target heterogeneous computing instance but not correctly registered, committed, or reclaimed. To solve this problem, a background cleanup task can be triggered when the node recovers, the Pod is re-detected, or the controller detects a Pod eviction event. For example, the background cleanup task can be executed by the persistent TPU container controller, the Rootfs persistence and state management module, or the background maintenance service. Specifically, the background cleanup task can scan and perform consistency checks on snapshot objects in the underlying storage system based on the instance identifier, historical snapshot identifier, persistent layer identifier, dataset path, and node local storage information to determine whether there are dangling snapshots corresponding to the target heterogeneous computing instance but not fully registered. If a dangling snapshot is detected, different processing can be performed based on its integrity and recoverability. If the dangling snapshot still has recovery value, its identification information can be supplemented and included in the snapshot metadata. If the dangling snapshot is corrupted, duplicated, or not recoverable, it can be deleted, reclaimed, or repaired to prevent invalid snapshots from occupying storage space for an extended period. Key parameters here may include instance identifier, node identifier, snapshot creation time, snapshot status flag, dataset path, verification result, and background cleanup task execution cycle.

[0076] For example, if a node suddenly crashes during the shutdown of a target heterogeneous computing instance, resulting in an intermediate snapshot being generated at the underlying level but the controller not yet completing the snapshotID registration, then after the node recovers, the background cleanup task can scan the ZFS dataset on that node, identify unregistered snapshot objects related to that instance, and further determine whether they can be repaired and reused. If they can be reused, their snapshot metadata is added; if they cannot be reused, cleanup is performed. This background cleanup mechanism improves the system's fault tolerance to abnormal interruption scenarios, prevents the continuous accumulation of dirty data and dangling snapshots, and ensures that the snapshot metadata relied upon by subsequent state recovery processes remains consistent and accurate.

[0077] In summary, in this optional embodiment, step 102 not only includes the container termination process triggered by the persistent TPU container controller, the root file system snapshot operation performed by the co-resident auxiliary container in the PreStop phase, and the snapshot metadata updated by the controller based on the snapshot identifier, but also further includes the identification of the active writable layer, the promotion mapping of atomic snapshots, the retention and alarm handling in snapshot failure scenarios, and the repair or cleanup of dangling snapshots in node anomaly scenarios. Through these collaborative mechanisms, the runtime environment state of the target heterogeneous computing instance can be saved as completely and reliably as possible before releasing heterogeneous computing resources, and a traceable, implementable, and consistent state foundation can be provided for the subsequent recovery process.

[0078] Step 103: If the status change request is a startup request, determine the target node for restoring the target heterogeneous computing instance based on the snapshot metadata.

[0079] Snapshot metadata is not the snapshot data itself, but rather a collection of data describing the snapshot. Snapshot metadata may include a snapshot identifier, snapshot data location, the topological fingerprint corresponding to the state being saved, and other descriptive information for subsequent recovery. The snapshot data location indicates the node, storage path, or storage medium location where the snapshot data is located. The topological fingerprint characterizes the interconnect bandwidth, interconnect latency, and physical connection structure between heterogeneous computing devices at the time of state saving, thus providing a basis for subsequent target node selection and instance recovery.

[0080] Upon receiving a startup request, a pre-screening process can be performed on candidate nodes at the API layer to filter out nodes whose resources do not meet instance specifications or whose health status is abnormal. This API layer pre-screening can quickly check the quantity of heterogeneous computing resources, node health status, and fault label information of candidate nodes to prevent invalid requests from entering the subsequent scheduling chain. Afterwards, the pre-screened candidate nodes are submitted to the scheduling layer for evaluation. In some implementations, the scheduling layer can be the native scheduling layer of a container orchestration platform, a scheduling extension, or a scheduling framework plugin.

[0081] During the scheduling and scoring process, factors such as the fragmentation level of heterogeneous computing resources, snapshot data migration costs, resource waiting costs, node health status, and the corresponding hardware topology information at the time of state saving can be comprehensively considered. The Bin Packing strategy can be understood as a candidate node selection strategy aimed at reducing the fragmentation of heterogeneous computing resources. Specifically, under the premise of meeting the resource requirements of the target heterogeneous computing instance, candidate nodes that can support the target heterogeneous computing instance with a higher degree of concentration are prioritized. This reduces the fragmentation level of the remaining heterogeneous computing resources after resource allocation and retains more continuous resources available for subsequent instances. Based on this strategy, the scheduling score of candidate nodes that can form a concentrated bin for the remaining heterogeneous computing resources can be improved. For candidate nodes that, while meeting the current recovery requirements, would lead to excessive dispersion of remaining resources, the scheduling score can be reduced, thereby reducing the fragmentation level of cluster resources and improving the schedulability of subsequent large-scale heterogeneous computing instances.

[0082] Furthermore, a spatiotemporal trade-off score can be determined based on the time-space trade-off, combining snapshot migration time and resource waiting time, to select between waiting for the original node to release resources and migrating to other nodes to restore the instance. For scenarios with topology sensitivity, the priority of candidate nodes can be determined based on the degree of matching between the topology fingerprint recorded in the snapshot metadata and the topology features of the candidate nodes' currently allocable resources, thereby selecting a suitable target node for restoration. The topology fingerprint can use a weighted adjacency matrix to represent the interconnection bandwidth and interconnection latency between heterogeneous computing devices, and combine it with the physical connection structure to generate a topology identifier, so that nodes with the same or similar topology as when the state was saved can be preferentially selected during restoration.

[0083] By performing step 103, resource availability, migration costs, and topology consistency can be comprehensively considered during instance recovery, reducing the risk of ineffective scheduling and recovery failures, and improving the performance stability and overall resource utilization of the recovered instance. Furthermore, by combining pre-screening and scoring, scheduling decision efficiency can be improved and resource waiting time reduced.

[0084] Step 104: Perform dynamic environment reconstruction on the target node and restore the root file system of the target heterogeneous computing instance based on the snapshot metadata.

[0085] Dynamic environment reconstruction refers to the process of rebuilding and injecting the context configuration required for instance operation during the instance recovery phase, based on the saved root file system state and combined with the latest user configuration and the current node environment. Here, the context configuration may include at least one of the following: cloud drive mounting configuration, large page memory configuration, shared memory configuration, and network port mapping configuration. Through dynamic environment reconstruction, the recovered instance can inherit the historical operating environment state while adapting to the current infrastructure conditions and configuration requirements.

[0086] After identifying the target node, the controller can dynamically construct or update the runtime configuration or Pod specification for launching the target heterogeneous computing instance based on the user's latest configuration, and inject context information such as network drive mounting configuration, large page memory configuration, shared memory configuration, and service port mapping configuration into it. This dynamic environment reconstruction corresponds to the context injection logic, which is used to rebuild parameters related to the runtime environment based on the user's current configuration when restoring the instance, rather than simply restoring the file system itself. After completing the context configuration injection, the root file system persistence and state management module on the target node can locate the target snapshot based on the snapshot metadata and restore the root file system of the target heterogeneous computing instance using on-demand loading, cloning, or other snapshot recovery methods.

[0087] As an optional embodiment, step 104, performing dynamic environment reconstruction on the target node, includes: obtaining instance configuration information corresponding to the target heterogeneous computing instance at startup, wherein the instance configuration information includes at least one of the user's latest configured cloud drive mounting parameters, memory parameters, and network port parameters; constructing a Pod specification for starting the target heterogeneous computing instance based on the instance configuration information, or updating an existing Pod specification; injecting context configuration corresponding to the instance configuration information into the Pod specification, wherein the context configuration includes at least one of cloud drive mounting configuration, HugePages configuration, shared memory configuration, and service port mapping configuration; and generating and updating the running configuration for starting the target heterogeneous computing instance based on the injected Pod specification.

[0088] The instance configuration information can be determined by the specification field of the persistent instance object, the latest configuration parameters submitted by the user, and the currently available infrastructure conditions of the target node. Here, the persistent instance object can be a custom resource object used to represent and manage the target heterogeneous computing power instance. After receiving the startup request, the controller can read the container template, target status, latest snapshot identifier, and the latest running parameters set by the user recorded in the custom resource object, thereby obtaining the instance configuration information corresponding to this startup. In addition to network disk mounting parameters, memory parameters, and network port parameters, the instance configuration information may also include at least one of the following: instance name, namespace, image basic information, resource quota information, startup command, or environment variables, so as to provide complete input for subsequent dynamic environment reconstruction.

[0089] Furthermore, the step of constructing a Pod specification for launching the target heterogeneous computing instance based on the instance configuration information, or updating an existing Pod specification, can be understood as regenerating container orchestration description information that matches the current operating conditions before the instance is restored. Specifically, the controller can create a new Pod specification based on the instance configuration information, or incrementally update fields such as volume mounts, resource limits, environment variables, port mappings, and startup parameters based on historical Pod templates. In this way, even if the root file system state corresponding to the target heterogeneous computing instance comes from a previously saved snapshot, the system can still reorganize the operating description by combining the current user configuration and the target node conditions, thereby avoiding the incompatibility issues caused by simply copying the historical environment configuration to the new infrastructure conditions.

[0090] In some implementations, the context configuration corresponding to the instance configuration information is injected into the Pod specification. This context injection logic not only focuses on restoring file system content but also on re-establishing the instance's runtime context. For example, for cloud storage mounting configuration, the controller can write the corresponding volume definition and mount point into the Pod specification based on the user's latest configured network storage address, mount path, access credentials, or read-only / write attributes, enabling the target heterogeneous computing instance to continue accessing the datasets, code repositories, or model files required by the user after recovery. For HugePages configuration, the controller can inject corresponding resource requests and resource limits into the Pod specification based on the user's currently specified large page memory requirements to meet the needs of high-performance computing or large model inference for specific memory page configurations. For shared memory configuration, the controller can adjust the mounted volumes or runtime parameters related to shared memory based on the user's currently set shared memory size, thereby meeting the requirements for shared memory space during multi-process training, data loading, or framework operation. For service port mapping configuration, the controller can write SSH ports, Jupyter ports, or other service ports into the Pod specification to ensure that the external access link remains consistent with the current user's usage after instance recovery.

[0091] For example, before stopping a target heterogeneous computing instance, a user has installed a specific version of the TPU driver, Python runtime environment, deep learning framework, and user training scripts in the root file system, and saved the working directory in a snapshot of the root file system. Upon subsequent startup, if the user changes the original cloud drive mount directory from the first path to the second path, adjusts the shared memory size from the first value to the second value, and adds a Jupyter access port, the controller, when performing dynamic environment reconstruction, will not only restore the historical root file system, but can regenerate the Pod specification based on the user's latest configuration. This Pod specification will update the cloud drive mount path, shared memory configuration, and Jupyter port mapping information. Therefore, the restored target heterogeneous computing instance can inherit the drivers, frameworks, dependency libraries, and user code from the original root file system, while also adapting to the user's new resource requirements and access methods.

[0092] Furthermore, the step of generating and updating the runtime configuration for launching the target heterogeneous computing instance based on the injected Pod specification can include generating a final resource object description for scheduling and launching by the container orchestration platform, and submitting this description to the corresponding control plane. Specifically, the controller can supplement the scheduling constraints, node binding information, container startup parameters, and runtime adaptation information related to the allocation of heterogeneous computing devices based on the injected Pod specification to form the final runtime configuration. Subsequently, the controller can submit the runtime configuration to the container orchestration platform, where the scheduler, node agent, and lightweight heterogeneous computing runtime interface module collaboratively complete the subsequent startup process. This ensures that the result of dynamic environment reconstruction not only remains at the configuration generation level but can be actually transformed into executable instance startup instructions.

[0093] After completing the aforementioned context configuration injection, the root file system persistence and state management module on the target node can further locate the target snapshot based on the snapshot metadata and perform root file system recovery. Snapshot metadata can include at least the snapshot identifier, snapshot data location, and the corresponding topology information at the time of state saving. The root file system persistence and state management module can first locate the corresponding snapshot based on the snapshot identifier, then determine the node, storage path, or underlying storage object where the snapshot data is located based on the snapshot data location information, and select an appropriate recovery method based on the access relationship between the target node and the snapshot location. In some implementations, if the snapshot is located locally on the target node, a local cloning recovery method can be used first. If the snapshot is located in remote storage or on other nodes, an on-demand loading method can be used to prioritize the recovery of data blocks required during the startup phase, and the remaining data blocks can be loaded gradually during instance operation; other block-level snapshot recovery methods can also be used depending on the capabilities of the underlying storage system.

[0094] For example, when the root file system snapshot corresponding to the target heterogeneous computing instance is stored in a ZFS dataset at the time of shutdown, and the target node can directly access this dataset, the root file system persistence and state management module can create a writable clone volume based on the snapshot, enabling the restored instance to obtain a mountable root file system in a short time. For critical data blocks that are prioritized for access during the startup phase, such as system directories, driver libraries, Python interpreters, core files of deep learning frameworks, and user entry scripts, priority can be given to recovery through on-demand loading or hotspot preheating, allowing the instance to enter a runnable state before the root file system is fully copied. Historical cache files, infrequently accessed dependency files, or large intermediate result files can be loaded gradually according to access requests during subsequent operation. These methods effectively shorten the cold start recovery time.

[0095] Therefore, the dynamic environment reconstruction in this embodiment is not simply restoring a historical snapshot. Instead, it rebuilds the context configuration required for instance operation based on the inherited historical state of the root file system, combined with the latest user configuration, the target node environment, and current infrastructure conditions, and executes this reconstruction collaboratively with the root file system recovery process. On the one hand, this approach ensures the continuity of the software environment of the target heterogeneous computing instance, avoiding repeated installation of drivers, frameworks, and dependencies by the user. On the other hand, it enables the restored instance to adapt to new network drive mounting requirements, memory configuration requirements, and port access requirements, thereby achieving a balance between environmental consistency and configuration flexibility, and improving the availability, adaptability, and recovery efficiency of the target heterogeneous computing instance recovery process.

[0096] In this application, when the target heterogeneous computing instance stops, the controller performs a snapshot of the root file system via the Sidecar to save the user's current runtime environment. When the user submits a startup request again, the system needs to restore the root file system on the new target node. If a traditional full recovery method is used, the entire root file system usually needs to be completely copied or rebuilt before the container can be started. This results in a long recovery time, especially when the root file system contains a large number of framework files, dependency packages, model files, or cache files.

[0097] Therefore, in this embodiment, ZFS cloning or on-demand loading mechanisms can be used to first restore the data blocks required for container startup, driver loading, and basic service startup, and then gradually supplement the remaining data blocks after the instance is running, so as to achieve second-level mounting or rapid availability. For example, ZFS cloning features or on-demand loading can be used to prioritize the restoration of data blocks required during the startup phase to shorten cold start time. ZFS cloning features can be understood as quickly generating a readable and writable file system instance based on a read-only snapshot. This readable and writable instance initially shares the underlying data blocks with the original snapshot, and only performs copy-on-write on the modified data blocks when subsequent writes occur. In some implementations, step 104, restoring the root file system of the target heterogeneous computing instance based on the snapshot metadata, may include: creating a writable clone volume based on the read-only snapshot corresponding to the snapshot, so that the writable clone volume shares the data blocks of the read-only snapshot in its initial state; after the writable clone volume is mounted, prioritizing the loading of key data blocks required during the instance startup phase, and gradually loading the remaining data blocks according to access requests during instance operation. The key data blocks may include at least one of the following: operating system startup files, heterogeneous computing power driver files, runtime dependency libraries, deep learning framework files, user entry scripts, and service configuration files. By employing this method, the target heterogeneous computing power instance can be brought into a runnable state first, without waiting for the full copy of the root file system to complete, thereby shortening recovery latency.

[0098] For example, suppose a target heterogeneous computing instance has the following components installed in its root file system before stopping: TPU driver and runtime, Python runtime environment, PyTorch or TensorFlow framework, user training scripts, and SSH or Jupyter service components. Upon stopping, the Sidecar generates a snapshot of the instance's currently active writable layer based on ZFS, for example, snapshot Snapshot_A. When a subsequent startup request is received, the Rootfs persistence and state management module on the target node can avoid performing a full disk copy of Snapshot_A, instead creating a new writable clone volume based on Snapshot_A, for example, Clone_A. Since Clone_A initially shares the underlying data blocks with Snapshot_A, the system can complete the root file system mounting in a very short time. Subsequently, the controller starts the Pod based on this clone volume, and the lightweight heterogeneous computing instance then completes the TPU device pass-through. At this point, the instance can first complete the following startup actions: loading the basic operating system environment, starting the container entry process, loading TPU driver dependencies, and starting the SSH or Jupyter service. During this process, only the data blocks that are actually accessed are read. If subsequent users modify files or perform new installation operations, only the modified data blocks are copied on write, without affecting the original snapshot. In this way, a working and writable recovery instance can be quickly obtained without pre-copying the entire rootfs.

[0099] In other scenarios, block-level hotspot preheating can be performed to further accelerate service availability after recovery. On-demand loading can be understood as not pre-transmitting or rebuilding all snapshot data during the recovery process, but instead prioritizing the recovery of data blocks necessary for instance startup. Data blocks not yet accessed can be loaded gradually during subsequent operation. Therefore, both serve the same technical goal: prioritizing the recovery of critical data and delaying the recovery of non-critical data, only their implementation paths differ. The former leans towards utilizing the rapid forking capabilities of the underlying file system, while the latter leans towards utilizing access-driven data recovery strategies.

[0100] By executing step 104, not only can the instance's file system state be restored, but also the critical context configurations related to operation can be restored simultaneously, thereby achieving a consistent environment reconstruction and avoiding instance anomalies caused by only restoring file data but missing runtime parameters. Furthermore, using on-demand loading or clone recovery methods can reduce the startup latency caused by full data copying.

[0101] Step 105: After the root file system is restored, allocate heterogeneous computing resources to the target heterogeneous computing instance and start the target heterogeneous computing instance.

[0102] After the root file system is restored, the runtime interface module on the target node can allocate the corresponding heterogeneous computing power devices to the target heterogeneous computing power instance according to the scheduling results, and complete device pass-through, resource limit configuration, and runtime environment enabling. For example, the specific heterogeneous computing power device files can be pass-through to the container through the lightweight heterogeneous computing power container runtime interface module, and the corresponding resource constraint information can be configured. For heterogeneous computing power devices, necessary status checks and initialization processes can also be performed before and after allocation to ensure device availability after the instance starts. In some implementations, the controller can also update the instance running status after the instance starts successfully and record the target node identifier and the identifier of the allocated heterogeneous computing power device. Then, the target heterogeneous computing power instance is started, and the instance status is updated to running status.

[0103] By executing step 105, the recovered instance can regain heterogeneous computing resources and quickly enter an available state, thus forming a complete closed loop from state preservation, node selection, environment reconstruction to instance recovery and startup, improving the continuity and stability of heterogeneous computing services. Furthermore, by re-establishing the association between instance status and device allocation information after recovery, the platform's observability and management capabilities over the instance lifecycle can be improved.

[0104] In this embodiment, the runtime environment state and user context of the target heterogeneous computing instance can be maintained while releasing heterogeneous computing resources, resulting in a heterogeneous computing instance with complete state preservation capabilities and support for rapid recovery. This reduces the idle cost of heterogeneous computing resources and the user environment reconstruction cost, enabling the system to start and stop heterogeneous computing instances on demand, quickly restore the root file system, and reconstruct the runtime environment in a consistent manner. This improves the continuity and stability of heterogeneous computing resource scheduling, reduces resource fragmentation, the probability of instance recovery failure, and the platform-side operation and maintenance scheduling overhead.

[0105] As an optional embodiment, in step 102, the snapshot metadata corresponding to the snapshot operation is recorded. This metadata is used not only to save the identification information of the root file system snapshot, but also to provide a state basis for the selection of the target node in the subsequent recovery phase. That is to say, this embodiment not only saves the root file system data of the target heterogeneous computing instance itself, but also further saves the hardware topology information and data location information related to the snapshot, thereby enabling the subsequent recovery process to have state-aware scheduling capabilities.

[0106] Based on this, recording the snapshot metadata corresponding to the snapshot operation in step 102 can be implemented as follows: Obtaining the TPU topology fingerprint corresponding to the state saving time to characterize the TPU device interconnection topology features corresponding to the target heterogeneous computing power instance at the time of state saving, wherein the TPU topology fingerprint includes a weighted adjacency matrix and a hash value generated based on the weighted adjacency matrix, the weighted adjacency matrix being used to characterize the interconnection bandwidth and interconnection latency between TPU devices; determining snapshot data location information to indicate the data storage location of the snapshot, the data storage location including at least one of a target node identifier and a storage path; writing the snapshot identifier, the TPU topology fingerprint, and the snapshot data location information into the snapshot metadata. Specifically, when determining the target node for restoring the target heterogeneous computing power instance based on the snapshot metadata, the reachability of the snapshot data is determined at least based on the snapshot data location information, and the degree of matching between the candidate node and the corresponding topology features at the time of state saving is determined based on the TPU topology fingerprint.

[0107] Specifically, after the controller recognizes that the state change request is a stop request, it can trigger the Sidecar container, which resides with the business container, to execute the PreStop lifecycle hook, so as to perform an atomic snapshot operation on the currently active writable layer before the business container terminates. For example, the ZFS file system can be used to identify the active writable layer corresponding to the target heterogeneous computing power instance, a snapshot can be taken of the active writable layer, and after the snapshot is completed, a persistent layer used for subsequent recovery can be formed through renaming or promotion operations. After the snapshot is generated, a unique snapshot identifier can be generated for this snapshot to distinguish the runtime environment state saved at different points in time.

[0108] After generating the snapshot identifier, in the above embodiments, the TPU topology fingerprint corresponding to the state saving can be further obtained. The TPU topology fingerprint is used to characterize the interconnection topology features between the TPU devices actually occupied by the target heterogeneous computing instance at the time of state saving. In fact, the TPU topology fingerprint can be a topology identifier formed by abstracting and encoding the interconnection relationship between the allocated TPU devices at the time of state saving, reflecting the hardware topology features that the instance depends on during runtime. For tasks requiring multi-card collaborative training or inference, the interconnection relationship between different TPU devices directly affects cross-card communication efficiency; therefore, simply recording the number of devices is often insufficient to support subsequent high-performance recovery. In this embodiment, each TPU device involved in the allocation at the time of state saving can be abstracted as a node in a graph structure, and the interconnection bandwidth and interconnection latency between each node can be used as edge weights to construct a corresponding weighted adjacency matrix. Furthermore, a hash value can be generated based on the weighted adjacency matrix and combined with device attributes to form a TPU topology fingerprint that is easy to store and quickly compare. Optionally, the hash value can be a digest identifier calculated based on the weighted adjacency matrix and related device attributes, used for compact representation, fast comparison, and unique identification of the topology structure.

[0109] In some implementations, a weighted adjacency matrix is ​​a data structure used to construct a TPU topological fingerprint. Nodes in the matrix correspond to individual TPU devices, and the weights of the matrix elements characterize the interconnect bandwidth and latency between devices. For example, if a target heterogeneous computing instance occupies four TPU devices when the state is saved, a 4x4 weighted adjacency matrix can be constructed, where each element describes the interconnect quality between two corresponding TPU devices. For instance, if two TPU devices have high bandwidth and low latency, the corresponding weights characterize a superior connection. Conversely, if communication between two TPU devices requires more intermediate paths or has lower bandwidth, the corresponding weights characterize a weaker connection. This method allows for the structured recording of hardware interconnect characteristics during state saving, enabling matching with the topological characteristics of currently allocable TPU resources for candidate nodes during subsequent recovery.

[0110] In addition to recording the TPU topology fingerprint, snapshot data location information can also be determined to indicate the actual storage location of the snapshot. This snapshot data location information may include at least one of the following: target node identifier and storage path. For example, if the snapshot is stored in the local ZFS dataset of the node where the snapshot was generated, the node identifier and the corresponding storage path of the dataset can be recorded. If the snapshot is stored in shared storage or a distributed storage system, the corresponding logical volume identifier, mount path, or object location can be recorded. The purpose of this snapshot data location information is that, during the subsequent recovery phase, this information can be used to determine the reachability and migration cost of candidate nodes accessing the snapshot data.

[0111] After completing the above information collection, the snapshot identifier, the TPU topology fingerprint, and the snapshot data location information can be uniformly written into the snapshot metadata. The snapshot metadata can serve as a set of descriptive information about the most recent state saving result of the target heterogeneous computing instance, and be associated with the instance identifier for subsequent initiation and recovery process calls.

[0112] In subsequent step 103, upon receiving the startup request, the snapshot data location information in the snapshot metadata can be read first to determine the reachability of the snapshot data. For example, if a candidate node is the same as the node where the snapshot data is located, or is close to the snapshot data in network distance, the data migration cost of the candidate node is low, and it can obtain a higher priority in the spatiotemporal trade-off score. Correspondingly, if a candidate node has sufficient heterogeneous computing resources, but the network path to access the snapshot data is long, the bandwidth is low, or the migration overhead is large, its recovery score can be relatively reduced.

[0113] Simultaneously, the system can also read the TPU topology fingerprint from the snapshot metadata and match it with the candidate topology information of the currently allocable TPU resources for each candidate node. In some implementations, it can prioritize determining whether the candidate topology of a candidate node completely matches the topology corresponding to the state at the time of saving; if it does not completely match, it can further determine whether the candidate topology contains the topology subgraph corresponding to the original topology; if the candidate topology differs significantly from the original topology, the topology matching score of the candidate node is reduced. In this way, target nodes with topologies consistent with or similar to those at the time of state saving can be prioritized during the recovery phase, thereby improving the computational performance stability of the instance after recovery.

[0114] For example, a target heterogeneous computing instance runs on node A when snapshotting stops, occupying four TPU devices forming a Ring interconnect structure, while snapshot data is stored locally on node A. Upon subsequent recovery, although node B has four idle TPUs, its candidate devices only have poor connectivity. Node C, although geographically closer to node A, can only provide a partially matching topology. In this case, the system can comprehensively consider snapshot data location information, TPU topology fingerprints, node resource status, and spatiotemporal tradeoff scores to decide whether to wait for node A to release resources or migrate to node C for recovery, rather than performing coarse-grained scheduling solely based on the number of devices.

[0115] Through the above methods, this embodiment enables snapshot metadata to serve not only snapshot location but also recovery scheduling decisions. On one hand, recording snapshot data location information improves the reachability assessment capability for snapshot recovery. On the other hand, recording TPU topology fingerprints improves performance consistency after recovery. Therefore, it can improve the accuracy of target node selection, reduce the risk of recovery failure, and improve the operational stability and overall resource utilization of heterogeneous computing instances after recovery.

[0116] As an optional embodiment, in step 103, determining the target node for restoring the target heterogeneous computing instance based on the snapshot metadata can be implemented as follows: after receiving the start request, pre-screening of candidate nodes is performed through API layer pre-scheduling to remove nodes whose resources do not meet the instance specification requirements and nodes that do not meet the health conditions; the pre-screened candidate nodes are submitted to the Kubernetes scheduling layer for scheduling scoring; and the target node is determined based on the scheduling scoring results.

[0117] In the above embodiments, the API layer pre-scheduling can be implemented collaboratively by the user service interface module, the controller, or the scheduling extension module. For example, after a user initiates a startup request through a RESTful API, command-line tool, or console page, the persistent instance controller can first read the instance specification information corresponding to the target heterogeneous computing power instance, the snapshot metadata formed by the most recent state saving, and the cluster cache status. Before entering the Kubernetes formal scheduling queue, it can quickly filter the candidate nodes in the cluster. Here, the quick filtering can at least check whether the type and quantity of available heterogeneous computing power devices on the candidate nodes meet the instance specification requirements, whether the nodes are in a healthy state, and whether the nodes carry fault tags or sub-health markers. In this way, nodes that obviously do not meet the recovery conditions can be eliminated in advance, avoiding invalid candidate nodes from entering the subsequent fine-grained scoring link.

[0118] Alternatively, when pre-screening candidate nodes through API layer pre-scheduling, fault label information corresponding to the candidate nodes can be obtained; candidate nodes with preset fault identifiers or in abnormal states can be removed from the candidate set, or the scheduling score can be reduced in the Kubernetes scheduling layer.

[0119] For example, the fault label information can be generated and reported by a node monitoring system, device health detection module, device plugin, or operation and maintenance management module to characterize the health status of candidate nodes or heterogeneous computing devices on candidate nodes. In some embodiments, the fault label information may include at least one of the following: device offline label, device reset anomaly label, memory anomaly label, temperature anomaly label, link fault label, communication timeout label, node unschedulable label, and resource service anomaly label. The preset fault identifier can be a predefined marker used by the system to indicate that a node is not suitable for carrying out the recovery task of the target heterogeneous computing instance, such as Faulty, Unhealthy, NotReady, LinkDegraded, or other equivalent fault markers.

[0120] In some implementations, when performing candidate node pre-screening, the API layer pre-scheduling can first read the fault tag information corresponding to each candidate node from the cluster cache, node status table, monitoring alarm information, or device plugin reporting results, and determine whether each candidate node has a preset fault identifier. If a candidate node has a preset fault identifier, or its current state meets preset abnormal conditions, the candidate node can be directly removed from the candidate set to avoid scheduling the recovery task to a node with hardware failure risk or operational instability risk. If a candidate node does not meet the direct removal conditions, but has minor abnormalities, incomplete historical alarms, or a low health score, the candidate node can not be immediately removed. Instead, its scheduling score can be lowered in the subsequent Kubernetes scheduling layer to give higher priority to candidate nodes with better health status.

[0121] For example, if a target heterogeneous computing instance requires four TPU cards for recovery, and node A currently provides four TPUs to meet the requirement, but the monitoring system detects abnormal link bandwidth on two of the TPUs and adds a link failure label to the node, then the API layer pre-scheduling can directly remove node A from the candidate set. As another example, although node B has not experienced a hardware failure, it has recently issued overheating alarms or has a high ECC error count, so the system can add a sub-health label to it. In this case, if node B still meets basic operating conditions, the API layer pre-scheduling can retain node B for subsequent scheduling and scoring processes, but demote it at the Kubernetes scheduling layer, thus prioritizing node C or node D, which are not marked as abnormal, for instance recovery.

[0122] Furthermore, in some implementations, the fault label information can also be used in conjunction with the instance specification requirements of the target heterogeneous computing instance. That is, different types of anomaly labels can have varying degrees of impact on different business scenarios. For multi-GPU training tasks that are highly sensitive to device interconnect quality, labels such as link degradation, increased interconnect latency, or communication anomalies in some devices can serve as strong filtering conditions. However, for lightweight inference tasks with lower topology sensitivity, some minor anomaly labels can only be used as deweighting factors in the scoring, without directly eliminating the corresponding candidate nodes. In this way, the fault label filtering mechanism can balance scheduling security and scheduling flexibility.

[0123] By introducing a pre-screening or de-weighting mechanism for the fault label information, problems that would otherwise only surface in the later stages of scheduling or even during runtime can be identified earlier, at the API layer pre-scheduling stage. This reduces ineffective scheduling, lowers the probability of recovery failure, and improves the operational stability and task success rate of the target heterogeneous computing instance after recovery. Furthermore, by removing obviously abnormal nodes in advance, the number of candidate nodes participating in the Kubernetes scheduling layer's evaluation can be reduced, thereby improving the overall scheduling decision efficiency.

[0124] Subsequently, in the above embodiments, the controller can submit the pre-screened candidate nodes to the Kubernetes native scheduling layer, scheduling extension, or scheduling framework plugin for further evaluation. For example, when the target heterogeneous computing instance requires 4 TPU cards and the nodes must be in a schedulable state, if node A has only 2 available TPUs, node B has 4 TPUs but has been marked as faulty by the monitoring system, and node C has 4 healthy TPUs, then nodes A and B can be directly filtered during the API layer pre-scheduling stage, and only node C or other nodes that meet the conditions will be submitted to the Kubernetes scheduling layer, thereby improving scheduling efficiency and shortening recovery waiting time.

[0125] Further optionally, in the above embodiments, the pre-screened candidate nodes are submitted to the Kubernetes scheduling layer for scheduling scoring, including performing resource de-fragmentation scoring using a bin packing strategy. The steps of resource de-fragmentation scoring include: obtaining the remaining heterogeneous computing power resource distribution information of each candidate node; determining the resource fragmentation degree corresponding to each candidate node based on the remaining heterogeneous computing power resource distribution information; determining whether the remaining heterogeneous computing power resources of each candidate node meet the centralized bin packing allocation conditions; if the resource fragmentation degree of the candidate node is lower than a set threshold, or the remaining heterogeneous computing power resources of the candidate node meet the centralized bin packing allocation conditions, then the scheduling score of the candidate node is increased; if the resource fragmentation degree of the candidate node is not lower than a set threshold, and the remaining heterogeneous computing power resources of the candidate node do not meet the centralized bin packing allocation conditions, then the scheduling score of the candidate node is decreased.

[0126] Specifically, the remaining heterogeneous computing power resource distribution information can include not only the number of remaining devices on each candidate node, but also the distribution location of the remaining devices within the nodes, the topological combinations that can be formed between the devices, and the continuity of the remaining resources after allocation. The degree of resource fragmentation can be characterized by the degree of dispersion of the remaining devices after they are divided, the ability to form large-scale device groups, or the number of complete resource blocks that meet the needs of subsequent multi-card tasks. Furthermore, the centralized binning allocation condition is set using a binning strategy. The centralized binning allocation condition can be understood as prioritizing the placement of tasks on nodes that can support the instance with a high degree of concentration, while meeting the recovery needs of the current target heterogeneous computing power instance, thereby reserving more complete idle nodes or complete resource blocks for subsequent large-scale tasks.

[0127] For example, in a cluster containing multiple 8-TPU nodes, if the target heterogeneous computing instance requires 4 TPU cards, and node D currently has 4 consecutive allocable TPUs remaining, while node E also has 4 TPUs remaining, but these 4 TPUs are scattered in multiple locations unfavorable for scheduling subsequent large-scale tasks, then node D can be considered to meet the binning allocation condition and receive a higher score. Node E, however, has a lower score because its remaining resources are more dispersed and allocation would likely further exacerbate fragmentation. By introducing the binning strategy, the system can restore the current instance while considering the overall cluster resource layout, reducing the probability of expensive heterogeneous computing resources being fragmented and improving the schedulability of subsequent large-scale training tasks.

[0128] It is understood that the increase or decrease in scheduling scores can be achieved using a rule-based adjustment method. For example, multiple score adjustment tiers can be pre-set. When a candidate node meets the centralized binning allocation conditions and its resource fragmentation level is significantly lower than a set threshold, the first-tier reward score is increased. When a candidate node meets only one of the conditions, the second-tier reward score is increased. When a candidate node neither meets the centralized binning allocation conditions nor has a high degree of fragmentation, the first-tier penalty score is deducted. When the fragmentation level of a candidate node further deteriorates, the second-tier penalty score is deducted. Through this tiered adjustment method, the differentiated characterization of the resource layout quality of candidate nodes can be achieved without significantly increasing the complexity of score calculation.

[0129] Furthermore, in some implementations, the improvement or reduction of the scheduling score can also be achieved using a continuous function approach. Specifically, the corresponding score increment or decrement can be calculated based on at least one of the following: changes in resource fragmentation before and after candidate node allocation, changes in the number of retainable complete resource blocks, and changes in the continuity index of remaining resources. If, after a candidate node hosts the target heterogeneous computing power instance, the number of retained complete idle resource blocks is relatively large and the fragmentation change is relatively small, the corresponding score increment is relatively large. If, after a candidate node hosts the target heterogeneous computing power instance, the number of complete idle resource blocks is significantly reduced and the fragmentation degree is significantly increased, the corresponding score decrement is relatively large. In this way, the resource defraction scoring can more precisely reflect the impact of different candidate nodes on the overall resource layout of the cluster.

[0130] Regarding the aforementioned threshold, in some implementations, the threshold can be determined based on at least one of the following: node device size, typical task specification distribution in the cluster, historical scheduling records, and preset resource management strategies. That is, the threshold is not fixed but can be configured or dynamically adjusted based on the actual cluster operating characteristics. For example, for a TPU cluster primarily consisting of 4-card and 8-card tasks, the ability to retain complete 4-card and 8-card resource blocks can be used as an important basis for determining the resource fragmentation threshold. For a cluster primarily consisting of small-scale inference tasks, the threshold can be appropriately relaxed to avoid overemphasizing centralized packing and affecting real-time resource utilization.

[0131] In some implementations, the set threshold can be determined using a static configuration method. Specifically, the system can pre-set the resource fragmentation threshold by maintenance personnel, the resource management strategy module, or the scheduling strategy configuration module. For example, the threshold can be set based on the proportional relationship between the total number of nodes and the dispersion of remaining devices, or based on whether nodes can still form a complete device group of a preset size after allocation. When the resource fragmentation index of a candidate node is lower than the threshold, it indicates that the remaining resource layout of the node is relatively optimal. When the resource fragmentation index is not lower than the threshold, it indicates that the remaining resource layout of the node has an adverse impact on subsequent large-scale tasks.

[0132] In other implementations, the set threshold can be determined using dynamic learning or statistical adjustment. Specifically, the resource fragmentation threshold can be updated by statistically analyzing changes in task scheduling success rate, average waiting time, large-scale task blocking probability, or resource utilization under different fragmentation levels based on historical scheduling results. For example, if the system consistently monitors that the scheduling success rate of 4-card or 8-card tasks significantly decreases after the remaining resource dispersion of a certain type of node exceeds a first value, then this first value can be determined as or updated as the set threshold. This approach allows the threshold to better match the actual business load characteristics.

[0133] For example, in an 8-TPU node cluster, if platform statistics show that when the remaining 4 TPUs on a single node can form a continuous allocable resource block, the average waiting time for subsequent 4-card training tasks is low. However, when the remaining 4 TPUs are divided into multiple discrete combinations, subsequent 4-card training tasks often need to wait for other nodes to release resources. In this case, the system can use "whether a complete 4-card resource block can be retained" as one of the centralized binning allocation conditions, and use "the number of complete resource blocks after allocation is lower than a predetermined number" or "the proportion of discrete resources after allocation exceeds a predetermined proportion" as the criteria for determining whether the degree of resource fragmentation is not lower than a set threshold. In this scenario, node D can receive a positive score adjustment because it can maintain a relatively optimal resource layout after hosting the current instance. Node E, because hosting the current instance would significantly disrupt the complete resource block structure, can be subject to a negative score adjustment.

[0134] Further optionally, in the above embodiments, submitting the pre-screened candidate nodes to the Kubernetes scheduling layer for scheduling scoring further includes: determining the snapshot data migration cost based on the snapshot data volume, network transmission bandwidth, and migration overhead; determining the waiting time based on the cluster queuing waiting time; determining the resource waiting cost based on the waiting time and the instance priority factor; and determining the spatiotemporal trade-off score corresponding to the candidate node based on the snapshot data migration cost and the resource waiting cost.

[0135] The snapshot data migration cost characterizes the time cost required to restore the root file system snapshot of the target heterogeneous computing instance from its current storage location to a candidate node. The resource waiting cost characterizes the time cost required to wait for the original node or other better nodes to release heterogeneous computing resources that meet the conditions. The migration overhead, in addition to network transmission time, may also include snapshot mount initialization time, underlying storage system processing overhead, and container recovery preparation overhead. The instance priority factor can reflect the sensitivity of different services to recovery latency; for example, the priority factor of an interactive development instance can be higher than that of a regular background task instance. The scheduling layer can construct a spatiotemporal tradeoff score based on the snapshot data migration cost and the resource waiting cost to choose between waiting for the original node to release resources and migrating to a new node to restore the instance.

[0136] For example, if the snapshot data of a target heterogeneous computing instance is 40GB, the available bandwidth between candidate node F and the storage location of the snapshot is high, the estimated migration and recovery time is about 8 seconds, and the estimated queuing time for the original saving node to release available 4 TPU resources is 180 seconds, then the system can prioritize migrating to node F to recover the instance. Conversely, if candidate node G has available resources, but the network distance between it and the location of the snapshot data is far and the migration time is estimated to exceed 300 seconds, while the original saving node is expected to provide resources that meet the conditions again after 20 seconds, then the system can determine that waiting for the original node is better. Through the aforementioned spatiotemporal trade-off scoring, the system can avoid simply fixing either local recovery or cross-node migration, and instead dynamically select the recovery path with the lower overall cost based on the actual scenario.

[0137] Further optionally, in the above embodiments, submitting the pre-screened candidate nodes to the Kubernetes scheduling layer for scheduling scoring further includes: obtaining the first TPU topology fingerprint corresponding to the target heterogeneous computing instance when its state is saved; obtaining the second TPU topology fingerprint corresponding to the allocatable heterogeneous computing resources on each candidate node; determining the topology matching score of each candidate node based on the degree of matching between the first TPU topology fingerprint and the second TPU topology fingerprint; and determining the target node based on the topology matching score.

[0138] In some implementations, the first TPU topology fingerprint can be derived from the snapshot metadata recorded in step 102, used to characterize the TPU device interconnection relationships actually occupied by the target heterogeneous computing instance during state saving. The second TPU topology fingerprint is used to characterize the interconnection relationships of the candidate node's currently allocable TPU resources. For scenarios requiring multi-card collaborative training, simply ensuring that the number of devices meets the requirements is usually insufficient to guarantee the performance after recovery; it is also necessary to ensure that the combination of TPU devices used for recovery is as consistent as possible with the interconnection structure as it was during state saving. For example, if the target heterogeneous computing instance used a 4-card Ring topology during state saving, and node H can currently provide 4 TPUs forming the same Ring structure, then node H's topology matching score can be high. If node I can also provide 4 TPUs, but its devices can only form a weak chain connection or there is higher cross-card communication latency, then node I's score can be relatively low. By introducing the topology matching score, the selection of recovery nodes can be improved from simple device quantity matching to state-aware scheduling based on hardware interconnection characteristics, thereby improving the consistency and stability of training performance after recovery.

[0139] Further optionally, in the above embodiments, the TPU topology fingerprint includes at least one of device interconnect bandwidth, device interconnect latency, and physical connection structure. Based on this, the topology matching score of each candidate node is determined according to the matching degree between the first TPU topology fingerprint and the second TPU topology fingerprint, including: obtaining the target topology information corresponding to the target heterogeneous computing power instance when the state is saved, wherein the target topology information includes a first weighted adjacency matrix, the first weighted adjacency matrix being used to characterize the interconnection bandwidth and interconnection latency between TPU devices; obtaining candidate topology information corresponding to the allocatable heterogeneous computing power resources on each candidate node, wherein the candidate topology information includes a second weighted adjacency matrix, the second weighted adjacency matrix being used to characterize the interconnection bandwidth and interconnection latency between allocatable TPU devices on the candidate node; determining whether the candidate topology information and the target topology information are completely matched; if the candidate topology information and the target topology information are completely matched, then a first topology matching score is determined for the corresponding candidate node; if the candidate topology information and the target topology information are not completely matched, then it is determined whether the candidate topology information contains the topology subgraph corresponding to the target topology information; if the candidate topology information contains the topology subgraph, then a second topology matching score is determined for the corresponding candidate node; wherein, the first topology matching score is higher than the second topology matching score.

[0140] Specifically, the first and second weighted adjacency matrices can abstract each TPU device as a node in a graph, and the interconnection links between TPU devices as weighted edges, where the edge weights can reflect interconnection bandwidth, transmission latency, or a combination of both. Through these weighted adjacency matrices, the originally complex physical connection structure of TPU devices can be transformed into computable and comparable structured data. A complete match can be understood as the topology formed by the TPU devices to be allocated in the candidate nodes satisfying preset consistency conditions in terms of the number of devices, connection relationships, and key connection weights with the target topology at the time of state saving. The inclusion of a topology subgraph can be understood as a candidate node, although not completely identical to the target topology, containing a substructure that covers the main structural features of the target topology in its currently allocable device topology. For example, if the target topology corresponding to the target heterogeneous computing power instance at the time of saving is a 4-card Ring structure, then its first weighted adjacency matrix can reflect the high-bandwidth, low-latency connection relationships between adjacent TPUs. If the four TPUs currently allocable on node J happen to form the same Ring structure, it can be considered a complete match and assigned a first topology matching score. If the currently allocatable resources on node K are an 8-card Mesh structure, and a subgraph with communication performance close to that of the 4-card Ring structure can be extracted, then this subgraph can be considered to contain the target topology and assigned a second topology matching score. Since a perfect match is more conducive to maintaining the original training performance after recovery, the first topology matching score should be higher than the second topology matching score. Through this method, an upgrade from coarse-grained resource matching to fine-grained topology semantic matching can be achieved.

[0141] Further optionally, in the above embodiments, after determining the topology matching score of each candidate node based on the matching degree between the first TPU topology fingerprint and the second TPU topology fingerprint, the method further includes: if no candidate node that fully matches the first TPU topology fingerprint is obtained within a preset time, obtaining the user-configured topology matching strategy; if the topology matching strategy is a strict matching strategy, then continuing to wait for candidate nodes that meet the full matching condition; if the topology matching strategy is a best-effort matching strategy, then determining the target node from candidate nodes whose topology matching score is lower than the full matching score and meets the operating conditions.

[0142] The topology matching strategy in the above embodiments can be used as a recovery preference parameter set by the user when creating or configuring the target heterogeneous computing power instance, in order to balance the relationship between recovery latency and performance consistency.

[0143] Among them, the strict matching strategy is suitable for training scenarios that are highly sensitive to cross-card communication performance, such as large model distributed training or long-cycle iterative tasks. In this case, if the system does not find a fully matching node within a preset time, it can continue to wait in order to ensure that the hardware topology conditions obtained after recovery are consistent with those at the time of saving.

[0144] For example, under a strict matching strategy, suppose a target heterogeneous computing instance occupied 4 TPU devices when it stopped saving, and these 4 TPU devices formed a Ring interconnect structure, with the corresponding topology fingerprint already written into the snapshot metadata. The service corresponding to this instance is a large-model distributed training task, with frequent parameter synchronization and gradient communication during training, making it sensitive to cross-card interconnect bandwidth and latency. When configuring this instance, the user set the topology matching strategy to strict matching. When a subsequent startup request is initiated, the system finds among the candidate nodes that although nodes A and B can both provide 4 TPU devices, node A can currently only form a chain connection structure, node B can form a Mesh structure but it is inconsistent with the original Ring structure saved, while node C is expected to release 4 TPU devices that perfectly match the original Ring structure in a short period of time. In this case, even if nodes A and B meet the basic operating conditions, they can still not be immediately selected as the target node, but instead continue to wait for node C to meet the perfect matching condition before executing the recovery. This approach ensures that the target heterogeneous computing instance continues to operate under the same hardware topology as when the state was saved after restoration, thereby reducing issues such as decreased cross-card communication efficiency, training throughput fluctuations, or unstable convergence performance caused by topology changes.

[0145] For example, a long-cycle iterative training task has been running for several days, relying on an 8-card high-bandwidth interconnect topology before being stopped and saved. If, during recovery, the task is scheduled to a candidate node that only meets the required number of devices but has poor topology quality, it may significantly increase the training time per step, thus extending the overall training cycle. In this scenario, a strict matching strategy allows the system to prioritize waiting for candidate nodes that meet the original topology conditions, rather than sacrificing long-term training performance to shorten short-term recovery time. Therefore, a strict matching strategy is more suitable for business scenarios with long training durations, high requirements for hardware topology consistency, and a high tolerance for short-term waiting times.

[0146] The best-fit matching strategy is suitable for scenarios that are more sensitive to recovery time, such as interactive development, debugging, or lightweight inference tasks. In this case, even if a fully matching node cannot be obtained in a short time, the target node can be selected from the candidate nodes that meet the basic operating conditions and have a high topology score, so as to prioritize ensuring that the instance is restored to availability as soon as possible.

[0147] For example, under a best-effort matching strategy, suppose a target heterogeneous computing instance corresponds to an interactive development environment, and the user wants to connect to SSH or Jupyter as soon as possible after recovery to continue debugging code. When the instance's state was saved, the corresponding TPU topology was a 4-card Ring structure. However, upon subsequent startup, the system failed to find a candidate node that fully matches the Ring structure within a preset time. Only node D was found to provide 4 TPU devices that meet basic operating conditions. Its topology differs somewhat from the original topology, but it can still support the current development and debugging tasks. Meanwhile, node E is expected to wait a considerable amount of time before releasing a fully matching 4-card Ring structure. In this situation, if the user pre-configures the topology matching strategy to a best-effort matching strategy, the system can directly select the target node from node D, which meets basic operating conditions and has a relatively high topology score, instead of waiting for node E. This prioritizes the recovery and availability of the target heterogeneous computing instance. In this way, the user can enter the development environment more quickly and continue executing code debugging, script verification, or lightweight inference tasks.

[0148] For example, in a lightweight inference scenario, a target heterogeneous computing instance is primarily used to load a pre-trained model and respond to inference requests. Although this instance uses a device combination with optimal topology when saving its state, the post-recovery workload has a relatively low dependence on cross-card communication and is more concerned with instance recovery speed and service availability. In this case, even if the system does not find a candidate node that perfectly matches the original topology within a preset time, it can prioritize the recovery of the instance by selecting a node with a higher topology score and lower recovery cost from multiple runnable candidate nodes under a best-effort matching strategy. This reduces service downtime and improves user experience.

[0149] For example, if a training instance requires a 4-card Ring topology, and within a preset waiting time of 30 seconds, only node L has a fully matching topology but is still in an unallocated state, while node M can provide a combination of devices containing the target topology subgraph, then when the user configures a strict matching policy, the system can continue to wait for node L to release resources. When the user configures a best-effort matching policy, the system can directly select node M to perform recovery. By introducing the aforementioned topology matching degradation mechanism, recovery scheduling can balance performance consistency and business timeliness, enhancing the system's adaptability and recovery flexibility in complex resource environments.

[0150] In summary, strict matching and best-effort matching strategies correspond to different business objectives. The former emphasizes post-recovery performance consistency and is suitable for scenarios such as large model training and long-term distributed training that are sensitive to topology structure. The latter emphasizes recovery timeliness and is suitable for interactive development, debugging, lightweight inference, and other scenarios with high requirements for rapid recovery in a short time. By allowing users to configure different topology matching strategies according to business type, the system can flexibly balance performance consistency and recovery timeliness, thereby improving the adaptability and practicality of recovery scheduling for target heterogeneous computing instances.

[0151] Reference Figure 2 , Figure 2 This illustration shows a schematic diagram of the interaction process of the heterogeneous computing instance scheduling and state management method in an embodiment of this application. Figure 2 The interactive entities can include users, API Servers, PTC controllers, TPU container Pods, ZFS Sidecars co-located with business containers, and the underlying ZFS storage. Figure 2 The key points illustrate the state preservation and resource release interaction process of the target heterogeneous computing instance after receiving a stop request. Correspondingly, the state restoration, resource scheduling, and environment reconstruction processes after receiving a start request can be completed collaboratively by various modules in the same control link. The relevant interaction processes are explained below in conjunction with steps 101 to 105.

[0152] As an embodiment of step 101, a state change request for the target heterogeneous computing power instance is received. In this embodiment, the user can modify the target state of the persistent TPU container object PTC through the RESTful API, command-line tool, or console page provided by the user service interface module. For example, the user can change the desiredState in the PTC object from Running to Stopped, or vice versa. When the user submits the state change request, the request first reaches the API Server, which performs object verification, state writing, and event distribution. After the PTC controller listens for the creation, update, or deletion event of the PTC object, it can further parse the instance identifier, instance specifications, current state, target state, and historical snapshot information corresponding to the target heterogeneous computing power instance, and decide whether to trigger the subsequent state saving process or the state recovery process based on the target state. Figure 2 As can be seen, when a user submits a stop request, the API Server receives the state modification operation of the PTC object and notifies the PTC controller of the state change, thereby providing a trigger entry point for the subsequent step 102.

[0153] As an embodiment of step 102, if the state change request is a stop request, a state saving process corresponding to the target heterogeneous computing power instance is triggered to perform a snapshot operation on the root file system of the target heterogeneous computing power instance before releasing the heterogeneous computing power resources, and to record the snapshot metadata corresponding to the snapshot operation. Combined with... Figure 2 After the PTC controller learns that the user has changed desiredState to Stopped, it can first update the instance state to Stopped and then delete the Pod corresponding to the target heterogeneous computing instance via the Kubernetes interface, or adjust the replica count of the associated workload to zero, thereby triggering the container termination process. Subsequently, the API Server or node-side runtime sends a termination signal to the TPU container Pod. Before the business container actually exits, the ZFS Sidecar co-located with the business container will respond to the PreStop lifecycle hook and begin executing the state saving logic. In some implementations, the ZFS Sidecar can first pause the business processing to reduce state write disturbances; then, it performs a ZFS Snapshot atomic snapshot operation on the currently active writable layer corresponding to the root file system, and performs a renaming or state promotion operation after the snapshot is completed, mapping the snapshot to the persistent layer used for subsequent recovery. After completing the above processing, the ZFS Sidecar can report the snapshotID corresponding to this snapshot to the PTC controller or write it to the shared state file. The PTC controller updates the snapshot metadata and instance state fields accordingly, such as updating lastSavedSnapshotID. After the PreStop process is complete, the Pod terminates, and the system releases heterogeneous computing resources such as TPUs. Further device cleanup operations can then be performed, including cleaning up residual processes, resetting device state, erasing residual data in GPU memory, and performing health checks. Finally, the PTC controller updates the PTC object state to Stopped after confirming Pod termination and resource release. Through this interaction, the root file system state can be solidified before releasing heterogeneous computing resources, avoiding the problem of runtime environment loss upon instance termination in existing technologies.

[0154] As an embodiment of step 103, if the state change request is a start request, the target node for restoring the target heterogeneous computing instance is determined based on the snapshot metadata. Although Figure 2 The focus is on demonstrating the shutdown process, but in conjunction with... Figure 2In the corresponding startup scenario, users can also initiate a startup request by modifying the desiredState of the PTC object to Running. After receiving the request, the PTC controller can read the instance specification information, the most recent snapshot metadata, and the current cluster status corresponding to the target heterogeneous computing instance. It first performs pre-scheduling filtering at the API layer, filtering out nodes with insufficient resources, abnormal health status, or fault labels. Then, the filtered candidate nodes are submitted to the Kubernetes scheduling layer or scheduling extension module for comprehensive scoring. This scoring can comprehensively consider resource de-fragmentation requirements, snapshot data migration costs, resource waiting costs, and the degree of TPU topology fingerprint matching at the time of state saving. For example, if the snapshot data is saved locally on the original node, and the original node is about to release TPU resources that meet the conditions, the system may tend to wait for the original node. If another candidate node can provide a TPU device combination that is consistent with or similar to the topology at the time of saving, and its cost of accessing the snapshot data is lower, the system can determine it as the target node. Thus, step 103 achieves state-aware scheduling based on snapshot metadata, rather than coarse-grained scheduling based solely on the number of devices.

[0155] As an embodiment of step 104, dynamic environment reconstruction is performed on the target node, and the root file system of the target heterogeneous computing instance is restored based on the snapshot metadata. After the PTC controller completes the target node selection, it can dynamically build or update the Pod specification for starting the target heterogeneous computing instance according to the latest user configuration and the current environment of the target node. The dynamic environment reconstruction may correspond to context injection logic, which is used to re-establish the context configuration required for instance operation in addition to restoring the root file system. In some implementations, the controller can inject network drive mounting configuration, HugePages configuration, shared memory configuration, and service port mapping configurations such as SSH and Jupyter into the Pod specification, so that the restored instance inherits the software environment state at the time of original saving and can adapt to the current infrastructure environment and the latest user configuration. After the context configuration injection is completed, the Rootfs persistence and state management module on the target node can locate the target snapshot according to the snapshot identifier, snapshot data location, and related persistence layer identifier in the snapshot metadata, and restore the root file system through ZFS Clone, on-demand loading, or other snapshot recovery methods. Especially when the root file system is large, critical data blocks required during the startup phase can be restored first to shorten the cold start latency.

[0156] As an embodiment of step 105, after the root file system is restored, heterogeneous computing resources are allocated to the target heterogeneous computing instance and the target heterogeneous computing instance is started. In some implementations, after the root file system on the target node is mounted or critical data blocks are restored, Kubelet can start the target container according to the updated Pod specification. The lightweight TPU container runtime interface module or device plugin can, according to the scheduling result, pass through the TPU device file allocated to the instance to the container and configure the corresponding access permissions and cgroup restrictions. If the platform adopts a combination of general CRIShim and OCI Hooks, TPU device initialization, driver mounting, and runtime environment injection can be completed through OCI Hooks before the container starts. If the platform adopts a direct integration with the TPU driver or management software, the runtime interface module can directly call the underlying tools to complete the TPU configuration and container enabling. After the target heterogeneous computing instance is successfully started, the PTC controller can update the instance status to Running and record the running node, allocated device identifier, and latest running status information. Through the above steps, the system completes the entire closed loop from stopping and saving to starting and restoring, enabling the target heterogeneous computing instance to achieve a virtual machine-like shutdown retention environment and a fast startup recovery experience.

[0157] Furthermore, from Figure 2 As can be seen from the stop interaction process and the start-up and recovery interaction processes in steps 101 to 105 above, this embodiment of the application achieves decoupled management of heterogeneous computing resources and runtime environment state through the collaboration between the PTC controller, the Rootfs persistence and state management module, the scheduling extension module, the lightweight TPU container runtime interface module, and the underlying storage system. That is, the root file system and snapshot metadata are retained during the stop phase, and node selection, environment reconstruction, and instance recovery are performed based on the saved state during the start phase. This supports both time-sharing reuse of expensive heterogeneous computing resources and ensures the continuity, consistency, and recoverability of the user's runtime environment.

[0158] The above describes a heterogeneous computing instance scheduling and state management method in the embodiments of this application. The following describes the heterogeneous computing instance scheduling and state management device that executes the above heterogeneous computing instance scheduling and state management method.

[0159] See Figure 3 ,like Figure 3 The diagram shows a structural schematic of a heterogeneous computing power instance scheduling and state management device. The heterogeneous computing power instance scheduling and state management device in this embodiment can achieve the above-mentioned... Figure 1The steps of the heterogeneous computing instance scheduling and state management method executed in the corresponding embodiments are described above. The functions implemented by the heterogeneous computing instance scheduling and state management device can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, and the modules can be software and / or hardware. The heterogeneous computing instance scheduling and state management device may include an input / output module 301 and a processing module 302. The functional implementation of the processing module 302 and the input / output module 301 can be found in [reference missing]. Figure 1 The operations performed in the corresponding embodiments will not be described in detail here. For example, the processing module 302 can be used to control the sending, receiving, and acquiring operations of the input / output module 301.

[0160] The input / output module 301 is configured to receive a state change request for a target heterogeneous computing power instance, the state change request including at least a stop request and a start request; The processing module 302 is configured to, if the state change request is a stop request, trigger a state saving process corresponding to the target heterogeneous computing instance, to perform a snapshot operation on the root file system of the target heterogeneous computing instance before releasing the heterogeneous computing resources, and record snapshot metadata corresponding to the snapshot operation; if the state change request is a start request, determine a target node for restoring the target heterogeneous computing instance based on the snapshot metadata; perform dynamic environment reconstruction on the target node, and restore the root file system of the target heterogeneous computing instance based on the snapshot metadata; after the root file system is restored, allocate heterogeneous computing resources to the target heterogeneous computing instance and start the target heterogeneous computing instance.

[0161] For the specific steps and functions of the above modules, as well as other related functions, please refer to the method embodiments above, which will not be described in detail here.

[0162] In this embodiment, the runtime environment state and user context of the target heterogeneous computing instance can be maintained while releasing heterogeneous computing resources, resulting in a heterogeneous computing instance with complete state preservation capabilities and support for rapid recovery. This reduces the idle cost of heterogeneous computing resources and the user environment reconstruction cost, enabling the system to start and stop heterogeneous computing instances on demand, quickly restore the root file system, and reconstruct the runtime environment in a consistent manner. This improves the continuity and stability of heterogeneous computing resource scheduling, reduces resource fragmentation, the probability of instance recovery failure, and the platform-side operation and maintenance scheduling overhead.

[0163] This application also relates to a chip system including at least one processor and an interface circuit. The processor includes a plurality of vector storage units. The processor is used to perform instruction and / or data interaction through the interface circuit, causing the chip system to perform the methods of any of the above embodiments.

[0164] In one possible implementation, the chip system may also directly include a memory in which computer programs or computer instructions are stored.

[0165] For example, the memory can be volatile memory or non-volatile memory, or may include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DRRAM).

[0166] This application also relates to a processor, which includes a plurality of storage units for calling computer programs or computer instructions stored in the memory to cause the processor to execute the methods described in any of the above embodiments.

[0167] For example, in the embodiments of this application, the processor is an integrated circuit chip with signal processing capabilities. For instance, the processor may be an FPGA, a general-purpose processor, a DSP, an ASIC, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, a SoC, a CPU, a network processor (NP), a microcontroller unit (MCU), a PLD, or other integrated chips, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. In one possible implementation, the embodiments of this application also provide a computer-readable storage medium storing program code, which, when executed on the computer, causes the computer to perform the above-described method embodiments.

[0168] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0169] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0170] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, apparatuses, or modules, and may be electrical, mechanical, or other forms.

[0171] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0172] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0173] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program is loaded and executed on a computer, it generates, in whole or in part, the processes or functions described in the embodiments of this application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, an optical medium, or a semiconductor medium, etc.

[0174] The technical solutions provided in the embodiments of this application have been described in detail above. Specific examples have been used in the embodiments of this application to illustrate the principles and implementation methods of the embodiments of this application. The description of the above embodiments is only for the purpose of helping to understand the methods and core ideas of the embodiments of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the embodiments of this application. Therefore, the content of this specification should not be construed as a limitation on the embodiments of this application.

Claims

1. A method for scheduling and managing the state of heterogeneous computing power instances, characterized in that, The method includes: Receive a state change request for a target heterogeneous computing power instance, the state change request including at least a stop request and a start request; If the state change request is a stop request, the state saving process corresponding to the target heterogeneous computing power instance is triggered to perform a snapshot operation on the root file system of the target heterogeneous computing power instance before releasing the heterogeneous computing power resources, and record the snapshot metadata corresponding to the snapshot operation. If the status change request is a startup request, the target node for restoring the target heterogeneous computing instance is determined based on the snapshot metadata. Perform dynamic environment reconstruction on the target node and restore the root file system of the target heterogeneous computing instance based on the snapshot metadata; After the root file system is restored, heterogeneous computing resources are allocated to the target heterogeneous computing instance and the target heterogeneous computing instance is started.

2. The heterogeneous computing power instance scheduling and state management method according to claim 1, characterized in that, The step of determining the target node for restoring the target heterogeneous computing instance based on the snapshot metadata includes: Upon receiving the startup request, candidate nodes are pre-screened through API layer pre-scheduling to remove nodes whose resources do not meet instance specification requirements and nodes that do not meet health conditions. The pre-screened candidate nodes are submitted to the Kubernetes scheduling layer for scheduling and scoring; The target node is determined based on the scheduling score results.

3. The heterogeneous computing power instance scheduling and state management method according to claim 2, characterized in that, The step of submitting the pre-screened candidate nodes to the Kubernetes scheduling layer for scheduling scoring includes performing resource de-fragmentation scoring using a bin packing strategy. The steps of resource de-fragmentation scoring include: Obtain information on the distribution of remaining heterogeneous computing power resources for each candidate node; The degree of resource fragmentation for each candidate node is determined based on the distribution information of remaining heterogeneous computing power resources. Determine whether the remaining heterogeneous computing power resources of each candidate node meet the centralized binning allocation conditions; the centralized binning allocation conditions are set using the Bin Packing strategy. If the resource fragmentation of a candidate node is lower than a set threshold, or if the remaining heterogeneous computing power of a candidate node meets the conditions for centralized binning and allocation, then the scheduling score of the candidate node will be increased. If the resource fragmentation level of a candidate node is not lower than a set threshold, and the remaining heterogeneous computing power resources of the candidate node do not meet the centralized binning allocation conditions, then the scheduling score of the candidate node will be reduced.

4. The heterogeneous computing power instance scheduling and state management method according to claim 2, characterized in that, The step of submitting the pre-screened candidate nodes to the Kubernetes scheduling layer for scheduling scoring also includes: Determine the snapshot data migration cost based on the snapshot data volume, network transmission bandwidth, and migration overhead. The waiting time is determined based on the cluster queuing time, and the resource waiting cost is determined based on the waiting time and the instance priority factor. The spatiotemporal trade-off score corresponding to the candidate node is determined based on the snapshot data migration cost and the resource waiting cost.

5. The heterogeneous computing power instance scheduling and state management method according to claim 2, characterized in that, The step of submitting the pre-screened candidate nodes to the Kubernetes scheduling layer for scheduling scoring also includes: Obtain the first TPU topology fingerprint corresponding to the target heterogeneous computing instance when its state is saved; Obtain the second TPU topology fingerprint corresponding to the allocable heterogeneous computing power resources on each candidate node; Based on the degree of matching between the first TPU topological fingerprint and the second TPU topological fingerprint, the topological matching score of each candidate node is determined; The target node is determined based on the topology matching score.

6. The heterogeneous computing power instance scheduling and state management method according to claim 5, characterized in that, The TPU topology fingerprint includes at least one of device interconnect bandwidth, device interconnect latency, and physical connection structure. The step of determining the topology matching score of each candidate node based on the matching degree between the first TPU topology fingerprint and the second TPU topology fingerprint includes: Obtain the target topology information corresponding to the target heterogeneous computing power instance when the state is saved, wherein the target topology information includes a first weighted adjacency matrix, the first weighted adjacency matrix being used to characterize the interconnection bandwidth and interconnection latency between TPU devices; Obtain candidate topology information corresponding to the allocatable heterogeneous computing power resources on each candidate node, wherein the candidate topology information includes a second weighted adjacency matrix, the second weighted adjacency matrix being used to characterize the interconnection bandwidth and interconnection latency between allocatable TPU devices on the candidate nodes; Determine whether the candidate topology information and the target topology information are completely matched; If the candidate topology information completely matches the target topology information, then the first topology matching score of the corresponding candidate node is determined; If the candidate topology information does not completely match the target topology information, then it is determined whether the candidate topology information contains the topology subgraph corresponding to the target topology information; If the candidate topology information contains the topology subgraph, then a second topology matching score for the corresponding candidate node is determined; wherein the first topology matching score is higher than the second topology matching score.

7. The heterogeneous computing power instance scheduling and state management method according to claim 6, characterized in that, After determining the topology matching score of each candidate node based on the matching degree between the first TPU topology fingerprint and the second TPU topology fingerprint, the method further includes: If no candidate node that completely matches the topology fingerprint of the first TPU is obtained within a preset time, the topology matching strategy configured by the user is obtained. If the topology matching strategy is a strict matching strategy, then continue to wait for a candidate node that meets the full matching condition; If the topology matching strategy is a best-effort matching strategy, then the target node is determined from the candidate nodes whose topology matching score is lower than the perfect matching score and which meet the operating conditions.

8. The heterogeneous computing power instance scheduling and state management method according to claim 1, characterized in that, The dynamic environment reconstruction performed on the target node includes: Obtain the instance configuration information corresponding to the target heterogeneous computing instance when it starts up. The instance configuration information includes at least one of the user's latest configured cloud drive mounting parameters, memory parameters, and network port parameters. Based on the instance configuration information, construct a Pod specification for launching the target heterogeneous computing instance, or update an existing Pod specification; Inject context configuration corresponding to the instance configuration information into the Pod specification. The context configuration includes at least one of the following: cloud drive mounting configuration, HugePages configuration, shared memory configuration, and service port mapping configuration. The runtime configuration for launching the target heterogeneous computing instance is generated and updated based on the injected Pod specification.

9. The heterogeneous computing power instance scheduling and state management method according to claim 1, characterized in that, The process of restoring the root file system of the target heterogeneous computing instance based on the snapshot metadata includes: The snapshot data corresponding to the root file system is determined based on the snapshot metadata. Using either on-demand loading or cloning, the snapshot data is loaded with priority during the initial startup of the container, and block-level hotspot preheating is performed on the startup data blocks to restore the root file system of the target heterogeneous computing instance.

10. The heterogeneous computing power instance scheduling and state management method according to claim 1, characterized in that, The step of performing a snapshot operation on the root file system of the target heterogeneous computing power instance before releasing the heterogeneous computing power resources includes: In response to the stop request, the persistent TPU container controller triggers a container termination process, which includes deleting the associated Pod via the Kubernetes interface or adjusting the number of associated workload replicas to zero. The snapshot operation is executed by triggering the PreStop lifecycle hook of the co-resident auxiliary container before the business container terminates; After the snapshot operation is completed, a snapshot identifier is recorded, and the snapshot metadata is updated based on the snapshot identifier.

11. The heterogeneous computing power instance scheduling and state management method according to claim 10, characterized in that, After triggering the PreStop lifecycle hook of the co-resident auxiliary container to execute the snapshot operation before the business container terminates, the method further includes: Identify the currently active writable layer of the root file system corresponding to the business container; The currently active writable layer is used to perform a solidified state operation to generate an atomic snapshot; A renaming operation is performed on the atomic snapshot to elevate its hierarchical level and map it to a persistent layer for subsequent recovery. The identification information corresponding to the persistent layer and the association between the identification information and the snapshot identifier corresponding to the atomic snapshot are written into the snapshot metadata, so that the root file system can be restored based on the snapshot metadata during subsequent startup.

12. The heterogeneous computing power instance scheduling and state management method according to claim 10, characterized in that, After triggering the PreStop lifecycle hook of the co-resident auxiliary container to execute the snapshot operation before the business container terminates, the method further includes: If the snapshot operation of the co-resident auxiliary container fails, a non-zero exit code is returned; The snapshot failure information corresponding to the non-zero exit code is captured by the persistent container controller; Based on the snapshot failure information, update the status of the target heterogeneous computing power instance to the snapshot failure status; Retain the Pod corresponding to the target heterogeneous computing power instance; Trigger an alarm message to notify operations and maintenance personnel to intervene and handle the situation.

13. The heterogeneous computing power instance scheduling and state management method according to claim 10, characterized in that, The response to the stop request triggering the container termination process further includes: If a node malfunctions and the PreStop lifecycle hooks are not executed, a background cleanup task will be triggered when the node recovers or the Pod is evicted. The background cleanup task checks whether there is a dangling snapshot corresponding to the target heterogeneous computing power instance. When the existence of the dangling snapshot is detected, a repair or cleanup operation is performed on the dangling snapshot.

14. The heterogeneous computing power instance scheduling and state management method according to claim 1, characterized in that, Before releasing the heterogeneous computing resources, the following is also included: Clean up processes that occupy heterogeneous computing power devices; Reset the status of heterogeneous computing power devices; Erase residual data on heterogeneous computing devices; Perform health checks on heterogeneous computing devices and update the device availability status based on the health check results.

15. The heterogeneous computing power instance scheduling and state management method according to claim 2, characterized in that, The pre-screening of candidate nodes through API layer pre-scheduling includes: Obtain the fault label information corresponding to the candidate node; Candidate nodes with preset fault identifiers or in an abnormal state are removed from the candidate set, or their scheduling scores are reduced in the Kubernetes scheduling layer.

16. The heterogeneous computing power instance scheduling and state management method according to claim 1, characterized in that, The snapshot metadata corresponding to the snapshot operation is recorded, including: The TPU topology fingerprint corresponding to the state saving is obtained to characterize the TPU device interconnection topology features corresponding to the target heterogeneous computing power instance when the state is saved. The TPU topology fingerprint includes a weighted adjacency matrix and a hash value generated based on the weighted adjacency matrix. The weighted adjacency matrix is ​​used to characterize the interconnection bandwidth and interconnection latency between TPU devices. Determine snapshot data location information to indicate the data storage location of the snapshot, wherein the data storage location includes at least one of the target node identifier and storage path; Write the snapshot identifier, the TPU topology fingerprint, and the snapshot data location information into the snapshot metadata; When determining the target node for restoring the target heterogeneous computing power instance based on the snapshot metadata, the reachability of the snapshot data is determined at least based on the snapshot data location information, and the degree of matching between the candidate node and the corresponding topological features when the state is saved is determined based on the TPU topological fingerprint.

17. A heterogeneous computing power instance scheduling and state management device, characterized in that, The device includes: The input / output module is configured to receive state change requests for a target heterogeneous computing power instance, the state change requests including at least a stop request and a start request; The processing module is configured to, if the state change request is a stop request, trigger a state saving process corresponding to the target heterogeneous computing instance, to perform a snapshot operation on the root file system of the target heterogeneous computing instance before releasing the heterogeneous computing resources, and record snapshot metadata corresponding to the snapshot operation; if the state change request is a start request, determine a target node for restoring the target heterogeneous computing instance based on the snapshot metadata; perform dynamic environment reconstruction on the target node, and restore the root file system of the target heterogeneous computing instance based on the snapshot metadata; after the root file system is restored, allocate heterogeneous computing resources to the target heterogeneous computing instance and start the target heterogeneous computing instance.

18. A computing device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the heterogeneous computing instance scheduling and state management method according to any one of claims 1-16.

19. A computer-readable storage medium, characterized in that, It includes instructions that, when executed on a computer, cause the computer to perform the heterogeneous computing instance scheduling and state management method as described in any one of claims 1-16.

20. A chip, characterized in that, The chip includes a processor coupled to a transceiver for executing the heterogeneous computing instance scheduling and state management method as described in any one of claims 1-16.

Images