Resource management method, apparatus, device, medium, and product

Abstract

The application discloses a resource management method and device, equipment, medium and product, and relates to the technical field of computers. The method comprises the following steps: acquiring running state information of each inference instance in a cluster and resource states of each host, the cluster comprising a plurality of hosts, and the inference instance being run by a GPU in the host; for a first inference instance in each inference instance, if the running state information indicates that the first inference instance is in an idle state, determining a migration strategy of first instance data of the first inference instance based on the resource state of a first host where the first inference instance is currently located, the migration strategy comprising at least one of intra-host migration and inter-host migration; migrating the first instance data based on the migration strategy, and releasing GPU memory occupied by the first instance data in the first host.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a resource management method, apparatus, equipment, medium and product. Background Technology

[0002] An inference instance is a service unit that loads a deep learning model into the memory of a graphics processing unit (GPU) and can independently receive inference requests and perform computations. During runtime, an inference instance requires a significant amount of GPU memory to store model parameters, intermediate activation values, and key-value cache (KV cache) data.

[0003] In related technologies, in order to improve the utilization of GPU memory, when an inference instance is in an idle state, its instance data is migrated to host memory or solid state disk (SSD) to release the GPU memory it occupies.

[0004] However, the storage capacity of host memory and local SSD is limited. As the number of idle inference instances increases or the data volume of a single instance grows, the available space of memory and SSD may be quickly exhausted, making it impossible for subsequent idle inference instances to be swapped out, and the video memory resources are difficult to be fully released. Summary of the Invention

[0005] This application provides a resource management method, apparatus, device, medium, and product. The technical solutions provided by this application include the following aspects.

[0006] On one hand, embodiments of this application provide a resource management method, the method comprising: Obtain the running status information of each inference instance in the cluster, as well as the resource status of each host. The cluster contains multiple hosts, and the inference instances are run by GPUs in the hosts. For the first inference instance among the various inference instances, when the running status information indicates that the first inference instance is in an idle state, a migration strategy for the first instance data of the first inference instance is determined based on the resource status of the first host where the first inference instance is currently located. The migration strategy includes at least one of intra-host migration and inter-host migration. The first instance data is migrated based on the migration strategy, and the GPU memory occupied by the first instance data in the first host is released.

[0007] On the other hand, embodiments of this application provide a resource management device, the device comprising: The acquisition module is configured to acquire the running status information of each inference instance in the cluster, as well as the resource status of each host. The cluster contains multiple hosts, and the inference instances are run by GPUs in the hosts. The determination module is configured to, for the first inference instance among the various inference instances, when the running status information indicates that the first inference instance is in an idle state, determine the migration strategy of the first instance data of the first inference instance based on the resource status of the first host where the first inference instance is currently located, the migration strategy including at least one of intra-host migration and inter-host migration. The migration module is configured to migrate the first instance data based on the migration strategy and release the GPU memory occupied by the first instance data in the first host.

[0008] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the above-described resource management method.

[0009] On the other hand, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and the computer program is loaded and executed by a processor to implement the above-described resource management method.

[0010] On the other hand, embodiments of this application provide a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor reads from the computer-readable storage medium and executes the computer program to implement the above-described resource management method.

[0011] The technical solution provided in this application can bring the following beneficial effects: First, the resource status of each host in the cluster and the running status information of the inference instances running on the host GPUs are obtained. Then, for the first inference instance among all inference instances, when its running status information indicates that the instance is in an idle state, the migration strategy for the instance's instance data is determined based on the resource status of the first host to which the first inference instance is currently located. Since the determined migration strategy includes intra-host migration and inter-host migration, the scope of instance data migration is expanded. Finally, the instance data is migrated according to the determined migration strategy, and the GPU memory it occupies is released. The released GPU memory resources are made available to other inference instances, realizing dynamic reuse of GPU memory resources and improving the overall resource utilization of the cluster. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating a resource management method provided in an illustrative embodiment of this application; Figure 2 This is a flowchart illustrating another illustrative embodiment of a resource management method provided in this application; Figure 3 This is a schematic diagram of a first instance data migration process provided in an illustrative embodiment of this application; Figure 4 This is a schematic diagram of the first instance data migration process provided in another illustrative embodiment of this application; Figure 5 This is a schematic diagram of the first instance data migration process provided in another illustrative embodiment of this application; Figure 6 This is a schematic diagram of an example data import process provided in an illustrative embodiment of this application; Figure 7 This is a schematic diagram of an example data import process provided in another illustrative embodiment of this application; Figure 8 This is a schematic diagram illustrating dynamic adjustment of video memory according to an illustrative embodiment of this application; Figure 9 This is a schematic diagram of the structure of a resource management device provided in an illustrative embodiment of this application; Figure 10 This is a schematic diagram of the structure of a resource management device provided in another illustrative embodiment of this application; Figure 11 This is a structural block diagram of a computer device provided in an illustrative embodiment of this application. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0014] Illustrative embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following illustrative embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0015] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0016] It should be understood that although the terms “first,” “second,” etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, a first parameter may also be referred to as a second parameter, and similarly, a second parameter may also be referred to as a first parameter. Depending on the context, the word “if” as used herein may be interpreted as “when…” or “in response to determination.”

[0017] It should be noted that this application may display a prompt interface, pop-up window, or output voice prompt information before collecting user, processor, computer device, and other related data, and during the process of collecting user-related data. This prompt interface, pop-up window, or voice prompt information is used to inform the user that their related data is being collected. This ensures that the application only begins executing the steps related to collecting user-related data after receiving confirmation from the user regarding the prompt interface or pop-up window; otherwise (i.e., without receiving confirmation from the user), the steps related to collecting user-related data end, meaning no user-related data is collected. In other words, all user data collected in this application is collected with the user's consent and authorization, and the collection, use, and processing of related user data must comply with the laws, regulations, and standards of relevant countries and regions.

[0018] First, the terms used in the embodiments of this application will be introduced.

[0019] A cluster is a computer cluster consisting of multiple computing nodes equipped with GPUs interconnected via a high-speed network. The nodes in a cluster work collaboratively to complete large-scale parallel computing tasks, such as deep learning model training, distributed inference, scientific simulations, and big data analysis. Clusters are typically deployed in data centers or cloud computing environments, supporting dynamic scaling and unified resource scheduling. Each node in a cluster is called a host.

[0020] An inference instance is a service unit that loads a deep learning model into GPU memory, capable of independently receiving inference requests and performing computations. Different inference instances occupy isolated memory spaces and do not affect each other. Each inference instance contains at least the following runtime data: model weights, inference engine context, task queue, key-value cache, intermediate activation values, and other temporary buffered data. Inference instances can be in an active or idle state, such as processing inference requests or waiting for tasks. Instance data from idle inference instances can be moved out of GPU memory to free up memory resources for other instances, thereby achieving dynamic reuse of GPU resources.

[0021] Memory isolation refers to dividing the physical GPU's memory resources into multiple independent memory regions, each allocated to a specific inference instance. Different inference instances cannot access each other's memory regions. Memory isolation can be achieved through a GPU isolation management module based on Multi-Instance GPU (MIG) technology. This involves dividing the physical GPU into multiple independent compute instances, each bound to its own dedicated memory quota, thus providing isolated memory resources for inference instances. Memory isolation avoids memory conflicts and interference between multiple inference instances, improving system security, stability, and resource predictability.

[0022] An inference instance is a service unit that loads a deep learning model into GPU memory, can independently receive inference requests, and perform computations. During runtime, inference instances require a significant amount of GPU memory to store model parameters, intermediate activation values, and key-value cache data. In related technologies, to improve GPU memory utilization, when an inference instance is idle, its instance data is migrated to host memory or SSD to release the GPU memory it occupies. However, the storage capacity of host memory and local SSDs is limited. As the number of idle inference instances increases or the data volume of a single instance grows, the available space in memory and SSDs may be quickly exhausted, preventing subsequent idle inference instances from being swapped out and hindering the full release of GPU memory resources.

[0023] To address the aforementioned issues, this application provides a resource management method. First, it acquires the resource status of each host in the cluster and the runtime status information of inference instances running on the host's GPU. Then, for the first inference instance among the inference instances, when its runtime status information indicates that the instance is idle, a migration strategy for the instance's instance data is determined based on the resource status of the first host where the first inference instance is currently located. Since the determined migration strategy includes intra-host migration and inter-host migration, the scope of instance data migration is expanded. Finally, the instance data is migrated according to the determined migration strategy, and the GPU memory it occupies is released. The released GPU memory resources are then made available to other inference instances, achieving dynamic reuse of GPU memory resources and improving the overall resource utilization of the cluster.

[0024] The resource management method provided in this application is applicable to clusters with resource management needs and can be executed by a resource management device within the cluster. This resource management device can be deployed independently outside the cluster or integrated into a host within the cluster. The host can be a terminal or a server. Terminals can be electronic devices such as mobile phones, tablets, in-vehicle terminals (vehicle infotainment systems), wearable devices, and PCs (Personal Computers). Servers can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud servers, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. In the following embodiments, for ease of description, the resource management method is illustrated using the execution of the resource management device as an example.

[0025] like Figure 1 As shown, a flowchart of a resource management method provided in an illustrative embodiment of this application is illustrated. The method can be executed by the resource management device described above, and the method includes at least one of the following steps.

[0026] Step 110: Obtain the running status information of each inference instance in the cluster, as well as the resource status of each host. The cluster contains multiple hosts, and the inference instances are run by GPUs in the hosts.

[0027] The cluster comprises multiple hosts, with inference instances running on GPUs within each host. Instance data from these instances occupies GPU memory, and the GPU memory used by different inference instances is isolated from each other. To move instance data from idle inference instances out of GPU memory, the resource management device first needs to identify the idle inference instances and their target addresses for data migration. Therefore, the resource management device needs to obtain the necessary criteria for this determination.

[0028] In some embodiments, the resource management device obtains the running status information of each inference instance in the cluster and the resource status of each host.

[0029] The runtime status information of the inference instance indicates whether the instance is currently idle or not. The resource management device can determine whether the inference instance is idle based on this runtime status information. The host's resource status includes the status information of various storage resources on the host, such as GPU memory, host memory, and local SSD. The resource management device determines the storage resource usage of each host based on its resource status, and then judges whether the host is suitable as a migration target for instance data.

[0030] Step 120: For the first inference instance among the various inference instances, if the running status information indicates that the first inference instance is in an idle state, based on the resource status of the first host where the first inference instance is currently located, determine the migration strategy of the first instance data of the first inference instance. The migration strategy includes at least one of intra-host migration and inter-host migration.

[0031] After obtaining the running status information of each inference instance in the cluster and the resource status of each host, the resource management device needs to determine the inference instances that are in an idle state and the target address for their instance data migration.

[0032] In some embodiments, for the first inference instance among various inference instances, when the running status information indicates that the first inference instance is in an idle state, the resource management device determines the migration strategy of the first instance data of the first inference instance based on the resource status of the first host where the first inference instance is currently located.

[0033] Among them, inference instances in an idle state do not perform inference tasks, and the GPU memory they occupy is not fully utilized. To improve the utilization of GPU memory, the resource management device treats idle instances as data migration targets, moves their instance data out of GPU memory, and releases the occupied memory resources for use by other inference instances.

[0034] In some embodiments, the migration strategy includes at least one of intra-host migration and inter-host migration, with intra-host migration having a higher priority than inter-host migration.

[0035] The migration strategies include intra-host migration and inter-host migration. Intra-host migration moves the instance data of the inference instance to a non-GPU memory space on the host where the inference instance resides. Inter-host migration moves the instance data of the inference instance to a non-GPU memory space in the cluster other than the host it resides on. The resource management device can migrate the instance data of the first inference instance, which is in an idle state, to the first host where the first inference instance resides, or to other hosts in the cluster other than the first host, thus expanding the range of instance data that can be migrated.

[0036] In addition, intra-host migration has a higher priority than inter-host migration. The resource management device prioritizes migrating instance data to intra-host storage space with faster recovery speed, so that it can be quickly migrated back when inference instances need to be woken up, which significantly improves migration and recovery efficiency.

[0037] Step 130: Migrate the first instance data based on the migration strategy and release the GPU memory occupied by the first instance data in the first host.

[0038] In some embodiments, after determining a migration strategy for the first instance data of the first inference instance, the resource management device marks the first instance data as migrateable, migrates the first instance data of the first inference instance based on the migration strategy, and releases the GPU memory occupied by the first instance data of the first inference instance in the first host for use by other inference instances in the first host.

[0039] For example, the resource management device issues a memory release command, marks the GPU memory occupied by the first instance data as releasable, and performs a memory release operation to release the GPU memory occupied by the first instance data for use by other inference instances in the first host. In summary, by first obtaining the resource status of each host in the cluster and the running status information of the inference instances running on the host GPU; then, for the first inference instance among the inference instances, when its running status information indicates that the instance is in an idle state, a migration strategy for the instance data is determined based on the resource status of the first host where the first inference instance is currently located. Since the determined migration strategy includes intra-host migration and inter-host migration, the migration scope of instance data is expanded. Finally, the instance data is migrated according to the determined migration strategy, and the GPU memory it occupies is released, making the released memory resources available for use by other inference instances, thus realizing dynamic reuse of GPU memory resources and improving the overall resource utilization of the cluster.

[0040] In some embodiments, the resource management device obtains the running status information of each inference instance in the cluster through a heartbeat mechanism or task queue check; and obtains the resource status of each host by periodically collecting indicators such as the video memory usage rate and remaining available video memory of each instance.

[0041] Optionally, the resource management module receives a heartbeat packet reported by each inference instance every second. The heartbeat packet contains the instance's status identifier (such as "active" or "idle") to determine whether the instance is idle, or checks the instance's task queue length. If there are no pending requests in the task queue and no inference tasks are currently being executed, the instance is ultimately confirmed to be idle. Simultaneously, the resource management device collects the memory usage and remaining available memory of each GPU instance in real time through an interface to determine the host's memory resource usage. This embodiment of the application does not impose limitations on this.

[0042] In some embodiments, the resource management device determines whether to perform intra-host migration or inter-host migration on the instance data based on the storage resource status of the first host where the first inference instance is currently located.

[0043] like Figure 2As shown, a flowchart of a resource management method provided by another illustrative embodiment of this application is illustrated. This method can be executed by the resource management device described above. Step 120 in the above-mentioned step can also be implemented as step 121 and step 122.

[0044] Step 121: When the running status information indicates that the first inference instance is in an idle state and the resource status of the first host indicates that the storage resources of the first host are sufficient, determine that the migration strategy of the first instance data of the first inference instance is intra-host migration.

[0045] In the case where the running status information indicates that the first inference instance is in an idle state, in order to release the GPU memory occupied by the first inference instance, the resource management device needs to migrate the instance data of the first inference instance out of the GPU memory; when the resource status of the first host indicates that the storage resources of the first host are sufficient, the first host where the first inference instance is located can be selected as the target for instance data migration, and the resource management device determines the migration strategy within the host, that is, to migrate the first instance data of the first inference instance to the first host where the first inference instance is located.

[0046] In some embodiments, the host has multiple storage spaces available for storing instance data. When determining to execute an in-host migration policy, the resource management device needs to further determine the specific target storage space within the host for the instance data migration.

[0047] For example, the host's storage space includes memory, SSD, etc. The resource management device can divide it into a first storage space and a second storage space, wherein the first storage space includes memory and the second storage space includes SSD, and the read / write speed of the first storage space is higher than that of the second storage space. During the migration process within the host, the resource management device determines whether to migrate instance data to the first storage space or the second storage space according to preset rules.

[0048] like Figure 3 The diagram illustrates a first instance data migration process according to an illustrative embodiment of this application. The diagram includes a host 310, which has GPU memory 311 and non-GPU memory storage space 312. The GPU memory 311 stores the first instance data 313. Before the migration, the non-GPU memory storage space 312 has a large available capacity and sufficient storage resources. Based on this, the resource management device determines to perform an intra-host migration, moving the first instance data 313 from the GPU memory 311 to the non-GPU memory storage space 312. Through this migration process, the GPU memory 311 occupied by the first instance data 313 is released and made available for use by other inference instances.

[0049] In some embodiments, the resource management device determines the target storage space for the first instance data to be migrated within the host based on the idle time of the first inference instance.

[0050] The idle time of the first inference instance can be used to characterize the recent demand for that instance. The shorter the idle time, the higher the probability of short-term memory usage demand, so the instance data should be stored in a storage space that is easy to retrieve quickly; the longer the idle time, the lower the probability of short-term memory usage demand, so the instance data can be stored in a storage space with a larger capacity but a relatively slower retrieval speed.

[0051] In some embodiments, the runtime status information of the inference instance also includes idle time. When the migration strategy for in-host migration is initially determined, the resource management device can further determine the target storage space for the first instance data migration based on the idle time of the inference instance.

[0052] In some embodiments, when the running status information indicates that the first inference instance is in an idle state, the idle time of the first inference instance reaches a first duration threshold, and the resource status of the first host indicates that the first storage space of the first host has sufficient resources, the resource management device determines that the migration strategy for the first instance data of the first inference instance is a first-level intra-host migration, which is used to migrate the instance data to the first storage space, where the first storage space refers to the storage space other than the GPU memory; when the running status information indicates that the first inference instance is in an idle state, the idle time of the first inference instance reaches a second duration threshold, and the resource status of the first host indicates that the second storage space of the first host has sufficient resources, the resource management device determines that the migration strategy for the first instance data of the first inference instance is a second-level intra-host migration, which is used to migrate the instance data to the second storage space, where the first duration threshold is less than the second duration threshold, the second storage space refers to the storage space other than the GPU memory, and the read / write speed of the first storage space is higher than the read / write speed of the second storage space.

[0053] When the first storage space has sufficient resources, the resource management device can migrate the first instance data to the first storage space. When the second storage space has sufficient resources, the resource management device can migrate the first instance data to the second storage space, with the first duration threshold being less than the second duration threshold. When the idle time of the first inference instance reaches the first duration threshold, it indicates that its idle time is short and the probability of short-term memory usage demand is high. Therefore, the resource management device migrates the first instance data to the first storage space with faster read / write speed so that GPU memory can be quickly recalled when demand arises. When the idle time of the first inference instance reaches the second duration threshold, it indicates that its idle time is long and the probability of short-term memory usage demand is low. Therefore, the resource management device migrates the first instance data to the second storage space with slower read / write speed but larger capacity to support the storage of instance data for a large number of idle inference instances.

[0054] For illustration, a first time limit threshold of 5 minutes and a second time limit threshold of 10 minutes are set. When the idle time of the first inference instance is greater than or equal to 5 minutes, the resource management device performs a first-level intra-host migration to migrate the instance data to the host memory; when the idle time of the first inference instance is greater than or equal to 10 minutes, a second-level intra-host migration is performed to migrate the instance data to the local SSD.

[0055] By migrating instance data with shorter idle times to a faster first storage space based on the idle duration of the first inference instance, GPU memory can be quickly recalled when needed. Instance data with longer idle times is migrated to a slower but larger second storage space to support the storage of large amounts of idle instance data. This tiered storage of instance data reduces the cost of using high-speed storage resources and enhances the overall throughput of the cluster.

[0056] In other embodiments, the resource management device may also determine the target storage space for migration based on the access frequency of the first instance data. For example, when the access frequency of the first instance data is high, the resource management device migrates it to the first storage space; when the access frequency is low, it migrates it to the second storage space.

[0057] In other embodiments, the resource management device may also determine the target storage space for migrating the first instance data based on the instance priority of the first inference instance. For example, when the instance priority of the first inference instance is high, the resource management device migrates its corresponding first instance data to the first storage space; when the instance priority is low, it migrates it to the second storage space. This application does not limit this approach.

[0058] Optionally, instance priority can be based on whether the inference instance is an online task. Illustratively, inference instances belonging to online tasks have higher instance priority, while inference instances belonging to offline tasks have lower instance priority. This application embodiment does not impose limitations on this.

[0059] In some embodiments, after determining the target storage space to be migrated within the host, the resource management device further determines the data content to be migrated based on the idle time of the first inference instance.

[0060] For example, when the idle time is short, only a portion of the instance data needs to be migrated; when the idle time is long, all or most of the instance data needs to be migrated. In other words, the longer the idle time, the larger the amount of data migrated, thereby freeing up more GPU memory.

[0061] In some embodiments, when determining the migration strategy for migration within the first-level host, the resource management device determines that the migration strategy for low-frequency instance sub-data in the first instance data is migration within the first-level host; when determining the migration strategy for migration within the second host, the resource management device determines that the migration strategies for low-frequency instance sub-data and high-frequency instance sub-data in the first instance data are migration within the second-level host, wherein the access frequency of low-frequency instance sub-data is lower than the access frequency of high-frequency instance sub-data.

[0062] The instance data includes low-frequency instance sub-data and high-frequency instance sub-data, with the low-frequency instance sub-data being accessed less frequently than the high-frequency instance sub-data. When the idle time is short, the inference instance may be woken up again in a short period of time. Therefore, only the low-frequency, non-urgent data needs to be moved out, while the high-frequency data is kept in the GPU memory or the first storage space with faster read and write speeds to achieve rapid recovery. When the idle time is long, the probability of the inference instance being woken up in a short period of time is low. All instance data can be moved out to maximize the release of GPU memory.

[0063] For example, low-frequency instance sub-data includes low-frequency access data such as model weights and intermediate results of completed inference tasks; high-frequency instance sub-data includes high-frequency access data such as currently requested KV cache and temporary activation values. When performing the first-level intra-host migration, the idle time is short, and only the low-frequency instance sub-data is migrated to memory; when performing the second-level intra-host migration, the idle time is long, and both the low-frequency and high-frequency instance sub-data are migrated to the local SSD.

[0064] like Figure 4The figure illustrates a schematic diagram of a first instance data migration process provided in another illustrative embodiment of this application. In the figure, the GPU memory 401 stores first instance data 404, which includes low-frequency instance sub-data 405 and high-frequency instance sub-data 406. According to a migration strategy, the resource management device migrates the low-frequency instance sub-data 405 to a first storage space 402 with faster read / write speed, or migrates both the low-frequency instance sub-data 405 and the high-frequency instance data 406 to a second storage space 403 with slower read / write speed.

[0065] By migrating low-frequency instance sub-data with low access frequency to the first storage space with faster read and write speeds during short periods of idle time, it is easier to quickly recover when there is demand. During long periods of idle time, high-frequency instance sub-data with high access frequency is migrated to the second storage space with slower read and write speeds, which makes it easier to free up high-speed storage resources to accommodate more inference instances. At the same time, cold data is stored for a long time using large-capacity storage media, thereby realizing hierarchical utilization of storage resources and optimization of overall system performance.

[0066] In other embodiments, the host contains only a single type of storage space for storing instance data. In this case, the resource management device does not need to further determine the specific migration target of the instance data within the host. This application does not limit this approach.

[0067] For example, the host may contain only memory or only SSDs as storage space for instance data. When the resource management device determines to execute an in-host migration policy, it directly migrates the instance data to this single storage space.

[0068] After determining the migration strategy, the resource management device needs to execute the determined migration strategy to migrate the first instance data.

[0069] In some embodiments, since the uncompressed instance data is large in size, direct migration would occupy a large amount of limited storage space. In order to save storage space, the resource management device first compresses the first instance data when migrating it, and then migrates the compressed data to the target storage space to improve the utilization of storage space, thereby accommodating more migrated instance data under the same storage resources.

[0070] In some embodiments, when the migration strategy is a first-level host migration, the resource management device compresses the first instance data using a first compression algorithm and migrates the compressed low-frequency instance sub-data from the GPU memory to the first storage space; when the migration strategy is a second-level host migration, the device compresses the low-frequency instance sub-data and high-frequency instance sub-data using a second compression algorithm, migrates the compressed low-frequency instance sub-data from the first storage space to the second storage space, and migrates the compressed high-frequency instance sub-data from the GPU memory to the second storage space. The compression ratio of the second compression algorithm is greater than that of the first compression algorithm.

[0071] Since the first-level host migration involves short idle times, it is necessary to minimize the latency of migration and subsequent recovery. Therefore, a lightweight compression algorithm with fast compression speed but low compression ratio is used to balance migration speed and space saving. The second-level host migration involves longer idle times, and the sensitivity to recovery latency is reduced. However, it is necessary to maximize the release of host memory resources and reduce the occupation of SSD capacity. Therefore, an algorithm with a higher compression ratio is used to save storage space.

[0072] For example, the resource management device employs the lightweight LZ4 compression algorithm (LZ4) when performing the first-level intra-host migration and the high-compression-ratio GNU zip compression algorithm (gzip) when performing the second-level intra-host migration. By using the first compression algorithm with a lower compression ratio for the first-level migration and the second compression algorithm with a higher compression ratio for the second-level migration, a flexible trade-off between migration efficiency and storage cost is achieved during different idle periods.

[0073] In addition, since low-frequency instance sub-data has been migrated to the first storage space when the idle time is short, when the idle time is long, the low-frequency instance sub-data and high-frequency instance sub-data need to be migrated to the second storage space together, and the data consistency between the two needs to be maintained.

[0074] By using a first compression algorithm with a lower compression ratio for migration within the first-level host, excessive decompression time is avoided to prevent recovery delays and ensure that instances can be woken up quickly. For migration within the second-level host, a second compression algorithm with a higher compression ratio is used to effectively reduce the storage space occupied by instance data. This allows more instance data of idle inference instances to be stored in a limited storage space, achieving a flexible trade-off between migration efficiency and storage capacity in different idle stages.

[0075] In other embodiments, the resource management device does not need to compress instance data. This application does not impose limitations on this aspect.

[0076] Step 122: If the running status information indicates that the first inference instance is in an idle state and the resource status of the first host indicates that the storage resources of the first host are insufficient, determine that the migration strategy for the first instance data is inter-host migration.

[0077] In the case where the running status information indicates that the first inference instance is in an idle state, in order to release the GPU memory occupied by the first inference instance, the resource management device needs to migrate the instance data of the first inference instance out of the GPU memory; when the resource status of the first host indicates that the storage resources of the first host are insufficient, the first host cannot store new instance data, and the first host where the first inference instance is located cannot be selected as the target for instance data migration. The resource management device needs to migrate the instance data to other hosts besides the first host and determine the migration strategy between hosts.

[0078] In some embodiments, the resource management device releases the GPU memory occupied by the first instance data back to the shared memory pool of the first host, and the memory space in the shared memory pool of the first host is shared by the inference instances in the host.

[0079] Optionally, the host's GPU is divided into a shared memory pool. The memory space in the shared memory pool is shared by inference instances in the host. When the resource management device releases the GPU memory occupied by the first instance data, it releases the GPU memory occupied by the first instance data back to the shared memory pool of the first host, so that the memory resources can be dynamically reused by other inference instances in the same host, avoiding resource idleness.

[0080] like Figure 5 The diagram illustrates a first instance data migration process according to another illustrative embodiment of this application. The diagram includes a first host 510 and a second host 520. The first host 510 has a first GPU memory 511 and a first non-GPU memory storage space 512, with the first GPU memory 511 storing the first instance data 513. The second host 520 has a second GPU memory 521 and a second non-GPU memory storage space 522, with the second GPU memory 521 storing the second instance data 523. Before the migration, the first non-GPU memory storage space 512 of the first host 510 has limited available capacity and insufficient storage resources, while the second non-GPU memory storage space 522 of the second host 520 has sufficient storage resources. Based on this, the resource management device determines to perform an inter-host migration, migrating the first instance data 513 from the first host 510 to the second non-GPU memory storage space 522 of the second host 520.

[0081] When the primary host has sufficient storage resources, intra-host migration is prioritized to move instance data to the local host's storage space. This allows for rapid migration back upon instance wake-up, reducing recovery latency. When the primary host's storage resources are insufficient, inter-host migration is performed to move instance data to the storage space of other hosts in the cluster, preventing migration failure due to insufficient local storage capacity. This approach ensures migration efficiency when local resources are plentiful while fully utilizing shared cluster storage resources as a supplement, improving the success rate of migration operations and the overall resource utilization of the system.

[0082] After determining the migration strategy, the resource management device needs to execute the determined migration strategy to migrate the first instance data.

[0083] In some embodiments, multiple candidate hosts in the cluster contribute a portion of their non-GPU memory storage space as a shared storage resource within the cluster. This shared cluster storage space can be used to store instance data migrated from GPU memory for a longer period.

[0084] Since there may be multiple hosts providing shared storage space for the cluster, the resource management device needs to select a target host to store the instance data to be migrated when determining the inter-host migration strategy.

[0085] In some embodiments, when the migration strategy is inter-host migration, the resource management device determines a second host from the candidate hosts based on the cluster shared storage space provided by the candidate hosts in the cluster. The cluster shared storage space is non-GPU memory space.

[0086] Because the shared storage resource utilization, network latency, and storage media type of each candidate host differ, the resource management device needs to select the most suitable host as the second host.

[0087] In some embodiments, the resource management device determines the second host based on the resource utilization of the candidate hosts. For example, the candidate host with the lowest non-video memory resource utilization is selected as the second host.

[0088] In other embodiments, the resource management device determines the second host based on the distance between the candidate host and the first host. For example, the candidate host with the lowest network hop count or the closest physical distance to the first host is selected as the second host.

[0089] In other embodiments, the resource management device determines the second host by combining the resource utilization rate of the candidate hosts and their distance from the first host. For example, the resource management device first obtains the cluster shared storage space utilization rate of each candidate host and excludes candidate hosts with a utilization rate exceeding 80%; then, among the remaining candidate hosts, it selects the host with the lowest network latency to the source host or the highest available bandwidth as the second host.

[0090] The embodiments of this application do not limit the specific method by which the resource management device determines the second host.

[0091] In some embodiments, after the second host is determined, the resource management device needs to migrate the first instance data to the cluster shared storage space provided by the second host.

[0092] In some embodiments, since the uncompressed instance data is large in size, direct migration would occupy a large amount of limited storage space. In order to save storage space, the resource management device first compresses the first instance data when migrating it, and then migrates the compressed data to the target storage space to improve the utilization of storage space, thereby accommodating more migrated instance data under the same storage resources.

[0093] In some embodiments, when migrating the first instance data to the second host, the resource management device first compresses the first instance data using a compression algorithm, and then migrates the compressed first instance data to the cluster shared storage space provided by the second host.

[0094] In other instances, the resource management device directly migrates the data of the first instance to the cluster-shared storage space provided by the second host. This application does not impose such limitations.

[0095] In some embodiments, when migrating the first instance data, the resource management device retains instance binding information in the GPU memory. This instance binding information indicates the mapping of GPU memory resources allocated to the first inference instance. Correspondingly, the GPU memory occupancy recorded in the metadata of the first inference instance is updated to a minimum occupancy state, indicating that only instance binding information is retained in the GPU memory. By retaining this instance binding information, the mapping relationship between the inference instance and GPU memory resources can be quickly reconstructed during subsequent recovery, thereby shortening instance wake-up time and improving recovery efficiency. When the local storage resources of the first host are insufficient, cross-host instance data migration is achieved by migrating the instance data to the shared storage space of other hosts in the cluster, effectively avoiding migration failures due to local storage capacity limitations. This enhances fault tolerance and resource utilization in scenarios with limited local storage, providing more data recovery paths for subsequent instance wake-ups.

[0096] Optionally, each inference instance in the cluster is configured with corresponding metadata to record the instance's running status information, instance data information, and migration-related information. The resource management device can determine the running information, instance data information, and migration information of the inference instance based on the inference instance's metadata.

[0097] In some embodiments, when migrating the first instance data of the first inference instance, the resource management device updates the metadata of the first inference instance, the metadata of the first inference instance including at least one of the storage location of the first instance data and a compression algorithm.

[0098] Specifically, when migrating the first instance data of the first inference instance, the metadata is updated, including the storage location of the first instance data and the compression algorithm, so that when the instance is restored in the future, the storage location of the data can be quickly located according to the metadata, and the original data can be restored by the corresponding decompression algorithm, thereby shortening the instance wake-up time.

[0099] For example, during the migration process, the metadata records the storage location of the first instance data in the memory of host A, and the compression algorithm used is LZ4. When the instance needs to be restored, the resource management device directly reads the first instance data from the memory of host A according to the storage address in the metadata, performs the corresponding decompression operation according to the recorded compression algorithm, and then loads the decompressed data into the GPU memory.

[0100] In other embodiments, the metadata of the first inference instance also includes transport information, such as data fragmentation information and transport checksums. The data fragmentation information indicates the fragmentation mapping relationship of data across different storage nodes or locations, and the transport checksum verifies the integrity and accuracy of the data during migration. This application does not limit the specific content included in the metadata.

[0101] By updating the metadata of the first instance when migrating the first instance data of the first inference instance, and recording the storage location and compression algorithm used for the first instance data in the metadata, the resource management device can directly locate the storage location of the data according to the storage location when the instance needs to be restored later, and directly use the corresponding decompression method to restore the data according to the compression algorithm identifier, without traversing and searching or renegotiating compression parameters. This significantly shortens the instance wake-up time, reduces the recovery latency, and improves the system's management efficiency and traceability of migrated data.

[0102] The following describes the data migration process.

[0103] If new requirements arise for the first inference instance after the first instance data is migrated out of GPU memory, the first instance data must be migrated back to GPU memory to reactivate the first inference instance.

[0104] In some embodiments, if the first inference instance meets the instance migration conditions, the first instance data of the first inference instance is migrated to the GPU memory of the target host. The first instance data was located in a storage space outside the GPU memory before the instance migration conditions were met.

[0105] The instance migration condition indicates that the first inference instance needs to be reactivated and migrated back to GPU memory. When the resource management device detects that this condition is met, it needs to migrate the first instance data from non-GPU memory space to GPU memory.

[0106] In some embodiments, instance migration conditions include the first inference instance receiving a new inference task or receiving an instance wake-up signal. In other embodiments, instance migration conditions include the first inference instance receiving cross-host task scheduling, requiring the instance to be migrated to another host to perform inference tasks; or receiving an instance migration instruction issued by the resource management device due to resource scheduling, requiring the instance to be migrated to another host with sufficient storage resources. This application does not limit the specific form of instance migration conditions.

[0107] By migrating the first instance data of the first inference instance from storage space outside the GPU memory to the GPU memory of the target host when the instance migration conditions are met, the inference instance that was previously swapped out due to idleness can quickly resume operation and respond to new inference requests or scheduling instructions in a timely manner. This avoids the problem of instance data remaining in external storage for a long time and being unable to be reactivated, thus ensuring the continuity of system services.

[0108] It's important to note that determining the target host, rather than directly migrating back to the original host, is necessary because the GPU memory resources of the original host may have been occupied by other instances during the instance data migration process, or the storage resource status of the original host may have changed, making it unable to accommodate the instance again. Therefore, the resource management device needs to dynamically select a suitable target host based on the current resource status of each host in the cluster to ensure the success rate of the migration operation and the performance of the instance.

[0109] In some embodiments, the resource management device periodically checks whether the first inference instance meets the instance migration conditions. If the first inference instance is detected to meet the instance migration conditions, the resource management device migrates the first instance data of the first inference instance to the GPU memory of the target host.

[0110] In other embodiments, the resource management device also periodically checks the integrity of the data stored in the non-GPU memory space and cluster shared memory space of the first host to avoid instance recovery failure due to data corruption.

[0111] In some embodiments, the resource management device determines the target host based on the metadata of the first inference instance and the resource status of each host. The metadata of the first inference instance includes the source host information of the first inference instance. The source host information is used to indicate the source host where the inference instance was initially deployed. The target host is the source host of the first inference instance or other hosts in the cluster.

[0112] The metadata of the first inference instance includes its source host information, which indicates the source host on which the instance was initially deployed. Since the source host may still retain some of the instance's data or cache, and network transmission overhead is relatively low, the source host is the preferred target for migrating back instance data. When the source host is unsuitable as the target host, the resource management device selects a suitable host from other hosts in the cluster as the target host based on the resource status of each host.

[0113] The target host is determined based on the metadata of the first inference instance and the resource status of each host in the cluster. The source host or other hosts in the cluster are used as candidate targets. On the one hand, the source host, as the migration target, can utilize the local cache, reserved GPU context or lower data transfer overhead that may exist in the source host to reduce the cost and latency of data migration. On the other hand, other hosts with sufficient resources in the cluster can be dynamically selected as target hosts to improve the success rate of instance wake-up and realize intelligent selection of target hosts.

[0114] In some embodiments, the resource management device determines whether the source host is suitable as the target host based on the video memory resources of the source host of the first inference instance. Before judging whether the video memory resources of the source host are sufficient, it is necessary to obtain the resource status of the source host.

[0115] In some embodiments, the resource management device obtains the resource status of the source host of the first inference instance from the resource status of each host based on the metadata of the first inference instance.

[0116] The metadata of the first inference instance records the source host information of the first inference instance. The resource management device can determine the source host of the first inference instance from the metadata of the first inference instance, and then obtain the resource status of the source host of the first inference instance from the resource status of each host.

[0117] In some embodiments, if the resource status indicates that the source host of the first inference instance meets the migration back conditions, the source host of the first inference instance is determined as the target host. After obtaining the resource status of the source host of the first inference instance, the resource management device can determine whether the source host is suitable as the target host for the data migration of the first instance based on the resource status of the source host.

[0118] In some embodiments, meeting the migration back condition means that the host has sufficient video memory resources. The resource management device determines whether the source host of the first inference instance is suitable as the target host based on whether the source host has sufficient video memory resources.

[0119] For example, when the resource status of the source host indicates that its video memory resources are sufficient, the source host is determined to be suitable as the target host; when the video memory resources of the source host are insufficient, it is determined to be unsuitable as the target host.

[0120] In other embodiments, meeting the migration back condition means that the host's load is lower than a preset load threshold, and the resource management device determines whether the source host is suitable as the target host based on the load of the source host in the first inference instance.

[0121] For example, when the resource status of the source host indicates that there are a large number of inference instances running on it, the source host is determined to be unsuitable as the target host; when there are a small number of inference instances running on it, the source host is determined to be suitable as the target host.

[0122] This application does not limit the specific content of the migration conditions or the specific method of determining the target host.

[0123] like Figure 6 The diagram illustrates an instance data migration process according to an illustrative embodiment of this application. The diagram includes a host 610, which has GPU memory 611 and non-GPU memory storage space 612. The process includes a migration phase and a migration-in phase. In the migration phase, the first instance data 613, originally stored in GPU memory 611, is migrated by the resource management device to the non-GPU memory storage space 612 of the same host 610. The host 610 is the source host for the inference instance corresponding to the first instance data. In the migration-in phase, the host 610 has a large available capacity in GPU memory 611, with sufficient GPU memory resources. The resource management device uses the host 610 (i.e., the source host) as the target host and migrates the first instance data 613 from the non-GPU memory storage space 612 back to GPU memory 611.

[0124] If the resource status of the source host of the first inference instance indicates that the source host of the first inference instance does not meet the migration back conditions, the target host is determined based on the resource status of other hosts in the cluster.

[0125] If the source host of the first inference instance is not suitable as the target host for the first instance data migration, the resource management device needs to determine the target host from other hosts in the cluster.

[0126] In some embodiments, the resource management device selects the host with the most available video memory as the target host based on the video memory resources of other hosts in the cluster. In other embodiments, the resource management device selects the host with the lowest load as the target host based on the load of other hosts in the cluster. In still other embodiments, the resource management device selects the host requiring the shortest transmission time or the lowest transmission latency as the target host based on the transmission distance or network latency of the data transmission link with each candidate host. This application does not impose any limitations on these embodiments.

[0127] like Figure 7 The diagram illustrates an instance data migration process provided in another illustrative embodiment of this application, including a first host 710 and a second host 720. The first host 710 is provided with a first GPU memory 711 and a first non-GPU memory storage space 712. The first GPU memory 711 stores first instance data 713. The second host 720 is provided with a second GPU memory 721 and a second non-GPU memory storage space 722. The process includes a migration stage and a migration-in stage. In the migration stage, the first instance data 713, originally stored in the first GPU memory 711, is migrated by the resource management device to the first non-GPU memory storage space 712 of the first host 710. At this time, the first host 710 is the source host for the inference instance corresponding to the first instance data 713. During the migration phase, due to insufficient allocable video memory in the first host 710, the resource management device does not use the first host 710 (i.e., the source host) as the target host, but instead selects the second host 720 with sufficient video memory resources as the target host, and migrates the first instance data 713 from the first non-video memory storage space 712 to the second GPU video memory 721 of the second host 720.

[0128] By obtaining the resource status of the source host based on the metadata of the first inference instance, and prioritizing whether the source host meets the migration back conditions, it is directly identified as the target host if it does, avoiding unnecessary cross-host data migration and reducing network transmission overhead and instance recovery latency. When the source host does not meet the migration back conditions, the target host can be dynamically re-identified from other hosts in the cluster, ensuring that instance data can be successfully migrated back to GPU memory and avoiding migration failure due to resource constraints on the source host. While prioritizing migration back efficiency, it also has fault tolerance and flexibility, improving the instance wake-up success rate and resource scheduling adaptability of the cluster in dynamic resource environments.

[0129] In some embodiments, since the resource management device does not select the target host based on the host's video memory resources, the allocatable video memory of the selected target host may be insufficient. If the allocatable video memory of the target host is less than the video memory requirement of the first inference instance, the second instance data of the second inference instance in the target host is migrated, and the GPU video memory occupied by the second instance data in the target host is released. The instance priority of the first inference instance is higher than the instance priority of the second inference instance.

[0130] The resource management device can determine the target host based on other criteria besides the availability of video memory resources. The determined target host may have insufficient video memory resources. In order to properly migrate the first instance data of the first inference instance, it is necessary to first release a portion of the GPU video memory in the target host to receive the first instance data.

[0131] In some embodiments, the resource management device releases GPU memory for migrating data from the first instance by migrating a second inference instance in the target host whose instance priority is lower than that of the first inference instance.

[0132] For example, a second inference instance is running on the target host, and the resource management device needs to migrate the first instance data of the first inference instance to the target host. If the instance priority of the first inference instance is higher than that of the second inference instance, the resource management device migrates the second instance data of the second inference instance to release the video memory space it occupies. After sufficient video memory resources are available, GPU video memory is allocated to the first inference instance, and the first instance data is migrated to that video memory.

[0133] In other embodiments, the resource management device may also determine which inference instances need to be released based on other factors such as the data size of the inference instance or its recent usage time. This application does not impose limitations on this aspect.

[0134] When the available GPU memory on the target host cannot meet the GPU memory requirements of the first inference instance, the data of the first instance can be successfully migrated by migrating the second inference instance with a lower instance priority and releasing the GPU memory it occupies. This ensures that the operation and recovery of the high-priority instance are prioritized, avoids instance wake-up failure or long-term blocking due to insufficient GPU memory, and improves the scheduling flexibility of the cluster in resource-constrained scenarios.

[0135] In some embodiments, when multiple instance data migration tasks exist simultaneously, the resource management device needs to determine the migration priority of each task.

[0136] In some embodiments, the resource management device determines the storage location of the first instance data based on the metadata of the first inference instance, the metadata of the first inference instance including the storage location of the first instance data; and migrates the first instance data based on the migration priority corresponding to the storage location of the first instance data, wherein the storage location includes a first storage space, a second storage space and a cluster shared storage space, the read / write speed of the first storage space is higher than that of the second storage space, the cluster shared storage space is non-GPU memory space provided by at least one candidate host in the cluster, the migration priority corresponding to the first storage space is higher than that corresponding to the second storage space, and the migration priority corresponding to the second storage space is higher than that corresponding to the cluster shared storage space.

[0137] Because the resource management device had previously migrated the first instance data to the first storage space, the second storage space, or the shared storage space on other hosts in the cluster, based on idle time and the storage resource status of the first host, the resource management device processed the migration tasks sequentially in descending order of recovery speed to restore the operation of the first inference instance as quickly as possible: first, the instance data stored in the first storage space was migrated; second, the instance data stored in the second storage space was migrated; and finally, the instance data stored in the cluster's shared storage space was migrated.

[0138] By determining the migration priority based on the storage location of the first instance data and executing data migration in descending order of priority, instance data in the first storage space with the fastest read and write speed can be migrated back to GPU memory first, thereby restoring the inference instance to operation as quickly as possible and significantly shortening the instance wake-up latency. Instance data in the second storage space is migrated second priority, and instance data in the cluster shared storage is migrated last. This fully utilizes the performance differences of different storage media. When multiple migration tasks exist simultaneously, resources can be reasonably allocated to prioritize the rapid online deployment of instances with fast recovery speeds, thereby improving the overall system response efficiency and service quality.

[0139] In other embodiments, when multiple instance data migration tasks exist simultaneously, the resource management device executes each migration task sequentially. This application does not impose such limitations.

[0140] In some embodiments, since the first instance data is compressed using a compression algorithm during migration, during migration, the resource management device determines the compression algorithm used for the first instance data based on the metadata of the first inference instance, the metadata of the first inference instance including the compression algorithm of the first instance data; decompresses the first instance data based on the decompression algorithm corresponding to the compression algorithm; and migrates the decompressed first instance data to the GPU memory of the target host.

[0141] The metadata of the first inference instance records the storage location and compression algorithm of the first instance data. The resource management device can determine the compression algorithm used by the first instance data during migration based on the metadata of the first inference instance, and then use the decompression algorithm corresponding to the compression algorithm to decompress the first instance data and migrate the decompressed first instance data to the GPU memory of the target host.

[0142] For example, if the first instance data is compressed using the LZ4 compression algorithm during migration, the compression algorithm is recorded as LZ4 in the metadata. During migration, the resource management device uses the decompression algorithm corresponding to LZ4 to decompress the data based on this record, and then migrates the decompressed data to the GPU memory of the target host.

[0143] By determining the original compression algorithm used for the first instance data based on metadata records during migration, and then using the corresponding decompression algorithm for decompression, the decompressed original data is migrated to GPU memory. This ensures that the data can be correctly recovered after compression migration, avoiding data corruption or recovery failure due to compression format incompatibility. Furthermore, the pre-recording of the compression algorithm in the metadata eliminates the need for additional negotiation or detection during the decompression process, improving migration efficiency.

[0144] Since GPU memory is allocated to inference instances, when migrating instance data from an inference instance that has already released GPU memory back to GPU memory, it is necessary to first allocate GPU memory to the inference instance and then migrate the instance data.

[0145] In some embodiments, the resource management device determines the video memory requirements of the first inference instance based on the metadata of the first inference instance, the metadata of the first inference instance including the video memory requirements of the first inference instance.

[0146] The metadata of the first inference instance includes the video memory requirements of the first inference instance. The resource management device can determine the video memory requirements of the first inference instance based on the metadata of the first instance data, thereby allocating GPU video memory to the first inference instance and migrating the first instance data to the allocated GPU video memory.

[0147] Optionally, the metadata of the first inference instance records its required video memory capacity information, including a preset video memory quota and a hard limit quota, whereby the hard limit quota is the maximum video memory capacity that the inference instance can occupy. The resource management device can directly determine the video memory requirements of the first inference instance by reading the preset video memory quota in the metadata, or use the hard limit quota as the video memory requirements of the first inference instance, thereby allocating a corresponding amount of GPU video memory to the target host.

[0148] In other embodiments, the resource management device calculates the peak or average video memory usage of the first inference instance based on its historical runtime data, using this as an estimate of the video memory requirement. For example, the maximum video memory usage during the instance's most recent runs is recorded and used as the current video memory requirement.

[0149] In other embodiments, the resource management device directly determines the video memory requirements of the first inference instance based on the model type or configuration file deployed thereon. For example, if the first inference instance loads a computing model, the total video memory requirements are calculated by adding a preset reserved space to the known parameters of the model.

[0150] The embodiments of this application do not limit the method for determining the memory requirements of the first inference instance.

[0151] In some embodiments, after determining the memory requirements of the first inference instance, the resource management device allocates GPU memory for the first inference instance in the target host based on the memory requirements.

[0152] As an illustration, the resource management device allocates a storage area of ​​a corresponding size in the GPU memory of the target host according to the memory requirements of the first inference instance.

[0153] In some embodiments, the resource management device allocates GPU memory for the first inference instance from the target host's shared memory pool based on memory requirements.

[0154] Optionally, the host's GPU is divided into a shared memory pool, and the memory space in the shared memory pool is shared by the inference instances in the host. When allocating GPU memory, the resource management device allocates GPU memory from the shared memory pool.

[0155] Optionally, when allocating GPU memory to the first inference instance, the resource management device can bind the allocated GPU memory to the first inference instance by modifying the instance binding information of the first inference instance, and set a hard limit on the GPU memory to prevent the first inference instance from over-consuming GPU memory resources.

[0156] By returning the released GPU memory to the shared memory pool of the host, the memory resources can be dynamically reused by other inference instances on the same host, avoiding resource idleness. At the same time, when allocating memory to the target host, it is allocated from the host's shared memory pool on demand, realizing unified pooled management and elastic scheduling of memory resources, improving the utilization efficiency and allocation flexibility of memory resources.

[0157] In some embodiments, after allocating GPU memory to a first inference instance, the resource management device migrates the first instance data of the first inference instance to the allocated GPU memory.

[0158] For example, after allocation, the resource management device determines the corresponding metadata based on the instance identifier (ID) of the first inference instance; according to the storage location of the first instance data recorded in the metadata, it reads the first instance data from the current storage space and migrates it to the allocated GPU memory area, thereby restoring the operation of the first inference instance.

[0159] By obtaining the GPU memory requirements from the metadata of the first inference instance and accurately allocating the corresponding amount of GPU memory space on the target host accordingly, and then performing data migration, the instance migration failure or runtime memory overflow caused by insufficient GPU memory allocation is avoided. It also prevents the waste of GPU memory resources caused by excessive allocation, so that the GPU memory allocation matches the actual needs of the instance, improves the accuracy and utilization of resource allocation, and ensures the stable operation of the instance after migration.

[0160] Optionally, each inference instance in the cluster is configured with corresponding metadata to record the instance's running status information, instance data information, and migration-related information. The resource management device can determine the running information, instance data information, and migration information of the inference instance based on the inference instance's metadata.

[0161] In some embodiments, when a first inference instance is migrated in, the resource management device updates the metadata of the first inference instance, the metadata of the first inference instance including at least one of the storage location of the first instance data and a compression algorithm.

[0162] Specifically, timely updating the metadata during the migration of the first inference instance ensures that the metadata accurately reflects the latest storage location and decompression status of the first instance's data. This guarantees that subsequent migrations or status queries of the instance can be performed based on accurate metadata, avoiding data location errors or unnecessary repeated decompression due to expired metadata.

[0163] For example, during the migration process, the resource management device decompresses the first instance data from the memory of host A and loads it into the GPU memory of the target host. After the migration is complete, the resource management device updates the metadata of the first inference instance: changing the storage location field from "host A memory" to "target host GPU memory", and changing the compression algorithm field from "LZ4" to "none" or clearing the field. When the instance needs to be migrated again in the future, the resource management device can directly read the data from the GPU memory based on the updated metadata without performing a decompression operation.

[0164] By updating the storage location and compression algorithm in the metadata of the first inference instance during migration, the metadata accurately reflects the latest storage and decompression status of the instance data. This avoids data location errors or unnecessary duplicate decompression operations during subsequent migrations or status queries of the instance due to expired metadata, ensuring the real-time nature and consistency of the metadata.

[0165] The following explains the dynamic adjustment process of GPU memory.

[0166] Optionally, the resource management device can also monitor the inference instances running on each host in the cluster, dynamically adjust the GPU memory occupied by the inference instances, and balance the memory usage of each inference instance.

[0167] In some embodiments, if the runtime status information of the first inference instance indicates that the first inference instance meets the conditions for increasing GPU memory, the resource management device allocates additional GPU memory to the first inference instance from the shared GPU memory pool of the first host; if the runtime status information of the first inference instance indicates that the first inference instance meets the conditions for decreasing GPU memory, the device releases a portion of the GPU memory space allocated to the first inference instance to the shared GPU memory pool of the first host; wherein the GPU memory space in the shared GPU memory pool of the first host is shared by the inference instances in the first host.

[0168] The running status information of the first inference instance indicates the running status of the first inference instance. The resource management device determines whether to perform video memory expansion or reduction based on the running status information of the first inference instance, so as to ensure that the video memory resources are dynamically adapted to the task load and reduce the possibility of out of memory (OOM) risk.

[0169] For example, when the video memory usage of the first inference instance reaches 90% of the allocated video memory, it is determined that the video memory expansion condition is met, and the resource management device allocates additional video memory for it from the video memory shared pool; when the video memory usage of the first inference instance is continuously lower than 30% of the allocated video memory and exceeds a preset time, it is determined that the video memory reduction condition is met, and the resource management device releases the excess video memory space back to the video memory shared pool.

[0170] By dynamically expanding and shrinking the GPU memory based on the running status of the first inference instance, performance degradation due to insufficient GPU memory or idle and wasted GPU memory resources is avoided. This achieves dynamic reuse of GPU memory resources, enabling multiple inference instances on the same host to share limited GPU memory resources more flexibly, thereby improving the overall utilization efficiency of the shared GPU memory pool.

[0171] In some embodiments, when the runtime status information of the first inference instance indicates that the GPU memory usage of the first inference instance has reached a first usage threshold, the resource management device allocates additional GPU memory for the first inference instance from the shared memory pool of the first host.

[0172] The first threshold is used to characterize that the video memory usage has reached a level that requires expansion. The resource management device needs to expand the video memory for the first inference instance and allocate additional GPU video memory for the first inference instance from the video memory shared pool of the first host.

[0173] For example, if the GPU memory utilization of the first inference instance reaches 95%, the resource management device allocates an additional 10GB of GPU memory to it from the shared GPU memory pool, thereby expanding the available GPU memory space of the first inference instance and preventing task interruption due to insufficient GPU memory.

[0174] In some embodiments, when the runtime status information of the first inference instance indicates that the first inference instance has received an inference task whose video memory requirement has reached a threshold, the resource management device allocates additional GPU video memory for the first inference instance from the video memory shared pool of the first host.

[0175] The demand threshold is used to indicate that the amount of video memory requested by a new task exceeds the currently available free quota, and that it needs to be expanded immediately to avoid task execution failure.

[0176] For example, if the first inference instance has been allocated 20GB of video memory, and the new inference task is estimated to require an additional 12GB of video memory, and the requirement threshold is set to 10GB, the video memory requirement of the new inference task is greater than the requirement threshold, and it is a high-load task. In this case, the resource management device will allocate an additional 12GB of video memory from the video memory shared pool to meet the task requirements.

[0177] In some embodiments, when multiple memory expansion tasks exist simultaneously, the resource management device performs memory expansion operations on the inference instance with the higher instance priority based on the instance priority of each inference instance.

[0178] Optionally, when the resource management device detects that an inference instance has a need for increased video memory, it suspends the inference instance from receiving new inference tasks. After the video memory expansion is completed, the inference instance resumes receiving new tasks. This avoids the failure of loading new tasks due to insufficient video memory resources during the expansion process, thereby improving the success rate of the expansion operation and the stability of the system operation.

[0179] In some embodiments, if the runtime status information of the first inference instance indicates that the GPU memory usage of the first inference instance is lower than a second usage threshold, the resource management device will release a portion of the memory space allocated to the first inference instance to the shared memory pool of the first host, wherein the first usage threshold is higher than the second usage threshold.

[0180] The first usage threshold is higher than the second usage threshold. The second threshold is used to characterize that the video memory usage has reached a level that requires scaling down. The resource management device needs to reduce the video memory for the first inference instance and release part of the video memory space allocated to the first inference instance to the video memory shared pool of the first host.

[0181] For example, if the second usage threshold is set to 30%, and the memory usage of the first inference instance remains below 30% for more than a preset time, the resource management device releases its excess 10GB of memory back to the shared pool for use by other instances.

[0182] In some embodiments, when the resource management device performs a video memory downsizing operation, if there is temporary cached data in the first inference instance, the temporary cached data is migrated to the reserved video memory area of ​​the first inference instance, and then the free video memory quota to be released is allocated from the currently allocated video memory and returned to the video memory shared pool of the first host, thereby avoiding data loss due to downsizing and ensuring the integrity of the instance's running state.

[0183] In some embodiments, when the resource management device indicates that the capacity of the shared memory pool of the first host has reached a first capacity threshold, it allocates GPU memory from the shared memory pool of the first host for the first inference instance.

[0184] The first capacity threshold is used to determine that there is enough remaining video memory available for allocation in the shared pool, thereby proactively expanding the capacity of the current inference instance to improve its running performance.

[0185] For example, if the first capacity threshold is set to 50GB of the total capacity of the shared pool and the current free capacity of the shared pool is 60GB, the resource management device will allocate an additional 5GB of video memory to the first inference instance to cope with possible load peaks, thereby avoiding idle GPU video memory resources.

[0186] In some embodiments, when performing a video memory expansion task, the resource management device determines the expansion amount based on the difference between the video memory requirements of the inference instance and the current video memory quota, and the expanded video memory quota does not exceed the hard upper limit of the inference instance's video memory. When performing a video memory reduction task, the reduction amount is determined based on the difference between the current video memory quota of the inference instance and the actual video memory requirements, and the reduced video memory quota is not lower than the minimum guaranteed quota of the inference instance.

[0187] For example, if a certain inference instance currently has a video memory quota of 20GB, a video memory requirement of 30GB, and a hard limit of 32GB, then the resource management device determines an expansion quota of 10GB, resulting in a video memory quota of 30GB, which does not exceed the hard limit of 32GB. If the inference instance currently has a video memory quota of 20GB, an actual video memory requirement of only 12GB, and a minimum guaranteed quota of 8GB, then the resource management device determines a reduction quota of 8GB, resulting in a video memory quota of 12GB, which is not less than the minimum guaranteed quota of 8GB.

[0188] In some embodiments, if the expanded video memory quota exceeds the hard limit of video memory for the inference instance, the resource management device refuses to perform the expansion operation, or only expands the video memory quota to the hard limit of video memory, and sends a prompt message to the upper-layer scheduler or user terminal indicating that the expansion failed or was partially expanded.

[0189] In some embodiments, if the reduced video memory quota is lower than the minimum guaranteed quota of the inference instance, the resource management device terminates the reduction operation, keeps the video memory quota at the minimum guaranteed quota, and triggers a video memory idle alarm. The upper-layer scheduler then decides whether to migrate or terminate the inference instance based on the running status of the inference instance.

[0190] like Figure 8 The figure illustrates a schematic embodiment of dynamic GPU memory adjustment provided in this application. In the figure, GPU memory 800 includes a shared memory pool 803, and allocated first instance data 801 and second instance data 802. The free capacity of the shared memory pool 803 exceeds 50GB. The resource management device triggers dynamic GPU memory adjustment, allocating more GPU memory from the shared memory pool 803 to the inference instance corresponding to the first instance data 801 and the inference instance corresponding to the second instance data 802, respectively, to cope with potential load peaks and thus avoid idle GPU memory resources.

[0191] By setting multiple memory expansion trigger conditions and combining them with memory reduction conditions, fine-grained and flexible adjustments to GPU inference instance memory quotas are achieved. When instance load increases or task demand surges, additional memory can be allocated from the shared pool in a timely manner to avoid task failures or performance degradation due to insufficient memory. When instance load decreases, idle memory is proactively reclaimed to the shared pool to prevent resource waste. At the same time, when the shared pool has sufficient free capacity, more memory can be proactively pre-allocated to instances to cope with future load peaks. Under the premise of ensuring instance operation stability, the utilization efficiency of the shared memory pool and the overall resource elastic scheduling capability of the cluster are significantly improved.

[0192] In some embodiments, if the resource management device indicates that the capacity of the shared memory pool of the first host is lower than a second capacity threshold, it migrates the third instance data of the third inference instance in the first host.

[0193] Optionally, in order to avoid the shared memory pool in the first host being too low, the resource management device needs to release a portion of the GPU memory occupied by the inference instances in the first host.

[0194] Among them, the third inference instance has a lower instance priority than other inference instances in the first host, and its data migration cost is the lowest. The resource management device migrates the third instance data of the third inference instance to free up GPU memory.

[0195] In some embodiments, after migrating the third instance data, the resource management device releases the GPU memory occupied by the third instance data and adds it to the shared memory pool of the first host. The second capacity threshold is lower than the first capacity threshold, and the instance priority of the third inference instance is lower than the instance priority of other inference instances in the first host.

[0196] After migrating the third instance data, the resource management device also needs to release the GPU memory occupied by the third inference instance, so that it returns to an allocatable state, in order to expand the capacity of the shared memory pool.

[0197] The second capacity threshold is lower than the first capacity threshold. The capacity threshold at which the resource management device actively allocates GPU memory from the shared memory pool to inference instances (i.e., the first capacity threshold) is higher than the capacity threshold at which the allocated GPU memory needs to be reclaimed to the shared memory pool (i.e., the second capacity threshold), thereby realizing a dynamic balancing mechanism where memory resources are actively allocated when there is sufficient free space and actively reclaimed when there is insufficient free space.

[0198] By proactively migrating the third instance data of the lowest-priority third inference instance in the first host when the free capacity of the shared memory pool falls below the second capacity threshold, the GPU memory occupied by the third inference instance is released and returned to the shared memory pool. This effectively avoids the problem of subsequent expansion failures or new instances failing to start due to shared memory depletion. Simultaneously, prioritizing the migration of the lowest-priority inference instance minimizes the impact on high-priority tasks in the cluster, ensuring dynamic balance of GPU memory resources and system stability.

[0199] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0200] like Figure 9As shown, it illustrates a schematic diagram of the structure of a resource management device provided in an illustrative embodiment of this application. The diagram includes a cluster 900, which includes a host 910, a monitoring module 920, a resource manager 930, and a switching module 940.

[0201] The host 910 includes a first GPU 901 and a second GPU 902. The first GPU 901 runs a first inference instance 903, and the second GPU 902 runs a second inference instance 904. The host 910 also includes memory 905 and storage 906.

[0202] The monitoring module 920 includes an instance statistics module 921 and a resource statistics module 922. The instance statistics module 921 is used to collect the running status information of each inference instance (such as idle / active status), and the resource statistics module 922 is used to collect the host's resource status information (such as video memory usage, remaining memory capacity, storage usage, etc.).

[0203] The resource manager 930 includes a status monitoring module 931, a swap triggering module 932, and a resource scheduling module 933. The status monitoring module 931 receives information from the monitoring module 920; the swap triggering module 932 determines whether to trigger memory swapping based on the runtime status information of each inference instance and the host's resource status information; the resource scheduling module 933 is used to decide on migration strategies, including intra-host migration or inter-host migration.

[0204] The swap module 940 includes a migration module 941 and a swap-in module 942. The migration module 941 is responsible for triggering the "swap-out" migration, which migrates the instance data of the inference instance from the GPU memory to memory 905 or storage 906, or further to the cluster shared storage; the swap-in module 942 is responsible for triggering the "swap-in" migration, which migrates the data from external storage back to the GPU memory when the instance is woken up.

[0205] In summary, through the collaborative work of the above modules, the cluster can achieve dynamic migration and recovery of GPU memory data based on the idle state of instances, thus completing the dynamic management of GPU resources.

[0206] like Figure 10 As shown, it illustrates a structural schematic diagram of a resource management device provided in another illustrative embodiment of this application, which includes an acquisition module 1010, a determination module 1020, and a migration module 1030.

[0207] The acquisition module 1010 is configured to acquire the running status information of each inference instance in the cluster, as well as the resource status of each host. The cluster contains multiple hosts, and the inference instances are run by GPUs in the hosts. The determination module 1020 is configured to, for the first inference instance among the various inference instances, when the running status information indicates that the first inference instance is in an idle state, determine the migration strategy of the first instance data of the first inference instance based on the resource status of the first host where the first inference instance is currently located. The migration strategy includes at least one of intra-host migration and inter-host migration. Migration module 1030 is configured to migrate the first instance data based on a migration strategy and release the GPU memory occupied by the first instance data in the first host.

[0208] The determination module 1020 is also configured to determine that the migration strategy for the first instance data of the first inference instance is intra-host migration when the running status information indicates that the first inference instance is in an idle state and the resource status of the first host indicates that the storage resources of the first host are sufficient. If the running status information indicates that the first inference instance is in an idle state and the resource status of the first host indicates that the storage resources of the first host are insufficient, the migration strategy for the first instance data is determined to be inter-host migration.

[0209] The determination module 1020 is further configured to determine the migration strategy of the first instance data of the first inference instance as first-level host migration when the running status information indicates that the first inference instance is in an idle state, the idle time of the first inference instance reaches a first time threshold, and the resource status of the first host indicates that the first storage space resources of the first host are sufficient. The first-level host migration is used to migrate the instance data to the first storage space, and the first storage space refers to the storage space other than the GPU memory. When the running status information indicates that the first inference instance is in an idle state, the idle time of the first inference instance reaches the second time threshold, and the resource status of the first host indicates that the second storage space of the first host has sufficient resources, the migration strategy for the first instance data of the first inference instance is determined to be the second-level host intra-migration. The second-level host intra-migration is used to migrate the instance data to the second storage space. The first time threshold is less than the second time threshold. The second storage space refers to the storage space other than the GPU memory, and the read and write speed of the first storage space is higher than that of the second storage space.

[0210] The determination module 1020 is also configured to determine the migration strategy for low-frequency instance sub-data in the first instance data as migration within the first-level host; The migration strategy for low-frequency instance sub-data and high-frequency instance sub-data in the first instance data is determined to be migration within the second-level host, with the access frequency of low-frequency instance sub-data being lower than that of high-frequency instance data.

[0211] The migration module 1030 is also configured to compress low-frequency instance sub-data using a first compression algorithm and migrate the compressed low-frequency instance sub-data from GPU memory to the first storage space when the migration strategy is migration within the first-level host. When the migration strategy is intra-host migration at level 2, low-frequency instance sub-data and high-frequency instance sub-data are compressed using a second compression algorithm. The compressed low-frequency instance sub-data is then migrated from the first storage space to the second storage space, and the compressed high-frequency instance sub-data is migrated from the GPU memory to the second storage space. The compression ratio of the second compression algorithm is greater than that of the first compression algorithm.

[0212] The migration module 1030 is also configured to determine a second host from the candidate hosts based on the cluster shared storage space provided by the candidate hosts in the cluster when the migration strategy is inter-host migration. The cluster shared storage space is non-video memory space. The data of the first instance is migrated from the GPU memory to the cluster shared storage space provided by the second host.

[0213] The migration module 1030 is also configured to update the metadata of the first inference instance when migrating the first instance data of the first inference instance, the metadata of the first inference instance including at least one of the storage location of the first instance data and a compression algorithm.

[0214] The migration module 1030 is also configured to migrate the first instance data of the first inference instance to the GPU memory of the target host when the first inference instance meets the instance migration conditions. The first instance data was located in a storage space outside the GPU memory before the instance migration conditions were met.

[0215] The determination module 1020 is also configured to determine the target host based on the metadata of the first inference instance and the resource status of each host. The metadata of the first inference instance includes the source host information of the first inference instance. The source host information is used to indicate the source host where the inference instance was initially deployed. The target host is either the source host of the first inference instance or other hosts in the cluster.

[0216] The determination module 1020 is also configured to obtain the resource status of the source host of the first inference instance from the resource status of each host based on the metadata of the first inference instance. If the resource status indicates that the source host of the first inference instance meets the migration conditions, the source host of the first inference instance will be determined as the target host. If the resource status of the source host of the first inference instance indicates that the source host of the first inference instance does not meet the migration back conditions, the target host is determined based on the resource status of other hosts in the cluster.

[0217] The migration module 1030 is also configured to migrate the second instance data of the second inference instance in the target host and release the GPU memory occupied by the second instance data in the target host when the allocatable GPU memory of the target host is less than the GPU memory requirement of the first inference instance. The instance priority of the first inference instance is higher than the instance priority of the second inference instance.

[0218] The migration module 1030 is also configured to determine the storage location of the first instance data based on the metadata of the first inference instance, wherein the metadata of the first inference instance includes the storage location of the first instance data. Based on the migration priority corresponding to the storage location of the first instance data, the first instance data is migrated. The storage location includes a first storage space, a second storage space, and a cluster shared storage space. The read / write speed of the first storage space is higher than that of the second storage space. The cluster shared storage space is a non-GPU memory space provided by at least one candidate host in the cluster. The migration priority corresponding to the first storage space is higher than that corresponding to the second storage space, and the migration priority corresponding to the second storage space is higher than that corresponding to the cluster shared storage space.

[0219] The migration module 1030 is also configured to determine the compression algorithm used for the first instance data based on the metadata of the first inference instance, wherein the metadata of the first inference instance includes the compression algorithm of the first instance data. Based on the decompression algorithm corresponding to the compression algorithm, the first instance data is decompressed; The decompressed first instance data is migrated to the GPU memory of the target host.

[0220] The migration module 1030 is also configured to determine the video memory requirements of the first inference instance based on the metadata of the first inference instance, the metadata of the first inference instance including the video memory requirements of the first inference instance. Based on the video memory requirements, allocate GPU video memory for the first inference instance on the target host; Migrate the first instance data of the first inference instance to the allocated GPU memory.

[0221] Migration module 1030 is also configured to release the GPU memory occupied by the first instance data back to the shared memory pool of the first host, and the memory space in the shared memory pool of the first host is shared by the inference instances in the host. Based on the memory requirements, allocate GPU memory for the first inference instance from the target host's shared memory pool.

[0222] The migration module 1030 is also configured to update the metadata of the first inference instance when the first inference instance is migrated in. The metadata of the first inference instance includes at least one of the storage location of the first instance data and compression algorithms.

[0223] The migration module 1030 is also configured to allocate additional GPU memory to the first inference instance from the shared memory pool of the first host when the runtime status information of the first inference instance indicates that the first inference instance meets the conditions for memory expansion. When the runtime status information of the first inference instance indicates that the first inference instance meets the conditions for reducing video memory, a portion of the video memory space allocated to the first inference instance will be released to the video memory shared pool of the first host. In this context, the video memory space in the shared video memory pool of the first host is shared by the inference instances in the first host.

[0224] The migration module 1030 is also configured to allocate additional GPU memory to the first inference instance from the shared memory pool of the first host when the GPU memory usage of the first inference instance reaches a first usage threshold, as indicated by the runtime status information of the first inference instance. When the runtime status information of the first inference instance indicates that the first inference instance has received an inference task whose video memory requirement has reached the threshold, additional GPU video memory is allocated to the first inference instance from the video memory shared pool of the first host. If the runtime status information of the first inference instance indicates that the GPU memory usage of the first inference instance is lower than the second usage threshold, a portion of the memory space allocated to the first inference instance will be released to the shared memory pool of the first host, wherein the first usage threshold is higher than the second usage threshold. If the resource status of the first host indicates that the capacity of the first host's shared memory pool has reached a first capacity threshold, GPU memory is allocated from the first host's shared memory pool for the first inference instance.

[0225] Migration module 1030 is also configured to migrate the third instance data of the third inference instance in the first host when the resource status of the first host indicates that the capacity of the shared memory pool of the first host is lower than the second capacity threshold. Release the GPU memory occupied by the third instance data and add it to the shared memory pool of the first host. The second capacity threshold is lower than the first capacity threshold, and the instance priority of the third inference instance is lower than the instance priority of other inference instances in the first host.

[0226] In summary, this approach first acquires the resource status of each host in the cluster and the runtime status information of the inference instances running on the host GPUs. Then, for the first inference instance among all inference instances, when its runtime status information indicates that the instance is idle, a migration strategy for the instance's instance data is determined based on the resource status of the first host to which the first inference instance is currently located. Since the determined migration strategy includes intra-host migration and inter-host migration, the scope of instance data migration is expanded. Finally, the instance data is migrated according to the determined migration strategy, and the GPU memory it occupies is released. The released GPU memory resources are then made available to other inference instances, achieving dynamic reuse of GPU memory resources and improving the overall resource utilization of the cluster.

[0227] Please refer to Figure 11 This illustration shows a structural block diagram of a computer device 1100 provided in an illustrative embodiment of this application. The computer device 1100 may be a terminal or server with storage capabilities, such as a smartphone, computer, or tablet computer. The computer device 1100 includes a processor 1101 and a memory 1102. The processor 1101 includes the resource management device shown in the above embodiment.

[0228] Processor 1101 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 1101 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 1101 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1101 may integrate a GPU, which is responsible for rendering and drawing the content required to be displayed on the screen.

[0229] The memory 1102 may include one or more computer-readable storage media, which may be tangible and non-transitory. The memory 1102 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices.

[0230] In some embodiments, the computer device 1100 may also optionally include a peripheral device interface 1103 and at least one peripheral device.

[0231] Those skilled in the art will understand that Figure 11 The structure shown does not constitute a limitation on the computer device 1100 and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0232] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0233] On the other hand, embodiments of this application provide a computer device, which includes a processor and a memory. The memory stores at least one instruction, which is loaded and executed by the processor to implement the resource management method provided in the embodiments of this application as described above.

[0234] On the other hand, embodiments of this application provide a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the resource management method provided in the embodiments of this application as described above.

[0235] On the other hand, embodiments of this application provide a computer program product, which includes computer instructions stored in a computer-readable storage medium. The processor of a computer device reads the computer instructions from the computer-readable storage medium, executes the computer instructions, and causes the processor of the computer device to load and execute the resource management methods provided in the above-described method embodiments.

[0236] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A resource management method, characterized in that, The method includes: Obtain the running status information of each inference instance in the cluster, as well as the resource status of each host. The cluster contains multiple hosts, and the inference instances are run by GPUs in the hosts. For the first inference instance among the various inference instances, when the running status information indicates that the first inference instance is in an idle state, a migration strategy for the first instance data of the first inference instance is determined based on the resource status of the first host where the first inference instance is currently located. The migration strategy includes at least one of intra-host migration and inter-host migration. The first instance data is migrated based on the migration strategy, and the GPU memory occupied by the first instance data in the first host is released.

2. The method according to claim 1, characterized in that, When the running status information indicates that the first inference instance is in an idle state, determining the migration strategy for the first instance data of the first inference instance based on the resource status of the first host where the first inference instance is currently located includes: If the running status information indicates that the first inference instance is in an idle state and the resource status of the first host indicates that the storage resources of the first host are sufficient, the migration strategy for the first instance data of the first inference instance is determined to be intra-host migration. If the running status information indicates that the first inference instance is in an idle state and the resource status of the first host indicates that the storage resources of the first host are insufficient, the migration strategy for the first instance data is determined to be the inter-host migration.

3. The method according to claim 2, characterized in that, The operational status information also includes idle time; The step of determining the migration strategy for the first instance data of the first inference instance as intra-host migration when the running status information indicates that the first inference instance is in an idle state and the resource status of the first host indicates that the storage resources of the first host are sufficient includes: When the running status information indicates that the first inference instance is in an idle state, the idle time of the first inference instance reaches a first time threshold, and the resource status of the first host indicates that the first storage space resources of the first host are sufficient, the migration strategy of the first instance data of the first inference instance is determined to be first-level host migration. The first-level host migration is used to migrate the instance data to the first storage space, and the first storage space refers to the storage space other than the GPU memory. When the running status information indicates that the first inference instance is in an idle state, the idle time of the first inference instance reaches a second duration threshold, and the resource status of the first host indicates that the second storage space resources of the first host are sufficient, the migration strategy for the first instance data of the first inference instance is determined to be a second-level intra-host migration. The second-level intra-host migration is used to migrate the instance data to the second storage space. The first duration threshold is less than the second duration threshold. The second storage space refers to the storage space other than the GPU memory, and the read / write speed of the first storage space is higher than the read / write speed of the second storage space.

4. The method according to claim 3, characterized in that, The migration strategy for determining the first instance data of the first inference instance is a first-level intra-host migration, including: The migration strategy for low-frequency instance sub-data in the first instance data is determined to be migration within the first-level host; The migration strategy for determining the first instance data of the first inference instance is a second-level intra-host migration, including: The migration strategy for the low-frequency instance sub-data and high-frequency instance sub-data in the first instance data is determined to be migration within the second-level host, wherein the access frequency of the low-frequency instance sub-data is lower than the access frequency of the high-frequency instance data.

5. The method according to claim 4, characterized in that, The migration of the first instance data based on the migration strategy includes: When the migration strategy is migration within the first-level host, the low-frequency instance sub-data is compressed using a first compression algorithm, and the compressed low-frequency instance sub-data is migrated from the GPU memory to the first storage space. When the migration strategy is migration within the second-level host, the low-frequency instance sub-data and the high-frequency instance sub-data are compressed using a second compression algorithm. The compressed low-frequency instance sub-data is then migrated from the first storage space to the second storage space, and the compressed high-frequency instance sub-data is migrated from the GPU memory to the second storage space. The compression ratio of the second compression algorithm is greater than that of the first compression algorithm.

6. The method according to claim 2, characterized in that, The migration of the first instance data based on the migration strategy includes: When the migration strategy is inter-host migration, a second host is determined from the candidate hosts based on the cluster shared storage space provided by the candidate hosts in the cluster, wherein the cluster shared storage space is non-GPU memory space; The data of the first instance is migrated from the GPU memory to the cluster shared storage space provided by the second host.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: In the case of migrating the first instance data of the first inference instance, the metadata of the first inference instance is updated, wherein the metadata of the first inference instance includes at least one of the storage location of the first instance data and a compression algorithm.

8. The method according to any one of claims 1 to 6, characterized in that, The method further includes: If the first inference instance meets the instance migration condition, the first instance data of the first inference instance is migrated to the GPU memory of the target host. The first instance data was located in storage space outside the GPU memory before the instance migration condition was met.

9. The method according to claim 8, characterized in that, The method further includes: Based on the metadata of the first inference instance and the resource status of each host, the target host is determined. The metadata of the first inference instance includes the source host information of the first inference instance. The source host information is used to indicate the source host where the inference instance was initially deployed. The target host is either the source host of the first inference instance or another host in the cluster.

10. The method according to claim 9, characterized in that, Determining the target host based on the metadata of the first inference instance and the resource status of each host includes: Based on the metadata of the first inference instance, the resource status of the source host of the first inference instance is obtained from the resource status of each host; If the resource status indicates that the source host of the first inference instance meets the migration back conditions, the source host of the first inference instance is determined as the target host. If the resource status of the source host of the first inference instance indicates that the source host of the first inference instance does not meet the migration back conditions, the target host is determined based on the resource status of the other hosts in the cluster.

11. The method according to claim 8, characterized in that, The method further includes: If the allocatable GPU memory of the target host is less than the GPU memory requirement of the first inference instance, the second instance data of the second inference instance in the target host is migrated, and the GPU memory occupied by the second instance data in the target host is released. The instance priority of the first inference instance is higher than the instance priority of the second inference instance.

12. The method according to claim 8, characterized in that, The step of migrating the first instance data of the first inference instance to the GPU memory of the target host includes: Based on the metadata of the first inference instance, the storage location of the first instance data is determined, wherein the metadata of the first inference instance includes the storage location of the first instance data; Based on the migration priority corresponding to the storage location of the first instance data, the first instance data is migrated. The storage location includes a first storage space, a second storage space, and a cluster shared storage space. The read / write speed of the first storage space is higher than that of the second storage space. The cluster shared storage space is a non-GPU memory space provided by at least one candidate host in the cluster. The migration priority corresponding to the first storage space is higher than that corresponding to the second storage space, and the migration priority corresponding to the second storage space is higher than that corresponding to the cluster shared storage space.

13. The method according to claim 8, characterized in that, The step of migrating the first instance data of the first inference instance to the GPU memory of the target host includes: Based on the metadata of the first inference instance, the compression algorithm used for the first instance data is determined, wherein the metadata of the first inference instance includes the compression algorithm of the first instance data. Based on the decompression algorithm corresponding to the compression algorithm, the first instance data is decompressed; The decompressed data of the first instance is migrated to the GPU memory of the target host.

14. The method according to claim 8, characterized in that, The step of migrating the first instance data of the first inference instance to the GPU memory of the target host includes: Based on the metadata of the first inference instance, the video memory requirement of the first inference instance is determined, wherein the metadata of the first inference instance includes the video memory requirement of the first inference instance. Based on the aforementioned video memory requirements, GPU video memory is allocated in the target host for the first inference instance. The first instance data of the first inference instance is migrated to the allocated GPU memory.

15. The method according to claim 14, characterized in that, Releasing the GPU memory occupied by the first instance data in the first host includes: The GPU memory occupied by the first instance data is released back to the shared memory pool of the first host, and the memory space in the shared memory pool of the first host is shared by the inference instances in the first host. The process of allocating GPU memory for the first inference instance in the target host based on the GPU memory requirements includes: Based on the aforementioned memory requirements, GPU memory is allocated for the first inference instance from the target host's shared memory pool.

16. The method according to claim 8, characterized in that, The method further includes: When the first inference instance is migrated in, the metadata of the first inference instance is updated, and the metadata of the first inference instance includes at least one of the storage location of the first instance data and compression algorithm.

17. The method according to any one of claims 1 to 16, characterized in that, The method further includes: If the running status information of the first inference instance indicates that the first inference instance meets the conditions for expanding the video memory, additional GPU video memory is allocated to the first inference instance from the video memory shared pool of the first host. If the running status information of the first inference instance indicates that the first inference instance meets the video memory reduction condition, a portion of the video memory space allocated to the first inference instance will be released to the video memory shared pool of the first host. In this context, the video memory space in the shared video memory pool of the first host is shared by the inference instances in the first host.

18. The method according to claim 17, characterized in that, When the runtime status information of the first inference instance indicates that the first inference instance meets the conditions for memory expansion, the additional allocation of GPU memory from the shared memory pool of the first host to the first inference instance includes at least one of the following: When the running status information of the first inference instance indicates that the GPU memory usage of the first inference instance has reached a first usage threshold, additional GPU memory is allocated to the first inference instance from the shared memory pool of the first host. When the running status information of the first inference instance indicates that the first inference instance has received an inference task whose video memory requirement has reached the requirement threshold, additional GPU video memory is allocated to the first inference instance from the video memory shared pool of the first host. The step of releasing a portion of the video memory space allocated to the first inference instance to the video memory shared pool of the first host when the running status information of the first inference instance indicates that the first inference instance meets the video memory reduction condition includes: If the running status information of the first inference instance indicates that the GPU memory usage of the first inference instance is lower than the second usage threshold, the portion of the memory space allocated to the first inference instance will be released to the memory sharing pool of the first host, wherein the first usage threshold is higher than the second usage threshold. The method further includes: When the resource status of the first host indicates that the capacity of the shared memory pool of the first host has reached a first capacity threshold, GPU memory is allocated from the shared memory pool of the first host for the first inference instance.

19. The method according to claim 18, characterized in that, The method further includes: If the resource status of the first host indicates that the capacity of the shared memory pool of the first host is lower than the second capacity threshold, migrate the third instance data of the third inference instance in the first host. Release the GPU memory occupied by the third instance data and add it to the shared memory pool of the first host. The second capacity threshold is lower than the first capacity threshold, and the instance priority of the third inference instance is lower than the instance priority of other inference instances in the first host.

20. A resource management device, characterized in that, The device includes: The acquisition module is configured to acquire the running status information of each inference instance in the cluster, as well as the resource status of each host. The cluster contains multiple hosts, and the inference instances are run by GPUs in the hosts. The determination module is configured to, for the first inference instance among the various inference instances, when the running status information indicates that the first inference instance is in an idle state, determine the migration strategy of the first instance data of the first inference instance based on the resource status of the first host where the first inference instance is currently located, the migration strategy including at least one of intra-host migration and inter-host migration. The migration module is configured to migrate the first instance data based on the migration strategy and release the GPU memory occupied by the first instance data in the first host.

21. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program that is loaded and executed by the processor to implement the resource management method as described in any one of claims 1 to 19.

22. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the resource management method as described in any one of claims 1 to 19.

23. A computer program product, characterized in that, The computer program product includes a computer program stored in a computer-readable storage medium, and a processor reads from and executes the computer program to implement the resource management method as described in any one of claims 1 to 19.

Images