Inference service system and method supporting multi-model hybrid deployment

Abstract

The present application belongs to the technical field of machine learning, and particularly relates to an inference service system and method supporting multi-model hybrid deployment. A plurality of inference instances and a scheduler are communicatively connected, and each inference instance is configured with a local storage space for caching model weight files. The scheduler analyzes a model set required by an inference request, and the local cache state and load condition of each inference instance, intelligently assigns the request to an optimal target inference instance; the target inference instance uses a dynamic downloading mechanism for a model not stored locally, and directly obtains a local model path through the cache for a locally stored model; and fine cache life cycle management is realized through reference counting and LRU strategy, so as to combine cache-aware scheduling and dynamic model loading, solve the problem that a traditional single-instance single-model deployment mode cannot well support multi-model hybrid deployment, and realize efficient use of resources and rapid response of requests.

Description

Inference service system and method supporting multi-model hybrid deployment Technical Field

[0001] This invention belongs to the field of machine learning technology, specifically relating to an inference service system and method that supports the hybrid deployment of multiple models. Background Technology

[0002] With the rapid development of artificial intelligence technology, multimodal generation tasks (such as text-to-image, image-to-video, and 3D generation) have become a hot topic in research and application. Unlike large language models that rely on a single model, multimodal tasks typically require multiple sub-models (such as text encoders, diffusion models, and image decoders) to be combined into a complex workflow for collaborative completion. These models can be derived into almost an infinite variety based on task type, function, accuracy, and fine-tuning versions, resulting in a massive model library that can reach terabyte levels. In real-world production environments, to meet diverse application needs, multimodal inference services must possess high versatility and be able to support flexible and ever-growing model combinations, posing a significant challenge to the underlying computing architecture.

[0003] Currently, traditional model inference services typically employ a single-instance, single-model deployment approach. In this architecture, model files are pre-statically placed on the local disk of the inference instance. After service startup, the instance is only responsible for loading the fixed model and processing corresponding requests. While this architecture is simple and easy to deploy and maintain, it has several shortcomings in practical applications. For example, the disk capacity of a single instance is limited (typically hundreds of GB), unable to accommodate a terabyte-scale global model library, preventing dynamic loading of model files on demand and limiting service flexibility and scalability. Furthermore, model download time is much longer than inference time; downloading the model for every request would result in unacceptable end-to-end latency. Additionally, the lack of effective cache sharing and lifecycle management mechanisms makes it impossible to guarantee the safety of model files in use under concurrent environments, and cache space utilization is low. Moreover, cross-instance request scheduling does not consider cache affinity, easily leading to the same request repeatedly downloading the model on different instances, resulting in a low overall cluster cache hit rate and severely impacting system performance and user experience.

[0004] In view of this, the present invention is hereby proposed. Summary of the Invention

[0005] One objective of this invention is to address the problems in multimodal AI scenarios, such as the inability to dynamically load models on demand, low cache hit rate, and unintelligent cross-instance scheduling, which are caused by the unlimited number of model types and the limited storage of a single instance. These problems result in low system resource utilization and high request response latency.

[0006] To achieve the above objectives, the present invention provides an inference service system supporting multi-model hybrid deployment, comprising:

[0007] Multiple inference instances, each configured with local storage space;

[0008] A scheduler, which communicates with the plurality of inference instances;

[0009] The scheduler is configured to allocate the inference request to the corresponding target inference instance based on the set of models required by the inference request and the local cache status and load of each inference instance.

[0010] The target inference instance is configured to dynamically download and cache models that are required but not stored in the local storage space; provide storage path information for models that are required and stored in the local storage space; and manage the lifecycle of cached models using a reference counting mechanism and manage the cache space based on an LRU strategy.

[0011] Furthermore, each inference instance is deployed with a workflow executor and a model cache management component connected via communication. The workflow executor is configured to, in response to receiving the inference request allocated by the scheduler, parse it and determine the set of models it depends on and the execution order; for each model in the set of models, initiate a retrieval request to the model cache management component; load the retrieved models in sequence and perform inference; and trigger a reference counting mechanism before and after model use. The model cache management component is configured to, in response to the retrieval request initiated by the workflow executor, dynamically download and cache models that are required but not stored in the local storage space; provide storage path information for models that are required and stored in the local storage space; and manage the lifecycle of cached models using a reference counting mechanism and manage the cache space based on an LRU strategy.

[0012] Furthermore, the model cache management component includes: a weight cache module, used to store model weight files locally, maintain cache metadata, record information of all cached items, and use a reference counting mechanism to distinguish between active and inactive cached items, and manage cache space based on an LRU strategy; a weight dynamic acquisition module, used to dynamically acquire the required model weight files according to the requests of the workflow executor, and update the weight cache module; and a weight downloader, used to download model weight files from a remote repository.

[0013] Furthermore, the weighted caching module is configured to: divide the cached models into a used set and a free set; when a model is loaded for inference, its reference count increases and it is placed in the used set; when a model is no longer in use, its reference count decreases, and if the reference count drops to zero, it is moved to the free set; when it is necessary to free up cache space for a new model, only models are selected from the free set for cleanup according to the LRU strategy.

[0014] Furthermore, the weight dynamic acquisition module is configured to: generate a unique cache key based on the model identifier; pre-allocate disk space corresponding to the model file size before downloading; store the downloaded file into the cache through atomic operations and initialize its reference count.

[0015] Furthermore, the step of allocating the request to the corresponding target inference instance based on the model set required by the inference request and the local cache status and load of each inference instance includes: for each inference instance, performing a weighted summation based on the cache hit rate on the model set required by the inference request and its current load to obtain a comprehensive score; and selecting the inference instance with the highest comprehensive score as the target inference instance.

[0016] Furthermore, the system also includes a request queue module connected between the user request entry and the scheduler; the request queue module is configured to receive and temporarily store inference requests initiated by the user, and provide the requests to the scheduler in a first-in-first-out manner; the scheduler is configured to sequentially obtain inference requests from the request queue module and perform query, scoring and assignment jobs.

[0017] In other embodiments, a method for supporting a multi-model hybrid deployment inference service system is provided, applicable to any of the above-mentioned inference service systems supporting multi-model hybrid deployment; the method includes: receiving an inference request and determining the set of models it depends on; selecting a target inference instance based on the cache hit rate of all inference instances to the model set and their current load; dynamically downloading required but not cached models on the target inference instance; obtaining the storage path information of the required and cached models; and managing the lifecycle of cached models on the target inference instance using a reference counting mechanism and managing cache space based on a least recently used strategy.

[0018] In some other embodiments, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is capable of implementing the method for supporting a multi-model hybrid deployment inference service system as described above when executing the computer program.

[0019] In other embodiments, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, it can be applied to the inference service system supporting multi-model hybrid deployment as described above and implement the method for the inference service system supporting multi-model hybrid deployment as described above.

[0020] Based on the foregoing description, those skilled in the art will understand that the present invention communicates with multiple inference instances and a scheduler, and each inference instance is configured with local storage space for caching model weight files. The scheduler analyzes the set of models required for the inference request and the local cache status and load of each inference instance, and intelligently allocates the request to the optimal target inference instance. The target inference instance uses a dynamic download mechanism for models not stored locally, and obtains the storage path information of models stored locally. Fine-grained cache lifecycle management is achieved through reference counting and LRU strategy, thereby combining cache-aware scheduling with dynamic model loading, solving the problem that the traditional single-instance single-model deployment mode cannot support the mixed deployment of multiple models, and achieving efficient resource utilization and fast request response. Attached Figure Description

[0021] The accompanying drawings, as part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments and descriptions of the invention are used to explain the invention, but do not constitute an undue limitation of the invention. Obviously, the drawings described below are merely some embodiments, and those skilled in the art can obtain other drawings based on these drawings without creative effort. In the drawings:

[0022] Figure 1 is a schematic diagram of the structure of an inference service system that supports multi-model hybrid deployment in some embodiments of the present invention;

[0023] Figure 2 is a timing diagram of the model cache execution of an inference service system that supports multi-model hybrid deployment in some embodiments of the present invention;

[0024] Figure 3 is a flowchart of a method for supporting multi-model hybrid deployment of an inference service system in some embodiments of the present invention.

[0025] 100. System; 110. Request Queue Module; 120. Scheduler; 130. Inference Instance; 131. Workflow Executor; 132. Dynamic Weight Acquisition Module; 133. Weight Cache Module; 134. Weight Downloader. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments will be clearly and completely described below with reference to the accompanying drawings. The following embodiments are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0027] Those skilled in the art should understand that the embodiments described below are merely a part of the embodiments of the present invention, and not all of the embodiments of the present invention. These partial embodiments are intended to explain the technical principles of the present invention and are not intended to limit the scope of protection of the present invention. Based on the embodiments provided by the present invention, all other embodiments obtained by those skilled in the art without creative effort should still fall within the scope of protection of the present invention.

[0028] The following describes in detail the inference service system and method supporting multi-model hybrid deployment in some embodiments of the present invention with reference to Figures 1 to 3. Figure 1 is a schematic diagram of the structure of the inference service system supporting multi-model hybrid deployment in some embodiments of the present invention; Figure 2 is a timing diagram of the model cache execution of the inference service system supporting multi-model hybrid deployment in some embodiments of the present invention; Figure 3 is a flowchart of the method for supporting multi-model hybrid deployment of the inference service system in some embodiments of the present invention.

[0029] As shown in Figures 1 and 2, in some embodiments of the present invention, an inference service system supporting multi-model hybrid deployment is provided to address the problems of dynamic model management and efficient scheduling in multimodal AI scenarios. The inference service system 100 supporting multi-model hybrid deployment includes multiple inference instances 130 and a scheduler 120. Each inference instance 130 is configured with local storage space, and the scheduler 120 communicates with the multiple inference instances 130. An inference instance 130 refers to an independent model inference service unit, which can be a physical server, a virtual machine, or a container, etc. In a cluster composed of multiple inference instances 130 (Instance i, j, k in Figure 1), each instance has its own independent disk storage area, i.e., local storage space, directly attached to that instance. This storage area is specifically represented as a WeightCache.

[0030] Local storage space refers to the disk storage area local to the instance, used for caching model weight files. A model set refers to a group of sub-models (such as a text encoder, diffusion model, etc.) required to complete a specific inference task (such as a text-generated graph). Reference counting is a resource management technique that uses a counter to record the number of times a resource is referenced; a count greater than zero indicates the resource is in use, and a count of zero indicates the resource can be released. The Least Recently Use (LRU) strategy is a cache eviction algorithm that prioritizes cleaning up data that has not been accessed for the longest time.

[0031] Scheduler 120 serves as the "brain" and "command center" of the multi-model inference service system 100. Its core function is to intelligently route user requests to the most suitable inference instance 130 to maximize overall system performance. By introducing cache affinity as a key dimension, it fundamentally changes the traditional simple round-robin or load-based scheduling strategies. Specifically, scheduler 120 receives user requests and allocates them to the corresponding target inference instance based on the set of models required for the inference request and the local cache status and load of each inference instance 130.

[0032] Since user requests may arrive suddenly, directly receiving user requests by scheduler 120 could potentially overwhelm the backend scheduler 120 and inference instance 130 due to a massive influx of requests. Therefore, a request queue module 110 is needed to form a "buffer" at the request queue module 110, decoupling request reception and processing, and ensuring that the system remains stable, orderly, and efficient under high concurrency and sudden traffic scenarios. Specifically, system 100 also includes a request queue module 110, which is connected between the user request entry point and scheduler 120. The request queue module 110 is configured to receive and temporarily store inference requests initiated by users, and provide requests to scheduler 120 in a first-in-first-out manner. Scheduler 120 is configured to sequentially retrieve inference requests from the request queue module 110 and execute query, scoring, and job allocation. User-initiated inference requests (1. User request) first arrive at the system 100 entry point. System 100 immediately places the request into the request queue module 110. All requests are queued sequentially in the request queue module 110. The request queue module 110 completely separates request acceptance from request processing. No matter how sudden or irregular the arrival of user requests, the request queue module 110 can absorb them, preventing traffic surges from directly impacting the backend scheduler 120. Then, the request queue module 110 adopts a first-in, first-out (FIFO) strategy by default, ensuring that requests are processed in the order of arrival. When the scheduler 120 is ready to process the next task, it will actively pull a request from the head of the request queue module 110. This "pull" action is performed sequentially, that is, the scheduler 120 will only retrieve the next request from the queue after processing one request, ensuring that the scheduler 120 can only focus on the scheduling decision of one request at a time. At the same time, this provides a stable time window for subsequent time-consuming query and scoring operations, avoiding the complexity of concurrent decision-making.

[0033] Scheduler 120 is configured to allocate requests to corresponding target inference instances based on the set of models required for the inference request and the local cache status and load of each inference instance 130. Upon receiving a request, scheduler 120 parses out all the models required for the request's execution; for example, a text image task might require three models: a "text encoder," a "diffusion model," and an "image decoder." Subsequently, scheduler 120 sends query requests to each inference instance 130 in the cluster (e.g., Instance i, j, k...). The queries include the cache status of the required models and the current device load status (i.e., CPU / memory usage). After receiving responses from all inference instances 130, scheduler 120 begins calculating the overall score for each inference instance 130. The formula for calculating the overall score for each inference instance 130 based on the cache status query of the required models and the current device load status is as follows:

[0034] Overall score = W hit *Cache hit rate +W load *(1-load).

[0035] The cache hit rate is used to calculate the overlap between the cached model of inference instance 130 and the model required by the request. A higher hit rate results in a higher score. The calculation formula is as follows:

[0036] Cache hit rate = |Set of models required for the request ∩ Set of cached models of instance i| / |Set of models required for the request|.

[0037] Load is the current resource utilization of this inference instance 130, normalized to between 0 and 1. The lower the load, the higher the score. hit and W load These are weighting coefficients, which are adjustable parameters used to determine whether to favor cache hit rate or load balancing. hit and W load The value can be set by the operator according to actual needs. For example, if the request requires model {A,B,C}, and instance i has cached {A,B}, then the hit rate is 2 / 3.

[0038] Through the aforementioned query and scoring steps, scheduler 120 no longer blindly allocates tasks, but precisely sends requests to the "most prepared" (highest cache hit rate) and "relatively idle" inference instance 130, thereby minimizing time-consuming model downloads and reducing request latency. Scheduler 120 compares the scores of all inference instances 130, selecting the instance with the highest score as the target inference instance for this request. If multiple instances have the same score, round-robin or random selection is used. Specifically, scheduler 120 is configured to, for each inference instance 130, perform a weighted summation based on the cache hit rate of the model set required by the inference request and its current load to obtain a comprehensive score. The instance with the highest comprehensive score is selected as the target inference instance, and the request is allocated to the target inference instance.

[0039] The target inference instance is configured to dynamically download and cache models that are needed but not stored in its local storage space; and to manage the lifecycle of cached models using a reference counting mechanism to ensure that models in use are not cleaned up, and to manage cache space based on the least recently used (LRU) policy.

[0040] Each inference instance 130 internally deploys a workflow executor 131 and a model cache management component. The workflow executor 131 communicates with the scheduler 120 and the model cache management component. The workflow executor 131 is configured to, in response to receiving an inference request allocated by the scheduler 120, parse the request and determine its dependent model set and execution order; for each model in the model set, initiate a retrieval request to the model cache management component; load the retrieved models in order and perform inference; and trigger a reference counting mechanism before and after model use. The model cache management component is configured to, in response to the retrieval request initiated by the workflow executor 131, dynamically download and cache models that are required but not stored in local storage; provide storage path information for models that are required and stored in local storage; manage the lifecycle of cached models using a reference counting mechanism; and manage cache space based on an LRU policy. The model cache management component includes a weight dynamic acquisition module, namely the weight cache manager 132 (WeightCacheManager), a weight cache module 133 (WeightCache), and a weight downloader 134 (WeightDownloader) shown in Figures 1 and 2.

[0041] The WorkflowExecutor 131, as the core execution unit of the target inference instance, communicates with the scheduler 120 and the weight dynamic acquisition module 132. It is responsible for parsing the set of models that the request depends on, preparing the models through interaction with the cache management component (cache lookup / dynamic download), performing the actual inference computation, managing the model usage lifecycle (ensuring model safety during use through a reference counting mechanism), and finally returning the inference result. It is the key execution hub connecting scheduling decisions and actual computation. Specifically, the WorkflowExecutor 131 is the parser and driver of the workflow. Multimodal tasks (such as text-to-image processing) typically require multiple steps to be executed in a specific order (e.g., text encoding -> diffusion -> image decoding), with each step corresponding to a sub-model. The WorkflowExecutor 131 understands and controls this execution order. When workflow executor 131 receives a request from scheduler 120, it parses the request to determine the model dependency chain required to complete the task, i.e., which models need to be called in sequence, ensuring that complex multimodal tasks can be executed correctly and in an orderly manner. Workflow executor 131 itself does not manage the cache, but it is the requester and user of model resources. As a "client" of the weight dynamic acquisition module 132 (WeightCacheManager), workflow executor 131 requests each model as needed. For each model dependency in the workflow, workflow executor 131 sends a request to the weight dynamic acquisition module 132 (e.g., a contains operation query, an acquire operation formally request usage). This request triggers the collaborative work of the entire cache management component; if the model is already cached, the path is directly obtained; if the model is not cached, a dynamic download process is triggered (executed by weight downloader 134). Then, the workflow executor 131 obtains the locally available path of the model from the weight dynamic acquisition module 132, interacts with the caching system through a standard interface, and does not need to care about the complex details of whether the model is already available locally or needs to be downloaded remotely, thus achieving decoupling of computation and storage. After ensuring that all required models are ready (available locally), the workflow executor 131 loads the model into the computing hardware (such as GPU memory) and performs the actual inference computation. Specifically, the workflow executor 131 loads the model weights into memory according to the model file path obtained from the cache management module; it sends the input data (such as text prompts) into the model, performs forward propagation computation, and generates the output result (such as an image). For complex workflows, the workflow executor 131 manages how the output of the previous model is used as the input of the next model, forming a computational pipeline.Furthermore, the workflow executor 131 triggers and collaborates with the cache management component to perform fine-grained management of the model lifecycle through simple "acquire" and "release" operations. Specifically, before use, the acquire operation increases the model's reference count, indicating that "this model is in use"; after use, the release operation decreases the model's reference count, indicating that "this model has been used up," thus providing feedback on the model's usage status to the cache management component. This ensures that models in use are not mistakenly deleted by the LRU policy, guaranteeing the safety and stability of the inference process.

[0042] The dynamic weight acquisition module, also known as the weight cache manager 132 (WeightCacheManager) shown in Figures 1 and 2, is the core coordinator connecting the workflow executor 131, the weight cache module 133, and the weight downloader 134. The dynamic weight acquisition module 132 communicates with the scheduler 120, the workflow executor 131, the weight cache module 133, and the weight downloader 134. The dynamic weight acquisition module 132 receives instructions, analyzes the situation (cache hit or not), and then precisely schedules various components such as cache lookup, space management, file download, and lifecycle management to work together. However, the dynamic weight acquisition module 132 (WeightCacheManager) does not directly store or download files; instead, it intelligently directs the entire "model preparation" process. Specifically, the dynamic weight acquisition module 132 dynamically acquires the required model weight files based on requests and updates the weight cache module 133. The dynamic weight acquisition module 132 is the sole interface between the workflow executor 131 (responsible for inference computation) and the underlying caching / downloading components. Whenever the workflow executor 131 needs any model, it sends a request to the dynamic weight acquisition module 132 (as shown in the `contains` and `acquire` operations in Figure 2). Upon receiving the request, the dynamic weight acquisition module 132 first queries the weight caching module 133 to determine whether the required model is "cached" or "not cached," abstracting the complex underlying caching and downloading details and providing the workflow executor 131 with a unified and simple model acquisition interface. When the query result indicates a cache hit, the dynamic weight acquisition module 132 executes an efficient path: it retrieves the model's metadata (such as the disk path) from the weight caching module 133; through the `acquire` operation, it increments the model's reference count (ref_count), marking it as "in use"; and it returns the model path to the workflow executor 131, enabling it to immediately load the model for inference. This path achieves extremely fast model loading, which is key to reducing latency and improving performance. When the query result is a cache miss, the weight dynamic acquisition module 132 initiates and directs a complex dynamic acquisition process, which includes space reservation management, triggering cleanup, starting download, and completing caching. Specifically, the space reservation management process: The weight dynamic acquisition module 132 first queries the weight cache module 133 for the required file size and performs a reserve operation to request pre-allocated disk space. The triggering cleanup process: If space is insufficient, the weight dynamic acquisition module 132 directs the weight cache module 133 to execute the LRU policy cleanup process (only cleaning up free models with a reference count of 0). The starting download process: After successful space reservation, the weight dynamic acquisition module 132 commands the weight downloader 134 to download the model file from the remote repository.The caching process is completed as follows: After the model download is finished, the weight dynamic acquisition module 132 instructs the weight caching module 133 to officially insert the file into the database through atomic operations and create or update cache metadata. The weight dynamic acquisition module 132 links the scattered steps of "space check-cleanup-download-caching" into an atomic and safe complete transaction, ensuring the reliability and data consistency of the entire dynamic acquisition process. Finally, at the beginning and end of model use, the weight dynamic acquisition module 132 is responsible for updating its state and working with the weight caching module 133 to manage reference counting. Specifically, when use begins, acquire is executed, and the reference count is incremented by 1; when use ends, release is executed, the reference count is decremented by 1, and if the count drops to 0, the weight caching module 133 is notified to move the model to the idle queue. Through precise reference counting management, the weight dynamic acquisition module 132 ensures that the model in use is absolutely safe and will not be mistakenly deleted by LRU or other cleanup strategies.

[0043] The WeightCache module 133 is a core component deployed within each inference instance 130. It communicates with the dynamic weight acquisition module 132 and is responsible for the physical storage of model weight files, maintenance of metadata, and fine-grained lifecycle management. Specifically, the WeightCache module 133 stores model weight files locally and maintains cache metadata, recording information for all cached items. Cache item information includes at least the storage status and usage of the cache item. The WeightCache module 133 uses a reference counting mechanism to distinguish between active and inactive cache items, ensuring that model weight files currently in use are not mistakenly deleted, and manages cache space based on a Least Recently Used (LRU) strategy to optimize resource utilization. The WeightCache module 133 is the physical repository for model weight files on the local disk of the inference instance 130. All model files obtained through the weight downloader 134 are ultimately stored here through atomic operations. Converting time-consuming remote downloads (minute-level) into fast local reads (second-level) is the fundamental guarantee for improving inference speed. The weight cache module 133 (WeightCache) also maintains cache metadata or a cache metadata table (as shown in Figure 2), recording key management information for each cached model file. Cache metadata includes, but is not limited to, cache key, storage path information, file attribute information, and lifecycle management information. The cache key is a unique identifier for the model, generated using a hash algorithm (such as MD5 or SHA256) based on the model identifier (e.g., the model's URL, unique name, etc.), for example, sfca_weight_[url_hash]. This avoids name conflicts and ensures that models with the same name from different sources can be correctly distinguished. Storage path information includes the base directory (base_dir) for storing model files, the subdirectories (sub_dir) for storing model files used for categorized management, and the actual filename of the model weight file (file_name). This information is used to locate the physical location of the model file on the disk for loading by the workflow executor 131 (WorkflowExecutor). File attribute information includes the size (in bytes) of the model file, used for space management. It calculates how much space is needed or how much space can be released when pre-allocating disk space and triggering LRU cleanup. The lifecycle management information is the most crucial part, including the number of inference tasks currently using the model file (ref_count) and the timestamp of the last time the model file was used or accessed (last_access). The reference count (ref_count) is used to distinguish whether the model is "in use" (ref_count>0) or "idle" (ref_count=0), ensuring that models with ref_count>0 (active cache items) are not cleaned up.The last access time (last_access) is the direct basis for implementing the LRU (Least Recently Used) eviction policy. When space needs to be cleaned up, models with the smallest last_access value (i.e., least recently used) and ref_count=0 are prioritized for eviction. This cache metadata table makes cache management no longer blind file operations, but intelligent decisions based on precise metadata. Meanwhile, the weighted cache module 133 (WeightCache) is also the core security mechanism of the module. Through reference counting (ref_count), the module divides cached items into two sets for management: the In Use Queue and the Idle Queue. The In Use Queue stores models with ref_count>0. These models are being used by one or more inference tasks and are absolutely protected, not to be cleaned up. The Idle Queue stores models with ref_count=0. These models are no longer used and are candidates for cache cleanup. When the WorkflowExecutor (131) needs to use a model, the module increments its `ref_count` by 1 and moves it to the currently used set. After use, `ref_count` is decremented by 1; if it reaches zero, it is moved to the idle set. This completely avoids the risk of accidentally deleting model files supporting online inference in a concurrent environment, ensuring service stability and high availability. When disk space is insufficient and space needs to be freed up for new models, the weight dynamic acquisition module (132, WeightCacheManager) performs cleanup based on the Least Recently Used (LRU) strategy. The cleanup operation only targets models in the idle set (ref_count=0) and prioritizes evicting the least recently accessed models, ensuring that the limited cache space always retains the most frequently used models, thereby maximizing the overall cache hit rate, reducing unnecessary downloads, and optimizing resource utilization.

[0044] The WeightDownloader 134 is the "executor" or "file transfer engine" in the model cache management component, communicating with the dynamic weight acquisition module 132. The core function of the WeightDownloader 134 is to securely and reliably download model weight files from a remote repository to the local machine of the inference instance 130, according to instructions. Specifically, the WeightDownloader 134 downloads model weight files from a remote repository. When the dynamic weight acquisition module 132 determines that a certain model is needed and is not cached locally, it commands the WeightDownloader 134 to execute the download task. The WeightDownloader 134 connects to a specified remote model repository (such as Hugging Face, an internal model server, etc.) and pulls model weight files (such as .safetensors or .bin files) to the local machine of the inference instance 130 via HTTP / HTTPS or other file transfer protocols, realizing on-demand dynamic model acquisition. It is the fundamental execution unit that breaks the static deployment mode of "single instance, single model" and enables multi-model hybrid deployment. The WeightDownloader 134 does not simply download; it needs to ensure reliability during large file transfers. The Weight Downloader 134 supports chunked downloading and resumeable downloading, allowing downloads to resume from where they left off after an interruption, preventing data loss due to network fluctuations. It also supports integrity verification, using a checksum (e.g., SHA256) to verify file integrity after download, preventing file corruption due to network errors or source file issues. Furthermore, it supports downloading to a temporary location (e.g., model.tmp) to ensure no "half-finished" files appear in the official cache area; other processes always see only complete files in the official cache directory. Since model files are typically large (GB-level), these reliable download mechanisms, including chunked downloading, resumeable downloading, integrity verification, and downloading to temporary locations, guarantee the correctness and availability of cached files, preventing inference failures or system instability caused by file corruption. Moreover, the Weight Downloader 134 is a crucial part of the entire cache update process and requires close cooperation with other components. Typically, the weight downloader 134 only begins the download job after the weight dynamic acquisition module 132 has completed disk space reservation, ensuring sufficient local space. After the download and verification process is complete, the weight dynamic acquisition module 132 performs an atomic move (rename) operation to instantly convert the temporary file into a formal cache file. This operation is performed by the manager calling system commands, but the downloader provides the correct source file. This collaborative mechanism of "temporary file + atomic move" ensures that the cache file seen by other workflows at any given time is complete and consistent, achieving data security in a concurrent environment.

[0045] After scheduler 120 assigns a request to the selected target inference instance, the workflow executor 131 in the target inference instance receives the request, performs preprocessing, determines the dependent weights (i.e., parses out which model files are needed to complete this inference task), and initiates a contains operation to the weight dynamic acquisition module 132 (whose logic is implemented by the weight cache manager WeightCacheManager), i.e., queries whether the currently required model is already in the local cache. The query result includes two cases: cache hit and cache miss.

[0046] A cache hit occurs when the weight dynamic acquisition module 132 finds that the model already exists in the weight cache module 133 (WeightCache). The weight dynamic acquisition module 132 directly returns the weight name and disk location to the workflow executor 131, which then begins using the model and managing reference counting. The target inference instance is also configured to obtain the model's storage path information for models required and stored in its local storage space; and to manage the lifecycle of cached models using a reference counting mechanism to ensure that models in use are not evicted, and to manage cache space based on a Least Recently Used (LRU) strategy. Specifically, the weight dynamic acquisition module 132 queries the weight cache module 133 to obtain the storage path information of the model; the reference count (ref_count) of the model is incremented to mark it as active; the model is loaded based on the returned storage path and inference calculation is performed; the reference count is kept greater than 0 during the model's use to ensure that the weight cache module 133 recognizes it as active and avoids it being mistakenly deleted by the LRU cleanup strategy; the reference count is decremented after inference is completed, and if the reference count drops to 0, the model is moved into the idle queue as a candidate for subsequent LRU cleanup.

[0047] A cache miss occurs when the dynamic weight acquisition module 132 detects that the model is not local and initiates the dynamic acquisition process. The target inference instance is configured to dynamically download and cache models that are needed but not stored in its local storage space; and to manage the lifecycle of cached models using a reference counting mechanism to ensure that models in use are not cleaned up, and to manage cache space based on the Least Recently Used (LRU) strategy. Specifically, the dynamic weight acquisition module 132 first performs an acquire operation, formally requesting the use of the model from the weight caching module 133. Since the model does not exist, the module needs to obtain the weight size and perform a reserve operation, requesting pre-allocation of corresponding disk space from the weight caching module 133. If the current disk space is insufficient, the weight caching module 133 will initiate a cleanup process. As shown in Figure 2, the scheduler 120, based on the LRU strategy, cleans up weights in the idle queue (these weights have a reference count of 0) and deletes files from the disk until the space requirement is met. This step ensures that models in use (reference count > 0) are absolutely safe. After successful space reservation, the weight downloader 134 is invoked to download the weight file from the remote repository to a temporary location. Once the download is complete, the weight dynamic acquisition module 132 atomically stores the file in the final location by performing an `add` operation, and creates or updates a cache metadata record in the weight caching module 133. This record is the cache item information mentioned in the first diagram. The cache item information includes at least: base directory (base_dir), subdirectory (sub_dir), weight name (file_name), disk location (base_dir, sub_dir), file size (size), reference count (ref_count) (initialized to 1 here, indicating that the model has been used by the current request), and last access time (last_access) (the model is marked as "active" and placed in the `inuse` set). Specifically, the weight dynamic acquisition module 132 is configured to generate a unique cache key based on the model identifier; pre-allocate disk space corresponding to the model file size before downloading; and atomically store the downloaded file in the cache and initialize its reference count.

[0048] After the inference task relying on the model is completed, the WorkflowExecutor 131 initiates a release operation. The weight dynamic acquisition module 132 notifies the weight cache module 133 to decrement the reference count of the model by 1, while checking if the reference count has dropped to 0. If the reference count > 0, it means that other tasks are using it, and the model remains in the in-use set. If the reference count = 0, it means that the model is completely idle. The weight cache module 133 will move it from the in-use set to the idle set. From this point on, the model enters the "inactive" state and becomes a candidate for future cleanup under the LRU policy. After the WorkflowExecutor 131 completes a request, it returns the request result (5.Result), and then the scheduler 120 generates a response (6.Response) and returns it to the user. After that, it returns to the request queue module 110, and then the scheduler 120 dispatches the next request (4.Dispatch) to the WorkflowExecutor 131. The WorkflowExecutor 131 begins preparing for the model dependencies of the next inference. Specifically, the weight caching module 133 is configured to divide cached models into a used set and a free set; when a model is loaded for inference, its reference count increases and it is placed in the used set; when a model is no longer in use, its reference count decreases, and if the reference count drops to zero, it is moved to the free set; it determines whether the available space on the current disk is greater than or equal to the size of the new model file. If the available space on the current disk is less than the size of the new file, the LRU cleanup policy is triggered, that is, when cache space needs to be freed up for a new model, only models are selected from the free set for cleanup according to the LRU policy. The LRU policy prioritizes the elimination of the least used and currently unused model files. After the LRU cleanup policy is triggered, the weight dynamic acquisition module 132 calculates the space to be released: space to be released = new file size - current available space. It checks whether there is sufficient free space: the system traverses the free set (cache_idle) and calculates the sum of the sizes of all model files in it (i.e., the total amount of space to be released). If the total available free space is less than the space to be freed, even clearing all idle models will not be enough, the reserve operation will return False, triggering an "insufficient cache space" error, and the download process will be aborted. If the total available free space is equal to or greater than the space to be freed, the LRU policy cleanup can meet the requirements, and the LRU policy eviction step will be executed. The LRU policy eviction step only targets model files in the free set (cache_idle), whose reference count (ref_count) is 0, indicating that they are not currently used by any inference task and are safe to be cleaned up.The weight dynamic acquisition module 132 checks the last access time (last_access) field of each model in the free set. Starting with the model with the smallest last_access value (i.e., the one that has not been accessed for the longest time), it deletes them sequentially until the accumulated freed disk space reaches or exceeds the "space to be released". For each selected model, its record is removed from the free set and the cache metadata table; a callback function (such as on_file_evicted) is called to delete its corresponding physical file on the disk; and the total cache space occupied is updated. After successful space release, the reserve operation returns True. The next download process continues.

[0049] As shown in Figure 3, in some other embodiments of the present invention, a method for supporting a multi-model hybrid deployment inference service system 100 is provided, which can be applied to the multi-model hybrid deployment inference service system 100 described above; the method includes:

[0050] Step S110: Receive inference requests and determine the set of models they depend on. The inference service system 100 supporting multi-model hybrid deployment includes a scheduler 120, which receives inference requests and determines the set of models the inference request depends on. Before step S110, "the scheduler 120 receives the inference request," the system further includes: receiving and temporarily storing user-initiated inference requests through a request queue module 110, and providing the requests to the scheduler 120 in a first-in-first-out manner.

[0051] Step S120: Based on the cache hit rate of all inference instances 130 on the model set and their current load, a target inference instance is selected. The scheduler 120 queries the local cache status and current load of each inference instance 130. Each inference instance 130 maintains a locally stored model weight file and manages the cache lifecycle using a reference counting mechanism. The scheduler 120 selects the inference instance 130 with the highest overall score as the target inference instance and allocates the inference request to the target inference instance. Specifically, the scheduler 120 obtains the cache status and load of each inference instance 130; it performs a weighted sum based on the cache hit rate and current load of the model set required by the inference request to obtain the overall score, calculated as: Overall Score = W hit *Cache hit rate +W load *(1-load), W hit and W load These are weighting coefficients, which are adjustable parameters used to determine whether to favor cache hit rate or load balancing. hit and W loadThe value can be set by the operator according to actual needs. The instance with the highest comprehensive score is selected as the target inference instance. If multiple inference instances have the same score, round-robin or random selection is used.

[0052] The steps by which the scheduler 120 obtains the cache status and load of each inference instance 130 include: for each inference instance 130, calculating its cache hit rate on the set of models required by the inference request, using the following formula:

[0053] Cache hit rate = |Set of models required for the request ∩ Set of cached models of instance i| / |Set of models required for the request|; where cache hit rate is the degree of overlap between the cached models of this inference instance 130 and the models required for the request. The higher the hit rate, the higher the score. Then, obtain the current load metric for each inference instance 130.

[0054] Step S130: On the target inference instance, dynamically download the required but not cached model; for the required and cached model, obtain the model's storage path information.

[0055] The steps for performing dynamic download and caching include: generating a unique cache key based on the model identifier; pre-allocating disk space corresponding to the model file size before downloading; storing the downloaded file into the cache through atomic operations and initializing its reference count.

[0056] The steps for pre-allocating disk space include: obtaining the size of the model file to be downloaded; determining whether the current remaining disk space meets the requirements; if the current remaining disk space is insufficient, cleaning up idle model files with a reference count of zero based on the LRU strategy until the released space meets the requirements.

[0057] Step S140: On the target inference instance, a reference counting mechanism is used to manage the lifecycle of the cached model, and the cache space is managed based on the least recently used strategy.

[0058] The steps of managing the lifecycle of cached models using a reference counting mechanism include: dividing the cached models into a set in use and a set in free; when a model is loaded for inference, its reference count is increased and it is placed in the set in use; when a model is no longer in use, its reference count is decreased; if the reference count drops to zero, the model is moved to the set in free.

[0059] The steps for managing cache space based on the LRU strategy include: when it is necessary to free up cache space for a new model, only models are selected from the free set according to the least recently used order for cleanup.

[0060] In some other embodiments of the present invention, the method further includes: after the target inference instance completes inference, updating its cache index and recording the information of the newly downloaded model weight file.

[0061] In other embodiments of the present invention, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor is capable of being applied to the inference service system 100 supporting multi-model hybrid deployment as described above and, when executing the computer program, implementing the method for the inference service system 100 supporting multi-model hybrid deployment as described above.

[0062] In other embodiments of the present invention, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, it can be applied to the inference service system 100 supporting multi-model hybrid deployment as described above and implement the method for the inference service system 100 supporting multi-model hybrid deployment as described above.

[0063] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, prediction models, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0064] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0065] Those skilled in the art will understand that the present invention communicates with multiple inference instances 130 and a scheduler 120, and each inference instance 130 is configured with local storage space for caching model weight files. The scheduler 120 analyzes the set of models required for inference requests and the local cache status and load of each inference instance, and intelligently allocates requests to the optimal target inference instance. The target inference instance uses a dynamic download mechanism for models not stored locally, and obtains the storage path information of models stored locally. Fine-grained cache lifecycle management is achieved through reference counting and LRU strategy, thereby combining cache-aware scheduling with dynamic model loading, solving the problem that the traditional single-instance single-model deployment mode cannot support multi-model mixed deployment, and achieving efficient resource utilization and fast request response.

[0066] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0067] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the claims, any of the claimed embodiments can be used in any combination.

[0068] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-described technical content to create equivalent embodiments without departing from the scope of the present invention. The implementation schemes in the above embodiments can be further combined or replaced. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A reasoning service system (100) supporting multi-model hybrid deployment, characterized in that, include: Multiple inference instances (130), each inference instance (130) is configured with local storage space; A scheduler (120) is communicatively connected to multiple inference instances (130). The scheduler (120) is configured to allocate the inference request to a corresponding target inference instance based on the model set required by the inference request and the local cache status and load of all inference instances (130). The target inference instance is configured to dynamically download and cache models that are required but not stored in the local storage space; obtain the storage path information of the models that are required and stored in the local storage space; and manage the lifecycle of cached models using a reference counting mechanism and manage the cache space based on an LRU policy. Each inference instance (130) is deployed with a workflow executor (131) and a model cache management component that are communicatively connected. The workflow executor (131) is configured to parse the inference request allocated by the scheduler (120) and determine the model set and execution order that it depends on; for each model in the model set, initiate a retrieval request to the model cache management component; and load and execute inference on the retrieved models in order. And trigger a reference counting mechanism before and after model use; the model cache management component is configured to, in response to the acquisition request initiated by the workflow executor (131), perform dynamic download and cache for models that are required but not stored in the local storage space; provide storage path information for models that are required and stored in the local storage space; and manage the lifecycle of cached models through a reference counting mechanism and manage cache space based on the LRU strategy; the step of allocating the inference request to the corresponding target inference instance according to the model set required by the inference request and the local cache status and load of all inference instances (130) includes: for each inference instance (130), performing a weighted summation of the cache hit rate on the model set required by the inference request and its current load to obtain a comprehensive score; The reasoning instance with the highest overall score (130) is selected as the target reasoning instance.

2. The inference service system (100) supporting multi-model hybrid deployment according to claim 1, characterized in that, The model cache management component includes: a weight cache module (133), which stores model weight files locally, maintains cache metadata, records information of all cached items, distinguishes between active and inactive cached items using a reference counting mechanism, and manages cache space based on an LRU strategy; a weight dynamic acquisition module (132), which dynamically acquires the required model weight files according to the request of the workflow executor (131) and updates the weight cache module (133); and a weight downloader (134), which downloads model weight files from a remote repository.

3. The inference service system (100) supporting multi-model hybrid deployment according to claim 2, characterized in that, The weighted cache module (133) is configured to: divide the cached models into a used set and a free set; when a model is loaded for inference, its reference count increases and it is placed in the used set; when a model is no longer in use, its reference count decreases, and if the reference count drops to zero, it is moved to the free set; when it is necessary to free up cache space for a new model, only models are selected from the free set for cleanup according to the LRU strategy.

4. The inference service system (100) supporting multi-model hybrid deployment according to claim 2, characterized in that, The weight dynamic acquisition module (132) is configured to: generate a unique cache key based on the model identifier; pre-allocate disk space corresponding to the model file size before downloading; store the downloaded file into the cache through atomic operations and initialize its reference count.

5. The inference service system (100) supporting multi-model hybrid deployment according to claim 1, characterized in that, The system (100) further includes a request queue module (110), which is connected between the user request entry and the scheduler (120). The request queue module (110) is configured to receive and temporarily store inference requests initiated by users and provide the requests to the scheduler (120) in a first-in-first-out manner. The scheduler (120) is configured to sequentially obtain inference requests from the request queue module (110) and perform query, scoring and assignment operations.

6. A method for supporting a multi-model hybrid deployment inference service system (100), characterized in that, An inference service system (100) supporting multi-model hybrid deployment, applicable to any one of claims 1 to 5, is provided. The method includes: receiving an inference request and determining the set of models it depends on; selecting a target inference instance based on the cache hit rate of all inference instances (130) on the set of models and their current load; dynamically downloading required but not cached models on the target inference instance; obtaining the storage path information of the models for required and cached models; and managing the lifecycle of cached models on the target inference instance using a reference counting mechanism and managing the cache space based on a least recently used strategy.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor is applied to the inference service system (100) supporting multi-model hybrid deployment as described in any one of claims 1 to 5 and executes a computer program, it implements the method for the inference service system (100) supporting multi-model hybrid deployment as described in claim 6.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it can be applied to the inference service system (100) supporting multi-model hybrid deployment as described in any one of claims 1 to 5 and implement the method for the inference service system (100) supporting multi-model hybrid deployment as described in claim 6.

Images