Model management system and method

Abstract

The embodiments of the present specification provide a model management system and method, wherein the model management system comprises a model service gateway and a model controller, comprising: the model service gateway is used for, in response to a calling request for a target business model, determining that the target business model is in a dormant state, modifying the expected state of a model resource object of the target business model to a wake-up state; the model controller is used for, in the case of detecting that the expected state of the model resource object is the wake-up state, allocating a target resource for the target business model, and sending a wake-up operation to a model instance of the target business model; modifying the actual state of the model resource object to the wake-up state according to the resource allocation result and the wake-up operation result; the model service gateway is used for, in the case of detecting that the actual state of the model resource object is the wake-up state, executing a model processing task corresponding to the calling request by using the target business model.

Description

[0001] This application claims priority to Singapore Patent Application No. 10202602448Q, filed on July 15, 2026, entitled “Model Management System and Method”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The embodiments in this specification relate to the field of model management technology, and particularly to model management systems and methods. Background Technology

[0003] With the development of computer and internet technologies, Large Language Models (LLMs) have become a core driving force in the field of artificial intelligence. However, providing efficient and low-cost inference services has become a key challenge for their commercialization. Cloud-native technologies, especially Kubernetes (K8s), have become the preferred infrastructure for deploying AI inference services due to their powerful container orchestration and automated management capabilities. Meanwhile, high-performance inference frameworks such as vLLM effectively manage the massive key-value cache in LLM inference through the PagedAttention algorithm, improving the request throughput and resource utilization of a single GPU. In terms of deployment paradigms, Serverless architecture, promising that users only need to focus on code logic while the underlying resources are automatically managed by the platform, has become an ideal choice. However, combining Serverless with large model inference requires addressing the fundamental contradiction between resource utilization and latency. While cloud-native solutions based on dynamic scaling are common in existing technologies, they face trade-offs such as high TTFT introduced by cold starts and the design of scaling-down thresholds. Integrated deployment solutions based on Kubernetes Operator or Ray optimize deployment processes and single-instance throughput, but fail to address issues such as long process initialization times, inability to reuse runtime instances, and the lack of intelligent dynamic preemption and fast data transfer mechanisms. Therefore, an effective solution is urgently needed to address these problems. Summary of the Invention

[0004] In view of the above, embodiments of this specification provide a model management system. One or more embodiments of this specification also relate to a model management method, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a model management system is provided, including a model service gateway and a model controller, comprising: The model service gateway is used to respond to a call request for a target business model and, if it is determined that the target business model is in a dormant state, modify the desired state of the model resource object of the target business model to a wake-up state. The model controller is configured to allocate target resources to the target business model and send a wake-up operation to the model instance of the target business model when the expected state of the model resource object is detected to be a wake-up state; and modify the actual state of the model resource object to a wake-up state based on the resource allocation result and the wake-up operation result. The model service gateway is used to execute the model processing task corresponding to the call request using the target business model when it detects that the actual state of the model resource object is awake.

[0006] According to a second aspect of the embodiments of this specification, a model management method is provided, applied to a model management system, the model management system including a model service gateway and a model controller, comprising: In response to a call request for a target business model, if the model service gateway determines that the target business model is in a dormant state, it modifies the desired state of the model resource object of the target business model to a wake-up state. When the model controller detects that the expected state of the model resource object is the awakened state, it allocates target resources for the target business model and sends a wake-up operation to the model instance of the target business model. Based on the resource allocation result and the wake-up operation result, it modifies the actual state of the model resource object to the awakened state. When the model service gateway detects that the actual state of the model resource object is awake, it uses the target business model to execute the model processing task corresponding to the call request.

[0007] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the above-described model management method.

[0008] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions that, when executed by a processor, implement the steps of the model management method described above.

[0009] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program or instructions that, when executed by a processor, implement the steps of the model management method described above.

[0010] The model management system provided in this embodiment includes a model service gateway and a model controller. When the model service gateway responds to a call request for a target business model and determines that the target business model is in a dormant state, it can modify the expected state of the model resource object of the target business model to a awakened state. Then, when the model controller detects that the expected state of the model resource object is awakened, it allocates target resources for the target business model and sends a wake-up operation to the model instance of the target business model. Afterwards, based on the resource allocation result and the wake-up operation result, it modifies the actual state of the model resource object to an awakened state. Finally, when the model service gateway detects that the actual state of the model resource object is awakened, it uses the target business model to execute the model processing task corresponding to the call request. This achieves dynamic management of model state and resource allocation through the model service gateway and model controller, modifying the expected state, allocating resources, and waking up the model when a dormant state is detected, thereby reducing model cold start latency and improving GPU resource utilization. It has the advantages of reducing model cold start latency and improving GPU resource utilization. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the structure of a model management system provided in one embodiment of this specification; Figure 2 This is a schematic diagram of the system architecture of a model management system provided in one embodiment of this specification; Figure 3 This is a flowchart illustrating a model management method provided in one embodiment of this specification; Figure 4 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0012] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0013] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0014] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0015] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0016] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0017] TTFT (Time to First Token): First Token Delay. This is a performance metric that measures the time elapsed from when a user submits a request to when the model generates and returns the first output token. A low TTFT is crucial for improving user experience because it directly impacts the perceived response speed.

[0018] vLLM: A high-performance inference framework for large language models. It is an open-source library that efficiently manages GPU memory using advanced algorithms such as PagedAttention, aiming to significantly improve the throughput and efficiency of inference for large language models.

[0019] K8s Operator (Kubernetes Operator): Kubernetes Operator. It is an extension of Kubernetes software that uses custom resource definitions (CRDs) and controller patterns to automate, package, and manage complex Kubernetes applications, encoding human operator expertise into the software.

[0020] Serverless: A serverless architecture. In this cloud computing execution model, the cloud service provider is responsible for configuring, managing, and scaling the underlying infrastructure. Users do not need to worry about server management; they only pay for the computing resources they actually use.

[0021] RDMA (Remote Direct Memory Access): This is a technology that allows networked computers to exchange data directly in main memory, bypassing the CPU and cache of both operating systems, thereby achieving high-throughput, low-latency data transfer.

[0022] LLM (Large Language Model): A deep learning model with a huge number of parameters (usually billions or more) that can understand and generate human language text and is widely used in natural language processing tasks.

[0023] KV Cache (Key-Value Cache): In the self-attention mechanism of large language models, it is used to store the key and value representations of previously generated tokens so that they can be reused when generating subsequent tokens, thereby improving inference efficiency.

[0024] K8s (Kubernetes): Kubernetes is an open-source platform for automating the deployment, scaling, and management (container orchestration) of containerized applications.

[0025] HPA (Horizontal Pod Autoscaler): Automatic scaling of horizontal pods. A mechanism in Kubernetes that automatically adjusts the number of pod replicas based on CPU utilization or custom metrics.

[0026] CR (Custom Resource): A Kubernetes API extension that allows users to store and manage custom structured data.

[0027] CRD (Custom Resource Definition): A custom resource definition. An API object that defines new types of custom resources in a Kubernetes cluster.

[0028] TP (Tensor Parallelism): A distributed training / inference technique that splits the model's tensors (data) across multiple devices for parallel processing.

[0029] PP (Pipeline Parallelism): Another distributed technique that assigns different layers / stages of a model to different devices, forming a processing pipeline.

[0030] CUDA IPC (Inter-Process Communication) is a high-efficiency transmission mechanism that allows different processes on the same machine to directly share GPU memory resources and events. It is used for fast copying of model data (wake-up operation) on the same node (same physical machine). By using CUDA-IPC to achieve efficient data copying, it significantly reduces model loading time and lowers the first token latency (TTFT).

[0031] This specification provides a model management system. One or more embodiments of this specification also relate to a model management method, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0032] In practical applications, existing technologies typically employ a strategy of "dynamically scaling down to zero replicas" to conserve GPU resources. When a new request arrives, it must go through a complete model loading process. This process includes not only the time required to load model weights from storage (such as disk, network storage, or cache) to GPU memory, but also a series of time-consuming initialization steps, such as creating new container instances, initializing process contexts, establishing network communication channels, configuring environment variables, and profiling the PagedAttention KV Cache. These additional times, not related to model loading itself, accumulate to a significant proportion, resulting in long TTFTs (Time To First Token Delay) for users, severely impacting the real-time response experience of Serverless services.

[0033] On the other hand, to avoid excessively frequent scaling up and down, existing solutions also set a fixed "idle timeout interval" (e.g., 5 minutes). A model is only scaled down to 0 replicas if there is no traffic within this fixed interval. This static threshold-based scaling down strategy is highly inefficient when deploying multiple models on a large scale. During the timeout period, a large amount of expensive GPU memory is continuously occupied by idle model instances and cannot be released to other services that require it, resulting in low utilization of the entire GPU cluster and increased operating costs. Therefore, an effective solution is urgently needed to address these issues.

[0034] The model management system provided in this embodiment includes a model service gateway and a model controller. When the model service gateway responds to a call request for a target business model and determines that the target business model is in a dormant state, it can modify the expected state of the model resource object of the target business model to a awakened state. Then, when the model controller detects that the expected state of the model resource object is awakened, it allocates target resources for the target business model and sends a wake-up operation to the model instance of the target business model. Afterwards, based on the resource allocation result and the wake-up operation result, it modifies the actual state of the model resource object to an awakened state. Finally, when the model service gateway detects that the actual state of the model resource object is awakened, it uses the target business model to execute the model processing task corresponding to the call request. This achieves dynamic management of model state and resource allocation through the model service gateway and model controller, modifying the expected state, allocating resources, and waking up the model when a dormant state is detected, thereby reducing model cold start latency and improving GPU resource utilization. In other words, the model management system provided in this embodiment can achieve the dual goals of low latency and high GPU resource utilization in sparse traffic scenarios.

[0035] See Figure 1 , Figure 1 This is a schematic diagram of the structure of a model management system provided in one embodiment of this specification. The model management system 100 includes a model service gateway 110 and a model controller 120.

[0036] The model service gateway 110 is used to respond to a call request for a target business model and, if it is determined that the target business model is in a dormant state, modify the desired state of the model resource object of the target business model to a wake-up state. The model controller 120 is configured to allocate target resources to the target business model and send a wake-up operation to the model instance of the target business model when the expected state of the model resource object is detected to be a wake-up state; and modify the actual state of the model resource object to a wake-up state according to the resource allocation result and the wake-up operation result. The model service gateway 110 is used to execute the model processing task corresponding to the call request using the target business model when it is detected that the actual state of the model resource object is awake.

[0037] When deploying large language model inference services in a cloud-native environment, due to the sparsity of request traffic, model instances are scaled down to a dormant state during idle periods. When a new request arrives, the model needs to be reloaded into a running state. This process involves operations such as process initialization, GPU resource allocation, and communication settings, increasing the time required to generate the first token. Furthermore, the scaling-down mechanism based on fixed time intervals cannot dynamically adapt to traffic changes, resulting in the ineffective use of GPU resources during idle periods and reducing overall resource utilization. Among these factors, the process initialization time when the model wakes up from dormancy becomes a key factor affecting system response speed, and the asynchronous execution of resource allocation and model activation further exacerbates the latency problem.

[0038] For example, in an online customer service application deployed on a Kubernetes cluster, the vLLM inference framework is used to provide large language model services. During off-peak hours at night, user query frequency decreases, and the system shrinks model instances to a dormant state. When a new user session request arrives, the model service gateway detects that the target model is in a dormant state and triggers a wake-up process. This process includes reallocating GPU resources, initializing model processes, and establishing communication channels, resulting in increased latency in the user's first token response. Furthermore, after the daytime peak traffic ends, because the shrinkage threshold is set to a fixed duration, model instances remain running during low-traffic periods, causing GPU computing units to remain idle and failing to release resources for other task scheduling. Thus, the system faces an inherent contradiction between resource allocation efficiency and response latency in sparse traffic scenarios.

[0039] If the above issues are not addressed, the initial token latency for the inference service will remain high, impacting the user experience. Simultaneously, inefficient use of GPU resources will limit the overall throughput of the cluster, increasing infrastructure costs. In scenarios with significant traffic fluctuations, the system may fail to respond promptly to sudden requests, reducing service availability. Specifically, the serial processing mechanism for process initialization and resource allocation will prolong the wake-up process, while static scaling-down strategies cannot match dynamic traffic demands, resulting in widespread resource idleness.

[0040] To address this, this embodiment provides a model management system, including a model service gateway and a model controller. When the model service gateway determines that the target business model is in a dormant state in response to a call request for that model, it can modify the expected state of the model resource object of the target business model to a awakened state. Then, when the model controller detects that the expected state of the model resource object is awakened, it allocates target resources for the target business model and sends a wake-up operation to the model instance of the target business model. Afterwards, based on the resource allocation result and the wake-up operation result, it modifies the actual state of the model resource object to an awakened state. Finally, when the model service gateway detects that the actual state of the model resource object is awakened, it can use the target business model to execute the model processing task corresponding to the call request. This achieves dynamic management of model state and resource allocation through the model service gateway and model controller, modifying the expected state, allocating resources, and waking up the model when a dormant state is detected, thereby reducing model cold start latency and improving GPU resource utilization. It has the advantages of reducing model cold start latency and improving GPU resource utilization.

[0041] Specifically, a model management system is responsible for the full lifecycle management of business models, including model registration, deployment, wake-up, hibernation, and resource allocation, aiming to optimize the performance and resource utilization efficiency of model services. The model service gateway, as the unified entry point for model services, is responsible for receiving external call requests and routing or triggering corresponding state transition processes based on the current state of the model. Its role is to shield the complexity of the underlying models and provide a unified access interface. The model controller is one of the core components of the model management system, responsible for monitoring the state changes of model resource objects and performing specific operations such as resource allocation and model instance wake-up based on the desired state to ensure that models can provide services on demand. The target business model refers to the specific business model that needs to be managed and provided with services, such as a large language model or image recognition model.

[0042] Correspondingly, a call request refers to a request sent by an external system or user to the model service gateway, requesting the target business model to perform a specific processing task. A dormant state means the target business model is currently not activated, and its occupied computing resources (such as GPU memory, CPU cores, etc.) have been released or are in a minimized state to save system resources. A model resource object refers to a logical entity in the model management system used to abstract and manage the target business model and its related resources. This object contains the model's expected state, actual state, and other metadata information. Expected state refers to the target state that the model resource object should achieve under system design or user intent. For example, when there is a call request, the model's expected state may be set to the awakened state. The awakened state means that the target business model has been activated, its required computing resources have been allocated, model data has been loaded, and it is ready to receive and process call requests. Target resources refer to the specific computing resources allocated to wake up the target business model, such as one or more GPU devices, CPU cores, memory space, etc. A model instance refers to a specific process or containerized deployment unit of the target business model during runtime, which carries the model's actual computing logic and data. A wake-up operation refers to the instruction sent by the model controller to the model instance, which triggers the model instance to load model data, initialize the runtime environment, etc., and transitions it from a dormant state to a woken state. Actual state refers to the current real-world state of the model resource object within the system, reflecting the actual running status and resource usage of the model instance. Model processing task refers to the specific computational process by which the target business model, after receiving a call request, processes the input data according to its internal logic and generates output results.

[0043] Based on this, the model management system provided in this embodiment aims to optimize the performance of large models in sparse traffic scenarios. This system achieves fine-grained management of the model lifecycle by introducing a model service gateway and a model controller. First, the model service gateway is configured to respond to invocation requests for a target business model. When such a request is received, the model service gateway detects the current state of the target business model. For example, the model service gateway can obtain its state information by querying a centralized state store or by communicating with the model instance. If it detects that the target business model is in a dormant state, i.e., it is currently inactive and its resources are in a released or minimized state, the model service gateway will modify the expected state of the model resource object corresponding to the target business model to a woken-up state. This modification can be achieved by updating a field of the model resource object in the database or by sending a state update message to a state management service.

[0044] Secondly, the model controller is configured to continuously monitor the desired state of model resource objects. Once the model controller detects that the desired state of a model resource object has been modified to a wake-up state, it will initiate a series of operations to transition the target business model from a dormant state to a wake-up state. Specifically, the model controller will allocate target resources to the target business model. For example, the model controller can select an available compute node from a pre-defined resource pool and allocate the required GPU memory, CPU cores, or system memory to it. After resource allocation is complete, the model controller will send a wake-up operation to the model instance of the target business model. This wake-up operation can be a remote procedure call (RPC) instructing the model instance to load its pre-trained model data and perform necessary initializations.

[0045] Next, the model controller modifies the actual state of the model resource object to the awakened state based on the resource allocation result and the wake-up operation result. For example, if the resource is successfully allocated and the model instance successfully responds to the wake-up operation, the model controller updates the actual state field in the model resource object to awakened. This step ensures that the actual state of the model resource object is consistent with the actual running state of the model.

[0046] Finally, the model service gateway is configured to execute the model processing task corresponding to the call request using the target business model when it detects that the actual state of the model resource object is awake. For example, the model service gateway can continuously query the actual state of the model resource object or receive state update notifications from the model controller. Once it is confirmed that the model is fully awake and ready, the model service gateway will route the previously received call request to the awakened model instance, which will then complete the specific model inference or processing task.

[0047] See Figure 2 The schematic diagram shown illustrates the model management system provided in this embodiment. It primarily includes a Gateway (model service gateway), which serves as the overall system's traffic entry and exit point. It mainly provides upper-layer capabilities for model registration and deletion, as well as user-side model call interfaces. Lower-layer interactions are achieved through Model CRDs and Model Operators to complete model registration and deletion, and obtain corresponding model instance addresses. It also includes a Model Operator (model controller), which is a model controller and GPU resource allocator implemented based on the Kubernetes Operator. It primarily schedules and creates model instances based on the model's CRD. This model controller mainly comprises three sub-modules, as follows: (1) The Scheduler (resource manager) is responsible for the GPU resource scheduling of model instances. It will select nodes based on the deployment information of the model, including the TP and PP of the model. After the decision is made, the current model will be scheduled to one or more nodes.

[0048] (2) ModelExecutor (Instance Manager) is the main execution module, which has several functions. First, it creates models. After obtaining the scheduling result, it executes the actual creation of Pods and Services in K8S and tracks the status of the corresponding instances until they are in the Ready state, at which point it can provide services to the outside world. Second, it is responsible for deleting models. When a model has no traffic, the corresponding Pod and Service can be deleted. Third, it performs swap-in and swap-out operations on the model instances that have been created. This is achieved by calling the corresponding interfaces of the underlying vLLM instance. Swapping in means swapping the current model instance into the GPU, which will actually occupy the GPU card. Swapping out means releasing the model-related data from the GPU and freeing up the GPU for other model instances to use.

[0049] (3) Dynamic Evict is a dynamic eviction module, which is mainly responsible for eviction of models without traffic. When the node where the model instance is located does not have a free GPU card, the current module will try to evict the model without traffic. Eviction means swapping the model out of the GPU. After successful eviction, if there are free GPU resources, the subsequent resource scheduling logic can be executed.

[0050] In addition, the model management system includes vLLM Instances, which are Kubernetes Pods of vLLM running on various GPU nodes. Each vLLM Pod carries all or part of the model's weights. The current version of vLLM has been modified to support fast loading of models from or to the Model Cache Service, which is crucial for reducing TTFT. The Model Cache Service is a distributed model caching service that provides fast model registration and loading for vLLM instances. All model weights are cached in pinned memory, supporting both IPC and RDMA communication methods. During model registration, the Cache Node where the vLLM instance resides is prioritized for registration, further improving model loading speed.

[0051] For example, suppose a large language model is deployed as the target business model in a cloud-native environment to handle text generation requests from user A. To conserve resources, this target business model is set to a dormant state when there are no requests; its corresponding model instance is inactive and does not consume GPU resources. When user A sends a text generation request to this large language model through a client, the request is first received by the model service gateway. Upon receiving the request, the model service gateway queries the status of the model resource object corresponding to the target business model. For example, the model service gateway can query a shared configuration center or database and find that the actual status of the target business model is currently dormant. Therefore, the model service gateway determines that the model needs to be woken up to respond to the request, and thus modifies the expected state of the model resource object to an awakened state. This operation can be a simple status field update. The model controller continuously monitors the expected states of all model resource objects. Once the model controller detects that the expected state of the target business model has changed to an awakened state, it begins the wake-up process. Specifically, the model controller first allocates target resources for the target business model. For example, the model controller will find an available compute node in the cluster and allocate a GPU device, along with corresponding memory and CPU resources, to it. After successful resource allocation, the model controller will send a wake-up operation to the model instance of the target business model. This wake-up operation may contain instructions requiring the model instance to load its pre-trained model weights and necessary runtime libraries into the allocated GPU memory and complete process initialization.

[0052] Furthermore, after the model instance successfully loads the model data and completes initialization, it returns a wake-up success response to the model controller. Upon receiving feedback of successful resource allocation and wake-up operation, the model controller modifies the actual state of the target business model's model resource object to the awakened state. This state update can also be reflected in the shared configuration center or database. At this time, the model service gateway checks the actual state of the target business model's model resource object again. When the model service gateway detects that the actual state has changed to the awakened state, it confirms that the target business model is fully ready. Therefore, the model service gateway routes the text generation call request initially sent by user A to the awakened model instance. After receiving the request, the model instance uses the loaded model data and the initialized runtime environment to execute the text generation task and returns the generation result to user A. Through the above process, the model management system can dynamically wake up large language models in a dormant state when user A makes a request, thereby effectively responding to user requests while ensuring efficient resource utilization.

[0053] In other words, traditional solutions, such as cloud-native solutions based on dynamic scaling, require a complete set of time-consuming steps, including process initialization, communication setup, and loading model weights into GPU memory, when a model is cold-started from zero replicas, resulting in a high First Token Latency (TTFT). In contrast, this embodiment introduces a state-driven dynamic wake-up mechanism by having the model service gateway and model controller work together. When the model service gateway detects a dormant state and modifies the desired state to wake-up, the model controller can intervene in advance, allocating target resources and sending a wake-up operation. This mechanism allows the resource preparation and initialization process of the model instance to begin before the actual processing tasks, effectively reducing the perceived cold-start latency.

[0054] Furthermore, in existing technologies, traditional scaling-down strategies based on fixed time intervals are often not intelligent enough, causing idle models to continue occupying valuable GPU resources before the scaling-down interval ends, resulting in low overall GPU utilization. This embodiment achieves fine-grained control over the model lifecycle by managing the expected and actual states of model resource objects. The model is only awakened when it receives a call request and is in a dormant state, and can be switched back to a dormant state after completing the task or after a long period of no traffic. This on-demand wake-up strategy avoids the model continuously occupying resources when idle, thereby significantly improving the overall utilization efficiency of computing resources.

[0055] In summary, this embodiment constructs an intelligent and efficient large-scale model management solution through the collaborative operation between the model service gateway and the model controller, as well as the effective management of the expected and actual states of model resource objects. This solution not only effectively reduces cold start latency in sparse traffic scenarios but also significantly improves the utilization rate of computing resources, providing important technical support for the commercialization of large-scale models.

[0056] Furthermore, existing model management systems lack efficient resource allocation and release mechanisms during model registration, leading to low resource utilization and potential startup delays. To address this, this implementation proposes that the model controller includes a resource manager and an instance manager. The model service gateway is also used to create a model resource object corresponding to the target business model in response to a model registration request submitted for the target business model. The instance manager is used to send a resource allocation request to the resource manager when the model resource object is detected; The resource manager is used to allocate resources to be released for the target business model in response to the resource allocation request; The instance manager is used to create model instances for the target business model after resource allocation, and to perform resource release operations for the resources to be released; when the resource release is completed, the target business model is switched to a dormant state.

[0057] Specifically, the model controller is the core component responsible for managing the model lifecycle and resource scheduling. It can be a standalone microservice or an Operator integrated into an existing platform (such as Kubernetes). The model controller can be implemented using various technologies, such as the Kubernetes Operator pattern, managing model resources through custom resource definitions (CRDs) and controllers; or using an asynchronous processing mechanism based on message queues to receive and process instructions from the model service gateway. Correspondingly, the resource manager is a component of the model controller, focusing on the scheduling and allocation of computing resources. The resource manager can select appropriate compute nodes and resource types (such as GPUs, CPUs, and memory) for allocation based on preset scheduling strategies (e.g., based on available resources, model priority, load balancing, etc.). Its implementation can include: maintaining a real-time view of a cluster resource pool, acquiring and allocating resources by interacting with underlying infrastructure (such as the Kubernetes scheduler, cloud platform APIs); or using AI-based intelligent scheduling algorithms to predict resource needs and optimize allocation.

[0058] Correspondingly, the instance manager is a component of the model controller, responsible for the creation, destruction, and state management of model instances. The instance manager can create or destroy model instances (e.g., Pods) by calling the API of a container orchestration platform (such as Kubernetes) based on resource information allocated by the resource manager. Its implementation can include: directly interacting with the container engine using the Container Runtime Interface (CRI); or managing the network traffic and lifecycle of model instances through a service mesh. Correspondingly, the model service gateway is the system's entry point, responsible for receiving external requests and interacting with internal services. The model service gateway can be implemented as an API Gateway, reverse proxy, or load balancer. Its implementation can include: custom development based on open-source components such as Nginx and Envoy; or utilizing API Gateway services provided by cloud service providers. Correspondingly, the model registration request is initiated by a user or system to introduce a new business model into the model management system. This request can contain model metadata (such as model name, version, required resource configuration, deployment parameters, etc.). Model registration requests can be submitted via RESTful API, gRPC interface, or message queues. A model resource object is an abstract representation of a model within the system, containing the model's configuration information, current state, and expected state. Model resource objects can be stored in Kubernetes as custom resources, or in a configuration center or database.

[0059] Correspondingly, a resource allocation request is a request issued by the instance manager to the resource manager to obtain the computing resources required to run a model instance. This request may include information such as the model ID, the type and quantity of required resources. Resource allocation requests can communicate via internal API calls, message queues, or shared memory. Resources to be released refer to resources reserved or allocated by the resource manager for the target business model, but not yet actually occupied by the model instance. This resource allocation strategy can improve resource utilization and avoid resource idleness. Resources to be released can be GPU memory, CPU cores, or memory blocks reserved on the compute node. Correspondingly, a model instance is a specific copy of the target business model at runtime, responsible for performing model inference tasks. Model instances are typically deployed on compute nodes in the form of containers (such as Docker containers). Correspondingly, a resource release operation refers to returning computing resources no longer needed by the model instance to the resource pool. Resource release operations can include: deleting the container or Pod corresponding to the model instance; or notifying the underlying infrastructure to reclaim GPU memory, CPU, and memory. A dormant state refers to a state where a model instance does not occupy computing resources, but its metadata and configuration information are still retained. In hibernation mode, the model can be quickly woken up, avoiding the full cold start process. Switching a model to hibernation mode can be achieved by unloading the model instance from GPU memory or stopping its running container without deleting its configuration.

[0060] Based on this, by subdividing the model controller into a resource manager and an instance manager, optimizations were achieved in model registration and resource management. When the model service gateway receives a model registration request for a target business model, it creates a corresponding model resource object. Upon detecting this model resource object, the instance manager sends a resource allocation request to the resource manager. The resource manager responds to this request, allocating resources to be released for the target business model. Subsequently, the instance manager uses these allocated resources to create a model instance and simultaneously performs resource release operations, unloading the model data from GPU memory. After resource release is complete, the target business model is switched to a dormant state. This mechanism ensures that the model is in a "ready but dormant" state after registration, meaning resources are reserved but not actually occupied, thus avoiding the immediate consumption of large amounts of resources after model registration, while allowing for rapid wake-up when needed. This division of labor and pre-allocation strategy makes the model registration process more efficient, improves resource utilization, and lays the foundation for subsequent rapid wake-up.

[0061] For example, a user registers a large language model with the Model Service Gateway by calling its registration interface, specifying parameters such as model name and TP / PP. Upon receiving the registration request, the Model Service Gateway writes a custom model resource (Model CR) named after the model to Kubernetes via a Kubernetes client. This resource contains model deployment information and sets the current state to initialization. The Scheduler in the Model Controller monitors the newly added Model CR and allocates GPU resources to the model based on its deployment information (such as TP / PP), scheduling it onto appropriate compute nodes. Subsequently, the ModelExecutor in the Model Controller creates the corresponding model instance (e.g., Pod and Service) in Kubernetes based on the scheduling results. Once the vLLM instance successfully loads model data from NFS, the Instance Manager performs a swap-out operation, releasing the model data from GPU memory and registering it with the model cache service. At this point, the model is in a dormant state. When the user queries the model status through the Model Service Gateway, it will display "Ready".

[0062] In summary, by implementing resource pre-allocation and model dormancy during the model registration phase, the model management system effectively solves the problem of low resource utilization after model registration in traditional solutions. This mechanism allows models to quickly enter a wakeable state after registration, avoiding the immediate consumption of large amounts of GPU resources and thus improving overall resource utilization. Furthermore, since resource reservation and data loading are completed while the model is dormant, it can be quickly woken up when subsequent calls are made, significantly reducing the first token latency (TTFT) and improving the user experience.

[0063] Furthermore, during state detection, the gateway does not dynamically adjust the processing flow based on the real-time state of the target business model. This may lead to unnecessary state modification steps or delayed task execution when processing call requests, thereby introducing additional processing latency and resource overhead. To address this, this embodiment proposes that the model service gateway, in response to a call request for the target business model, further detect the actual state of the model resource object of the target business model; if the actual state is dormant, execute a step to modify the desired state of the model resource object of the target business model to a wake-up state; if the actual state is wake-up, execute a step to utilize the target business model to perform the model processing task corresponding to the call request.

[0064] Specifically, "responding to a call request for the target business model" refers to the model service gateway receiving a request initiated by an external system or user, intended for processing using a specific business model. This request can be an API call sent via HTTP / HTTPS, a message event received via a message queue (such as Kafka or RabbitMQ), or a service request received via a Remote Procedure Call (RPC) framework (such as gRPC). "Detecting the actual state of the model resource object of the target business model" refers to the model service gateway obtaining the current actual running state of the target business model in the system. This can be achieved by querying a centralized state registry, reading the `status` field of the corresponding model custom resource (CRD) in Kubernetes (K8s), or accessing cached state information maintained internally by the model service gateway. "Executing the step of modifying the desired state of the model resource object of the target business model to a wake-up state" means that when a model is detected to be in a dormant state, the model service gateway will issue an instruction to the system indicating that it hopes the model will be activated. This can be achieved by updating the `spec.desiredState` field of the corresponding model CRD in Kubernetes to "wake up," or by calling the API provided by the model controller to trigger the model wake-up process. The phrase "execute the step of using the target business model to execute the model processing task corresponding to the call request" refers to the model service gateway forwarding the received call request to the activated and ready target business model instance when the model is in a wake-up state, to execute actual model inference, data processing, or other business logic. This can be achieved by routing the request to the access address of the model instance through an internal load balancer, or by directly calling the inference service interface provided by the model instance.

[0065] Based on this, by introducing state detection and conditional execution mechanisms, the model service gateway can intelligently process call requests according to the current actual state of the target business model. When the model service gateway receives a call request for the target business model, it first checks the actual state of the model resource object of the target business model. This detection step ensures that the model service gateway can make decisions based on real-time state information, avoiding blindly executing state modification operations. If the actual state is detected as dormant, the model service gateway will execute the step of modifying the expected state of the model resource object of the target business model to the awakened state, thereby triggering the model controller (such as the model controller mentioned above) to perform resource allocation and wake-up operations of the model instance. This process works in conjunction with the model controller in the model management system mentioned above to ensure that the model is correctly woken up from the dormant state. However, if the actual state is detected as already awakened, the model service gateway will skip the state modification step and directly execute the model processing task corresponding to the call request using the target business model. This design allows the model service gateway to dynamically switch processing paths, avoiding unnecessary expected state modification operations when the model is already awakened, thereby reducing redundant system interactions and potential latency. In this way, the solution in this embodiment further optimizes the request processing flow and improves the system's response efficiency to changes in model state, based on the aforementioned model management system.

[0066] For example, suppose a user sends a Chat message specifying the model name as "Model A" through the Chat interface provided by the Model Service Gateway. Upon receiving this request, the Model Service Gateway immediately queries the `status.actualState` field of the custom resource (CRD) corresponding to "Model A" in Kubernetes to obtain its current actual state. If the query result shows `status.actualState` as "awakened," the Model Service Gateway will directly route the Chat request to the access address of the vLLM instance corresponding to "Model A" to generate the result. If the query result shows `status.actualState` as "dormant," the Model Service Gateway will modify the `spec.desiredState` field of the CRD corresponding to "Model A" in Kubernetes to "awakened" to trigger subsequent model wake-up processing operations.

[0067] In summary, the aforementioned model service gateway can dynamically adjust the request processing flow based on the real-time actual state of the target business model. This avoids performing unnecessary expected state modification steps when the model is already in an active state, thus significantly reducing request processing latency, especially for frequently invoked hot models, providing faster response times. Simultaneously, it reduces unnecessary internal system state update operations, lowers the load on the model controller and related components, and improves the resource utilization efficiency and stability of the entire model management system.

[0068] Furthermore, resource allocation and model loading may be inefficient, leading to increased wake-up latency. To address this, this embodiment proposes a resource manager to detect whether available resources exist on the computing node corresponding to the target business model; if so, it identifies the target resource from the available resources and allocates the target resource to the target business model. The instance manager is used to send a model wake-up request to the model instance of the target business model, wherein the model wake-up request carries resource information corresponding to the target resource; The model instance is used to respond to the model wake-up request, load the model data corresponding to the target business model into the target resource according to the resource information, and modify the actual state of the model resource object to the wake-up state according to the data loading result.

[0069] Specifically, the resource manager is a module or service responsible for managing and allocating computing resources. Its function is to find and allocate suitable resources on available computing nodes according to the model's needs. For example, the resource manager can be an independent microservice that interacts with the model controller via an API interface; alternatively, it can be a component within the model controller that directly manages the resource pool of computing nodes. The resource manager detects whether there are available resources on the computing nodes corresponding to the target business model. This process can be implemented in various ways. For example, the resource manager can periodically query the resource monitoring agent on the computing nodes to obtain real-time usage and idle status of resources such as CPU, GPU, and memory; or, the computing nodes can proactively report their resource status to the resource manager. If available resources exist, the resource manager will identify the target resource from these available resources and allocate it to the target business model. Identifying the target resource can be based on a preset scheduling strategy, such as selecting resources on the computing node with the lowest load, or selecting resources that best match the target business model type. Allocating the target resource can be done by updating the resource status database, marking that the resource is occupied, and generating a resource allocation certificate.

[0070] Correspondingly, the instance manager is a module or service responsible for managing the lifecycle of model instances. Its function is to perform operations such as creation, wake-up, and hibernation on model instances based on the desired state of the model resource object. For example, the instance manager can be a Kubernetes Operator that manages the Pods of model instances by listening for changes in custom resources (CRs); or it can be a standalone process that operates model instances by calling the underlying container runtime interface. The instance manager sends a model wake-up request to the model instance of the target business model. This request is an instruction that triggers the model instance to load model data and enter a working state. The sending method can be through Remote Procedure Call (RPC), message queues (such as Kafka, RabbitMQ), or directly calling the RESTful API provided by the model instance. The model wake-up request carries resource information corresponding to the target resource, which is necessary for the model instance to load model data, such as the ID of the allocated GPU device, the size of the video memory, and the network configuration. This information ensures that the model instance can accurately identify and use the allocated resources.

[0071] Correspondingly, the model instance is the carrier in which the target business model actually runs on the computing node. For example, the model instance can be a vLLM service process running in a Docker container, or a standalone Python process. The model instance responding to a model wake-up request means that it has received and parsed the wake-up instruction from the instance manager. Loading the model data corresponding to the target business model into the target resource according to the resource information is crucial for the model to transition from a dormant state to a working state. For example, the model instance will load the model weight file from storage media (such as disk) into the specified GPU memory based on the GPU ID carried in the request, and initialize the necessary KV Cache structure. Modifying the actual state of the model resource object to a woken state based on the data loading result ensures that the state of the model resource object is consistent with the actual running state of the model instance. For example, after successfully loading the model data, the model instance will send a status update message to the model controller, informing it that it has been successfully woken up.

[0072] Based on this, by optimizing the collaborative process of resource allocation and model loading, the latency of model wake-up is effectively reduced. Specifically, before allocating resources, the resource manager first checks whether there are available resources on the computing node corresponding to the target business model. This ensures the effectiveness of resource allocation, avoids invalid operations when there are no resources, and thus saves unnecessary waiting time. If available resources exist, the target resource is identified and allocated. This process optimizes resource selection efficiency and reduces the time for resource search and allocation by intelligently selecting the most suitable resource. When the instance manager sends a model wake-up request to the model instance of the target business model, it synchronously carries the resource information corresponding to the target resource. This allows the model instance to directly obtain the detailed configuration required for loading without additional query or negotiation steps, thus simplifying the model loading process. After receiving the wake-up request, the model instance can immediately load model data in the specified target resource according to the provided resource information. This direct loading method reduces configuration and initialization time. Finally, the actual state of the model resource object is modified to the wake-up state based on the data loading result, ensuring accurate and timely updates of the model state and reflecting the true situation that the model is ready for use. Through the aforementioned closely collaborative process, the solution in this embodiment significantly improves the efficiency of the model from sleep to wake-up, effectively solving the wake-up delay problem caused by low resource allocation and model loading efficiency in traditional solutions.

[0073] For example, when the model service gateway detects that the model resource object of a target business model is in a dormant state and there is a call request to wake up the model, the resource manager in the model controller will first check whether the compute node where the target business model is located has sufficient GPU resources. For example, the resource manager can query a centralized resource scheduler that maintains the GPU usage of all compute nodes. If available GPU resources exist, the resource manager will determine a target GPU according to a preset scheduling strategy (e.g., selecting the GPU with the highest idle rate) and allocate it to the target business model. Subsequently, the instance manager will send a model wake-up request to the model instance of the target business model (e.g., a container running a vLLM service). This request will contain the ID of the target GPU and other necessary resource configuration information. Upon receiving the request, the model instance will immediately utilize the allocated GPU resources to load the weight data of the target business model from storage into the GPU memory and initialize the KV cache required by vLLM. Once the model data loading is complete, the model instance will report to the model controller that it has been successfully woken up, and the model controller will then update the actual state of the model resource object to the woken state.

[0074] In summary, the above processing in this embodiment enables efficient resource allocation and rapid model loading for the target business model, significantly reducing the latency of the model transitioning from a dormant to a waking state. This allows the system to respond to user requests more quickly, improving the user experience, while also optimizing the utilization efficiency of computing resources and avoiding prolonged resource idleness.

[0075] Based on this, when computing nodes lack available resources, resource allocation is impossible, leading to target business model wake-up failure or delay, impacting system performance and resource utilization. To address this, this embodiment proposes that the model controller also include a dynamic eviction module. The resource manager is also configured to send a resource eviction request to the dynamic eviction module if there are no available resources in the computing node corresponding to the target business model. The dynamic eviction module is configured to, in response to the resource eviction request, detect the traffic field in the candidate model resource object corresponding to at least one candidate model; determine, based on the detection result, that the traffic field is a target candidate model without traffic identification among the at least one candidate model; execute a resource eviction task for the target candidate model; and, if the available resources in the computing node meet the allocation conditions based on the task execution result, execute the step of allocating target resources for the target business model.

[0076] Specifically, the dynamic eviction module is a functional unit of the model controller. Its main responsibility is to intelligently identify and release occupied but underutilized (e.g., no traffic) resources on the current computing node when system resources are scarce, making them available for other high-priority or resource-urgent models. This module can be a standalone software service or process that communicates with the resource manager through an internal API or message queue; alternatively, it can be a submodule or function set within the model controller that directly calls the resource manager's interface to perform resource operations. It provides the system with a proactive resource reclamation mechanism, avoiding business interruptions or delays caused by insufficient resources. When the resource manager detects that no available resources exist on the computing node corresponding to the target business model, it sends a resource eviction request to the dynamic eviction module. This request serves as a signal to trigger the dynamic eviction process, indicating that the current system needs to reclaim existing resources to meet new resource allocation requirements. The resource manager can send a request containing information about the target compute node and the amount of resources required by calling the RPC interface (such as gRPC or RESTful API) provided by the dynamic eviction module; alternatively, the resource manager can encapsulate the resource eviction request into a message and publish it to a message queue (such as Kafka or RabbitMQ), and the dynamic eviction module can subscribe to the message queue and process it.

[0077] Correspondingly, the dynamic eviction module responds to resource eviction requests by detecting the traffic field in the candidate model resource objects corresponding to at least one candidate model. This is the core step in identifying models that can be evicted. By detecting the traffic field, the system can determine which models are currently idle or under low load, thus avoiding the eviction of active models that are providing services. The dynamic eviction module can query a centralized model status database or configuration service, which stores the real-time status of all model resource objects, including the traffic field; or, the dynamic eviction module can directly query the current traffic statistics of the model service gateway or model instance through the API interface and use it as the traffic field for evaluation. Based on the detection results, the module identifies target candidate models with no traffic flags among at least one candidate model. This step, based on the detection results of the traffic field, precisely filters out candidate models that are not currently processing any requests or have extremely low traffic. These models are given priority for eviction. The dynamic eviction module can set a traffic threshold, and models below this threshold are marked as having no traffic flags; or, the system can maintain a model activity list, dynamically updating the traffic field based on the number of requests or connections of the model over a period of time, and determining the no-traffic flag accordingly. Performing a resource eviction task on the target candidate model is the actual operation of releasing resources. By executing the eviction task, the computing resources (such as GPU memory, CPU, memory, etc.) occupied by the target candidate model will be reclaimed, thereby freeing up space for the target business model. The eviction task may include sending a hibernation command to the model instance of the target candidate model, causing it to release GPU memory and computing resources, and updating the state of its model resource object; alternatively, the eviction task may also involve directly terminating the container or process corresponding to the target candidate model, and having the resource scheduling system reclaim the resources it occupies.

[0078] Accordingly, if the available resources in the compute node meet the allocation conditions based on the task execution results, the step of allocating target resources to the target business model is executed. This step ensures the effectiveness of resource eviction. Only when the eviction task successfully releases enough resources to make the compute node meet the allocation requirements of the target business model will subsequent resource allocation be performed, avoiding invalid allocation attempts. After completing the eviction task, the dynamic eviction module will re-query the available resources of the compute node and compare them with the resources required by the target business model; alternatively, the execution result of the eviction task will be directly returned to the resource manager, which will determine whether the allocation conditions are met based on the result and continue to execute the allocation operation.

[0079] Therefore, existing model management systems often fail to effectively allocate resources when computing node resources are insufficient, leading to model wake-up failures or delays. This embodiment introduces a dynamic eviction module, which works in conjunction with the resource manager, to provide an intelligent resource scheduling mechanism. When the model service gateway responds to a call request, detects that the target business model is in a dormant state, and instructs the model controller to wake it up, the resource manager in the model controller attempts to allocate target resources to the target business model. If the resource manager detects that there are no available resources in the computing node corresponding to the target business model, it no longer simply reports insufficient resources, but actively sends a resource eviction request to the dynamic eviction module. Upon receiving this request, the dynamic eviction module immediately initiates a resource reclamation process. It detects the traffic field in the model resource object corresponding to at least one candidate model on the current computing node. By analyzing these traffic fields, the dynamic eviction module can intelligently identify target candidate models with "no traffic identifiers" that currently have no traffic or extremely low traffic. These models are usually idle or have extremely low utilization and are prioritized for eviction. Once the target candidate models are determined, the dynamic eviction module performs resource eviction tasks for these models. This task aims to safely and efficiently release the computing resources currently occupied by the target candidate model, such as GPU memory, CPU, or RAM. The execution result of the resource eviction task is crucial; the dynamic eviction module determines whether the available resources in the computing node meet the allocation conditions for the target business model based on the result. Only when the resource quantity meets the allocation conditions will the system continue to execute the step of allocating target resources to the target business model. Through the above mechanism, the solution in this embodiment enables the model management system to proactively and intelligently optimize resources when facing resource shortages, rather than passively waiting. The close collaboration between the resource manager and the dynamic eviction module forms a closed-loop resource scheduling and reclamation system. This not only solves the problem of model wake-up failure or delay when computing nodes have no available resources, but also minimizes the impact on active services by prioritizing the eviction of models with no traffic, significantly improving the overall resource utilization and response speed of the system. This combination of dynamic eviction strategy and model wake-up process enables the system to achieve more efficient resource management and lower TTFT (Time To First Token) latency in scenarios with large models and sparse traffic, thereby optimizing the user experience.

[0080] For example, when the model service gateway receives a call request for model A and finds that model A is in a dormant state, it notifies the model controller to wake it up. The resource manager in the model controller attempts to allocate the required GPU resources for model A on compute node X. Suppose the resource manager detects that there are currently insufficient available GPU resources on compute node X. At this point, the resource manager sends a resource eviction request to the dynamic eviction module in the model controller. This request includes the identifier of compute node X and the amount of GPU resources required by model A. Upon receiving the request, the dynamic eviction module queries the model resource objects corresponding to all currently running candidate models (e.g., model B, model C, model D) on compute node X. It checks the traffic fields in these model resource objects, which may record the model's request QPS (queries per second) or connection count over a past period. Suppose the traffic fields of models B and C show a QPS of 0, while model D has a higher QPS. The dynamic eviction module then determines models B and C as target candidate models with no traffic identifier in their traffic fields. Subsequently, the dynamic eviction module executes resource eviction tasks for models B and C. This may involve invoking the hibernation interface of model instances B and C, instructing them to release their occupied GPU memory and computing resources, and updating their state to hibernation. After the resources of models B and C are successfully released, the dynamic eviction module reassesses the amount of available GPU resources on compute node X. If the amount of available GPU resources on compute node X now meets the allocation conditions for model A, the dynamic eviction module notifies the resource manager, which then proceeds to allocate the target GPU resources for model A.

[0081] In summary, this embodiment effectively solves the problem of target service model wake-up failure or delay caused by the inability to allocate resources when computing nodes lack available resources. Specifically, when the resource manager detects insufficient resources, it no longer passively waits but actively triggers the dynamic eviction module to reclaim resources. The dynamic eviction module can intelligently identify and prioritize the eviction of idle models with no traffic, thereby avoiding impact on active services. This mechanism ensures that even under resource constraints, the system can still allocate the necessary resources to the target service model in a timely manner, significantly reducing the cold start latency (TTFT) of model wake-up and improving the overall resource utilization and response speed of the system. At the same time, by accurately identifying and evictioning models with no traffic, it avoids service interruptions that may be caused by blind eviction, ensuring system stability and user experience.

[0082] Furthermore, existing model management systems often face issues of inefficient or untimely resource release when dealing with sparse traffic scenarios and insufficient existing resources are needed to allocate resources for new business models. This may result in resources not being explicitly returned to the resource manager after eviction or inadequate assurance of resource availability, thus affecting the resource manager's ability to quickly reallocate resources and ultimately reducing system response speed and resource utilization. To address this, this embodiment proposes a dynamic eviction module, which also calls the hibernation interface of the candidate model instance of the target candidate model. Based on the hibernation interface, the target candidate model is switched to a hibernation state, and the allocated resources of the target candidate model are returned to the resource manager. The resource manager is further configured to, based on the resource return results, determine that the amount of available resources in the computing node meets the allocation conditions, and then execute the step of allocating target resources for the target business model.

[0083] Specifically, when the dynamic eviction module needs to release resources, it no longer simply terminates the model instance. Instead, it implements controlled hibernation of the target candidate model by calling the hibernation interface of the candidate model instance. This hibernation interface can be a predefined API endpoint, allowing external systems to send instructions to the model instance to put it into a low-power or inactive state. For example, this interface could be an HTTP / RPC interface. Upon receiving a hibernation instruction, the model instance will perform a series of resource release operations, such as unloading model weights and KV cache from GPU memory, and releasing CPU cores and memory resources. As another implementation, this hibernation interface can also be triggered by updating the Kubernetes CustomResource (CR) status field corresponding to the model instance. The controller inside the model instance will listen for this status change and execute the corresponding hibernation logic.

[0084] Correspondingly, switching the target candidate model to a dormant state based on the hibernation interface means that after receiving the hibernation command, the model instance will actively release the computing resources it occupies, but its process or container may still be alive, just in an inactive state. This state switching can ensure that resources are released in an orderly and complete manner, avoiding resource leaks or state inconsistencies that may be caused by forced termination. For example, the model instance can persist its current state for quick wake-up later.

[0085] Correspondingly, the dynamic eviction module returns the allocated resources of the target candidate model to the resource manager. This step is crucial for resource reclamation, ensuring that the resource manager can promptly identify available resources. This can be achieved by the dynamic eviction module sending a resource release notification message to the resource manager, containing detailed information about the released resources, such as GPU ID, memory size, and number of CPU cores. Alternatively, the dynamic eviction module can directly update the resource list maintained by the resource manager, marking released resources as available.

[0086] Accordingly, after receiving the resource return results, the resource manager updates its internal resource availability view based on this information and determines whether the amount of available resources in the compute nodes meets the allocation criteria. These allocation criteria may include the minimum GPU memory, CPU core count, and memory size required by the target business model. Only when it is confirmed that the available resources meet the allocation requirements of the target business model will the resource manager execute the step of allocating the target resources for the target business model. This ensures the accuracy and effectiveness of resource allocation and avoids invalid allocations when resources are insufficient.

[0087] Based on this, when the model controller needs to allocate resources to a target business model but the computing node resources are insufficient, the dynamic eviction module will select a target candidate model for eviction according to a preset strategy (e.g., identifying target candidate models without traffic identifiers based on traffic fields). Unlike directly terminating a model instance, the dynamic eviction module will call the hibernation interface of the candidate model instance of the target candidate model, prompting the model instance to actively and gracefully release its occupied resources and enter a hibernation state. Once the resources are released, the dynamic eviction module will explicitly return this released resource information to the resource manager. After receiving these resource return results, the resource manager will immediately update its resource pool status and determine whether the available resources in the current computing node meet the allocation conditions based on the resource requirements of the target business model. If so, the resource manager will quickly allocate the target resources to the target business model. This series of coordinated operations makes the flow of resources from idle models to new models more efficient and controllable.

[0088] For example, when the model service gateway responds to a call request for a target business model and detects that the target business model is in a dormant state, it triggers a resource allocation process. If the resource manager on the compute node finds that there are not enough available resources to wake up or deploy the target business model, it sends a resource eviction request to the dynamic eviction module. The dynamic eviction module identifies a candidate model that is idle and has no traffic on the current compute node based on the traffic field, such as an LLM instance that has not been called for a long time. The dynamic eviction module then sends a " / hibernate" or " / sleep" command by calling the RESTful API interface of the LLM instance. After receiving the command, the LLM instance executes its internal resource cleanup logic, such as unloading its model weights and KV cache in GPU memory to host memory, and releasing the GPU computing units and some CPU cores it occupies. After completing the resource release, the LLM instance sends a confirmation message to the dynamic eviction module, which then encapsulates the released GPU memory block ID and CPU core ID, among other resource information, into a message and sends it to the resource manager. Once the resource manager receives this information, it adds it to the available resource pool and reassesses the resource allocation request of the target business model. If the total amount of available resources meets the needs of the target business model, the resource manager selects a suitable target resource from the available resource pool and allocates it to the target business model, thereby allowing the target business model to be woken up or loaded.

[0089] In summary, this embodiment effectively solves the problem of inefficient resource allocation caused by untimely and ambiguous resource release in traditional solutions. By introducing a hibernation interface for model instances and a clear resource return mechanism, it ensures that resources can be reclaimed in an orderly and efficient manner, with timely feedback to the resource manager. This allows the resource manager to make allocation decisions quickly based on real-time resource availability, significantly shortening resource allocation waiting time and improving the system's response speed to the deployment or revival of new business models. Simultaneously, this controlled resource reclamation method avoids the additional overhead and instability that may result from forcibly terminating model instances, thereby improving overall resource utilization and system stability.

[0090] Furthermore, existing model management systems, when processing model invocation requests, include waking up the model and loading model data. However, during this process, there is a lack of efficient strategies for copying business model data, resulting in high data transmission latency when the model cache is distributed across different computing nodes, thus impacting overall performance. To address this, this embodiment proposes copying the business model data of the target business model according to an inter-process communication strategy when the model cache corresponding to the model instance is distributed across the same computing node. If the model cache corresponding to the model instance is not distributed on the same computing node, the business model data of the target business model is copied according to the remote memory access strategy.

[0091] Specifically, inter-process communication (IPC) strategies refer to mechanisms for data exchange and sharing between different processes within the same computing node. These strategies typically utilize operating system-provided mechanisms such as shared memory, message queues, semaphores, and pipes to achieve direct data transfer at the memory level, avoiding network transmission overhead and protocol processing latency. For example, a shared memory mechanism can be used to directly map the business model data of the target business model into the address spaces of multiple processes, thereby achieving zero-copy data sharing; alternatively, specific hardware acceleration interfaces, such as CUDA-IPC, can be used to allow different CUDA contexts (usually corresponding to different processes) on the same physical GPU to directly access each other's video memory, achieving efficient GPU memory data copying. Remote memory access (RDMA) strategies refer to the ability of one computing node to directly access the memory of another computing node without the intervention of the target node's CPU. This strategy typically relies on network hardware supporting RDMA (such as InfiniBand network cards or RoCE-enabled Ethernet cards), significantly reducing data transmission latency and CPU overhead and increasing data transmission bandwidth by bypassing the operating system kernel and CPU. For example, GPU-Direct RDMA technology can be used to allow the GPU to send data directly to or receive data from the memory of a remote GPU without going through the buffers of the host CPU and system memory, thereby enabling efficient cross-node data transfer in a distributed environment.

[0092] Based on this, the solution described in this embodiment aims to optimize the efficiency of the model management system when waking up models and loading business model data, especially considering the distribution of model caches. When the model service gateway responds to the call request and the model controller allocates resources and sends a wake-up operation to the model instance, the model instance needs to load the business model data of the target business model. At this time, the system dynamically selects the most suitable data copying strategy based on the actual distribution of the model cache corresponding to the model instance. Specifically, if the model cache corresponding to the model instance and the target business model data are located on the same computing node, the system will use an inter-process communication strategy for data copying. This selection is based on the low latency and high bandwidth characteristics of data transmission within the same node, ensuring that data can be quickly and efficiently transmitted from the cache to the model instance by avoiding network transmission overhead. Conversely, if the model cache corresponding to the model instance and the target business model data are distributed on different computing nodes, the system will switch to a remote memory access strategy. This strategy can effectively address the challenges of cross-node data transmission by bypassing the CPU overhead of traditional network protocol stacks and using dedicated hardware to achieve high-speed, low-latency data transmission. Through this intelligent strategy selection, the solution in this embodiment ensures that business model data can be copied in the most optimized way regardless of how the model cache is distributed, thereby significantly shortening the data loading time during the model wake-up process. This mechanism is closely integrated with the resource allocation, model instance creation, and wake-up process of the aforementioned model management system, together forming an efficient on-demand model loading and running environment. Especially in sparse traffic scenarios, it can effectively reduce the TTFT (Time To First Token) latency of model cold starts and improve the user experience.

[0093] For example, when a model controller needs to wake up a target business model and instruct its model instance to load business model data, the model instance first determines whether its required model cache (e.g., model weights and KV cache) is located in the GPU memory of the current compute node (i.e., the node where the model instance is running) or in the GPU memory of another compute node. For instance, if the model cache corresponding to the model instance is distributed across the same compute node, the system can use CUDA-IPC (CUDA Inter-Process Communication) for efficient data copying. In this case, the model's own data (e.g., weights and KV cache) can be pre-stored in page-locked memory. When the model instance needs to load it, through the CUDA-IPC mechanism, different CUDA contexts (which may correspond to different processes) can directly access and copy the GPU memory data on the same physical GPU, fully utilizing PCIe bandwidth to achieve extremely low-latency data transfer. On the other hand, if the model cache corresponding to the model instance is not distributed across the same compute node, i.e., the model cache is located on a remote compute node, the system can use GPU-DirectRDMA (GPU Direct Remote Memory Access) for data copying. In this scenario, by utilizing network hardware that supports RDMA, such as InfiniBand network cards, the GPU can directly transfer data from the video memory of a remote compute node to the video memory of the local compute node, or vice versa, without requiring multiple copies through the host CPU and system memory. This approach fully utilizes the bandwidth of the IB network, ensuring efficient and low-latency data transmission even in a distributed environment.

[0094] In summary, through the above processing, this embodiment can intelligently select the most suitable data copying strategy based on the actual distribution of the model cache, thereby significantly optimizing the transmission efficiency of target business model data during the wake-up process. When the model caches corresponding to model instances are distributed on the same computing node, the inter-process communication strategy utilizes local high-speed data transmission mechanisms to avoid unnecessary network overhead and significantly reduce data copying time. When the model caches corresponding to model instances are not distributed on the same computing node, the remote memory access strategy effectively addresses the challenges of cross-node data transmission, achieving high-speed, low-latency data transmission by bypassing the limitations of traditional network protocol stacks. This mechanism of dynamically matching cache location and copying strategy, combined with the rapid wake-up and resource allocation capabilities of the aforementioned model management system, ensures that model instances can quickly and efficiently load the required data, thereby significantly reducing the TTFT latency of model cold starts and improving the response speed of model processing tasks and user experience. Especially in scenarios with large models and sparse traffic, this efficient data copying mechanism is a key foundation for the efficient operation of runtime instance reuse and dynamic eviction functions, ensuring that the underlying vLLM instances can be quickly woken up, thus improving the resource utilization and overall performance of the entire model management system.

[0095] The model management system provided in this embodiment includes a model service gateway and a model controller. When the model service gateway responds to a call request for a target business model and determines that the target business model is in a dormant state, it can modify the expected state of the model resource object of the target business model to a awakened state. Then, when the model controller detects that the expected state of the model resource object is awakened, it allocates target resources for the target business model and sends a wake-up operation to the model instance of the target business model. Afterwards, based on the resource allocation result and the wake-up operation result, it modifies the actual state of the model resource object to an awakened state. Finally, when the model service gateway detects that the actual state of the model resource object is awakened, it uses the target business model to execute the model processing task corresponding to the call request. This achieves dynamic management of model state and resource allocation through the model service gateway and model controller, modifying the expected state, allocating resources, and waking up the model when a dormant state is detected, thereby reducing model cold start latency and improving GPU resource utilization. It has the advantages of reducing model cold start latency and improving GPU resource utilization.

[0096] Corresponding to the above system embodiments, this specification also provides embodiments of model management methods. Figure 3 A flowchart illustrating a model management method provided in one embodiment of this specification is shown. Figure 3 As shown, this method is applied to a model management system, which includes a model service gateway and a model controller, and includes: In step S302, in response to a call request for a target business model, if the model service gateway determines that the target business model is in a dormant state, it modifies the desired state of the model resource object of the target business model to a wake-up state.

[0097] In step S304, when the model controller detects that the expected state of the model resource object is the awakened state, it allocates target resources for the target business model and sends a wake-up operation to the model instance of the target business model. Based on the resource allocation result and the wake-up operation result, it modifies the actual state of the model resource object to the awakened state.

[0098] In step S306, when the model service gateway detects that the actual state of the model resource object is awake, it uses the target business model to execute the model processing task corresponding to the call request.

[0099] In one optional embodiment, the model controller includes a resource manager and an instance manager; The model service gateway is also used to create a model resource object corresponding to the target business model in response to a model registration request submitted for the target business model. The instance manager is used to send a resource allocation request to the resource manager when the model resource object is detected; The resource manager is used to allocate resources to be released for the target business model in response to the resource allocation request; The instance manager is used to create model instances for the target business model after resource allocation, and to perform resource release operations for the resources to be released; when the resource release is completed, the target business model is switched to a dormant state.

[0100] In an optional embodiment, the model service gateway is further configured to, in response to a call request for a target business model, detect the actual state of the model resource object of the target business model; if the actual state is a dormant state, perform a step of modifying the desired state of the model resource object of the target business model to a wake-up state; and if the actual state is a wake-up state, perform a step of using the target business model to execute the model processing task corresponding to the call request.

[0101] In an optional embodiment, the resource manager is configured to detect whether there are available resources on the computing node corresponding to the target business model; if so, to determine the target resource from the available resources and to allocate the target resource for the target business model; The instance manager is used to send a model wake-up request to the model instance of the target business model, wherein the model wake-up request carries resource information corresponding to the target resource; The model instance is used to respond to the model wake-up request, load the model data corresponding to the target business model into the target resource according to the resource information, and modify the actual state of the model resource object to the wake-up state according to the data loading result.

[0102] In an optional embodiment, the model controller further includes a dynamic expulsion module; The resource manager is also configured to send a resource eviction request to the dynamic eviction module if there are no available resources in the computing node corresponding to the target business model. The dynamic eviction module is configured to, in response to the resource eviction request, detect the traffic field in the candidate model resource object corresponding to at least one candidate model; determine, based on the detection result, that the traffic field is a target candidate model without traffic identification among the at least one candidate model; execute a resource eviction task for the target candidate model; and, if the available resources in the computing node meet the allocation conditions based on the task execution result, execute the step of allocating target resources for the target business model.

[0103] In an optional embodiment, the dynamic eviction module is further configured to call the hibernation interface of the candidate model instance of the target candidate model, switch the target candidate model to a hibernation state based on the hibernation interface, and return the allocated resources of the target candidate model to the resource manager; The resource manager is further configured to, based on the resource return results, determine that the amount of available resources in the computing node meets the allocation conditions, and then execute the step of allocating target resources for the target business model.

[0104] In one optional embodiment, when the model caches corresponding to the model instances are distributed on the same computing node, the business model data of the target business model is copied according to the inter-process communication strategy; If the model cache corresponding to the model instance is not distributed on the same computing node, the business model data of the target business model is copied according to the remote memory access strategy.

[0105] The model management system provided in this embodiment includes a model service gateway and a model controller. When the model service gateway responds to a call request for a target business model and determines that the target business model is in a dormant state, it can modify the expected state of the model resource object of the target business model to a awakened state. Then, when the model controller detects that the expected state of the model resource object is awakened, it allocates target resources for the target business model and sends a wake-up operation to the model instance of the target business model. Afterwards, based on the resource allocation result and the wake-up operation result, it modifies the actual state of the model resource object to an awakened state. Finally, when the model service gateway detects that the actual state of the model resource object is awakened, it uses the target business model to execute the model processing task corresponding to the call request. This achieves dynamic management of model state and resource allocation through the model service gateway and model controller, modifying the expected state, allocating resources, and waking up the model when a dormant state is detected, thereby reducing model cold start latency and improving GPU resource utilization. It has the advantages of reducing model cold start latency and improving GPU resource utilization.

[0106] The above is an illustrative scheme of a model management method according to this embodiment. It should be noted that the technical solution of this model management method and the technical solution of the model management system described above belong to the same concept. For details not described in detail in the technical solution of the model management method, please refer to the description of the technical solution of the model management system described above.

[0107] Figure 4 A structural block diagram of a computing device 400 according to one embodiment of this specification is shown. The components of the computing device 400 include, but are not limited to, a memory 410 and a processor 420. The processor 420 is connected to the memory 410 via a bus 430, and a database 450 is used to store data.

[0108] The computing device 400 also includes an access device 440, which enables the computing device 400 to communicate via one or more networks 460. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 440 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0109] In one embodiment of this specification, the aforementioned components of the computing device 400 and Figure 4 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 4 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0110] The computing device 400 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 400 can also be a mobile or stationary server.

[0111] The processor 420 is used to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described model management method.

[0112] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the model management method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the model management method described above.

[0113] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the model management method described above.

[0114] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the model management method described above belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the model management method described above.

[0115] An embodiment of this specification also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the model management method described above.

[0116] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the model management method described above belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the model management method described above.

[0117] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0118] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0119] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0120] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0121] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described in this specification. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification.

Claims

1. A model management system, characterized in that, Includes model service gateway and model controller, including: The model service gateway is used to respond to a call request for a target business model and, if it is determined that the target business model is in a dormant state, modify the desired state of the model resource object of the target business model to a wake-up state. The model controller is configured to allocate target resources to the target business model and send a wake-up operation to the model instance of the target business model when the expected state of the model resource object is detected to be a wake-up state; and modify the actual state of the model resource object to a wake-up state based on the resource allocation result and the wake-up operation result. The model service gateway is used to execute the model processing task corresponding to the call request using the target business model when it detects that the actual state of the model resource object is awake.

2. The model management system according to claim 1, characterized in that, The model controller includes a resource manager and an instance manager; The model service gateway is also used to create a model resource object corresponding to the target business model in response to a model registration request submitted for the target business model. The instance manager is used to send a resource allocation request to the resource manager when the model resource object is detected; The resource manager is used to allocate resources to be released for the target business model in response to the resource allocation request; The instance manager is used to create model instances for the target business model after resource allocation, and to perform resource release operations for the resources to be released; when the resource release is completed, the target business model is switched to a dormant state.

3. The model management system according to claim 1, characterized in that, The model service gateway is further configured to, in response to a call request for a target business model, detect the actual state of the model resource object of the target business model; if the actual state is a dormant state, execute the step of modifying the desired state of the model resource object of the target business model to a wake-up state; and if the actual state is a wake-up state, execute the step of using the target business model to perform the model processing task corresponding to the call request.

4. The model management system according to claim 2, characterized in that, The resource manager is used to detect whether there are available resources on the computing node corresponding to the target business model; If it exists, determine the target resource from the available resources and allocate the target resource for the target business model; The instance manager is used to send a model wake-up request to the model instance of the target business model, wherein the model wake-up request carries resource information corresponding to the target resource; The model instance is used to respond to the model wake-up request, load the model data corresponding to the target business model into the target resource according to the resource information, and modify the actual state of the model resource object to the wake-up state according to the data loading result.

5. The model management system according to claim 4, characterized in that, The model controller also includes a dynamic expulsion module; The resource manager is also configured to send a resource eviction request to the dynamic eviction module if there are no available resources in the computing node corresponding to the target business model. The dynamic eviction module is used to respond to the resource eviction request by detecting the traffic field in the candidate model resource object corresponding to at least one candidate model; and determining, based on the detection result, the target candidate model with no traffic identifier among the at least one candidate models. A resource eviction task is executed for the target candidate model. If the available resources in the computing node meet the allocation conditions based on the task execution results, the step of allocating target resources for the target business model is executed.

6. The model management system according to claim 5, characterized in that, The dynamic eviction module is also used to call the hibernation interface of the candidate model instance of the target candidate model, switch the target candidate model to hibernation state based on the hibernation interface, and return the allocated resources of the target candidate model to the resource manager; The resource manager is further configured to, based on the resource return results, determine that the amount of available resources in the computing node meets the allocation conditions, and then execute the step of allocating target resources for the target business model.

7. The model management system according to claim 1, characterized in that, When the model caches corresponding to the model instances are distributed on the same computing node, the business model data of the target business model is copied according to the inter-process communication strategy; If the model cache corresponding to the model instance is not distributed on the same computing node, the business model data of the target business model is copied according to the remote memory access strategy.

8. A model management method, characterized in that, Applied to a model management system, the model management system includes a model service gateway and a model controller, including: In response to a call request for a target business model, if the model service gateway determines that the target business model is in a dormant state, it modifies the desired state of the model resource object of the target business model to a wake-up state. When the model controller detects that the expected state of the model resource object is the awakened state, it allocates target resources for the target business model and sends a wake-up operation to the model instance of the target business model. Based on the resource allocation result and the wake-up operation result, it modifies the actual state of the model resource object to the awakened state. When the model service gateway detects that the actual state of the model resource object is awake, it uses the target business model to execute the model processing task corresponding to the call request.

9. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the method of claim 8.

10. A computer-readable storage medium, characterized in that, It stores computer-executable instructions that, when executed by a processor, implement the steps of the method of claim 8.

11. A computer program product, characterized in that, It includes a computer program or instructions that, when executed by a processor, implement the steps of the method of claim 8.

Images