A model inference service scheduling method, a multi-model management system, and a storage medium

Abstract

The specification provides a model inference service scheduling method, a multi-model management system and a storage medium. The model inference service scheduling method comprises: in response to a user request, a service routing layer determines a target model in a preset model cluster based on the user request, and obtains the running state of the target model; when the target model is in a dormant state, the service routing layer sends a wake-up request to the inference framework layer; the inference framework layer executes the operation of writing the weight data of each network layer of the target model from local storage to video memory based on the wake-up request, and after the weight data of the first network layer is written, it starts to execute the inference calculation operation of the target task according to the order of each network layer of the target model from front to back. The model inference service scheduling method can start inference after the model is partially loaded into the video memory, effectively shortening the TTFT, providing a multi-model management system that reacts faster, provides a better user experience, and has lower operating costs.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and in particular to a scheduling method for model inference services, a multi-model management system, and a storage medium. Background Technology

[0002] With the development of Artificial Intelligence (AI) technology, many enterprises or departments with high data security requirements typically deploy Large Language Models (LLMs) privately and use Model-as-a-Service (MaaS) to provide AI services to employees or customers. To provide more comprehensive and diverse services, multiple LLMs with different capabilities and parameter sets are usually deployed privately to provide corresponding AI services in different scenarios. However, due to resource constraints during private deployment, it is difficult to deploy multiple LLMs simultaneously. Therefore, how to schedule and manage multiple LLMs to balance resource utilization and user experience is a problem that needs to be solved.

[0003] Currently, private MaaS multi-model management primarily employs a manual command-driven scheduling approach. The multi-model management system provides an Application Programming Interface (API) for model start / stop, allowing administrators or users to explicitly invoke start or stop commands based on actual needs. When the API receives a start command, the system recreates the corresponding inference container and loads the complete model weights into GPU memory, making the model serviceable. When the API receives a stop command, the system destroys the inference container and releases the occupied computing resources to conserve system resources.

[0004] However, the aforementioned multi-model management approach suffers from problems such as high operating costs, poor user experience, and waste of computing resources in practical applications.

[0005] The background information is merely information known only to the inventor and does not imply that such information had entered the public domain before the date of this application, nor does it imply that it could be considered prior art in this disclosure. Summary of the Invention

[0006] This specification provides a scheduling method for model inference services, a multi-model management system, and a storage medium, enabling inference to begin immediately after a model is partially loaded into GPU memory. This effectively shortens the Time To First Token (TTFT) generation time, thus providing a multi-model management system with faster response, better user experience, and lower operating costs.

[0007] To achieve the above objectives, the embodiments in this specification adopt the following technical solutions: Firstly, this specification provides a method for scheduling model inference services, applied to a MaaS-based multi-model management system. The multi-model management system includes a service routing layer and an inference framework layer. The method includes: responding to a user request, the service routing layer determines a target model in a preset model cluster based on the user request and obtains the running status of the target model. The user request is used to request inference of a target task based on the target model, and the running status includes an active state and a dormant state. When the target model is in a dormant state, the service routing layer sends a wake-up request to the inference framework layer. The wake-up request is used to request that the target model be put into an active state. Based on the wake-up request, the inference framework layer performs an operation to write the weight data of each network layer of the target model from local storage to video memory. After the weight data of the first network layer is written, the inference calculation operation of the target task is performed layer by layer in the order of each network layer of the target model from front to back.

[0008] In some embodiments, the inference framework layer performs an operation of writing the weight data of each network layer of the target model from local storage to video memory based on the wake-up request, including: the inference framework layer performs an operation of writing the weight data of each network layer from local storage to video memory layer by layer in the order of each network layer of the target model from front to back based on the wake-up request.

[0009] In some embodiments, the inference framework layer performs an operation based on the wake-up request to write the weight data of each network layer of the target model from local storage to video memory layer by layer, in the order of each network layer from front to back. This includes: the inference framework layer determining the amount of weight data for each network layer of the target model; determining a target storage space in the video memory for storing the weight data of each network layer of the target model based on the amount of data and the remaining storage space in the video memory; and writing the weight data of each network layer of the target model from local storage to the target storage space in the order of each network layer from front to back.

[0010] In some embodiments, determining the target storage space for storing the weight data of each network layer of the target model in the video memory based on the amount of data and the remaining storage space of the video memory includes: if the amount of data is less than the remaining storage space of the video memory, determining the target storage space for storing the weight data of each network layer of the target model in the remaining storage space; and if the amount of data is greater than the remaining storage space of the video memory, determining the candidate model and the target network layer from which the candidate model is swapped out, and determining the target storage space for storing the weight data of each network layer of the target model based on the remaining storage space of the video memory and the storage location of the weight data of the target network layer.

[0011] In some embodiments, determining the candidate model and the target network layer from which the candidate model is swapped out if the data volume is greater than the remaining storage space of the video memory includes: if the data volume is greater than the remaining storage space of the video memory, determining the candidate model from the preset model cluster based on the swapping out strategy of the service routing layer; and determining the swapping out space size based on the difference between the data volume and the remaining storage space of the video memory, and determining the target network layer from which the candidate model is swapped out based on the swapping out space size.

[0012] In some embodiments, the target network layer to be replaced is the last N network layers of the candidate model, and the total amount of weight data of the last N network layers is greater than the size of the replacement space, where N is an integer greater than or equal to 1.

[0013] In some embodiments, determining the target storage space for storing the weight data of each network layer of the target model in the video memory based on the data volume and the remaining storage space of the video memory includes: if the data volume is greater than the remaining storage space of the video memory, then determining a candidate model, and replacing the weight data layer by layer in the order of the network layers of the candidate model from back to front, until the size of the replacement space meets the data volume requirement.

[0014] In some embodiments, the swap-out strategy includes at least: a swap-out strategy based on usage frequency, a swap-out strategy based on least recently used, or a swap-out strategy based on the order in which video memory is written.

[0015] In some embodiments, the running state further includes a partial dormant state; the method further includes: when the target model is in a partial dormant state, the service routing layer sends a wake-up request to the inference framework layer, the wake-up request being used to request that the target model be put into an active state; and the inference framework layer performs the following first and second operations in parallel: based on the wake-up request, performing a first operation to write the weight data of the network layer from which the target model is swapped out to the video memory, and a second operation to perform the inference calculation of the target task layer by layer in the order of each network layer of the target model from front to back.

[0016] In some embodiments, the multi-model management system further includes a service management layer, and the method further includes: during the model service deployment phase, the service management layer performs pre-startup preprocessing on each model in the preset model cluster to generate pre-stored model preprocessing data; the model preprocessing data includes at least one of: a pre-initialized execution environment, pre-parsed weight data, and pre-compiled computation data; and the pre-initialized execution environment includes a pre-initialized container environment and a general parallel computing architecture context; the pre-parsed weight data includes weight data conversion data after format conversion based on the target data format of the video memory; the pre-compiled computation data includes binary operator data compiled based on the model network structure.

[0017] In some embodiments, before the inference framework layer performs the operation of writing the weight data of each network layer of the target model from local storage to video memory based on the wake-up request, the method further includes: the inference framework layer obtaining and loading the pre-initialized execution environment corresponding to the target model from the local storage; after the pre-initialized execution environment is loaded, obtaining the pre-compiled computation data corresponding to the target model from the local storage and loading it into the video memory; and the inference framework layer performing the operation of writing the weight data of each network layer of the target model from local storage to video memory based on the wake-up request includes: writing the pre-parsed weight data corresponding to each network layer of the target model recorded in the local storage to the video memory.

[0018] In some embodiments, after the target model is in an active state, the method further includes: if the idle time of the target model reaches a preset sleep threshold, or if a sleep instruction is received; performing a sleep operation on the target model, the sleep operation including: storing the key-value cache data in the target model to local storage, and setting the running state of the target model to a sleep state.

[0019] Secondly, this specification also provides a multi-model management system, comprising: at least one storage medium storing at least one instruction set for managing models that provide inference services; and at least one processor communicatively connected to the at least one storage medium, wherein, when the multi-model management system is running, the at least one processor reads the at least one instruction set and implements the method provided in the first aspect according to the instructions of the at least one instruction set.

[0020] Thirdly, this specification also provides a computer-readable non-volatile storage medium, wherein the computer-readable non-volatile storage medium stores at least one instruction set, which, when executed by at least one processor, implements the method provided in the first aspect.

[0021] The scheduling methods for the model inference service, the multi-model management system, and other functions of the storage medium provided in this specification are partially listed in the following description. The inventive aspects of the scheduling methods for the model inference service, the multi-model management system, and the storage medium provided in this specification can be fully understood through practice or by using the methods, apparatus, and combinations described in the detailed examples below. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A schematic diagram illustrating an application scenario of the scheduling method for the model inference service provided according to embodiments of this specification is shown. Figure 2 A computer system 200 provided according to an embodiment of this specification is shown; Figure 3 A flowchart is shown of a scheduling method for a model inference service provided according to an embodiment of this specification; Figure 4 A schematic diagram of a model service deployment phase provided according to an embodiment of this specification is shown; Figure 5 A schematic diagram of a model service application phase and a model inference computation phase provided according to an embodiment of this specification is shown; Figure 6 This diagram illustrates an inference computation operation according to an embodiment of the present specification, in which weight data of each network layer is written from local storage to video memory layer by layer and a target task is executed. Figure 7A schematic diagram of a target network layer to be replaced according to an embodiment of this specification is shown; and Figure 8 A schematic diagram of another target network layer provided according to an embodiment of this specification is shown. Detailed Implementation

[0024] The following description provides specific application scenarios and requirements for this specification, intended to enable those skilled in the art to make and use the contents of this specification. Various partial modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments and applications without departing from the spirit and scope of this specification. Therefore, this specification is not limited to the embodiments shown, but rather to the widest scope consistent with the claims.

[0025] The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not restrictive. For example, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” used herein may also include the plural forms. When used in this specification, the terms “comprising,” “including,” and / or “containing” mean that the associated integers, steps, operations, elements, and / or components are present, but do not exclude the presence of one or more other features, integers, steps, operations, elements, components, and / or groups, or that other features, integers, steps, operations, elements, components, and / or groups may be added to the system / method.

[0026] Considering the following description, these and other features of this specification, as well as the operation and function of the related components of the structure, and the economy of assembly and manufacture of the parts, can be significantly improved. All of these form part of this specification with reference to the accompanying drawings. However, it should be clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of this specification. It should also be understood that the drawings are not drawn to scale.

[0027] The flowcharts used in this specification illustrate operations implemented according to some embodiments of this specification. It should be clearly understood that the operations in the flowcharts may not be implemented in a sequential order. Instead, the operations may be implemented in reverse order or simultaneously. Furthermore, one or more additional operations may be added to the flowcharts. One or more operations may be removed from the flowcharts.

[0028] For ease of description, the terms that will appear later in this manual will be explained first.

[0029] Term 1: Model-as-a-Service (MaaS): A deployment model that provides large language models to users as services. Users can call the model for inference through API interfaces without having to worry about the underlying implementation and deployment details of the model.

[0030] Term 2: Model Weights: These refer to the trainable parameters in a large language model, typically stored as a floating-point matrix. Model weights determine the model's behavior and performance, and usually require tens to hundreds of gigabytes of storage space.

[0031] Term 3: Time To First Token (TTFT): This refers to the time interval from when the user sends a request to when the model generates the first output token. TTFT is an important metric for measuring the response speed of the model service.

[0032] Term 4: Key-Value Cache (KVCache): A caching mechanism used during the inference process of large language models to store the intermediate results (key and value vectors) of the attention mechanism that have already been computed. KVCache can reduce the repeated computation of the same input sequence and improve inference efficiency.

[0033] Term 5: KVCache Persistence: A technique that saves KVCache cached data to persistent storage (such as disk) so that the cache is not lost during model hibernation and activation, and can be quickly loaded and reused after activation.

[0034] In the embodiments of this specification, the Large Language Model (LLM) can also be simply referred to as the large model. A large language model is a natural language processing model based on deep learning technology. It has a high number of parameters, typically reaching billions to hundreds of billions or even higher, and possesses powerful language understanding and generation capabilities. Large language models can employ the Transformer architecture or its variants (such as GPT, BERT, etc.). This architecture utilizes an attention mechanism to achieve global modeling of sequential data, efficiently handling long-distance dependencies and thus performing excellently in natural language tasks. Large language models can be used to perform various tasks such as text generation, language translation, text summarization, and question answering reasoning.

[0035] In some embodiments, large language models can also have the ability to understand and generate data from other modalities (such as visual, audio, etc.). In this case, large language models can also be called multimodal large language models (MLLMs). MLLMs provide a richer and more natural interactive experience by integrating multiple types of input and output, such as text, images, and sound.

[0036] It should be noted that the key technologies of large language models can be found in the detailed description in the paper "A Survey of Large Language Models" (paper number: arXiv:2303.18223v18, published on March 8, 2026), and will not be repeated here.

[0037] The following section introduces the application scenarios of this manual.

[0038] The technical solutions provided in this specification are applicable to MaaS scheduling and management scenarios involving multiple large language models in private deployments. Specifically, MaaS scheduling and management of multiple large language models in private deployments refers to the process of deploying multiple large language models with different parameter values ​​and functionalities locally, and uniformly managing the lifecycle, computing resource allocation, and activation and dormancy status of each model through a MaaS architecture to provide inference services for the target model. For example, financial institutions commonly use private deployment models to build AI service platforms to ensure customer data security and compliance; this AI service platform exemplifies a scenario employing MaaS scheduling and management of multiple large language models.

[0039] In this scenario, different types of large language models can provide different AI capabilities, offering a variety of AI services to employees or customers. For example, the AI ​​service platform can provide multiple types of large language models, such as general dialogue models, contract review models, risk warning models, and code assistance models, to provide intelligent services in various scenarios, including credit approval, compliance review, and daily office work.

[0040] However, in related solutions, the multi-model scheduling and management under the private MaaS architecture still has many limitations. Specifically, due to the limited graphics processing unit (GPU) memory resources in the private environment, existing platforms usually require operations and maintenance personnel to manually issue activation and hibernation commands for models based on the model load forecast, in order to switch multiple models online and offline.

[0041] For example, when the existing platform receives a model activation command, it recreates the inference container and loads the complete model weights into the GPU memory. When the existing platform receives a model hibernation command, it directly destroys the container and clears all model runtime data. This scheduling method is highly dependent on manual operation and maintenance, cannot automatically adjust the model's running state according to the model's real-time load, and is difficult to adapt to dynamic fluctuations in model load. Furthermore, the startup process from hibernation to activation in the existing solution is time-consuming, which cannot meet the user's real-time interactive needs. Frequent container reconstruction and full weight loading generate a large amount of repetitive computation and data read / write overhead, resulting in inefficient use of GPU memory resources. This leads to problems such as high operation and maintenance costs, large service response latency, low utilization of computing resources, and poor user experience.

[0042] To this end, this specification provides a scheduling method for model inference services, applied to a MaaS-based multi-model management system, which includes a service routing layer and an inference framework layer. The method includes: responding to a user request for inference of a target task based on a target model; the service routing layer determining the target model in a preset model cluster based on the user request and obtaining the running status of the target model, including an active state and a dormant state; when the target model is in a dormant state, the service routing layer sending a wake-up request to the inference framework layer, the wake-up request requesting that the target model be activated; the inference framework layer performing the operation of writing the weight data of each network layer of the target model from local storage to video memory based on the wake-up request, and after the weight data of the first network layer is written, starting to execute the inference calculation operation of the target task layer by layer in the order of the network layers of the target model from front to back.

[0043] In the solution provided in this specification, the service routing layer of the multi-model management system responds to user requests and determines the target model from a preset model cluster based on the user requests. It then obtains the running status of the target model through the inference framework layer of the multi-model management system. When the target model is in a dormant state, the service routing layer sends a wake-up request to the inference framework layer. Based on the wake-up request, the inference framework layer writes the weight data of each network layer of the target model from local storage to video memory. After the weight data of the first network layer is written, it begins to execute the inference calculation operation of the target task layer by layer in the order of the network layers of the target model from front to back. In this way, the multi-model management system can complete the transition from dormant to active models without manually issuing model start / stop commands, automatically adapting to dynamic changes in requirements. Furthermore, by decoupling model loading from inference execution, the multi-model management system can improve the execution logic of existing technologies that require waiting for the full loading of weight data of all network layers before starting inference, shortening the time for the model to switch from a dormant state to an active state and output the first inference result. Furthermore, the multi-model management system parallelizes the writing of weight data to the subsequent network layers with the inference computation, further improving the overall execution efficiency of the inference service. It can support the deployment and operation of more large language models with different capabilities and parameter amounts under limited GPU memory resources, improving the utilization of GPU memory resources, reducing the operating cost of multiple large language models under private deployment, and improving the user experience.

[0044] Figure 1 A schematic diagram illustrating an application scenario of the scheduling method for the model inference service provided according to embodiments of this specification is shown. For example... Figure 1 As shown, the application scenario 100 may include a multi-model management system 11 and a terminal device 12.

[0045] refer to Figure 1 The multi-model management system 11 can be deployed on a local device or a cluster of local devices with data processing capabilities. For example, the multi-model management system 11 can be deployed on a server, server cluster, or cloud server. It should be noted that the server, server cluster, or cloud server on which the multi-model management system 11 is deployed can be a private server, a private server cluster, or a private cloud. In this case, the physical device corresponding to the multi-model management system 11 can deploy multiple large language models with different parameter amounts and capabilities, and can provide AI services externally. The multi-model management system 11 stores data or instructions for executing the scheduling method for the model inference service described in this specification, and can execute or be used to execute said data or instructions. In some embodiments, the multi-model management system 11 may include a hardware device with data information processing capabilities and the necessary programs required to drive the hardware device.

[0046] As an example, terminal device 12 may be an electronic device capable of communicating with the multi-model management system 11. For example, terminal device 12 may include mobile devices, tablets, laptops, built-in devices in motor vehicles, or similar content, or any combination thereof. In some embodiments, the mobile device may include wearable devices, camera devices, smart mobile devices, virtual reality devices, augmented reality devices, or similar devices, or any combination thereof. In some embodiments, wearable devices include smartwatches, smart bracelets, smart glasses, etc. In some embodiments, the smart mobile device may include smartphones, personal digital assistants, gaming devices, navigation devices, etc., or any combination thereof. In some embodiments, the virtual reality device or augmented reality device may include head-mounted displays, virtual reality headsets, virtual reality glasses, virtual reality patches, augmented reality headsets, augmented reality glasses, augmented reality patches, or similar content, or any combination thereof.

[0047] In some embodiments, the terminal device 12 may have one or more applications (APPs) installed. The APPs provide the ability or interface to access AI services provided by multiple large language models in the multi-model management system 11. The APPs include, but are not limited to: web browser APPs, search APPs, chat APPs, shopping APPs, office APPs, video APPs, financial management APPs, instant messaging tools, email clients, social media platform software, etc. In some embodiments, the terminal device 12 may have a target APP installed. The target APP is an application corresponding to the multi-model management system 11.

[0048] In some embodiments, the target app can respond to user operations by generating and sending user requests to the multi-model management system 11. The multi-model management system 11 determines the running status of the corresponding target model based on the user request. When the target model is dormant, the multi-model management system 11 can activate the target model using the scheduling method of the model inference service and use the target model to process the user request, obtaining the inference result corresponding to the user request. Finally, the multi-model management system 11 can return the inference result corresponding to the user request to the target app and display the inference result through the target app.

[0049] It should be understood that Figure 1 The number of multi-model management system 11 and terminal devices 12 shown is merely illustrative. Depending on implementation needs, any number of multi-model management system 11 and terminal devices 12 can be included.

[0050] This specification also provides a computer system 200. Figure 2 A computer system 200 according to an embodiment of this specification is shown. The computer system 200 can serve as... Figure 1 The multi-model management system in [the context of the text is missing].

[0051] In some embodiments, one or more computer systems 200 may perform one or more steps of one or more methods described or illustrated herein. In some embodiments, one or more computer systems 200 may also provide functionality described or illustrated herein, such as the ability to retrieve one or more engines in system 11 (or other arbitrary functionality).

[0052] This specification considers any suitable number of computer systems 200. The computer systems 200 are considered to take any suitable physical form. By way of example and not limitation, the computer system 200 may be an embedded computer system, a system-on-a-chip (SOC), a single-board computer system (SBC) (e.g., a computer-on-module (COM) or system-on-module (SOM)), a desktop computer system, a notebook or laptop system, an interactive kiosk, a mainframe, a grid of computer systems, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, or any suitable combination thereof. Where appropriate, one or more computer systems 200 may be single or distributed; may span multiple locations or multiple machines; may span one or more data centers; or reside in the cloud, which may include one or more cloud components and communicate through one or more networks. Where appropriate, one or more computer systems 200 may perform one or more steps of one or more methods described or illustrated herein without substantial spatial or temporal limitations, and may perform in real-time or batch mode, or at different times and / or different locations.

[0053] In some embodiments, computer system 200 includes: processor 202, memory 204, storage 206, input / output (I / O) interface 208, communication interface 210, and interconnect structure 212. Although this specification describes and illustrates an example computer system 200 having a specific number of specific components in a specific arrangement, this specification contemplates any suitable number of any suitable components and any suitable arrangement.

[0054] In some embodiments, processor 202 is used to execute instructions. Processor 202 can retrieve instructions through internal registers, internal caches, memory 204, or storage 206, decode and execute them, and write one or more results to internal registers, internal caches, memory 204, or storage 206. Where appropriate, processor 202 may include one or more internal caches. Processor 202 may also include one or more internal registers, one or more arithmetic logic units (ALUs), and one or more processor cores. Processor 202 may be a multi-core processor and may include, but is not limited to: microcontrollers, microprocessors, RISC, ASICs, ASIPs, CPUs, GPUs, PPUs, DSPs, FPGAs, ARMs, and programmable logic devices (PLDs), or other circuits or processors capable of performing one or more functions, or any combination thereof.

[0055] In some embodiments, memory 204 may be used to store instructions to be executed by processor 202 and / or data to be operated by processor 202. Memory 204 may include main memory and may include random access memory (RAM). Where appropriate, memory 204 may be volatile memory (e.g., dynamic RAM (DRAM) or static RAM (SRAM)), or other suitable memory types. Processor 202 may couple to memory 204 via one or more memory paths for instruction and data access. Where appropriate, memory 204 may include one or more memory cells.

[0056] In some embodiments, storage 206 may be used to provide non-volatile storage for data and / or instructions. Storage 206 may include, but is not limited to: hard disk drive (HDD), solid-state drive (SSD), flash memory, ROM, erasable ROM, optical media (e.g., optical disc), magnetic media (e.g., magnetic disk or magnetic tape), USB storage device, or combinations thereof. Where appropriate, storage 206 may be located inside or outside computer system 200 and may include one or more storage control units to facilitate communication between processor 202 and storage 206.

[0057] In some embodiments, I / O interface 208 may be used to provide an interface between computer system 200 and one or more I / O devices. I / O interface 208 may include hardware, software, or a combination of both. I / O devices enable communication between a person and computer system 200 or between computer system 200 and external devices. Exemplary I / O devices may include, but are not limited to: keyboard, keypad, microphone, display, mouse, printer, scanner, speaker, camera, touch input device, stylus, trackball, tablet, and other suitable I / O devices; and I / O devices may also include one or more sensors. Where appropriate, I / O interface 208 may include one or more device or software drivers to enable processor 202 to control one or more associated I / O devices.

[0058] In some embodiments, communication interface 210 may be used to provide a communication interface between computer system 200 and one or more other computer systems 200 and / or one or more networks. Communication interface 210 may include hardware, software, or a combination of both. Communication interface 210 may support wired or wireless communication and may include a network interface controller (NIC) or a wireless adapter (e.g., a Wi-Fi adapter). Exemplary networks may include, but are not limited to: ad hoc networks, personal area networks (PANs), local area networks (LANs), wide area networks (WANs), metropolitan area networks (MANs), one or more portions of the Internet, and so on. Where appropriate, communication interface 210 may include one or more communication interfaces 210.

[0059] In some embodiments, the interconnect structure 212 is used to interconnect the components of the computer system 200. The interconnect structure 212 may include, but is not limited to: bus, switching structure, crossbar switch, on-chip interconnect network, point-to-point interconnect or other suitable interconnect structure, or any combination thereof.

[0060] This specification also provides a computer-readable non-volatile storage medium that stores a computer program. When executed by a processor, the computer program can be used to perform one or more steps of one or more methods described or shown herein, or to implement one or more functions described or shown herein. The computer-readable non-volatile storage medium may include one or more semiconductor or other integrated circuit (IC) based memories (e.g., FPGA or ASIC built-in memory / configuration memory), HDDs, HHDs, optical discs, optical disc drives (ODDs), magneto-optical discs, magneto-optical drives, floppy disks, floppy disk drives (FDDs), magnetic tapes, SSDs, RAM drives, secure digital cards, or combinations thereof, or other suitable media.

[0061] Figure 3A flowchart illustrating a scheduling method for a model inference service according to an embodiment of this specification is shown. As previously described, the computer system 200 can execute the scheduling method for the model inference service as a multi-model management system.

[0062] like Figure 3 As shown, the scheduling method for the model inference service may include: S310: In response to a user request, the service routing layer determines the target model in the preset model cluster based on the user request and obtains the running status of the target model. The user request is used to request inference of the target task based on the target model. The running status includes active state and dormant state.

[0063] In some embodiments, a multi-model management system may include a service management layer, a service routing layer, and an inference framework layer.

[0064] The service management layer, also known as the service management module, is a module in the multi-model management system used for global control and scheduling. The service management layer manages the entire lifecycle of model instances in the model cluster and deploys new models to the cluster according to their service modes.

[0065] The service routing layer, also known as the service routing module, is the unified entry point for user requests in a multi-model management system and a scheduling module used to wake up models and distribute requests. The service routing layer can detect user requests, determine the target model, activate the target model, and respond to and distribute user requests to the target model.

[0066] The inference framework layer, also known as the inference framework module, is a module in a multi-model management system used to perform inference computations. The inference framework layer can utilize activated models to perform inference computations for the target task corresponding to the user's request.

[0067] In some embodiments, the scheduling method for model inference services can be divided into three stages: model service deployment stage, model service application stage, and model inference computation stage.

[0068] During the model service deployment phase, the service management layer can pre-start and preprocess each model in the pre-defined model cluster to generate pre-stored model preprocessing data. This model preprocessing data includes at least one of the following: a pre-initialized execution environment, pre-parsed weight data, and pre-compiled computation data. For example, the pre-initialized execution environment may include a pre-initialized container environment and a general parallel computing architecture context; the pre-parsed weight data may include weight data converted from the target data format in video memory; and the pre-compiled computation data may include binary operator data compiled based on the model network structure.

[0069] It should be noted that a general-purpose parallel computing architecture refers to a computing architecture that supports offloading general-purpose computing tasks to parallel computing devices for execution, providing a unified programming model, runtime environment, and memory management mechanism. Common general-purpose parallel computing architectures may include, but are not limited to, Compute Unified Device Architecture (CUDA), Radeon Open Compute (ROCm) architecture, oneAPI architecture, and Compute Architecture for Neural Networks (CANN). The general-purpose parallel computing architecture context refers to a global container that manages the computing resources, execution state, and runtime environment of parallel computing devices. The general-purpose parallel computing architecture context can be determined based on the specific type of GPU device.

[0070] Figure 4 A schematic diagram of a model service deployment phase provided according to an embodiment of this specification is shown.

[0071] In some embodiments, reference Figure 4 During the model service deployment phase, the multi-model management system can utilize the service management layer to deploy models to the model cluster. For example, the multi-model management system can receive basic metadata, runtime environment dependency information, model structure and weight information, inference service configuration information, and scheduling strategy configuration information of the models to be deployed.

[0072] The basic metadata can include a globally unique identifier for the model to be deployed, version information, model attributes, and status information. Model attributes can include model name, model type (large language model, multimodal model, code model, etc.), number of model parameters, training framework, deployer information, and deployment time. Runtime environment dependencies can include inference framework dependencies, general parallel computing architecture dependencies, system and software dependencies, and hardware resource requirements. Model structure and weight information can include network structure parameters, weight file metadata, and weight information for each network layer. Inference service configuration information can include basic inference configuration, KVCache configuration, logging and monitoring configuration, and security and compliance configuration. Scheduling strategy configuration information can include service mode, lifecycle scheduling strategy, memory scheduling strategy, routing strategy configuration, and scaling strategy.

[0073] The service mode can include persistent mode or on-demand mode. When the service mode of the model to be deployed is persistent mode, the model will remain active after deployment. The scheduling method of the model inference service provided in this specification can ignore models running in persistent mode, that is, the model will not participate in the swapping out, hibernation and other related operations in the scheduling method of the model inference service provided in this specification.

[0074] Continue to refer to Figure 4 When the service mode of the model to be deployed is on-demand mode, the multi-model management system can use the service management layer to pre-start and pre-process the model to be deployed.

[0075] For example, the multi-model management system pre-initializes the runtime environment based on the basic metadata and runtime environment dependency information of the models to be deployed. Specifically, the multi-model management system can obtain a pre-initialized container environment based on the inference framework dependency, system and software dependency, and hardware resource requirements in the basic metadata and runtime environment dependency information; and obtain a general parallel computing architecture context based on the general parallel computing architecture dependency in the runtime environment dependency information.

[0076] The multi-model management system can also perform pre-analysis based on the model structure and weight information of the model to be deployed, obtaining pre-analyzed weight data. Furthermore, the multi-model management system can perform pre-compilation based on inference service configuration information, obtaining pre-compiled computation data.

[0077] Once the multi-model management system has completed the pre-startup and pre-processing of the model to be deployed, it can deploy the model to the model cluster in an on-demand mode. At the same time, the model pre-processing data corresponding to the model to be deployed is stored in the local storage of the multi-model management system.

[0078] In this embodiment, during the model service deployment phase, the multi-model management system utilizes the service management layer to perform pre-startup preprocessing on each model in the model cluster and generate pre-stored model preprocessing data. By pre-completing the computational overhead required for model operation at the time of each model deployment and caching the model preprocessing data in local storage, when a model is awakened, there is no need to perform time-consuming operations such as real-time compilation, format parsing, and environment initialization. The model can be activated simply by loading the pre-cached model preprocessing data from local storage into the video memory. This effectively shortens the cold start time for models to switch from a dormant state to an active state, improving the service response speed and user experience of the multi-model management system.

[0079] Figure 5 A schematic diagram of a model service application phase and a model inference computation phase provided according to an embodiment of this specification is shown.

[0080] In some embodiments, reference Figure 5When the multi-model management system receives a user request, it can respond to the user request using the service routing layer and enter the model service application stage.

[0081] As an example, user requests can be described in natural language or recorded using structured data. For example, see [reference]. Figure 1 In the scenario illustrated, the terminal device can display a target page for user input requests. This target page can correspond to a model in the model cluster. The terminal device can respond to user input and sending operations, packaging the globally unique identifier of the model corresponding to the target page and the user's input text data into a user request, and then sending the user request to the multi-model management system. The multi-model management system can use the service routing layer to parse the user request, obtain the globally unique identifier carried in the user request and the target task corresponding to the user request, and then determine the model in the model cluster corresponding to the globally unique identifier as the target model.

[0082] Alternatively, the terminal device can respond to user input and sending operations, converting the received user input into structured data. This structured data includes at least the task type and task description input by the user, and is then sent as a user request to the multi-model management system. The multi-model management system can use the service routing layer to parse the user request, obtain the target task and its type, and then determine the model in the model cluster capable of performing that type of task as the target model.

[0083] In some embodiments, referring to the foregoing example, each model in the model cluster maintains its own basic metadata after deployment, which includes the model's status information. The status information records the model's deployment and running status. When the multi-model management system determines the target model based on a user request, it can read the status information from the target model's corresponding basic metadata to determine whether the target model is active or dormant.

[0084] Among them, reference Figure 5 When the target model is active, the multi-model management system can directly distribute the target task to the inference framework layer, entering the model inference computation phase. In this case, during the model inference computation phase, the inference framework layer can directly input the target task into the target model, utilize the target model to perform the inference computation of the target task, and obtain the corresponding inference result. Then, the multi-model management system sends the inference result to the terminal device via the service routing layer and displays it on the target page on the terminal device.

[0085] S320: When the target model is in a dormant state, the service routing layer sends a wake-up request to the inference framework layer. The wake-up request is used to request that the target model be put into an active state.

[0086] In some embodiments, when the multi-model management system determines that the target model is in a dormant state, it can generate a wake-up request for the target model through the service routing layer. The wake-up request may include the globally unique identifier of the target model and the target task.

[0087] When the inference framework layer receives a wake-up request, it can load the model preprocessing data of the corresponding model into the video memory based on the globally unique identifier in the wake-up request, and update the state information of the target model in the model cluster.

[0088] As an example, the inference framework layer can retrieve and load the pre-initialized execution environment corresponding to the target model from local storage. After the pre-initialized execution environment is loaded, the inference framework layer can retrieve the pre-compiled computation data corresponding to the target model from local storage and load it into the video memory.

[0089] Once the multi-model management system has completed the above operations, it indicates that the target model has been activated. Then, the multi-model management system can load the weight information of the target model into the video memory according to the method shown in S330, and perform the inference calculation operation of the target task.

[0090] S330: The inference framework layer executes the operation of writing the weight data of each network layer of the target model from local storage to video memory based on the wake-up request. After the weight data of the first network layer is written, it starts to execute the inference calculation operation of the target task layer by layer in the order of each network layer of the target model from front to back.

[0091] In some embodiments, the weight data of each network layer refers to the pre-parsed weight data corresponding to each network layer. That is, the operation of writing the weight data of each network layer of the target model from local storage to video memory based on the wake-up request means that the inference framework layer writes the pre-parsed weight data corresponding to each network layer of the target model recorded in local storage to video memory.

[0092] In some embodiments, when the inference framework layer receives a wake-up request, it can determine whether to enter the model inference computation phase. (See reference) Figure 5 During the model inference computation phase, the inference framework layer can execute the operation of writing the weight data of each network layer from local storage to video memory layer by layer, according to the order of each network layer of the target model from front to back, based on the wake-up request.

[0093] Figure 6 The illustration shows a schematic diagram of an inference computation operation that, according to an embodiment of this specification, writes the weight data of each network layer from local storage to video memory layer by layer and performs the target task.

[0094] In some embodiments, reference Figure 6The inference framework layer can first determine the amount of weight data for each network layer of the target model (i.e., the GPU memory usage of the target model). Specifically, Figure 6 The diagram shows the target model's network layers 0 through 5, a total of 6 layers. The sum of the pre-parsed weight data for each of the 6 network layers represents the weight data for each layer of the target model, which is also the GPU memory usage of the target model.

[0095] Then, the inference framework layer can determine the target storage space in the video memory for storing the weight data of each network layer of the target model, based on the amount of data and the remaining storage space in the video memory.

[0096] As an example, if the amount of data (the GPU memory usage of the target model) is less than the remaining GPU memory space, then the target storage space for storing the weight data of each network layer of the target model is determined from the remaining storage space. For example, refer to... Figure 6 If the remaining storage space in the video memory is greater than the video memory usage of the target model, then the inference framework layer can determine a storage space in the video memory that is not less than the video memory usage of the target model as the target storage space.

[0097] Finally, the inference framework layer can write the weight data of each network layer from local storage to the target storage space of the video memory layer by layer, in the order of each network layer of the target model from front to back.

[0098] As an example, refer to Figure 6 The inference framework layer can initiate write requests in parallel to write the weight data of each network layer of the target model from local storage to the target memory. Then, the inference framework layer can assign priority to each write request according to the order of the network layers, with earlier layers having higher priority. For example, refer to... Figure 6 Among them, network layer 0 has the highest priority in the target model, while network layer 5 has the lowest priority.

[0099] As an example, when the general parallel computing architecture is CUDA, the inference framework layer can use different CUDA multi-asynchronous copy streams to copy the weight data of each network layer of the target model in local storage, and perform copy synchronization as needed before the layer computation begins.

[0100] For example, the inference framework layer creates an independent CUDA asynchronous copy stream for each network layer of the target model. All CUDA asynchronous copy streams execute independently and in parallel. Each CUDA asynchronous copy stream reads the weight data of the corresponding network layer from local storage and asynchronously copies it to the address corresponding to that network layer in the target storage space of the GPU memory.

[0101] As an example, refer to Figure 6Assuming the number of network layers M=6, in order for the target model to perform the inference computation operation of the target task, the inference framework layer can execute the CUDA asynchronous copy stream corresponding to network layer i before the inference computation task of network layer i starts, and copy the weight data corresponding to network layer i to the corresponding address in the video memory, where i∈[0,5].

[0102] In some embodiments, reference Figure 5 During the model inference computation phase, the inference framework layer can also begin to perform the inference computation operation of the target task layer by layer in the order of each network layer of the target model from front to back after the weight data of the first network layer has been written.

[0103] For example, refer to the examples above and Figure 6 The first network layer of the target model is network layer 0. The inference framework layer can prioritize executing the CUDA asynchronous copy stream corresponding to network layer 0. Once the inference framework layer determines that the weight data of network layer 0 has been written, it can use network layer 0 to begin executing the inference computation operation of the target task. Simultaneously, the inference framework layer can also execute the CUDA asynchronous copy stream corresponding to network layer 1.

[0104] Once the inference computation operation corresponding to network layer 0 is completed, if the inference framework layer determines that the weight data of network layer 1 has been written, it can continue to use network layer 1 to perform the inference computation operation for the target task. If the weight data of network layer 1 has not yet been written, the inference framework layer can wait for the weight data of network layer 1 to be written before using network layer 1 to perform the inference computation operation for the target task.

[0105] The inference framework layer can execute CUDA asynchronous copy stream and inference computation operations multiple times in the above manner until all network layers have executed the inference computation operation of the target task, and the last network layer outputs the inference result corresponding to the target task.

[0106] In this embodiment, the inference framework layer in the multi-model management system can execute the operation of writing the weight data of each network layer of the target model from local storage to video memory based on the wake-up request, and immediately start inference computation after the weight data of the first network layer is written. In this way, the multi-model management system can start performing inference computation when some network layers have been loaded, and execute the operation of writing the weight data of subsequent network layers to video memory in parallel with the inference computation operation of the preceding network layers. This achieves parallel execution of inference computation and weight swapping, overlapping the weight data transmission time with the inference computation time, thereby shortening the time from waking up the target model from the dormant state to outputting the first inference result, reducing the TTFT after the model wakes up, and making the process of activating and waking up the model in the multi-model management system faster and more efficient, thus improving the user experience.

[0107] In some embodiments, when the remaining storage space in GPU memory is less than the GPU memory usage of the target model, the inference framework layer needs to release some space in GPU memory so that the remaining storage space in GPU memory is greater than or equal to the GPU memory usage of the target model. If the data volume (GPU memory usage of the target model) is greater than the remaining storage space in GPU memory, the inference framework layer can determine the candidate model and the target network layer from which the candidate model is replaced, and determine the target storage space for storing the weight data of each network layer of the target model based on the remaining storage space in GPU memory and the storage location of the weight data of the target network layer.

[0108] As an example, if the amount of data exceeds the remaining storage space in the GPU memory, the inference framework layer can determine candidate models from a pre-defined model cluster based on the service routing layer's swap-out strategy. Furthermore, the inference framework layer can determine the swap-out space size based on the difference between the amount of data and the remaining storage space in the GPU memory, and determine the target network layer from which the candidate model will be swapped out based on the swap-out space size.

[0109] In some embodiments, when the inference framework layer determines candidate models from a preset model cluster based on the swap-out strategy of the service routing layer, the swap-out strategy that can be adopted includes at least: a swap-out strategy based on usage frequency, a swap-out strategy based on least recently used, or a swap-out strategy based on the order of writing to video memory.

[0110] For example, when the replacement strategy is based on usage frequency, assuming model A is called 12 times / hour, model B is called 3 times / hour, and model C is called 1 time / hour, the inference framework layer can determine model C, which has the lowest usage frequency, as the candidate model.

[0111] For example, when the replacement strategy is based on the least recently used strategy, assuming that model A was last called 5 minutes ago, model B was last called 30 minutes ago, and model C was last called 2 hours ago, the inference framework layer can determine model C, which has not been used for the longest time, as a candidate model.

[0112] For example, when the swapping strategy is based on the order in which GPU memory is written, assuming that the first write time of model A was 2 hours ago, the first write time of model B was 1 hour ago, and the first write time of model C was 3 hours ago, then the inference framework layer can determine that model C, which was the earliest to write to GPU memory, is the candidate model.

[0113] Figure 7 A schematic diagram of a target network layer to be replaced according to an embodiment of this specification is shown.

[0114] In some embodiments, the target network layers to be replaced are the last N network layers of the candidate model, and the total amount of weight data of the last N network layers is greater than the size of the replacement space, where N is an integer greater than or equal to 1.

[0115] As an example, refer to Figure 7 The target model's GPU memory usage exceeds the remaining storage space. Assuming the target model's GPU memory usage is 8GB and the remaining storage space is 2GB, the swap space size is 6GB. If the candidate model's network layer 7 has a GPU memory usage of 3GB and network layer 8 has a GPU memory usage of 4GB, then the total data size of the last two network layers of the candidate model is 7GB, which is greater than the 6GB swap space. The inference framework layer determines that the target network layers to be swapped out are the last two network layers of the candidate model (network layers 7 and 8). Therefore, network layers 7 and 8 of the candidate model need to be swapped out together to obtain the target storage space that satisfies the GPU memory usage requirement of the target model.

[0116] Then, the inference framework layer can swap out network layers 7 and 8 of the candidate model, leaving 9GB of free memory. Next, the inference framework layer can utilize... Figure 6 The method shown in the figure determines the target storage space in the remaining storage space of the video memory, writes the weight data of each network layer of the target model into the target storage space, and at the same time, uses the network layers that have been written to perform the inference calculation operation of the target task layer by layer to obtain the inference result corresponding to the target task.

[0117] In this embodiment, when the amount of weight data for each network layer in the target model exceeds the remaining storage space in the GPU memory, the multi-model management system can first determine candidate models based on a preset swapping strategy, then determine the target network layers to be swapped out from the candidate models according to the swapping space size, and swap out the target network layers so that the weight data of each network layer in the target model can be written to the GPU memory for executing the target task. The multi-model management system only swaps out a portion of the network layers in the candidate models to write the weight data of each network layer in the target model to the GPU memory and execute the target task. This fine-grained swapping method avoids swapping out the entire dormant model to meet the loading requirements of the new model, reduces the amount of data transfer during the swapping process, and enables the multi-model management system to quickly respond to model wake-up requests, improving the flexibility of multi-model start-up and shutdown and the overall utilization efficiency of GPU memory resources. Meanwhile, this fine-grained swapping method only removes some network layer weight data at the tail of the candidate model, while retaining the weight data of its first network layer and the complete runtime environment context. When the candidate model needs to be woken up, it can restore the complete inference capability simply by reloading the swapped-out tail network layer weight data, reducing the amount of data transmission during the candidate model wake-up process, further improving the cold start speed of candidate model wake-up, and further improving the overall efficiency of multi-model elastic scheduling.

[0118] Figure 8 A schematic diagram of another target network layer to be replaced according to an embodiment of this specification is shown.

[0119] In some embodiments, if the amount of data (the memory usage of the target model) is greater than the remaining storage space of the video memory, the inference framework layer can determine the candidate model based on the method for determining the candidate model in the above example.

[0120] Then, the inference framework layer can replace the weight data layer by layer according to the candidate model's network layers from back to front, until the replacement space size meets the data volume requirements.

[0121] As an example, see reference Figure 8 The target model's network layers include layers 0 through 5. The inference framework layer can start from layer 0 of the target model and sequentially write the weight data of each network layer of the target model into the GPU memory. (Reference) Figure 8 When the inference framework layer writes the weight data corresponding to network layers 0, 1, and 2 of the target model into the GPU memory, there is insufficient remaining storage space in the GPU memory to write the weight data corresponding to network layer 3 of the target model. In this case, the inference framework layer can remove the weight data corresponding to the last network layer in the candidate model (i.e., network layer 8 of the candidate model) and then write the weight data corresponding to network layer 3 of the target model into the GPU memory.

[0122] Continue to refer to Figure 8 When the weight data corresponding to the network layer 3 of the target model is written to the video memory, and the remaining storage space in the video memory is insufficient to continue writing the weight data corresponding to the network layer 4 of the target model, the inference framework layer can remove the weight data corresponding to the last network layer (i.e., the network layer 7 of the candidate model) among the remaining multiple network layers of the candidate model, and write the weight data corresponding to the network layers 4 and 5 of the target model to the video memory, thus completing the replacement of the target model.

[0123] In this embodiment, when the target model's memory usage exceeds the remaining memory space, the multi-model management system can swap out weight data layer by layer from the candidate model's network layers in reverse order until the swapped-out space meets the target model's memory usage. The multi-model management system achieves dynamic and refined swapping out of memory resources in this way. This swapping method does not require pre-calculating the number of network layers to be swapped out; instead, it determines whether to swap out candidate model network layers based on the real-time changes in the remaining memory space during the swapping process. This better adapts to the dynamic changes in remaining memory space, effectively reducing interference from the remaining memory space on the swapping process, making the swapping process more flexible and controllable, reducing the probability of errors during the swapping process, and improving the stability and reliability of memory scheduling.

[0124] It should be noted that the running state also includes a partially dormant state. When a candidate model has had some network layers removed, its running state can be marked as partially dormant. A partially dormant state means that only some of the tail layer weights have been released, while the weights of the first network layer and the running environment context are still retained.

[0125] In some embodiments, reference Figure 5 When the target model is in a partially dormant state, the service routing layer sends a wake-up request to the inference framework layer. The wake-up request is used to request that the target model be put into an active state. The inference framework layer executes the following first and second operations in parallel: based on the wake-up request, the first operation is to write the weight data of the network layers that have been swapped out of the target model from local storage to video memory; and the second operation is to perform the inference calculation of the target task layer by layer in the order of each network layer of the target model from front to back.

[0126] As an example, suppose Figure 7 and Figure 8 The candidate models that were replaced entered a partially dormant state. (Reference) Figure 5 Once the service routing layer determines that the candidate model is the new target model, it can send a wake-up request to the inference framework layer. After receiving the wake-up request, the inference framework layer can determine to enter the model inference calculation stage. In the model inference calculation stage, the original candidate model is regarded as the target model.

[0127] refer to Figure 7 and Figure 8 Since the weight data of the first network layer of the target model and the runtime environment context are still stored in the GPU memory, the first network layer of the target model can be directly used to perform inference computation of the target task (i.e., the second operation).

[0128] At the same time, refer to Figure 7 and Figure 8 Because the tail layer weights (layers 7 and 8) of the target model have been removed, the inference framework layer needs to reload the missing layer weights into GPU memory. For instructions on how to reload the missing layer weights into GPU memory, please refer to [link to documentation / reference]. Figure 7 or Figure 8 The method shown in the diagram will not be elaborated upon here.

[0129] Furthermore, this mechanism only swaps out the minimum number of tail network layer weights that meet the loading requirements of the target model, while retaining the first network layer weights of the candidate model in the GPU memory. When the candidate model needs to be woken up on demand, only the swapped-out small number of tail network layer weights need to be reloaded to restore the complete inference capability, which greatly reduces the amount of data transmission during the model wake-up process and further improves the cold start speed of model wake-up.

[0130] In some implementations, if there are some dormant models in the model cluster, the inference framework layer can prioritize the dormant models as candidate models when determining candidate models.

[0131] In this embodiment, the multi-model management system introduces a partial dormancy state. This partial dormancy state, while retaining the model's runtime environment and some of the first network layer weights, releases the GPU memory resources occupied by the weights of some of the model's later network layers for use by other models. When a model in a partial dormancy state is awakened, it does not need to load the runtime environment and the weights of the first network layer and can be directly used to execute the target task. Simultaneously, after being awakened, a model in a partial dormancy state only needs to load the missing network layer weights to restore its full inference capability. In this way, the multi-model management system can achieve a better balance between GPU memory utilization and model awakening speed, further overlapping data transfer time and inference computation time, reducing the amount and time spent on data transfer during model awakening, further improving the model's cold start speed, shortening the time-to-threshold transition (TTFT), and thus providing a more efficient and smoother user experience.

[0132] Furthermore, when the multi-model management system has some dormant models in the model cluster, it prioritizes identifying the partially dormant models as candidate models. Since some dormant models have already had some network layer weight data swapped out, prioritizing the swapping of some dormant models not only reduces interference to models that will be activated or dormant models, but also shortens the total time of the memory swapping process. This allows the target model to complete weight loading and start inference services faster, further improving the model's wake-up and cold start speed.

[0133] In some implementations, when the target model is active, if the idle time of the target model reaches a preset sleep threshold, or if a sleep command is received for the target model, the inference framework layer can perform a sleep operation on the target model. The sleep operation includes: storing the key-value cache data in the target model to local storage, and setting the running state of the target model to a sleep state.

[0134] In this context, the key-value cache data in the target model is called the KVCache of the target model. The process of storing the KVCache of the target model to local storage can also be called the persistence of the KVCache of the target model. The KVCache of the target model can include the KVCache corresponding to each network layer of the target model.

[0135] In some embodiments, when the inference framework layer performs the operation of writing the weight data of each network layer from local storage to video memory in the order of each network layer of the target model from front to back based on the wake-up request, if the target model has a corresponding KVCache, the inference framework layer can also write the KVCache corresponding to the network layer to video memory when writing the weight data of each network layer from local storage to video memory.

[0136] In this embodiment, the multi-model management system persistently stores the key-value cache of the dormant model when it is in sleep mode. When the model is awakened, the network layer weight data and the corresponding key-value cache of the model are loaded into the GPU memory. In this way, the multi-model management system can completely store the key-value cache data corresponding to each network layer of the target model in local persistent storage when performing a sleep operation on the target model, instead of directly clearing the cache. This allows the key-value cache data to be retained during the model's sleep and wake-up state transitions. After the target model is awakened, the key-value cache data loaded into the GPU memory can be directly reused for inference, avoiding the recalculation of historical attention intermediate results, reducing computational power consumption, shortening inference response time, further reducing the TTFT after model wake-up, thereby improving the operating efficiency of the multi-model management system and enhancing the user experience of continuous interaction.

[0137] In summary, the model inference service scheduling method provided in this specification involves a multi-model management system whose service routing layer responds to user requests and determines the target model from a pre-defined model cluster based on these requests. The system then obtains the target model's running status through the multi-model management system's inference framework layer. When the target model is in a dormant state, the service routing layer sends a wake-up request to the inference framework layer. Based on this wake-up request, the inference framework layer writes the weight data of each network layer of the target model from local storage to video memory. After the weight data of the first network layer is written, it begins executing the inference calculations for the target task layer by layer in the order of the target model's network layers from front to back. In this way, the multi-model management system can automatically adapt to dynamic changes in requirements without manually issuing model start / stop commands, transitioning the model from dormant to active. Furthermore, by decoupling model loading from inference execution, the multi-model management system improves upon existing technologies that require waiting for the full loading of all network layer weight data before inference can begin, shortening the time required for the model to switch from a dormant state to an active state and output the first inference result. Furthermore, the multi-model management system parallelizes the writing of weight data to the subsequent network layers with the inference computation, further improving the overall execution efficiency of the inference service. It can support the deployment and operation of more large language models with different capabilities and parameter amounts under limited GPU memory resources, improving the utilization of GPU memory resources, reducing the operating cost of multiple large language models under private deployment, and improving the user experience.

[0138] This specification, in another aspect, provides a computer-readable non-transitory storage medium storing at least one instruction set for managing models that provide inference services. When the at least one instruction set is executed by a processor, it instructs the processor to implement the steps of the scheduling method for the model inference service described in this specification. In some possible embodiments, various aspects of this specification can also be implemented as a program product comprising program code. When the program product is run on a computer system 200, the program code causes the computer system 200 to perform the steps of the scheduling method for the model inference service described in this specification. The program product for implementing the above method may employ a portable compact disk read-only memory (CD-ROM) containing program code and may run on the computer system 200. However, the program product of this specification is not limited thereto. In this specification, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system. The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. The computer-readable storage medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can send, propagate, or transmit programs for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof. Program code for performing the operations described herein can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on computer system 200, partially on computer system 200, as a standalone software package, partially on computer system 200 and partially on a remote computing device, or entirely on a remote computing device.

[0139] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0140] In summary, after reading this detailed disclosure, those skilled in the art will understand that the foregoing detailed disclosure is presented by way of example only and is not restrictive. Although not explicitly stated herein, those skilled in the art will understand that this specification requires various reasonable changes, improvements, and modifications to the embodiments. These changes, improvements, and modifications are intended to be made by this specification and are within the spirit and scope of the exemplary embodiments described herein.

[0141] Furthermore, certain terms in this specification have been used to describe embodiments of this specification. For example, "an embodiment," "an embodiment," and / or "some embodiments" mean that a particular feature, structure, or characteristic described in connection with that embodiment may be included in at least one embodiment of this specification. Therefore, it is to be emphasized and understood that two or more references to "an embodiment" or "an embodiment" or "alternative embodiment" in various parts of this specification do not necessarily refer to the same embodiment. Moreover, specific features, structures, or characteristics may be suitably combined in one or more embodiments of this specification.

[0142] It should be understood that in the foregoing description of the embodiments in this specification, various features are combined in a single embodiment, drawing, or description for the purpose of simplifying the description and to aid in understanding a feature. However, this does not mean that the combination of these features is necessary, and those skilled in the art, upon reading this specification, may readily identify some of the devices as separate embodiments. That is, the embodiments in this specification can also be understood as an integration of multiple secondary embodiments. And the content of each secondary embodiment is valid even if it contains fewer than all the features of a single foregoing disclosed embodiment.

[0143] Every patent, patent application, publication of a patent application, and other material such as articles, books, specifications, publications, documents, articles, etc., cited herein, except for those inconsistent with or conflicting with this document, or those having a restrictive effect on the widest scope of the claims, may be incorporated herein by reference for all purposes now or hereafter associated with this document. Furthermore, in the event of any inconsistency or conflict between the description, definition, and / or use of relevant terms in any material and the description, definition, and / or use of relevant terms in this document, the terms in this document shall prevail.

[0144] Finally, it should be understood that the embodiments disclosed herein are illustrative of the principles of the embodiments described in this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are merely examples and not limitations. Those skilled in the art can implement the applications described in this specification using alternative configurations based on the embodiments in this specification. Therefore, the embodiments in this specification are not limited to the embodiments precisely described in the applications.

Claims

1. A method for scheduling model inference services, applied to a multi-model management system based on Model-as-a-Service (MaaS), the multi-model management system comprising: The service routing layer and inference framework layer, the method includes: In response to a user request, the service routing layer determines the target model in a preset model cluster based on the user request and obtains the running status of the target model. The user request is used to request inference of the target task based on the target model, and the running status includes an active state and a dormant state. When the target model is in a dormant state, the service routing layer sends a wake-up request to the inference framework layer. The wake-up request is used to request that the target model be put into an active state. Based on the wake-up request, the inference framework layer performs the operation of writing the weight data of each network layer of the target model from local storage to video memory. After the weight data of the first network layer is written, the inference calculation operation of the target task is performed layer by layer in the order of each network layer of the target model from front to back.

2. The method according to claim 1, wherein, The inference framework layer performs the operation of writing the weight data of each network layer of the target model from local storage to video memory based on the wake-up request, including: The inference framework layer executes an operation based on the wake-up request, writing the weight data of each network layer from local storage to video memory layer by layer in the order of each network layer of the target model from front to back.

3. The method according to claim 2, wherein, The inference framework layer, based on the wake-up request, performs the operation of writing the weight data of each network layer from local storage to video memory layer by layer in the order of each network layer of the target model from front to back, including: The inference framework layer determines the amount of weight data for each network layer of the target model; Based on the data volume and the remaining storage space in the video memory, a target storage space is determined in the video memory for storing the weight data of each network layer of the target model; and The weight data of each network layer of the target model are written from local storage to the target storage space of the video memory layer by layer, in the order from front to back.

4. The method according to claim 3, wherein, The step of determining the target storage space in the video memory for storing the weight data of each network layer of the target model, based on the amount of data and the remaining storage space in the video memory, includes: If the amount of data is less than the remaining storage space of the video memory, then a target storage space for storing the weight data of each network layer of the target model is determined from the remaining storage space; and If the amount of data is greater than the remaining storage space of the video memory, then the candidate model and the target network layer from which the candidate model is replaced are determined. Based on the remaining storage space of the video memory and the storage location of the weight data of the target network layer, the target storage space for storing the weight data of each network layer of the target model is determined.

5. The method according to claim 4, wherein, If the amount of data is greater than the remaining storage space of the video memory, then determining the candidate model and the target network layer from which the candidate model is replaced includes: If the amount of data exceeds the remaining storage space of the video memory, a candidate model is determined from the preset model cluster based on the swap-out policy of the service routing layer; and Based on the difference between the amount of data and the remaining storage space of the video memory, the size of the swap space is determined, and based on the size of the swap space, the target network layer from which the candidate model is swapped out is determined.

6. The method according to claim 5, wherein, The target network layers to be replaced are the last N network layers of the candidate model, and the total amount of weight data of the last N network layers is greater than the size of the replacement space, where N is an integer greater than or equal to 1.

7. The method according to claim 3, wherein, The step of determining the target storage space in the video memory for storing the weight data of each network layer of the target model, based on the amount of data and the remaining storage space in the video memory, includes: If the amount of data is greater than the remaining storage space of the video memory, a candidate model is determined, and the weight data of each network layer of the candidate model is replaced in reverse order until the size of the replaced space meets the data volume requirement.

8. The method according to claim 5, wherein, The swapping strategy includes at least: a swapping strategy based on usage frequency, a swapping strategy based on least recently used, or a swapping strategy based on the order in which video memory is written.

9. The method according to claim 1, wherein, The operating state also includes: partial dormancy state; The method further includes: When the target model is in a partially dormant state, the service routing layer sends a wake-up request to the inference framework layer, the wake-up request being used to request that the target model be put into an active state; and The inference framework layer performs the following first and second operations in parallel: based on the wake-up request, it performs the first operation of writing the weight data of the network layer to be swapped out of the target model from local storage to video memory, and the second operation of performing the inference calculation of the target task layer by layer in the order of each network layer of the target model from front to back.

10. The method according to claim 1, wherein, The multi-model management system further includes a service management layer, and the method further includes: During the model service deployment phase, the service management layer performs pre-startup preprocessing on each model in the preset model cluster, generating pre-stored model preprocessing data; the model preprocessing data includes at least one of: a pre-initialized execution environment, pre-parsed weight data, and pre-compiled computation data; and The pre-initialized execution environment includes a pre-initialized container environment and a general parallel computing architecture context; the pre-parsed weight data includes weight data conversion data after format conversion based on the target data format of the video memory; the pre-compiled computation data includes binary operator data compiled based on the model network structure.

11. The method according to claim 1, wherein, Before the inference framework layer performs the operation of writing the weight data of each network layer of the target model from local storage to video memory based on the wake-up request, it further includes: The inference framework layer retrieves and loads the pre-initialized execution environment corresponding to the target model from the local storage; after the pre-initialized execution environment is loaded, it retrieves the pre-compiled computation data corresponding to the target model from the local storage and loads it into the video memory; and The inference framework layer performs the operation of writing the weight data of each network layer of the target model from local storage to video memory based on the wake-up request, including: Write the pre-parsed weight data corresponding to each network layer of the target model, which is recorded in the local storage, from the local storage to the display memory.

12. The method according to claim 1, wherein, After the target model is in an active state, the method further includes: If the idle time of the target model reaches a preset sleep threshold, or if a sleep command is received, a sleep operation is performed on the target model. The sleep operation includes: storing the key-value cache data in the target model to local storage, and setting the running state of the target model to a sleep state.

13. A multi-model management system, comprising: At least one storage medium storing at least one instruction set for managing models that provide inference services; as well as At least one processor is communicatively connected to the at least one storage medium, wherein, when the multi-model management system is running, the at least one processor reads the at least one instruction set and implements the method as described in any one of claims 1-12 according to the instructions of the at least one instruction set.

14. A computer-readable non-volatile storage medium, wherein, The computer-readable non-volatile storage medium stores at least one instruction set, which, when executed by at least one processor, implements the method as described in any one of claims 1-12.

Images