A model scheduling method, system and electronic device

Abstract

The application discloses a model scheduling method and system and electronic equipment, and relates to the technical field of artificial intelligence. When model deployment is needed, a first model scheduling instance is used to realize rapid deployment of a target model. In the process of responding to a model calling request of the target model based on the first model scheduling instance, if the target model enters a high load mode, that is, a hot state switching condition is met, the target model is deployed based on a second model scheduling instance, and after the second model scheduling instance completes the deployment of the target model, the model calling request of the target model is responded to based on the second model scheduling instance. In this way, the model deployment time is shortened, and the utilization rate of model calculation resources is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a model scheduling method and system and an electronic device. BACKGROUND

[0002] With the wide application of large language models (LLMs) in natural language processing, code generation, intelligent question answering and other fields, how to efficiently deploy and schedule these models to provide stable inference services has become a key technical problem in artificial intelligence infrastructure.

[0003] In related technologies, a high-performance inference engine is usually used as a model scheduling tool. The high-performance inference engine is particularly suitable for high-load services to improve model performance. However, the model deployment time of the high-performance inference engine is relatively long, and the model computing resource demand is relatively high, while the model load is dynamically changing. If in the case of non-high load, the utilization rate of model computing resources will be reduced. SUMMARY

[0004] The present application provides a model scheduling method, system and electronic device to at least solve the problem of long model deployment time of the high-performance inference engine in related technologies, and reduced utilization rate of model computing resources in the case of non-high load.

[0005] The present application provides a model scheduling method, comprising:

[0006] obtaining model calling information of a user;

[0007] determining a target model to be scheduled according to the model calling information;

[0008] in the case that the target model is not deployed, deploying the target model based on a first model scheduling instance, and marking the state of the target model as a warm state;

[0009] sending a model calling request input by the user to the first model scheduling instance, and the first model scheduling instance responding to the model calling request based on the target model to obtain a model response result;

[0010] when the current load of the target model meets a preset hot state switching condition, deploying the target model based on a second model scheduling instance, and marking the state of the target model as a hot state;

[0011] sending the model calling request to the second model scheduling instance, and the second model scheduling instance responding to the model calling request based on the target model to obtain a model response result;

[0012] The model deployment speed of the first model scheduling instance is faster than the deployment speed of the second model scheduling instance, and the request response efficiency of the second model scheduling instance is higher than the request response efficiency of the first model scheduling instance.

[0013] The application further provides a model scheduling device, comprising:

[0014] The application further provides a model scheduling device, comprising:

[0015] The application further provides a model scheduling device, comprising:

[0016] The application further provides a model scheduling device, comprising:

[0017] The application further provides a model scheduling device, comprising:

[0018] The application further provides a model scheduling device, comprising:

[0019] The application further provides a model scheduling device, comprising:

[0020] The model deployment speed of the first model scheduling instance is faster than the deployment speed of the second model scheduling instance, and the request response efficiency of the second model scheduling instance is higher than the request response efficiency of the first model scheduling instance.

[0021] The application further provides a model scheduling system, comprising: a first model scheduling instance, a second model scheduling instance and a scheduler.

[0022] The first model scheduling instance and the second model scheduling instance are used for deploying a target model and responding to a model calling request based on the target model.

[0023] The scheduler schedules the first model scheduling instance and the second model scheduling instance based on any one of the model scheduling methods.

[0024] The application further provides an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any one of the model scheduling methods.

[0025] The application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program.

[0026] The application further provides a computer program product, comprising a computer program, and the computer program is executed by a processor to implement the steps of any one of the model scheduling methods.

[0027] According to the application, when model deployment is needed, the first model scheduling instance is used to implement rapid deployment of a target model, in the process of responding to a model calling request of the target model based on the first model scheduling instance, if the target model enters a high load mode, that is, the hot state switching condition is met, the second model scheduling instance is used to deploy the target model, and after the second model scheduling instance completes the deployment of the target model, the second model scheduling instance is used to respond to the model calling request of the target model, thereby shortening the model deployment time and improving the utilization rate of model computing resources. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the application, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0029] Figure 1 The flowchart of the model scheduling method provided by the embodiments of the application is shown in the figure.

[0030] Figure 2 The structural diagram of the model scheduling device provided by the embodiments of the application is shown in the figure.

[0031] Figure 3 The structural diagram of the model scheduling system provided by the embodiments of the application is shown in the figure.

[0032] Figure 4 The structural diagram of the scheduler provided by the embodiments of the application is shown in the figure.

[0033] Figure 5 The structural diagram of the electronic device provided by the embodiments of the application is shown in the figure. DETAILED DESCRIPTION

[0034] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only some of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the application.

[0035] It should be noted that in the description of the present application, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or apparatus including a series of elements includes not only those elements, but also other elements not explicitly listed, or inherent to such a process, method, article or apparatus. The terms "first", "second" and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0036] Large model inference services are usually run in server environments with high-performance GPU resources. Users submit inputs through APIs, and the system returns model inference results. To ensure service quality, models need to be long-term resident in GPU memory to reduce response delay, but this also brings higher computing resource occupation cost.

[0037] The current mainstream deployment methods include lightweight local deployment tools (such as Ollama) and high-performance inference engines (such as vLLM). Ollama is a model running tool for local development and testing scenarios, supporting one-click model service launch, easy deployment, low resource overhead, and is suitable for handling occasional and low-concurrency inference requests. However, its inference throughput and response delay are not good in high-concurrency scenarios. In contrast, vLLM (vectorized Large Language Model) uses memory optimization techniques such as PagedAttention to significantly improve batch processing capacity and generation speed, making it suitable for production-level high-load services. However, its startup time is relatively long (usually tens of seconds to load the model to the GPU), and it continuously occupies a large amount of memory and computing resources during resident operation, making it unsuitable for long-term retention of low-frequency models.

[0038] Therefore, in a multi-user, multi-model inference service platform, there is an urgent need for a technical mechanism that can quickly respond to low-frequency requests and improve the performance of high-frequency model services on demand.

[0039] To address the aforementioned technical problems, embodiments of this application provide a model scheduling method, system, and electronic device. The method includes: acquiring model call information from a user; determining a target model to be scheduled based on the model call information; deploying the target model based on a first model scheduling instance when the target model is not deployed, and marking the target model's state as warm; sending a model call request input by the user to the first model scheduling instance, the first model scheduling instance responding to the model call request based on the target model to obtain a model response result; when the current load of the target model meets preset hot state switching conditions, deploying the target model based on a second model scheduling instance, and marking the target model's state as hot; sending a model call request to the second model scheduling instance, the second model scheduling instance responding to the model call request based on the target model to obtain a model response result; wherein, the model deployment speed of the first model scheduling instance is faster than that of the second model scheduling instance, and the request response efficiency of the second model scheduling instance is higher than that of the first model scheduling instance. The method provided by the above scheme achieves rapid deployment of the target model by using a first model scheduling instance when model deployment is required. If the target model enters a high-load mode, i.e., meets the hot state switching condition, during the process of responding to the model call request of the target model based on the first model scheduling instance, the target model is deployed based on the second model scheduling instance. After the second model scheduling instance completes the deployment of the target model, it responds to the model call request of the target model based on the second model scheduling instance. This shortens the model deployment time and improves the utilization rate of model computing resources.

[0040] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0041] This application provides a model scheduling method for performing model scheduling on a multi-user, multi-model inference service platform. The execution subject of this application embodiment is an electronic device, such as a server, desktop computer, laptop computer, tablet computer, or other electronic devices that can be used for model scheduling.

[0042] like Figure 1 The diagram shown is a flowchart of the model scheduling method provided in an embodiment of this application. The method includes:

[0043] Step 101: Obtain the user's model call information.

[0044] Specifically, when a user logs into the inference service platform, it can be determined whether the user's current operation meets the preset model scheduling trigger conditions, thereby obtaining the user's model call information. This model call information may include the user's call records, etc.

[0045] Step 102: Determine the target model to be scheduled based on the model call information.

[0046] Specifically, the model most frequently invoked by the user can be selected as the target model based on the user's model invocation information.

[0047] The target model can be a large language model, which refers to a deep learning model trained with a large amount of text data, enabling the model to generate natural language text or understand the meaning of language text.

[0048] Step 103: If the target model is not deployed, deploy the target model based on the first model scheduling instance and mark the state of the target model as warm.

[0049] The first model scheduling instance can be a lightweight model deployment instance, such as an ollama instance. The first model scheduling instance has the characteristics of fast loading and low resource consumption, which is suitable for low-frequency, bursty request scenarios, but the response latency is high and the concurrency capability is limited.

[0050] Step 104: The user-inputted model call request is sent to the first model scheduling instance. The first model scheduling instance responds to the model call request based on the target model and obtains the model response result.

[0051] Specifically, the first model scheduling instance performs inference calculations based on the deployed target model to respond to model call requests, generate model response results such as dialogue text, and feed the obtained model response results back to the user.

[0052] Step 105: When the current load of the target model meets the preset hot state switching conditions, deploy the target model based on the second model scheduling instance and mark the state of the target model as hot state.

[0053] The second model scheduling instance can use a high-performance inference engine, such as a vLLM instance. The second model scheduling instance uses PagedAttention memory optimization technology, which has high throughput and low response latency characteristics. It is suitable for high-frequency and continuous request scenarios (high-load scenarios), but it takes a long time to start up and consumes a lot of video memory and computing resources when running in the background.

[0054] Step 106: The model invocation request is sent to the second model scheduling instance. The second model scheduling instance responds to the model invocation request based on the target model and obtains the model response result.

[0055] Among them, the deployment speed of the first model scheduling instance is faster than that of the second model scheduling instance, and the request response efficiency of the second model scheduling instance is higher than that of the first model scheduling instance.

[0056] Specifically, after the second model scheduling instance completes the deployment of the target model, i.e., is ready, all new target model call requests are redirected to the second model scheduling instance. For requests that are still being processed in the first model scheduling instance at the moment of switching, the first model scheduling instance is allowed to continue processing and return results to avoid request interruption. After all old requests have been processed, the second model scheduling instance can unload the target model to release the memory resources occupied by the target model in the second model scheduling instance.

[0057] Among these, heartbeat detection or health checks can be used to confirm whether the second model scheduling instance is ready.

[0058] Specifically, the switching between the second and first model scheduling instances enables seamless service switching and a smooth transition. Users are unaware of changes in request routing, preventing request loss and sudden increases in response latency, thus ensuring service continuity. By using the second model scheduling instance to guarantee performance under high load and the first model scheduling instance to save resources under low load, model performance is maintained while avoiding waste of model computing resources.

[0059] Specifically, by setting call frequency and time window status judgment rules, the system automatically identifies models entering the high-frequency call phase and migrates them to a high-performance second model scheduling instance to ensure service quality. It supports dynamic management of multiple instances and seamless service switching. After the second model scheduling instance is created and before the first model scheduling instance is shut down, the scheduler keeps both services running in parallel, completing health checks and request routing transitions to ensure that user requests are not lost or interrupted, achieving seamless service switching.

[0060] Based on the above embodiments, as an implementable approach, in one embodiment, determining the target model to be scheduled according to model invocation information includes:

[0061] Step 1021: Based on the user's model usage history represented by the model call information, determine the model that the user uses most frequently;

[0062] Step 1021: Select the model most frequently used by users as the target model to be scheduled.

[0063] Specifically, the model most frequently used by the user can be identified as the model most frequently invoked by the user, thus determining the target model to be scheduled. For example, if the user model uses historical records to indicate that the user has invoked the dialogue model 20 times and the code generation model 5 times, then the dialogue model is identified as the target model to be scheduled. If the code generation model is also invoked 20 times, but the dialogue model has been invoked more times in the past hour than the code generation module, then the dialogue model is still considered the target model to be scheduled.

[0064] Specifically, when a user is logging into the inference service platform for the first time, the system retrieves the user's core requirement tags, which include tags for code developers and users with whom they engage in daily conversations. Furthermore, based on a set of recently used models compiled from platform-wide user call data statistics and the user's core requirement tags, the system determines the user's target model. This ensures the accuracy of the target model determination and lays the foundation for improving the user's model response efficiency.

[0065] Based on the above embodiments, as an implementable approach, in one embodiment, deploying the target model based on the first model scheduling instance includes:

[0066] Step 1031: Apply for the first model running area on the preset model running device;

[0067] Step 1032: Based on the first model scheduling instance, load the model file of the target model into the corresponding first model runtime area;

[0068] Step 1033: Start the request and response service of the first model scheduling instance and configure the request and response port corresponding to the target model;

[0069] Step 1034: Register the request-response port for routing so that the first model scheduling instance can receive model invocation requests through the request-response port.

[0070] The preset model running devices include accelerator cards such as GPUs. The first model running area is a portion of the GPU's video memory used to store the model file of the target model, which includes the model parameters of the target model.

[0071] Specifically, after the model file is loaded, the first model scheduling instance starts the request-response service (REST API service) and assigns an independent request-response port to the target model to ensure that subsequent requests received through this port can directly trigger the inference calculation of the target model. It also establishes the association between the target model, the first model runtime area, and the request-response port, storing this association information in a routing table. When a user requests a call to the target model, the routing table is queried to determine whether the request should be forwarded to the request-response port, allowing the first model scheduling instance to receive the model call request through the request-response port.

[0072] Based on the above embodiments, as an implementable approach, in one embodiment, the method further includes:

[0073] Step 201: During the process of the first model scheduling instance responding to model call requests based on the target model, the call time of each model call request is recorded;

[0074] Step 202: Determine the number of times the target model is called within the first monitoring period based on the call time of each model call request;

[0075] Step 203: When the number of calls to the target model within the first monitoring period indicates that the current load of the target model has reached the preset high load judgment standard, determine that the current load of the target model meets the preset hot state switching condition.

[0076] It should be noted that the method provided in this application embodiment is applied to a scheduler, which includes a status judgment module.

[0077] Specifically, during the process of the first model scheduling instance processing the target model call request, the scheduler synchronously records the timestamp of the received request through the status judgment module to record the call time of each model call request. Based on the call counter within the first monitoring period T1 (e.g., within 5 minutes), the number of calls to the target model within the first monitoring period is counted. If the number of calls to the target model within the first monitoring period is greater than a preset first threshold (e.g., 10 times), it is determined that the current load of the target model meets the preset hot state switching conditions.

[0078] This approach avoids premature switching, which wastes resources by starting a second model scheduling instance during low load, or switching too late and still using the first model scheduling instance during high load, thus degrading the user experience. By triggering a hot state switch only when the model actually enters a high load, the approach avoids premature switching, which wastes resources by starting a second model scheduling instance during low load, or switching too late and still using a first model scheduling instance during high load.

[0079] It should be further noted that the scheduler also supports model sharing judgment in a multi-user environment. For example, if two users frequently use the same model, even if the individual call frequency does not reach the hot state threshold, the system can combine global call statistics to comprehensively determine whether to start the deployment of the first model scheduling instance in advance, avoiding the delay caused by repeated warming-up processes. The call count statistics in this embodiment include the number of calls to the target model by a single user and the total number of calls to the target model by multiple users globally.

[0080] The scheduler supports user-based model preference recording, which can preload users' frequently used models into the first model scheduling instance when they log in, enabling personalized preheating; it also supports global call statistics, which can deploy popular models in a shared second model scheduling instance, avoiding the repeated independent upgrades of the same model by multiple users.

[0081] Based on the above embodiments, as an implementable approach, in one embodiment, the method further includes:

[0082] Step 301: During the process of the second model scheduling instance responding to model call requests based on the target model, the call time of each model call request is recorded;

[0083] Step 302: Determine the number of times the target model is called within the second monitoring period based on the call time of each model call request;

[0084] Step 303: When the number of calls to the target model in the second monitoring cycle indicates that the current load of the target model has reached the preset temperature load judgment standard, it is determined that the current load of the target model meets the preset temperature state switching condition.

[0085] Step 304: Re-deploy the target model based on the first model scheduling instance, and mark the state of the target model as warm.

[0086] Step 305: The model invocation request is sent to the first model scheduling instance. The first model scheduling instance responds to the model invocation request based on the target model and obtains the model response result.

[0087] Specifically, during the period when the second model scheduling instance acts as a request-response end (during the process of the second model scheduling instance responding to model call requests based on the target model), the scheduler continuously monitors the target's call behavior through the status judgment module. If the number of calls is lower than a preset second threshold (e.g., 5 times) within the second monitoring period T2 (e.g., within 30 minutes), it is determined that the current load of the target model meets the preset warm state switching condition, thus triggering the degradation process. The scheduler first sends a target loading instruction to the first model scheduling instance to enable the first model scheduling instance to redeploy the target model, ensuring that it can resume operation in a warm state environment. After the first model scheduling instance successfully loads the target and returns a ready signal, the scheduler reroutes subsequent requests back to the first model scheduling instance; then it sends a termination instruction to the second model scheduling instance, shutting down the instance and releasing the computing resources such as GPUs it occupies; and updates the target model's status to a warm state.

[0088] The scheduler maintains a status table entry for the target model locally, marking it as either currently deployed in the first model scheduling instance with a "warm" status, or currently deployed in the second model scheduling instance with a "hot" status. It also records the first call time and the number of calls within the corresponding period. The call count is based on a call counter, which is initialized to 1.

[0089] Furthermore, in one embodiment, after sending the model invocation request to the second model scheduling instance, a model unload command can be sent to the second model scheduling instance to unload the target model from the second model scheduling instance.

[0090] Specifically, by sending a model unload command to the second model scheduling instance, the target model is unloaded from the second model scheduling instance, thereby releasing the resources occupied by the target model in the second model scheduling instance.

[0091] Accordingly, in one embodiment, when the current load of the target model meets the preset cold state switching conditions, a model unload command is sent to the first model scheduling instance to unload the target model from the first model scheduling instance.

[0092] Specifically, in the process of responding to model call requests based on the target model, the call time of each model call request is recorded in the first model scheduling instance as the request response end; the number of times the target model is called in the third monitoring period is determined based on the call time of each model call request; when the number of times the target model is called in the third monitoring period indicates that the current load of the target model has reached the preset cold load judgment standard, the current load of the target model is determined to meet the preset cold state switching condition.

[0093] For example, if a target model in a warm state is not accessed by any user within the third monitoring period T3, the scheduler determines it to be a long-term idle model and performs a complete unloading. A model unloading command is sent to the first model scheduling instance to unload the target model from the first model scheduling instance, releasing the resources occupied by the target model in the first model scheduling instance and clearing the model's status information in the scheduler, that is, clearing the model's status table entry in the scheduler.

[0094] By setting a warm idle timeout mechanism (T3), the model is only unloaded from the first model scheduling instance after it has not been used for a long time. This not only prevents repeated cold loading caused by excessive release, but also avoids unnecessary long-term residence.

[0095] Specifically, in one embodiment, when the target model is unloaded from the first model scheduling instance or the first model scheduling instance, the metadata of the target model is retained; when the first model scheduling instance or the first model scheduling instance redeploys the target model, the target model is deployed according to the retained metadata of the target model.

[0096] The metadata of the target model includes a unique model identifier (such as Model_A_ID), model type (such as dialogue model, code generation model), lightweight runtime file storage path, and deployment resource specifications.

[0097] Specifically, the scheduler sends a model unload command to the first model scheduling instance. This command only requires releasing the resources of the first model runtime area occupied by the target model. Simultaneously, the scheduler's instance management module extracts the model's metadata from the first model scheduling instance and stores it in the scheduler's local model metadata database, binding it to the target model's unique identifier to form a mapping relationship. When the target model needs to be deployed again on the first model scheduling instance, the scheduler queries the model metadata database to retrieve the previously retained metadata based on the target model's unique identifier. The scheduler then sends a model load command to the first model scheduling instance, directly injecting reused metadata so that the first model scheduling instance loads the lightweight model file from the storage path specified by the metadata, avoiding re-downloading or path lookup. Furthermore, it requests a first model runtime area of ​​the appropriate size according to the resource specifications recorded in the metadata, eliminating the need to reassess the target model's resource requirements and shortening the time consumed in redeploying the target model.

[0098] The model scheduling method provided in this application embodiment obtains user model call information; determines the target model to be scheduled based on the model call information; if the target model is not deployed, deploys the target model based on a first model scheduling instance and marks the target model's state as warm; sends the user-input model call request to the first model scheduling instance, which responds to the model call request based on the target model and obtains a model response result; when the current load of the target model meets the preset hot state switching conditions, deploys the target model based on a second model scheduling instance and marks the target model's state as hot; sends the model call request to the second model scheduling instance, which responds to the model call request based on the target model and obtains a model response result; wherein, the model deployment speed of the first model scheduling instance is faster than that of the second model scheduling instance, and the request response efficiency of the second model scheduling instance is higher than that of the first model scheduling instance. The method provided by the above scheme achieves rapid deployment of the target model by using a first model scheduling instance when model deployment is required. If the target model enters a high-load mode, i.e., meets the hot state switching condition, during the process of responding to the model call request of the target model based on the first model scheduling instance, the target model is deployed based on the second model scheduling instance. After the second model scheduling instance completes the deployment of the target model, it responds to the model call request of the target model based on the second model scheduling instance. This shortens the model deployment time and improves the utilization rate of model computing resources.

[0099] Furthermore, for models used infrequently, a first model scheduling instance is used for warm deployment, which consumes less resources than the second model scheduling instance. This allows for more flexible deployment of multiple models on a single GPU and supports rapid deployment and decommissioning. By loading models only when needed, GPU resource idleness caused by reserving the second model scheduling instance for potential long-term use is avoided. When a user logs in or calls a model for the first time, the system immediately loads the model in the first model scheduling instance for rapid warm-up; once usage frequency increases, it smoothly migrates to the second model scheduling instance. This two-tier deployment strategy ensures acceptable response latency for users on their first visit (lower than the cold start loading time of a pure second model scheduling instance), avoiding the long wait caused by directly launching the second model scheduling instance from disk. The method provided in this application achieves dynamic resource allocation, on-demand performance enhancement, and user experience assurance for large-scale language model inference services under limited computing power, resolving the technical contradiction between low resource utilization and poor response performance in traditional static deployment schemes.

[0100] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0101] The embodiments of this application also provide a model scheduling apparatus for executing the model scheduling method provided in the above embodiments.

[0102] like Figure 2 The diagram shown is a schematic representation of the model scheduling device provided in an embodiment of this application. The model scheduling device 20 includes: an acquisition module 201, a determination module 202, a first deployment module 203, a first response module 204, a second deployment module 205, and a second response module 206.

[0103] The system comprises the following modules: an acquisition module for acquiring user model call information; a determination module for determining the target model to be scheduled based on the model call information; a first deployment module for deploying the target model based on a first model scheduling instance when the target model is not deployed, and marking the target model's status as warm; a first response module for sending the user-input model call request to the first model scheduling instance, which then responds to the model call request based on the target model and obtains the model response result; a second deployment module for deploying the target model based on a second model scheduling instance when the target model's current load meets preset hot state switching conditions, and marking the target model's status as hot; and a second response module for sending the model call request to the second model scheduling instance, which then responds to the model call request based on the target model and obtains the model response result.

[0104] Among them, the deployment speed of the first model scheduling instance is faster than that of the second model scheduling instance, and the request response efficiency of the second model scheduling instance is higher than that of the first model scheduling instance.

[0105] For a description of the features in the embodiment corresponding to the model scheduling device, please refer to the relevant description of the embodiment corresponding to the model scheduling method, which will not be repeated here.

[0106] Embodiments of this application also provide a model scheduling system for executing the model scheduling method provided in the above embodiments.

[0107] like Figure 3 The diagram shown is a structural schematic of the model scheduling system provided in an embodiment of this application. The model scheduling system includes: a first model scheduling instance, multiple second model scheduling instances that can be started and stopped independently, and a scheduler.

[0108] The first model scheduling instance and the second model scheduling instance are used to deploy the target model and to call the model based on the target model's model call request; the scheduler schedules the first model scheduling instance and the second model scheduling instance based on the model scheduling method provided in the above embodiments.

[0109] Specifically, the second model scheduling instance starts and stops in environments such as bare metal, containerized environment, and business scheduling platform according to the scheduler's instructions; the first model scheduling instance is a resident instance; external requests and user login information are passed to the scheduler from external systems, and the user login information includes the user's model call information, while the external request is the model call request input by the user.

[0110] Specifically, such as Figure 4 The diagram shown illustrates the structure of the scheduler instantiated in this application. The scheduler includes a request receiving module, a user login information collection module, a routing and forwarding module, a deployment decision module, an instance management module, a status monitoring module, and a model status determination module. In the above embodiment, the status determination module includes both a status monitoring module and a model status determination module. The modules within the scheduler collaborate through internal function calls. The user management module receives login events and triggers preloading; the status monitoring module periodically collects model call data; the deployment decision module determines state transitions based on a rule engine; the instance management module executes lifecycle operations for the first and second model scheduling instances; and the request receiving module and the routing and forwarding module complete the receiving, forwarding, and response return of requests.

[0111] The scheduler is the control center of the entire system. It receives user requests through the network interface and maintains the running status information of all models, including but not limited to the last call time of each model, the call frequency per unit time, the current deployment location (Ollama or vLLM), and the preferences or usage history of the user to which it belongs.

[0112] An Ollama instance is a long-running, lightweight model service node with model loading, unloading, and basic inference capabilities. Ollama is suitable for low-frequency, bursty request scenarios with low deployment overhead, but its response latency is relatively high and its concurrency capabilities are limited. This instance supports receiving model loading (online) and unloading (offline) commands from the scheduler via a REST API or a local command interface.

[0113] vLLM instances are a group of high-performance inference service processes that can be dynamically created and destroyed, with each instance dedicated to running a single large language model. vLLM instances employ the PagedAttention mechanism to optimize attention computation, significantly improving throughput and response speed, making them suitable for high-frequency, continuous request scenarios. However, each vLLM instance has a long startup time, and its persistent operation continuously consumes significant GPU resources. Each vLLM instance can be deployed via containerization (such as Docker) or as a process, supporting remote start / stop control and health status reporting.

[0114] For a description of the features in the embodiment corresponding to the model scheduling system, please refer to the relevant descriptions in the embodiment corresponding to the model scheduling method, which will not be repeated here.

[0115] Embodiments of this application also provide an electronic device, such as... Figure 5 The diagram shown is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, including a processor 501 and a memory 502. The memory 502 stores a computer program, and the processor 501 is configured to run the computer program to execute the steps in any of the above-described model scheduling method embodiments.

[0116] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described model scheduling method embodiments at runtime.

[0117] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0118] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described model scheduling method embodiments.

[0119] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described model scheduling method embodiments.

[0120] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0121] The above provides a detailed description of a model scheduling method, system, and electronic device provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A model scheduling method, characterized in that, include: Obtain the user's model call information; Based on the model invocation information, determine the target model to be scheduled; If the target model is not deployed, the target model is deployed based on the first model scheduling instance, and the state of the target model is marked as warm. The user-inputted model call request is sent to the first model scheduling instance, and the first model scheduling instance responds to the model call request based on the target model to obtain the model response result; When the current load of the target model meets the preset hot state switching conditions, the target model is deployed based on the second model scheduling instance, and the state of the target model is marked as hot state; The model invocation request is sent to the second model scheduling instance, and the second model scheduling instance responds to the model invocation request based on the target model to obtain the model response result; The deployment speed of the first model scheduling instance is faster than that of the second model scheduling instance, and the request response efficiency of the second model scheduling instance is higher than that of the first model scheduling instance.

2. The model scheduling method according to claim 1, characterized in that, The step of determining the target model to be scheduled based on the model invocation information includes: Based on the user's model usage history represented by the model call information, determine the model that the user uses most frequently; The model used most frequently by the user will be selected as the target model to be scheduled.

3. The model scheduling method according to claim 1, characterized in that, The deployment of the target model based on the first model scheduling instance includes: Apply for the first model running area on the preset model running device; Based on the first model scheduling instance, the model file of the target model is loaded into the corresponding first model running area; Start the request-response service of the first model scheduling instance and configure the request-response port corresponding to the target model; The request-response port is registered for routing so that the first model scheduling instance can receive model invocation requests through the request-response port.

4. The model scheduling method according to claim 1, characterized in that, The method further includes: During the process of the first model scheduling instance responding to the model call request based on the target model, the call time of each model call request is recorded; Based on the call time of each model call request, determine the number of times the target model is called within the first monitoring period; When the number of calls to the target model within the first monitoring period indicates that the current load of the target model has reached a preset high load judgment standard, it is determined that the current load of the target model meets the preset hot state switching condition.

5. The model scheduling method according to claim 1, characterized in that, The method further includes: During the process of the second model scheduling instance responding to the model call request based on the target model, the call time of each model call request is recorded; Based on the call time of each model call request, determine the number of times the target model is called within the second monitoring period; When the number of times the target model is called within the second monitoring cycle indicates that the current load of the target model has reached the preset temperature load judgment standard, it is determined that the current load of the target model meets the preset temperature state switching condition. Re-deploy the target model based on the first model scheduling instance, and mark the state of the target model as warm. The model invocation request is sent to the first model scheduling instance, and the first model scheduling instance responds to the model invocation request based on the target model to obtain the model response result.

6. The model scheduling method according to claim 1, characterized in that, After sending the model invocation request to the second model scheduling instance, the method further includes: Send a model unload command to the first model scheduling instance to unload the target model from the first model scheduling instance.

7. The model scheduling method according to claim 1, characterized in that, The method further includes: When the current load of the target model meets the preset cold state switching conditions, a model unload command is sent to the first model scheduling instance to unload the target model from the first model scheduling instance.

8. The model scheduling method according to claim 6 or 7, characterized in that, The method further includes: When the target model is unloaded from the first model scheduling instance or the second model scheduling instance, the metadata of the target model is retained; When the first model scheduling instance or the second model scheduling instance redeploys the target model, the target model is deployed according to the retained metadata of the target model.

9. A model scheduling system, characterized in that, include: First model scheduling instance, second model scheduling instance, and scheduler; The first model scheduling instance and the second model scheduling instance are used to deploy the target model and respond to model invocation requests based on the target model; The scheduler schedules the first model scheduling instance and the second model scheduling instance based on the model scheduling method as described in any one of claims 1 to 8.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for implementing the steps of the model scheduling method as described in any one of claims 1 to 8 when executing the computer program.

Images