Model scheduling method, electronic device, and computer program product

Abstract

The present disclosure provides a model scheduling method, an electronic device and a computer program product, relating to the technical field of artificial intelligence, and the method comprises: obtaining operation and maintenance monitoring information of a plurality of models in a preset period; performing calling heat analysis on the operation and maintenance monitoring information of the plurality of models from each strategy dimension in at least one preset strategy dimension, to obtain a heat analysis report of each model in the plurality of models; generating a model scheduling strategy table of the plurality of models based on the heat analysis report of each model in the plurality of models; and performing model scheduling on each model in the plurality of models based on the model scheduling strategy table of the plurality of models and the current operation and maintenance monitoring information of each model in the plurality of models. The model scheduling is used to migrate the model into or out of a reasoning resource pool. The method is used to improve the utilization rate of processor resources of a beam model.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a model scheduling method, electronic device, and computer program product. Background Technology

[0002] In application scenarios of model inference services, the call frequency of different models is uneven, exhibiting a typical long-tail distribution characteristic. That is, a few popular models undertake the vast majority of inference requests, while a large number of unpopular models are only called occasionally.

[0003] In related technologies, when deploying models, regardless of whether the model is popular or unpopular, a model-exclusive processor (e.g., graphics processing unit, GPU) resource allocation method is adopted. Once the processor resource is allocated, it will continue to occupy the resource unless the model terminates the service.

[0004] However, if a large number of unpopular models occupy processor resources for a long time, it will lead to low utilization of processor resources. Summary of the Invention

[0005] This disclosure provides a model scheduling method, electronic device, and computer program product that can improve the utilization of processor resources for deploying models.

[0006] In a first aspect, this disclosure provides a model scheduling method, which includes: acquiring operation and maintenance monitoring information of multiple models within a preset period; performing call popularity analysis on the operation and maintenance monitoring information of multiple models from each of at least one preset strategy dimension to obtain a popularity analysis report of each model in the multiple models; generating a model scheduling strategy table for multiple models based on the popularity analysis reports of each model in the multiple models; and performing model scheduling on each model in the multiple models based on the model scheduling strategy table of multiple models and the current operation and maintenance monitoring information of each model in the multiple models; the model scheduling is used to migrate models into or out of the inference resource pool.

[0007] Secondly, this disclosure provides a model scheduling device, which includes: an acquisition module and a processing module; the acquisition module is used to acquire operation and maintenance monitoring information of multiple models within a preset period; the processing module is used to perform call popularity analysis on the operation and maintenance monitoring information of multiple models from each of at least one preset strategy dimension to obtain a popularity analysis report of each model in the multiple models; based on the popularity analysis reports of each model in the multiple models, a model scheduling strategy table of multiple models is generated; based on the model scheduling strategy table of multiple models and the current operation and maintenance monitoring information of each model in the multiple models, model scheduling is performed on each model in the multiple models; model scheduling is used to migrate models into or out of the inference resource pool.

[0008] Thirdly, this disclosure provides an electronic device comprising: a processor and a memory; the memory storing instructions executable by the processor; the processor being configured to, when executing the instructions, cause the electronic device to perform the method provided in the first aspect above.

[0009] Fourthly, this disclosure provides a computer-readable storage medium, including a non-transitory computer-readable storage medium on which computer instructions are stored; when the computer instructions are executed in an electronic device, they cause the electronic device to perform the method provided in the first aspect above.

[0010] Fifthly, this disclosure provides a computer program product including computer instructions that, when executed on an electronic device, cause the electronic device to perform the method provided in the first aspect above.

[0011] This disclosure utilizes continuous collection of operation and maintenance monitoring information and performs heat analysis within a preset period. Based on the heat analysis report, a model scheduling strategy table is generated for model scheduling, migrating models into or out of the inference resource pool. Compared to the static binding of models and processor resources in existing technologies, model scheduling through the model scheduling strategy table breaks the static binding between models and processor resources. For less popular models that are in a low-call or idle state, they can be proactively migrated out of the inference resource pool to release processor resources. For popular models that are experiencing a call peak, they can be proactively migrated into the inference resource pool to ensure inference services. This achieves the reuse and transfer of processor resources among different models, thereby improving the utilization rate of processor resources for deployed models. Attached Figure Description

[0012] The accompanying drawings are provided to further understand the technical solutions of the present invention and constitute a part of the specification. They are used together with the embodiments of this disclosure to explain the technical solutions of the present invention and do not constitute a limitation on the technical solutions of the present invention.

[0013] Figure 1 A schematic diagram of the composition of the model scheduling system provided in the embodiments of this disclosure; Figure 2 A schematic diagram of a cache resource pool provided in an embodiment of this disclosure; Figure 3 This is a schematic diagram illustrating the execution of the scheduling strategy of the executor provided in an embodiment of this disclosure; Figure 4 A flowchart for generating scheduling strategies provided in this embodiment of the disclosure; Figure 5 A flowchart illustrating the model scheduling method provided in this embodiment of the disclosure; Figure 6Another flowchart illustrating the model scheduling method provided in this embodiment of the disclosure; Figure 7 This is a schematic diagram of model state switching provided in an embodiment of the present disclosure; Figure 8 Another flowchart illustrating the model scheduling method provided in this embodiment of the disclosure; Figure 9 This is a schematic diagram of the initial loading process of the model provided in the embodiments of this disclosure; Figure 10 This is a schematic diagram of the model migration operation process provided in the embodiments of this disclosure; Figure 11 This is a schematic diagram of the secondary migration operation process of the model provided in the embodiments of this disclosure; Figure 12 A schematic diagram of the model termination workflow provided in this embodiment of the disclosure; Figure 13 A schematic diagram of the composition of the model scheduling device provided in the embodiments of this disclosure; Figure 14 This is a schematic diagram illustrating the composition of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0014] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this disclosure.

[0015] Unless the context otherwise requires, throughout the specification and claims, the term "comprise" and other forms such as the third-person singular "comprises" and the present participle "comprising" are interpreted as open-ended and encompassing, meaning "including, but not limited to." In the description of the specification, terms such as "one embodiment," "some embodiments," "exemplary embodiments," "example," "specific example," or "some examples" are intended to indicate that a particular feature, structure, material, or characteristic associated with that embodiment or example is included in at least one embodiment or example of this disclosure. The illustrative representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics mentioned may be included in any suitable manner in any one or more embodiments or examples.

[0016] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this disclosure, unless otherwise stated, "a plurality of" means two or more.

[0017] In this disclosure, the terms "exemplarily" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplarily" or "for example" in this disclosure should not be construed as being more preferred or advantageous than other embodiments or designs. Rather, the use of the terms "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0018] In addition, the use of “based on” implies openness and inclusivity, because processes, steps, calculations or other actions “based on” one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated.

[0019] Current processor (such as GPU) hardware exhibits a divergence in virtualization capabilities. Some processors support virtualization technology, enabling the shared deployment of multiple models on a single or multi-processor system. However, processors that do not support virtualization require the exclusive use of the entire processor's computing resources, even when deploying models with only a low number of parameters.

[0020] Meanwhile, in application scenarios of model (such as Large Language Model (LLM)) inference services, the call frequency of different models is uneven, exhibiting a typical long-tail distribution characteristic. That is, a few popular models undertake the vast majority of inference requests, while a large number of unpopular models are only called occasionally.

[0021] In related technologies, when deploying models, regardless of whether the model is popular or unpopular, a dedicated processor resource allocation method is adopted. Once the processor resource is allocated, it will continue to occupy the resource unless the model terminates its service.

[0022] However, if a large number of unpopular models occupy processor resources for a long time, it will lead to low processor resource utilization.

[0023] Based on this, embodiments of this disclosure provide a model scheduling method, electronic device, and computer program product, which can perform call popularity analysis on models and dynamically generate scheduling strategies to migrate models into or out of the inference resource pool as needed, thereby avoiding the ineffective occupation of inference resources by unpopular models and improving the utilization rate of inference resources.

[0024] The following description is provided in conjunction with the accompanying drawings.

[0025] Figure 1 This is a schematic diagram illustrating the composition of the model scheduling system provided in an embodiment of this disclosure. Figure 1 As shown, the model scheduling system may include: an inference resource pool 100, a cache resource pool 200, an inference service system 300, a message session capability recording center 400, a model scheduling policy generation center 500, and a model scheduler 600. The inference service system 300 loads multiple models ( Figure 1 (Taking models 1 to k as examples), the models are used to provide inference services to external parties.

[0026] Inference resource pool 100 is used to provide inference resources for the model. For example, such as... Figure 1 As shown, the inference resource pool 100 may include multiple extensible processing units (XPUs). Figure 1Examples of XPUs are given below: XPU0, XPU1, ..., XPUn. Specifically, an XPU may include a Graphics Processing Unit (GPU), a Central Processing Unit (CPU), a Neural Processing Unit (NPU), a Data Processing Unit (DPU), or a Tensor Processing Unit (TPU), etc. This disclosure does not limit the specific types of processors providing inference resources in the inference resource pool 100.

[0027] The cache resource pool 200 is created based on memory. The cache resource pool 200 stores the model file information and model environment configuration information of each of the aforementioned models. Figure 1 (The example shown is the model information from model 1 to model k).

[0028] For example, Figure 2 This is a schematic diagram of a cache resource pool provided in an embodiment of this disclosure. Figure 2 As shown, the cache resource pool 200 can store the model environment configuration information of model A. Figure 2 (Taking A1 to A4 as examples), model file information of model A ( Figure 2 (Taking A5 to A7 as examples), Model B's model environment configuration information ( Figure 2 (Examples are shown below, using B1 to B4 as examples) and model file information for model B ( Figure 2 (Examples are shown for models B5 to B7). During model scheduling, the model environment configuration information of model A and model B in cache resource pool 200 can be loaded into the executor of inference service system 300, and the model file information of model A and model B in cache resource pool 200 can be loaded into the XPU of inference resource pool 100.

[0029] The inference service system 300 is used to receive inference requests and perform inference on the inference requests based on the models loaded in the inference service system 300 to generate inference responses.

[0030] The inference service system 300 includes multiple executors. Figure 1 The example shown is executor 1 to executor n. Each executor has one or more models loaded.

[0031] In some embodiments, the executor is also used to perform model scheduling operations on the model. As an example, the executor is used to receive scheduling instructions sent by the model scheduler 600 and perform model scheduling operations on the model based on the scheduling instructions.

[0032] For example, taking the scheduling instruction as the model migration instruction, the executor can call the interface that connects to the cache resource pool 200 to obtain the model file information and model environment configuration information, load the model environment configuration information, and send the model file information to the XPU in the inference resource pool 100 to complete the model migration or loading.

[0033] For example, taking the scheduling instruction as a model migration instruction, the executor can unload the model environment configuration information of the loaded model to the cache resource pool 200, and initiate a model unloading operation to the XPU in the inference resource pool 100, unloading the model file information of the model to the executor, and unloading the model file information to the cache resource pool 200, so as to complete the model migration or unloading.

[0034] For example, Figure 3 This is a schematic diagram illustrating the execution of the scheduling strategy of the executor provided in an embodiment of this disclosure. Figure 3 As shown, the executor and cache resource pool 200 are deployed in the same Dynamic Random-Access Memory (DRAM). The executor sends the collected operation and maintenance monitoring information to the model scheduler 600, and the model scheduler 600 sends scheduling instructions to the executor. The executor interacts with the XPUs in the cache resource pool 200 and the inference resource pool 100, thereby migrating model file information from the cache resource pool 200 to the inference resource pool 100, and migrating model environment configuration information from the cache resource pool 200 to the executor; or, migrating model file information from the inference resource pool 100 to the cache resource pool 200, and migrating model environment configuration information deployed in the executor to the cache resource pool 200.

[0035] In some embodiments, before model scheduling, a first communication link between the cache resource pool 200 and the inference resource pool 100 and a second communication link between the cache resource pool 200 and the executor may be established.

[0036] For example, the first communication link may include a first sub-link between the cache resource pool (DRAM) and the executor (DRAM) (such as a Memory Bus Compute Express Link (CXL) or a High Bandwidth Memory (HBM) interface, etc. This disclosure does not limit the specific implementation of the first sub-link), and a second sub-link between the executor (DRAM) and the inference resource pool (XPU) (such as a Peripheral Component Interconnect Express (PCIe) or CXL, etc. This disclosure does not limit the specific implementation of the second sub-link).

[0037] In some embodiments, the actuator is also used to collect operation and maintenance monitoring information of multiple models and send the collected operation and maintenance monitoring information to the message session capability recording center 400 through the model scheduler 600.

[0038] The message session capability recording center 400 is used to obtain the operation and maintenance monitoring information of multiple models and send the operation and maintenance monitoring information of multiple models to the model scheduling strategy generation center 500.

[0039] The model scheduling strategy generation center 500 is used to perform call heat analysis on the operation and maintenance monitoring information of multiple models from each of the at least one preset strategy dimension, obtain heat analysis reports of each model in the multiple models, generate model scheduling strategy tables for multiple models based on the heat analysis reports of each model in the multiple models, and then send the model scheduling strategy tables of multiple models to the model scheduler 600.

[0040] For example, Figure 4 A flowchart for generating the scheduling strategy provided in the embodiments of this disclosure is shown. Figure 4 As shown, the message session capability recording center 400 can send operation and maintenance monitoring information of multiple models to the model scheduling strategy generation center 500. The model scheduling strategy generation center 500 includes a model scheduling strategy selection module. The model scheduling strategy selection module analyzes the heat index information in the operation and maintenance monitoring information and generates an analysis report. Based on the analysis report, the model scheduling strategy generation center 500 generates a model scheduling strategy table and sends the model scheduling strategy table to the model scheduler 600.

[0041] The model scheduler 600 is used to schedule models based on model scheduling strategy tables of multiple models and current operation and maintenance monitoring information of each model in multiple models.

[0042] As an example, the model scheduler 600 can generate scheduling instructions based on the current operation and maintenance monitoring information and model scheduling strategy table of each model, and send the scheduling instructions to the executor in the inference service system 300 so that the executor can perform model scheduling operations based on the scheduling instructions.

[0043] The execution entity of the model scheduling method proposed in this disclosure is a model scheduling device, which can be an electronic device with computing processing capabilities, such as a server or a computer. In some embodiments, the model scheduling device can also be a processor (e.g., CPU) in the aforementioned electronic device; or, the model scheduling device can also be a software system deployed in the aforementioned electronic device (e.g., the one described above). Figure 1 The model scheduling device may be the model scheduling system or platform shown in the diagram; or it may be a functional module in the aforementioned electronic device used to execute the model scheduling method. This disclosure does not limit the scope of the embodiments.

[0044] For the sake of simplicity, the following description will take the model scheduling device as the execution subject of the model scheduling method provided in the embodiments of this disclosure as an example.

[0045] Figure 5 This is a flowchart illustrating the model scheduling method provided in an embodiment of this disclosure. Figure 5 As shown, the model scheduling method may include the following steps: S101. Obtain operation and maintenance monitoring information for multiple models within a preset period.

[0046] The preset period can be pre-set in the model scheduling device. For example, the preset period can be set to 8 hours, 1 day, 3 days, 7 days, 15 days, or 30 days, etc. This embodiment of the disclosure does not limit the specific duration of the preset period. Multiple models can be artificial intelligence (AI) models, such as large language models or other machine learning models. This embodiment of the disclosure does not limit the specific types of multiple models.

[0047] As an example, the preset period can be set based on the size of the inference resources in the inference resource pool.

[0048] For example, in scenarios where the inference resource pool has sufficient inference resources (or the computing power exceeds the computing power threshold) and can tolerate a certain degree of inference resource limitation, a longer preset period (such as 15 days or 30 days) can be set to strive for higher scheduling stability and reduce the overhead caused by frequent model migration in and out.

[0049] For example, in scenarios where the inference resource pool is underutilized (or the computing power is below the computing power threshold), a shorter preset period (e.g., 1 day or 3 days) can be set to more sensitively capture the changing trend of model usage popularity, schedule models in a timely manner, and improve the utilization rate of inference resources.

[0050] As another example, the preset period can be set during the deployment phase of the model scheduling system.

[0051] For example, in the initial deployment phase of a model scheduling system, where there is a lack of operation and maintenance monitoring information and the fluctuation range is large, a shorter preset period (such as 1 day or 3 days) can be set to continuously and sensitively schedule the model.

[0052] For example, during the stable operation phase of the model scheduling system, the fluctuation range of operation and maintenance monitoring information is stable, and a longer preset period (such as 7 days or 15 days) can be set to reduce model scheduling overhead. As an example, operation and maintenance monitoring information may include the information collection time, the name of the actuator corresponding to the model, resource load information, inference performance information, inference task information, etc.

[0053] For example, the specific content included in the operation and maintenance monitoring information can be shown in Table 1 below: Table 1

[0054] S102. Perform call popularity analysis on the operation and maintenance monitoring information of multiple models from each of the at least one preset strategy dimension, and obtain a popularity analysis report for each model in the multiple models.

[0055] As an example, the preset strategy dimensions may include at least one of the following: model invocation time dimension, model resource load in the inference resource pool dimension, model inference performance dimension, and model inference task dimension.

[0056] As an example, a popularity analysis report may include popularity metrics information under at least one strategy dimension. Popularity metrics information under at least one strategy dimension may include at least one of the following: peak scheduling periods, off-peak scheduling periods, resource load information of the model in the inference resource pool, inference performance information of the model, and statistical information of the model's inference tasks.

[0057] For example, the specific content included in a heat analysis report can be as shown in Table 2 below: Table 2

[0058] S103. Based on the heat analysis report of each model in multiple models, generate a model scheduling strategy table for multiple models.

[0059] In this context, a model scheduling strategy table for multiple models refers to a single table corresponding to multiple models. The model scheduling strategy table may include one or more model scheduling strategies. As a possible implementation, the model scheduling strategy table may also include strategy identifiers and priority information for each model scheduling strategy. Model scheduling strategies may include at least one of the following: tidal strategy, resource load strategy, inference performance strategy, and inference task strategy.

[0060] Tidal strategies may include: performing a migration operation on the model when the information collection time in the operation and maintenance monitoring information is during the peak call period, and / or performing a migration operation on the model when the information collection time is during the low call period.

[0061] The peak scheduling period can be understood as the time period during which the number of inference tasks executed exceeds a threshold. The off-peak scheduling period can be understood as the time period during which the number of inference tasks executed is less than a threshold.

[0062] In the model scheduling method provided in this embodiment, the tidal strategy triggers scheduling actions based on the matching relationship between the information collection time of operation and maintenance monitoring information and the pre-determined peak and off-peak call periods. This ensures that the migration in and out operations of models are synchronized with the historical call time patterns of the models themselves. During off-peak call periods, models are migrated out of the inference resource pool to reclaim idle inference resources for other models to use; during peak call periods, models are migrated into the inference resource pool to obtain inference resources in a timely manner to process inference requests. This enables on-demand use of inference resources in the time dimension, thereby ensuring the inference service capacity of models during peak call periods while improving the resource utilization rate of inference resources during off-peak call periods.

[0063] Resource load strategies may include: performing a migration operation on the model when the resource load information of the model in the inference resource pool is lower than the load threshold.

[0064] As an example, resource load information can be GPU load, and the load threshold can be set to 25%, 20%, 15%, or 10% of the GPU's maximum load.

[0065] As an example, resource load information can be host memory utilization, and the load threshold can be set to 25%, 20%, 15%, or 10%, etc. This disclosure does not limit the specific value of the load threshold.

[0066] In the model scheduling method provided in this embodiment, the resource load strategy can use the resource load information of the model in the inference resource pool as a basis. When the resource load information of the model in the inference resource pool is detected to be lower than the load threshold, it indicates that only a small part of the inference resources currently occupied by the model are actually used to execute inference tasks, and the majority of the rest are idle. In this case, moving the model out of the inference resource pool can release most of the idle inference resources, thereby reducing the situation where inference resources are ineffectively occupied by low-load models and improving the utilization rate of inference resources.

[0067] Inference performance strategies may include: performing a model eviction operation when the model's inference performance information does not meet the preset performance requirements of the model's inference.

[0068] As an example, preset performance requirements may include inference latency being less than or equal to a latency threshold, which may be set to 500 milliseconds (ms), 800 ms, 1000 ms, or 1500 ms, etc. This disclosure does not limit the specific value of the latency threshold.

[0069] As an example, the preset performance requirement may include an inference throughput greater than or equal to a throughput threshold, which may be set to 40 Tokens / second, 60 Tokens / second, 80 Tokens / second, 15 inference requests / second, 20 inference requests / second, 25 inference requests / second, or 30 inference requests / second. This disclosure does not limit the specific value of the throughput threshold.

[0070] In the model scheduling method provided in this embodiment, the inference performance strategy can take whether the model's inference performance information meets the preset performance requirements of the model inference as the strategy condition. When it is detected that the model's inference performance information does not meet the preset performance requirements, it indicates that the model can no longer complete the inference task with a qualified service level in the inference resource pool. At this time, the model is moved out of the inference resource pool. On the one hand, the model can be removed from the inference service to prevent the model from continuing to receive inference requests and return inference responses that do not meet the preset performance requirements. On the other hand, the inference resources released after the model is moved out of the inference resource pool can be used by other loaded models, thereby preventing the inference resources from being continuously occupied by degraded models. This improves the service quality of the inference service while increasing the utilization rate of inference resources.

[0071] The inference task strategy may include: performing a migration operation on the model when the model's inference task statistics meet the aforementioned preset task backlog condition, and / or performing a migration operation on the model when the model's inference task statistics meet the aforementioned preset task idle condition.

[0072] For example, the preset task backlog conditions may include at least one of the following: the request queue backlog ratio (i.e., the number of inference requests in the request queue / the maximum capacity of the request queue) is greater than or equal to a first ratio threshold; QPS is greater than or equal to a first QPS threshold.

[0073] For example, the preset task idle condition may include at least one of the following: the request queue backlog ratio (i.e., the number of inference requests in the request queue / the maximum capacity of the request queue) is less than the second ratio threshold; the QPS is less than the second QPS threshold.

[0074] The second proportional threshold is less than or equal to the first proportional threshold. For example, the first proportional threshold can be set to 70%, 75%, 80%, or 85%, etc. The second proportional threshold can be set to 30%, 25%, 20%, or 15%, etc. The specific values ​​of the first and second proportional thresholds are not limited in this embodiment. The second QPS threshold is less than or equal to the first QPS threshold. For example, the first QPS threshold can be set to 40, 50, 60, or 70, etc. The second QPS threshold can be set to 15, 10, 8, 5, or 3, etc. The specific values ​​of the first and second QPS thresholds are not limited in this embodiment.

[0075] In the model scheduling method provided in this embodiment, the inference task strategy can use whether the model's inference task statistics meet the preset task backlog condition or the preset task idle condition as the strategy condition, so that the model scheduling operation is associated with the model's task load. When the inference task statistics meet the preset task backlog condition, it indicates that the model's task load is high. At this time, a migration operation is performed on the model to increase the supply of inference resources, thereby alleviating the pressure of task load. When the inference task statistics meet the preset task idle condition, it indicates that the model's task load is low. At this time, a migration operation is performed on the model to reclaim idle inference resources, which are convenient for other models to call. Thus, the inference resources in the inference resource pool are dynamically adjusted with the changes in the model's task load, ensuring the timely response of the inference service to the inference task and reducing unnecessary inference resource occupation, thereby improving the resource utilization rate of the inference resource pool.

[0076] For example, the specific contents of the model scheduling strategy table can be shown in Table 3 below: Table 3

[0077] S104. Based on the model scheduling strategy table of multiple models and the current operation and maintenance monitoring information of each model in the multiple models, perform model scheduling for each model in the multiple models.

[0078] Model scheduling is used to move models into or out of the inference resource pool.

[0079] As an example, the model file information of a model can be moved from the storage device to the XPU in the inference resource pool to move the model into the inference resource pool; or, the model file information of a model can be moved from the XPU in the inference resource pool to the storage device to move the model out of the inference resource pool.

[0080] The scheduling method provided in this disclosure can continuously collect operation and maintenance monitoring information and perform heat analysis within a preset period. Based on the heat analysis report, a model scheduling strategy table is generated to schedule models, moving them into or out of the inference resource pool. Compared with the static binding of models and processor resources in existing technologies, model scheduling through a model scheduling strategy table breaks the static binding between models and processor resources. For unpopular models that are in a low-call or idle state, they can be actively moved out of the inference resource pool to release processor resources. For popular models that are experiencing a call peak, they can be actively moved into the inference resource pool to ensure inference services. This realizes the reuse and flow of processor resources among different models, thereby improving the utilization rate of processor resources for deploying models.

[0081] The implementation process of S103 is described below.

[0082] In some embodiments, Figure 6 This is another flowchart illustrating the model scheduling method provided in an embodiment of this disclosure. Figure 6 As shown, the above S103 may specifically include the following steps: S1031. Based on the heat index information of at least one strategy dimension contained in the heat analysis report of each model, determine the strategy type corresponding to each model, and establish the mapping relationship between each model and the strategy type.

[0083] As mentioned above, the popularity index information under at least one strategy dimension may include at least one of the following: peak scheduling time period, off-peak scheduling time period, resource load information of the model in the inference resource pool, inference performance information of the model, and statistical information of the model's inference task.

[0084] For example, if the heat analysis report of the first model shows a peak scheduling period (or a period in which the number of times the inference task is executed exceeds the number threshold for a duration exceeding the duration threshold) and / or a trough scheduling period (or a period in which the number of times the inference task is executed is less than the number threshold for a duration exceeding the duration threshold), and the peak scheduling period and / or the trough scheduling period are regularly distributed within a preset period (or repeat within at least two sub-periods within the preset period), then the strategy type corresponding to the first model can be determined to be a tidal strategy.

[0085] The number of occurrences threshold can be preset in the model scheduling device. For example, the number of occurrences threshold can be set to 100, 200, 500, or 1000 times, etc. This embodiment of the present disclosure does not limit the specific value of the number of occurrences threshold. The duration threshold can be set to 5 minutes, 10 minutes, 30 minutes, 1 hour, 3 hours, 5 hours, 10 hours, or 12 hours, etc. This embodiment of the present disclosure does not limit the specific value of the duration threshold.

[0086] For example, if the duration for which the resource load information of the second model in the inference resource pool is below the load threshold in the heat analysis report of the second model exceeds the duration threshold, then the strategy type corresponding to the second model can be determined to be the resource load strategy.

[0087] The load threshold can be set in the model scheduling device. For example, taking GPU load as an example, the load threshold can be set to 25%, 20%, 15%, or 10% of the maximum GPU load; or taking host memory utilization as an example, the load threshold can be set to 25%, 20%, 15%, or 10%, etc. This disclosure does not limit the specific value of the load threshold.

[0088] For example, if the inference performance information of the third model in the heat analysis report does not meet the preset performance requirements of the model inference for a period of time exceeding the time threshold, then the strategy type corresponding to the third model can be determined to be an inference performance strategy.

[0089] For example, the preset performance requirements may include at least one of the following: inference latency is less than or equal to a latency threshold; inference throughput is greater than or equal to a throughput threshold.

[0090] The latency threshold can be set in the model scheduling device. For example, the latency threshold can be set to 500 milliseconds (ms), 800 ms, 1000 ms, or 1500 ms, etc. This embodiment of the present disclosure does not limit the specific value of the latency threshold. The throughput threshold can also be set in the model scheduling device. For example, the throughput threshold can be set to 15 inference requests / second, 20 inference requests / second, 25 inference requests / second, or 30 inference requests / second, etc. This embodiment of the present disclosure does not limit the specific value of the throughput threshold.

[0091] For example, if the duration for which the statistical information of the inference task of the fourth model meets the preset task backlog condition and / or preset task idle condition in the heat analysis report of the fourth model exceeds the duration threshold, then the strategy type corresponding to the fourth model can be determined to be the inference task strategy.

[0092] For example, the preset task backlog condition may include at least one of the following: the request queue backlog ratio (i.e., the number of inference requests in the request queue / the maximum capacity of the request queue) is greater than or equal to a first ratio threshold; the QPS is greater than or equal to a first QPS threshold. The preset task idle condition may include at least one of the following: the request queue backlog ratio is less than a second ratio threshold; the QPS is less than a second QPS threshold.

[0093] The second proportional threshold is less than or equal to the first proportional threshold. The first and second proportional thresholds can be set in the model scheduling device. For example, the first proportional threshold can be set to 70%, 75%, 80%, or 85%, etc. The second proportional threshold can be set to 30%, 25%, 20%, or 15%, etc. The specific values ​​of the first and second proportional thresholds are not limited in this embodiment. The second QPS threshold is less than or equal to the first QPS threshold. The first and second QPS thresholds can be set in the model scheduling device. For example, the first QPS threshold can be set to 40, 50, 60, or 70, etc. The second QPS threshold can be set to 15, 10, 8, 5, or 3, etc. The specific values ​​of the first and second QPS thresholds are not limited in this embodiment.

[0094] S1032. Construct a model scheduling strategy table based on the model scheduling strategy and mapping relationship corresponding to each strategy type.

[0095] As an example, the step of constructing a model scheduling strategy table in S1032 above, based on the model scheduling strategies and mapping relationships corresponding to each strategy type, may specifically include: for each model: determining the target strategy type corresponding to the model based on the mapping relationship; based on the target strategy type corresponding to the model, analyzing the change in the heat index information related to the target strategy type in the model's heat analysis report, and determining the threshold value of the heat index information in the strategy conditions corresponding to the target strategy type based on the change; and generating a model scheduling strategy table containing the model scheduling strategies of the target strategy type based on the strategy conditions corresponding to the target strategy type, the correspondence between the strategy conditions and scheduling operations, and the threshold value.

[0096] For example, taking the tidal strategy as the target strategy type corresponding to the first model mentioned above, if the number of times the start time of the peak scheduling period falls within the first time period in the heat analysis report of the first model reaches a threshold, then a time can be selected from the first time period (e.g., the start time of the first time period) as the start time of the peak scheduling period in the tidal strategy. If the number of times the end time of the peak scheduling period falls within the second time period in the heat analysis report of the first model reaches a threshold, then a time can be selected from the second time period (e.g., the end time of the second time period) as the end time of the peak scheduling period in the tidal strategy. If the number of times the start time of the trough scheduling period falls within the third time period in the heat analysis report of the first model reaches a threshold, then a time can be selected from the third time period (e.g., the start time of the third time period) as the start time of the trough scheduling period in the tidal strategy. If the number of times the end time of the trough scheduling period falls within the fourth time period in the heat analysis report of the first model reaches a threshold, then a time can be selected from the fourth time period (e.g., the end time of the fourth time period) as the end time of the trough scheduling period in the tidal strategy.

[0097] For example, taking the resource load strategy as the target strategy type corresponding to the second model mentioned above, if the number of times the historical distribution value of the resource load information in the heat analysis report of the second model falls within the target value range reaches the threshold, then a value can be selected from the target value range (for example, the lower limit of the target value range) as the threshold value of the load threshold in the resource load strategy.

[0098] The target numerical range can be determined based on the lowest A quantile interval of the resource load information in the second model within a preset period, where A is a positive integer greater than 1, such as 2, 3, 4, or 5. This embodiment does not limit the specific value of A.

[0099] For example, taking the inference performance strategy as the target strategy type corresponding to the third model mentioned above, if the number of times the inference latency in the inference performance information of the third model falls within the target latency range reaches a threshold, then a value can be selected from the target latency range (e.g., the upper limit of the target latency range) as the threshold value of the latency threshold in the inference performance strategy; and / or, if the number of times the inference throughput in the inference performance information of the third model falls within the target throughput range reaches a threshold, then a value can be selected from the target throughput range (e.g., the lower limit of the target throughput range) as the threshold value of the throughput threshold in the inference performance strategy.

[0100] The target latency range can be determined based on the highest B quantile interval of the inference latency of the third model within a preset period, where B is a positive integer greater than 1, such as 10, 9, 8, 7, 6, or 5. This embodiment of the present disclosure does not limit the specific value of B. The target throughput range can be determined based on the lowest C quantile interval of the inference throughput of the third model within a preset period, where C is a positive integer greater than 1, such as 10, 9, 8, 7, 6, or 5. This embodiment of the present disclosure does not limit the specific value of C.

[0101] For example, taking the target strategy type corresponding to the fourth model mentioned above as the inference task strategy, if the number of times the request queue backlog ratio in the inference task statistics of the fourth model's popularity analysis report falls within the first ratio range reaches a threshold, then a value can be selected from the first ratio range (e.g., the upper limit of the first ratio range) as the threshold value of the first ratio threshold in the inference task strategy; and / or, if the number of times the QPS in the inference task statistics of the fourth model's popularity analysis report falls within the first QPS range reaches a threshold, then a value can be selected from the first QPS range (e.g., the upper limit of the first QPS range) as the inference task strategy. The threshold value of the first QPS threshold; and / or, if the number of times the request queue backlog ratio in the inference task statistics of the fourth model's heat analysis report falls within the second ratio range reaches the threshold value, then a value can be selected from the second ratio range (e.g., the lower limit of the second ratio range) as the threshold value of the second ratio threshold in the inference task strategy; and / or, if the number of times the QPS in the inference task statistics of the fourth model's heat analysis report falls within the second QPS range reaches the threshold value, then a value can be selected from the second QPS range (e.g., the lower limit of the second QPS range) as the threshold value of the second QPS threshold in the inference task strategy.

[0102] The first ratio range can be determined based on the highest D quantile range of the request queue backlog ratio of the fourth model within a preset period, where D is a positive integer greater than 1, such as 10, 9, 8, 7, 6, or 5. This embodiment does not limit the specific value of D. The first QPS range can be determined based on the highest E quantile range of QPS of the fourth model within a preset period, where E is a positive integer greater than 1, such as 10, 9, 8, 7, 6, or 5. This embodiment does not limit the specific value of E. The second ratio range can be determined based on the lowest F quantile range of the request queue backlog ratio of the fourth model within a preset period, where F is a positive integer greater than 1, such as 10, 9, 8, 7, 6, or 5. This embodiment does not limit the specific value of F. The second QPS range can be determined based on the lowest G quantile range of QPS of the fourth model within a preset period, where G is a positive integer greater than 1, such as 10, 9, 8, 7, 6, or 5. This embodiment does not limit the specific value of G.

[0103] For example, prompt words can be directly input into a large language model, which can then analyze the changes in heat index information related to the target strategy type in the model's heat analysis report and output threshold values.

[0104] The prompt words are used to indicate the amount of heat index information related to the target strategy type in the heat analysis report of the large language model analysis, and output the threshold value of the heat index information in the model scheduling strategy corresponding to the target strategy type.

[0105] The model scheduling method provided in this disclosure can determine the strategy type corresponding to each model based on the popularity index information of at least one strategy dimension contained in the popularity analysis report of each model, and establish a mapping relationship between each model and the strategy type. It can automatically match a strategy type suitable for the model's calling pattern based on the popularity index information exhibited by each model in a specific strategy dimension, such as matching a tidal strategy to a model exhibiting a clear tidal pattern, or a resource load strategy to a model with long-term low resource load. Then, a model scheduling strategy table is constructed based on the model scheduling strategies corresponding to each strategy type and the mapping relationship. This ensures that the scheduling strategy configured for each model in the final generated model scheduling strategy table is not a uniform fixed strategy, but a differentiated strategy matched to the model's own calling pattern, thereby providing a more targeted decision-making basis for subsequent model scheduling operations and improving the accuracy of scheduling operations.

[0106] Furthermore, the model scheduling method provided in this disclosure can determine the target strategy type corresponding to the model based on the mapping relationship, and determine the threshold value of the heat index information related to the target strategy type in the heat analysis report of the model based on the change of the heat index information related to the target strategy type. This ensures that the threshold value of the strategy condition in the model scheduling strategy corresponding to each model is not a preset fixed value, but is dynamically determined according to the actual historical change of the heat index information of the model itself within a preset period. This improves the accuracy of the triggering timing of the model scheduling strategy and reduces the problem of scheduling too early or too late due to improper threshold setting.

[0107] In some embodiments, the system may also directly receive a model scheduling strategy table for multiple models input by operations and maintenance personnel.

[0108] The implementation process of S104 described above will be introduced below.

[0109] In some embodiments, before scheduling models among multiple models, the method further includes: creating a cache resource pool in memory, and establishing a first communication link between the cache resource pool and the inference resource pool, and a second communication link between the cache resource pool and the executor. The cache resource pool is used to cache model file information and model environment configuration information. In this case, the step of scheduling models among multiple models in S104 above may specifically include: migrating the model file information of the model to be migrated out of the inference resource pool to the cache resource pool based on the first communication link, and migrating the model environment configuration information of the model to be migrated out of the executor to the cache resource pool based on the second communication link; or, migrating the model file information of the model to be migrated in of the cache resource pool to the inference resource pool based on the first communication link, and migrating the model environment configuration information of the model to be migrated in of the cache resource pool to the executor based on the second communication link.

[0110] For example, model parameter information may include at least one of the following: model weights, key-value cache (KVcache), miscellaneous information, activation information, and optimizer status information. Model environment configuration information may include at least one of the following: session context, model runtime parameters, and model deployment location.

[0111] For example, the migration in and migration out based on the first communication link and the second communication link can be specifically shown in Table 4 below: Table 4

[0112] The model scheduling method provided in this disclosure can pre-create a cache resource pool in memory and use a first communication link and a second communication link to collaboratively complete model migration in and out. Compared with related technologies that schedule model information from external storage devices (such as solid-state drives or hard disk drives), this application, because the cache resource pool is located in memory, ensures that model file information and model environment configuration information always reside in high-bandwidth, low-latency DRAM media during the scheduling process, without involving read / write operations on external storage devices. When a model needs to be migrated out, the model file information in the inference resource pool is migrated to the cache resource pool via the first communication link, and the model environment configuration information in the executor is migrated to the cache resource pool via the second communication link. Both migration steps are completed within the same memory address space or the high-speed bus between memory and XPU memory, avoiding low-speed I / O operations to external storage devices. When a model needs to be migrated in again, the executor can directly restore the model environment configuration information from the memory cache resource pool via the second communication link, and transfer the model file information from memory to the inference resource pool via the executor or directly through the first communication link. There is no need to reload the model weight file from external storage or reinitialize the runtime environment, thereby improving model scheduling efficiency.

[0113] In some embodiments, the step of scheduling each model among multiple models in S104 above may specifically include: for each model to be migrated out among multiple models that is determined to be migrated out: waiting for the model to be migrated out to process the currently accepted inference request; and performing the migration out operation on the model to be migrated out after the model to be migrated out has finished processing the inference request.

[0114] The model scheduling method provided in this disclosure can perform a migration operation on the model to be migrated out after the model to be migrated out has finished processing the inference request. This can avoid the processing interruption caused by the model to be migrated out directly before it has finished processing the inference request, and avoid affecting the user's experience of using the inference service.

[0115] In some embodiments, the step of scheduling each model among multiple models in S104 above may specifically include: determining the model priority and / or model deployment time of at least one model to be migrated out, and determining the migration order of at least one model to be migrated out based on the model priority and / or model deployment time; and performing migration operations on at least one model to be migrated out in sequence based on the migration order.

[0116] For example, taking models A, B, and C to be migrated as an example, assuming model A has a priority of 1 (low priority), model B has a priority of 3 (medium priority), and model C has a priority of 5 (high priority), then the models can be sorted based on their priorities. Models with lower priorities are migrated out first, releasing the inference resources occupied by lower-priority models first. Therefore, the migration order is determined as: Model A, Model B, Model C. For example, taking models A, B, and C to be migrated out as an example, assuming that the deployment time of model A is April 20, 2026, the deployment time of model B is April 21, 2026, and the deployment time of model C is April 22, 2026, then the models can be sorted based on their deployment time, with the models with earlier deployment times being migrated out first, thus determining the migration order as: model A, model B, model C.

[0117] For example, consider models A, B, and C to be migrated out. Assume model A has a priority of 1 (low priority), model B has a priority of 1 (low priority), and model C has a priority of 5 (high priority). Model A is deployed on April 20, 2026, model B on April 21, 2026, and model C on April 22, 2026. We can first sort them based on their priorities to determine the initial migration order as: Model A / Model B, Model C. Since Model A and Model B have the same priority, we further sort them based on their deployment times. Model A was deployed earlier than Model B, thus determining the final migration order as: Model A, Model B, Model C.

[0118] The model scheduling method provided in this disclosure can determine the migration order based on the model priority and / or deployment time of at least one model to be migrated out, and execute the migration operation sequentially according to the migration order. The migration operation is not performed on all models to be migrated out simultaneously, but is executed in an orderly manner according to a preset sorting rule. Given that multiple models need to be migrated out of the inference resource pool simultaneously, the migration operation is executed in an orderly manner according to a clear sorting rule, avoiding the impact of disordered migration on the inference service. This reduces the disturbance to the inference service caused by scheduling operations while ensuring the efficiency of inference resource recovery.

[0119] In some embodiments, S104 may specifically include the following steps: For each model: Based on the current operation and maintenance monitoring information of the model, perform policy condition matching in the model scheduling policy table to obtain the model scheduling policy currently matched by the model; If multiple matched model scheduling policies are determined from the model scheduling policy table, and the scheduling operations of the multiple matched model scheduling policies conflict (or are different), perform model scheduling on the model based on the model scheduling policy with the highest priority among the multiple matched model scheduling policies.

[0120] For example, taking the current operation and maintenance monitoring information collection time of model A as 02:30 and the request queue backlog as 80%, assuming that 02:30 meets the scheduling off-peak period (22:00 to 9:00) in the tidal strategy, and that the request queue backlog of 80% meets the preset task backlog condition in the inference task strategy. That is, model A simultaneously meets the requirement of migration out during the off-peak period in the tidal strategy and the preset task backlog condition in the inference task strategy, requiring migration in. In this case, it can be determined that the scheduling operations of these two scheduling strategies conflict. Assuming that the priority 5 of the inference task strategy is higher than the priority 1 of the tidal strategy, the inference task strategy can be executed to migrate (or maintain) model A into the tidal strategy.

[0121] The model scheduling method provided in this disclosure can preset priorities for each model scheduling strategy, and when there is a conflict in the scheduling operations of multiple matched model scheduling strategies, the model scheduling strategy with the highest priority is selected to perform the scheduling operation, thereby avoiding scheduling operation conflicts between different scheduling strategies, which would lead to repeated switching of scheduling instructions and avoid frequent invalid migration in and out of the model in the inference resource pool, thereby improving the reliability of the model scheduling method.

[0122] In some embodiments, the above S104 may specifically include the following steps: based on the model scheduling strategy table of multiple models and the current operation and maintenance monitoring information of each model in the multiple models, determine the model to be scheduled and the scheduling operation of the model to be scheduled from the multiple models; obtain the current running status of the model to be scheduled, and identify whether the current running status of the model to be scheduled supports the scheduling operation based on the first preset correspondence; if the current running status of the model to be scheduled supports the scheduling operation, perform the scheduling operation on the model to be scheduled.

[0123] The first preset correspondence is the correspondence between the running status and the supported scheduling operations. The running status of the model to be scheduled is used to characterize the deployment status of the model to be scheduled in the inference resource pool.

[0124] For example, the operating state of the model may include at least one of the following: initial state, working state, migration state, and termination state.

[0125] The initial state indicates that the model instance of the model to be scheduled has been created. The scheduling operations supported in the initial state include migration in. The working state indicates that the model to be scheduled has been migrated into the inference resource pool. The scheduling operations supported in the working state include migration out. The migration out state indicates that the model to be scheduled has been migrated out of the inference resource pool. The scheduling operations supported in the migration out state include migration in. The termination state indicates that the model file information and model environment configuration information of the model to be scheduled have been deleted from the cache resource pool. The scheduling operations supported in the termination state are empty.

[0126] For example, taking the current running state of model A as the working state, the scheduling operations supported by the working state include the checkout operation. Assuming that the scheduling operation for model A is determined to be a checkout operation based on the scheduling policy, it can be determined that the current running state (working state) of model A supports the checkout operation, and thus the checkout operation is performed on model A.

[0127] For example, taking the current running state of model A as the working state, the scheduling operations supported by the working state include the migration out operation. Assuming that the scheduling operation for model A is determined to be the migration in operation based on the scheduling policy, it can be determined that the current running state (working state) of model A does not support the migration in operation, and therefore the migration in operation will not be performed on model A, and a policy error message will be output.

[0128] The model scheduling method provided in this disclosure can further determine the model to be scheduled and its scheduling operation based on the model scheduling policy table. First, it obtains the current running state of the model to be scheduled. Based on a preset correspondence between the running state and supported scheduling operations, it identifies whether the current running state of the model to be scheduled supports the scheduling operation. The scheduling operation is only executed on the model to be scheduled if the running state supports the scheduling operation. In this way, scheduling operations not supported by the running state can be intercepted, thereby verifying the scheduling operations generated by the model scheduling policy. Invalid or conflicting scheduling instructions caused by delays in model scheduling policy generation, state synchronization delays, or policy configuration errors are intercepted, avoiding frequent invalid migrations in and out of the model inference resource pool, thus improving the reliability of the model scheduling method.

[0129] In some embodiments, the model scheduling method may further include the following steps: in response to a state switching event, switching the running state of the model to be scheduled based on the current running state of the model to be scheduled, the state switching event, and a second preset correspondence.

[0130] The second preset correspondence is used to represent the correspondence between the current running state, the state switching event, and the running state after the switch.

[0131] For example, Figure 7 This is a schematic diagram illustrating model state switching provided in an embodiment of this disclosure. Figure 7 As shown, for each model among multiple models, the running state of the model can be switched in the following ways: When the model is in the terminated state, in response to the start of the model-related inference process, the model switches from the terminated state to the initial state. When the model is in the initial state, in response to receiving a model-related inference request, the model switches from the initial state to the working state. When the model is in the working state, in response to the termination of the model-related inference process, the model switches from the working state to the terminated state. When the model is in the working state, in response to performing a checkout operation on the model, the model switches from the working state to the checkout state. When the model is in the checkout state, in response to the termination of the model-related inference process, the model switches from the checkout state to the terminated state. When the model is in the checkout state, in response to performing a checkin operation on the model, the model switches from the checkout state to the working state.

[0132] The scheduling method provided in this embodiment can also respond to a state switching event by switching the running state of the model to be scheduled in a timely manner based on the current running state of the model to be scheduled, the state switching event, and the second preset correspondence, thereby facilitating subsequent visualization for operation and maintenance personnel and verification of the scheduling operations generated by the model scheduling strategy based on the running state.

[0133] Based on the understanding of the above embodiments, in the exemplary embodiments, Figure 8 This is another flowchart illustrating the model scheduling method provided in this embodiment of the disclosure. Figure 8 As shown, the model scheduling method may include the following steps: S201. The inference service system creates an executor based on the model deployment requirements of the target model.

[0134] S202. The model instance of the target model has been created and the target model has entered the initial state.

[0135] S203. The model scheduling strategy generation center sends the scheduling startup strategy of the target model to the model scheduler according to the preset deployment blueprint.

[0136] S204. The model scheduler sends the model migration instruction of the target model to the executor.

[0137] S205. The executor initiates a loading operation to the XPU to load the model file information of the target model.

[0138] S206, XPU feeds back to the actuator that the model loading is complete.

[0139] S207, The target model in the actuator enters the working state.

[0140] S208. The model scheduling strategy generation center sends the generated model scheduling strategy table to the model scheduler.

[0141] S209. The model scheduler sends the target model's model migration instruction to the executor according to the model scheduling policy table.

[0142] S210, The actuator initiates an unloading operation of the target model to the XPU.

[0143] S211 and XPU unload the model file information of the target model in the video memory to the cache resource pool and send feedback to the executor to indicate that the unloading is complete.

[0144] S212, The target model in the actuator enters the migration state.

[0145] S213. The inference service system terminates the inference process of the target model and deletes the executor.

[0146] S214, The target model has entered the termination state.

[0147] Based on the understanding of the above embodiments, in the exemplary embodiments, Figure 9 This is a schematic diagram of the initial loading process of the model provided in an embodiment of this disclosure. Figure 9 As shown, the initial model loading process may include the following steps: S301. The inference service system creates an executor based on the model deployment requirements of the target model.

[0148] S302. The model instance of the target model has been created and the target model has entered the initial state.

[0149] S303, The model scheduling strategy generation center transmits the model scheduling strategy table to the model scheduler.

[0150] S304. The model scheduler sends the model migration instruction of the target model to the executor according to the model scheduling strategy table.

[0151] S305. The executor loads the model environment configuration information of the target model according to the initial deployment default strategy of the model.

[0152] S306. The executor initiates a loading operation to the XPU to load the model file information of the target model.

[0153] S307 and XPU report that the model loading is complete to the actuator.

[0154] S308, The target model in the actuator enters the model working state.

[0155] S309. The user sends an inference request to the target model, and the target model begins to process the inference request.

[0156] Based on the understanding of the above embodiments, in the exemplary embodiments, Figure 10 This is a schematic diagram of the model migration operation process provided in an embodiment of this disclosure. Figure 10 As shown, the model migration operation can include the following steps: S401, The model scheduling strategy generation center generates the model scheduling strategy table.

[0157] S402. The model scheduling strategy generation center sends the model scheduling strategy table to the model scheduler.

[0158] S403. The model scheduler sends the target model's model migration instruction to the executor according to the model scheduling policy table.

[0159] S404, The actuator initiates an unloading operation of the target model to the XPU.

[0160] S405 and XPU unload the model file information of the target model in the video memory to the cache resource pool and send feedback to the executor to indicate that the unloading is complete.

[0161] S406. The executor unloads the model environment configuration information of the target model to the cache resource pool.

[0162] S407. The cache resource pool stores the model file information and model environment configuration information of the target model in the specified cache pool area.

[0163] S408: The cache resource pool successfully stored the target model's model file information and model environment configuration information back to the executor.

[0164] S409. The target model's running state in the executor is changed to the removed state, and it stops providing services to the outside world.

[0165] Based on the understanding of the above embodiments, in the exemplary embodiments, Figure 11 This is a schematic diagram of the secondary migration operation process of the model provided in the embodiments of this disclosure. Figure 11 As shown, the model secondary migration operation can include the following steps: S501, The model scheduling strategy generation center sends the generated model scheduling strategy table to the model scheduler.

[0166] S502. The model scheduler sends the model migration instruction of the target model to the executor according to the model scheduling strategy table.

[0167] S503, The target model is currently in a migration-out state.

[0168] S504. The executor sends a request to the cache resource pool to load the model environment configuration information of the target model.

[0169] S505, the cache resource pool sends the environment configuration information of the target model to the executor.

[0170] S506, The actuator initiates a target model loading operation to the XPU.

[0171] S507 and XPU load the model file information of the target model from the cache resource pool and send a feedback message to the executor indicating that loading is complete.

[0172] S508, The target model has been reloaded and is now in working condition.

[0173] S509: The user sends an inference request to the target model, and the target model begins to process the inference request.

[0174] Based on the understanding of the above embodiments, in the exemplary embodiments, Figure 12 This is a schematic diagram of the model termination workflow provided in an embodiment of this disclosure. Figure 12 As shown, the model termination workflow may include the following steps: S601, The user sends a termination command for the target model to the inference service system.

[0175] S602, The inference service system sends a termination command to the executor.

[0176] S603, the executor unloads the model environment configuration information to the cache resource pool.

[0177] S604, The cache resource pool reports the unloading completion result to the executor.

[0178] S605, The actuator initiates an unloading operation of the target model to the XPU.

[0179] The S606 and XPU unload the model file information of the target model from the video memory to the cache resource pool and send feedback to the executor to indicate that the unloading is complete.

[0180] S607, The actuator sends a message to the inference service system that the model information clearing is complete.

[0181] S608. The inference service system terminates the inference process of the target model and deletes the executor.

[0182] S609, The target model has entered the termination state.

[0183] The foregoing primarily describes the solutions provided by the embodiments of this disclosure from a methodological perspective. To achieve the aforementioned functions, each device, such as a model scheduling device, includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the algorithmic steps of the examples described in the embodiments disclosed herein, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Experts may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0184] This disclosure embodiment can divide the model scheduling device into functional modules according to the above method embodiment. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one functional module. The integrated module can be implemented in hardware or software. It should be noted that the module division in this disclosure embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods. The following description uses the example of dividing each function into a separate functional module.

[0185] In an exemplary embodiment, this disclosure provides a model scheduling apparatus. Figure 13 This is a schematic diagram illustrating the composition of the model scheduling device provided in an embodiment of this disclosure. Figure 13 As shown, the model scheduling device may include an acquisition module 1301 and a processing module 1302.

[0186] The acquisition module 1301 is used to acquire operation and maintenance monitoring information of multiple models within a preset period.

[0187] The processing module 1302 is used to perform call popularity analysis on the operation and maintenance monitoring information of multiple models from each of the at least one preset strategy dimension to obtain the popularity analysis report of each model in the multiple models; based on the popularity analysis report of each model in the multiple models, generate a model scheduling strategy table for multiple models; based on the model scheduling strategy table of multiple models and the current operation and maintenance monitoring information of each model in the multiple models, perform model scheduling for each model in the multiple models; model scheduling is used to migrate models into or out of the inference resource pool.

[0188] In some embodiments, the processing module 1302 is specifically used to determine the strategy type corresponding to each model based on the heat index information under at least one strategy dimension contained in the heat analysis report of each model, and to establish a mapping relationship between each model and the strategy type; wherein, the heat index information under at least one strategy dimension includes at least one of the following: peak scheduling time period, off-peak scheduling time period, resource load information of the model in the inference resource pool, inference performance information of the model, and inference task statistics of the model; and to construct a model scheduling strategy table based on the model scheduling strategy corresponding to each strategy type and the mapping relationship; wherein, the model scheduling strategy includes at least one of the following: tidal strategy, resource load strategy, inference performance strategy, and inference task strategy.

[0189] In some embodiments, the processing module 1302 is specifically configured to, for each model: determine the target strategy type corresponding to the model based on the mapping relationship; based on the target strategy type corresponding to the model, analyze the change in the heat index information related to the target strategy type in the heat analysis report of the model, and based on the change, determine the threshold value of the heat index information in the strategy conditions corresponding to the target strategy type; and generate a model scheduling strategy table containing the model scheduling strategy of the target strategy type based on the strategy conditions corresponding to the target strategy type, the correspondence between the strategy conditions and the scheduling operation, and the threshold value.

[0190] In some embodiments, the processing module 1302 is specifically used for each model to: perform policy condition matching in the model scheduling policy table based on the current operation and maintenance monitoring information of the model to obtain the model scheduling policy currently matched by the model; and when multiple matched model scheduling policies are determined from the model scheduling policy table, and the scheduling operations of the multiple matched model scheduling policies conflict, the model is scheduled based on the model scheduling policy with the highest priority among the multiple matched model scheduling policies.

[0191] In some embodiments, the processing module 1302 is further configured to create a cache resource pool in memory before performing model scheduling on each model among multiple models, and establish a first communication link between the cache resource pool and the inference resource pool, and a second communication link between the cache resource pool and the executor; the cache resource pool is used to cache model file information and model environment configuration information; the processing module 1302 is specifically configured to migrate the model file information of the model to be migrated out of the inference resource pool to the cache resource pool based on the first communication link, and migrate the model environment configuration information of the model to be migrated out of the executor to the cache resource pool based on the second communication link; or, migrate the model file information of the model to be migrated in of the cache resource pool to the inference resource pool based on the first communication link, and migrate the model environment configuration information of the model to be migrated in of the cache resource pool to the executor based on the second communication link.

[0192] In some embodiments, the processing module 1302 is specifically used to: wait for the model to be migrated to process the currently accepted inference request; and perform the migration operation on the model to be migrated after it has finished processing the inference request.

[0193] In some embodiments, the processing module 1302 is specifically used to determine the model priority and / or model deployment time of at least one model to be migrated out, and to determine the migration order of at least one model to be migrated out based on the model priority and / or model deployment time; and to perform migration operations on at least one model to be migrated out in sequence based on the migration order.

[0194] In some embodiments, the processing module 1302 is specifically used to determine the model to be scheduled and the scheduling operation of the model to be scheduled from multiple models based on the model scheduling strategy table of multiple models and the current operation and maintenance monitoring information of each model in the multiple models; obtain the current running status of the model to be scheduled, and identify whether the current running status of the model to be scheduled supports the scheduling operation based on a first preset correspondence; the first preset correspondence is the correspondence between the running status and the supported scheduling operation; the running status of the model to be scheduled is at least used to characterize the deployment status of the model to be scheduled in the inference resource pool; if the current running status of the model to be scheduled supports the scheduling operation, perform the scheduling operation on the model to be scheduled.

[0195] In some embodiments, the processing module 1302 is further configured to, in response to a state switching event, switch the running state of the model to be scheduled based on the current running state of the model to be scheduled, the state switching event, and a second preset correspondence; wherein the second preset correspondence is used to represent the correspondence between the current running state, the state switching event, and the running state after the switch.

[0196] It should be noted that the above Figure 13 Modules in a module can also be called units; for example, a processing module can be called a processing unit. Additionally, in... Figure 13 In the embodiments shown, the names of the modules may not be the same as those shown in the figure. For example, the acquisition module may also be called the transceiver module or the communication module.

[0197] Figure 13If the various modules in the present disclosure are implemented as software functional modules and sold or used as independent products, they can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this disclosure, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a mobile phone, personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this disclosure. Storage media for storing computer software products include: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0198] In an exemplary embodiment, this disclosure also provides an electronic device. Figure 14 This is a schematic diagram illustrating the composition of an electronic device provided in an embodiment of this disclosure. For example... Figure 14 As shown, the electronic device includes a processor 1402, a communication interface 1403, and a bus 1404. As an example, the electronic device may also include a memory 1401.

[0199] Processor 1402 may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with this disclosure. Processor 1402 may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with this disclosure. Processor 1402 may also be a combination of functions implementing computation, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0200] Communication interface 1403 is used to connect to other devices via a communication network. This communication network can be Ethernet, wireless access network, wireless local area network (WLAN), etc.

[0201] The memory 1401 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0202] As one possible implementation, the memory 1401 can exist independently of the processor 1402. The memory 1401 can be connected to the processor 1402 via a bus 1404 and is used to store instructions or program code. When the processor 1402 calls and executes the instructions or program code stored in the memory 1401, it can implement the model scheduling method provided in the embodiments of this disclosure.

[0203] In another possible implementation, the memory 1401 can also be integrated with the processor 1402.

[0204] The 1404 bus can be an extended industry standard architecture (EISA) bus, etc. The 1404 bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 14 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0205] Through the above description of the implementation methods, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the model scheduling device can be divided into different functional modules to complete all or part of the functions described above.

[0206] In an exemplary embodiment, this disclosure also provides a computer-readable storage medium, which includes a non-transitory computer-readable storage medium storing computer instructions; when the computer instructions are executed in an electronic device, the electronic device causes the electronic device to perform the method as described in the above method embodiments.

[0207] In an exemplary embodiment, this disclosure also provides a computer program product including computer instructions that, when executed on an electronic device, cause the electronic device to perform the methods described in the above method embodiments.

[0208] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer-executable instructions. When these computer-executable instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer-executable instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer-executable instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, Bluetooth, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape) or an optical medium (e.g., DVD), etc.

[0209] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0210] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

[0211] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A model scheduling method, characterized in that, The method includes: Obtain operation and maintenance monitoring information for multiple models within a preset period; The operation and maintenance monitoring information of the multiple models is analyzed for call popularity from each of the at least one preset strategy dimension, and a popularity analysis report of each of the multiple models is obtained. Based on the heat analysis report of each model in the multiple models, a model scheduling strategy table for the multiple models is generated; Based on the model scheduling strategy table of the multiple models and the current operation and maintenance monitoring information of each model in the multiple models, model scheduling is performed on each model in the multiple models; the model scheduling is used to migrate models into or out of the inference resource pool.

2. The method according to claim 1, characterized in that, The generation of a model scheduling strategy table based on the popularity analysis report of each of the multiple models includes: Based on the popularity index information under at least one strategy dimension contained in the popularity analysis report of each model, determine the strategy type corresponding to each model, and establish the mapping relationship between each model and the strategy type. The popularity index information under at least one strategy dimension includes at least one of the following: peak scheduling time period, off-peak scheduling time period, resource load information of the model in the inference resource pool, inference performance information of the model, and inference task statistics of the model. Based on the model scheduling strategies corresponding to each strategy type and the mapping relationship, the model scheduling strategy table is constructed; The model scheduling strategy includes at least one of the following: tidal strategy, resource load strategy, inference performance strategy, and inference task strategy.

3. The method according to claim 2, characterized in that, The process of constructing the model scheduling policy table based on the model scheduling policies corresponding to each policy type and the mapping relationship includes: For each model: determine the target policy type corresponding to the model based on the mapping relationship; Based on the target strategy type corresponding to the model, analyze the changes in the heat index information related to the target strategy type in the heat analysis report of the model; Based on the amount of change, determine the threshold value of the heat index information in the strategy conditions corresponding to the target strategy type; Based on the policy conditions corresponding to the target policy type, the correspondence between the policy conditions and scheduling operations, and the threshold value, a model scheduling policy table containing model scheduling policies of the target policy type is generated.

4. The method according to claim 1, characterized in that, The model scheduling strategy table based on the multiple models and the current operation and maintenance monitoring information of each model in the multiple models are used to schedule the models, including: For each model: Based on the current operation and maintenance monitoring information of the model, perform policy condition matching in the model scheduling policy table to obtain the model scheduling policy currently matched for the model; If multiple matching model scheduling strategies are determined from the model scheduling strategy table, and the scheduling operations of the multiple matching model scheduling strategies conflict, the model is scheduled based on the model scheduling strategy with the highest priority among the multiple matching model scheduling strategies.

5. The method according to claim 1, characterized in that, Before performing model scheduling on each of the plurality of models, the method further includes: A cache resource pool is created in memory, and a first communication link is established between the cache resource pool and the inference resource pool, as well as a second communication link between the cache resource pool and the executor; the cache resource pool is used to cache model file information and model environment configuration information. The model scheduling of each of the plurality of models includes: Based on the first communication link, the model file information of the model to be migrated out in the inference resource pool is migrated to the cache resource pool, and based on the second communication link, the model environment configuration information of the model to be migrated out in the executor is migrated to the cache resource pool; or, Based on the first communication link, the model file information of the model to be imported from the cache resource pool is migrated to the inference resource pool, and based on the second communication link, the model environment configuration information of the model to be imported from the cache resource pool is migrated to the executor.

6. The method according to claim 1, characterized in that, The model scheduling of each of the plurality of models includes: For each model to be migrated out among the multiple models: wait for the model to be migrated out to process the currently accepted inference request; After the model to be migrated out has processed the inference request, the migration operation is performed on the model to be migrated out.

7. The method according to claim 1, characterized in that, The model scheduling of each of the plurality of models includes: Determine the model priority and / or model deployment time of at least one model to be migrated out, and determine the migration order of the at least one model to be migrated out based on the model priority and / or the model deployment time; Based on the migration order, the migration operation is performed sequentially on at least one model to be migrated.

8. The method according to claim 1, characterized in that, The model scheduling strategy table based on the multiple models and the current operation and maintenance monitoring information of each model in the multiple models are used to schedule the models, including: Based on the model scheduling strategy table of the multiple models and the current operation and maintenance monitoring information of each model in the multiple models, the model to be scheduled and the scheduling operation of the model to be scheduled are determined from the multiple models. The current running state of the model to be scheduled is obtained, and whether the current running state of the model to be scheduled supports the scheduling operation is identified based on a first preset correspondence; the first preset correspondence is the correspondence between the running state and the supported scheduling operation; the running state of the model to be scheduled is used to characterize the deployment status of the model to be scheduled in the inference resource pool. If the current running state of the model to be scheduled supports the scheduling operation, the scheduling operation is performed on the model to be scheduled.

9. The method according to claim 8, characterized in that, The method further includes: In response to a state switching event, the running state of the model to be scheduled is switched based on the current running state of the model to be scheduled, the state switching event, and the second preset correspondence. The second preset correspondence is used to represent the correspondence between the current running state, the state switching event, and the running state after the switch.

10. An electronic device, characterized in that, The electronic device includes: a memory and a processor; The memory stores instructions that the processor can execute; When the processor is configured to execute the instructions, the electronic device performs the method as described in any one of claims 1-9.

11. A computer program product, characterized in that, include: Computer instructions; When the computer instructions are executed in an electronic device, the electronic device causes the electronic device to perform the method as described in any one of claims 1-9.

Images