Processing method and device for target model reasoning service, equipment, medium and program product

Abstract

The invention relates to a processing method and device for a target model reasoning service, equipment, a medium and a program product. The method comprises the steps of obtaining at least one to-be-distributed reasoning task for the target model reasoning service; under the condition that a preset condition is met, aiming at each instance in the at least one instance, according to at least one of an execution progress condition of at least one distributed reasoning task of each instance and performance information of a model corresponding to each instance, determining a scheduling weight of each instance, the scheduling weight represents the scheduling priority of each instance in the at least one instance; and distributing the at least one reasoning task to be distributed to the at least one instance according to the scheduling weight of the at least one instance. Therefore, by dynamically adjusting the scheduling weight of the model reasoning service instance, the high availability, the response timeliness and the instance resource utilization rate of the model reasoning service can be improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to a processing method and device for a target model inference service, equipment, a medium and a program product. BACKGROUND

[0002] In a large model inference service, multiple instances are usually required to provide services together to meet the demand of a large number of inference task requests, and the instances of the inference service can process and output answers for the inference tasks, wherein the instance can be understood as a container in which a large model is deployed, and one container usually deploys one large model.

[0003] In order to realize high availability and good load balancing of the model inference service, a common solution is to use a load balancer as a traffic entrance to distribute inference task requests to multiple instances of the inference service backend. For example, the mainstream load balancer currently usually uses the following algorithms for traffic distribution: (1) Round Robin algorithm, that is, according to the time sequence of request arrival, requests are evenly distributed to each backend instance in turn. (2) Weighted Round Robin algorithm, that is, according to the performance configuration of the backend instance, different weights are assigned to each instance, and requests are distributed in proportion to the weights. (3) Least Connections algorithm, that is, new requests are assigned to the instance with the least number of currently established connections. (4) IP Hash algorithm, that is, according to the client IP address to calculate the hash value, requests with the same IP address are fixedly scheduled to the same backend instance, which can realize session stickiness. In addition, some commercial load balancers also provide intelligent routing algorithms based on delay, bandwidth and other monitoring indicators, or can also use a combination of different algorithms to obtain better load balancing effect. SUMMARY

[0004] When the above existing load balancing algorithm is applied to the distribution of inference tasks of the large model inference service, the inference task distribution is still not balanced, which causes problems such as low availability, slow response and low instance utilization efficiency of the inference service.

[0005] Therefore, the present disclosure provides a processing method and device for a target model inference service, equipment, a medium and a program product, which dynamically adjusts the scheduling weight of the instance in the target model inference service, and improves the high availability, response timeliness and instance utilization efficiency of the target model inference service.

[0006] According to an aspect of the present disclosure, a processing method for a target model inference service is provided. The target model inference service corresponds to at least one instance. The processing method comprises: obtaining at least one to-be-distributed inference task for the target model inference service; in a case where a preset condition is met, for each instance in the at least one instance, determining a scheduling weight of the each instance according to at least one of an execution progress of at least one distributed inference task of the each instance and performance information of a model corresponding to the each instance, wherein the scheduling weight represents a priority of the each instance in the at least one instance in terms of scheduling; and distributing the at least one to-be-distributed inference task to the at least one instance according to the scheduling weights of the at least one instance.

[0007] In some embodiments, the execution progress comprises at least a task duration of each distributed inference task, and the determining the scheduling weight of the each instance according to at least one of the execution progress of the at least one distributed inference task of the each instance and the performance information of the model corresponding to the each instance comprises: for the each instance, determining, according to at least one task duration corresponding to the at least one distributed inference task, a representative value representing a task duration level of the at least one distributed inference task as a whole; and determining the scheduling weight of the each instance according to the representative value, wherein the scheduling weight of the each instance is negatively correlated with the representative value.

[0008] In some embodiments, the determining, for the each instance, the representative value representing the task duration level of the at least one distributed inference task as a whole according to at least one task duration corresponding to the at least one distributed inference task comprises: determining, according to a size relationship between the task duration of the at least one distributed inference task and one or more duration thresholds, a number of distributed inference tasks falling into each time interval of a plurality of time intervals, wherein the plurality of time intervals are divided by the one or more duration thresholds; and determining the representative value according to the number of distributed inference tasks falling into each time interval.

[0009] In some embodiments, the execution progress of the at least one dispatched inference task further comprises a total number of tasks of the at least one dispatched inference task, and the determining the representative value representing the task duration level of the at least one dispatched inference task as a whole according to the at least one task duration corresponding to the at least one dispatched inference task comprises: determining whether the total number of tasks of the at least one dispatched inference task exceeds a total number of tasks threshold; and in a case where the total number of tasks exceeds the total number of tasks threshold, determining the representative value representing the task duration level of the at least one dispatched inference task as a whole according to the at least one task duration corresponding to the at least one dispatched inference task.

[0010] In some embodiments, the performance information of the model comprises at least one of a first word latency, a throughput, and an error rate, and the determining the scheduling weight of each instance according to at least one of the execution progress of the at least one dispatched inference task of each instance and the performance information of the model corresponding to each instance comprises: determining an evaluation value representing the performance level of the model corresponding to each instance according to at least one of the first word latency, the throughput, and the error rate of the model corresponding to each instance, wherein the evaluation value is negatively correlated with the first word latency and the error rate and is positively correlated with the throughput; and determining the scheduling weight of each instance according to the evaluation value, wherein the scheduling weight of each instance is positively correlated with the evaluation value.

[0011] In some embodiments, the distributing the at least one to-be-distributed inference task to the at least one instance according to the scheduling weight of the at least one instance comprises: determining a distribution priority of each to-be-distributed inference task, wherein the distribution priority indicates a priority of the distribution of each to-be-distributed inference task; and distributing the at least one to-be-distributed inference task to the at least one instance based on the distribution priority of the at least one to-be-distributed inference task and the scheduling weight of the at least one instance.

[0012] In some embodiments, the determining the distribution priority of each to-be-distributed inference task comprises: obtaining task information of each to-be-distributed inference task, wherein the task information comprises a task priority and a task response requirement of each to-be-distributed inference task, and the task priority represents an importance of each to-be-distributed inference task; and determining the distribution priority of each to-be-distributed inference task according to the task priority and the task response requirement of each to-be-distributed inference task.

[0013] In some embodiments, the target model inference service corresponds to a plurality of instances, and there are a plurality of the to-be-distributed inference tasks, and the distributing the at least one to-be-distributed inference task to the at least one instance based on the distribution priority of the at least one to-be-distributed inference task and the scheduling weight of the at least one instance comprises: determining at least one candidate instance from the plurality of instances based on the scheduling weights of the plurality of instances; determining at least one target inference task from the plurality of to-be-distributed inference tasks based on the distribution priorities of the plurality of to-be-distributed inference tasks; and distributing the at least one target inference task to the at least one candidate instance.

[0014] In some embodiments, different types of models are deployed in the plurality of instances, and the distributing the at least one target inference task to the at least one candidate instance comprises: for any target inference task, determining a target instance matching the target inference task from the at least one candidate instance, wherein the type of the model deployed in the target instance matches the type of the model required by the target inference task; and distributing the target inference task to the target instance.

[0015] In some embodiments, the method further comprises: obtaining inference task flow and service performance data of the target model inference service, wherein the inference task flow represents the number of to-be-distributed inference tasks received for the target model inference service per unit time; and the service performance data comprises average model performance and / or average resource usage corresponding to the at least one instance; and adjusting the number of instances corresponding to the target model inference service according to the inference task flow and the service performance data.

[0016] In some embodiments, the adjusting the number of instances corresponding to the target model inference service according to the inference task flow and the service performance data comprises: in a case where the inference task flow exceeds a first flow threshold, and / or the average resource usage exceeds a first resource threshold, and / or the average model performance is lower than a first performance threshold, determining a first number of instances to be added, and starting new instances of the first number of instances; and in a case where the inference task flow is lower than a second flow threshold, and / or the average resource usage is lower than a second resource threshold, and / or the average model performance is higher than a second performance threshold, determining a second number of instances to be closed, and closing existing instances of the second number of instances.

[0017] According to another aspect of the present disclosure, a processing apparatus for a target model inference service is provided, the target model inference service corresponding to at least one instance, the processing apparatus comprising: an obtaining module configured to obtain at least one to-be-distributed inference task for the target model inference service; a determining module configured to, when a preset condition is met, determine, for each instance of the at least one instance, a scheduling weight of the each instance according to at least one of an execution progress of at least one distributed inference task of the each instance and performance information of a model corresponding to the each instance, wherein the scheduling weight represents a priority of the each instance in the at least one instance; and a distributing module configured to distribute the at least one to-be-distributed inference task to the at least one instance according to the scheduling weights of the at least one instance.

[0018] According to another aspect of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored on the memory, the processor executing the computer program to implement the steps of the above method.

[0019] According to another aspect of the present disclosure, a non-volatile computer readable storage medium is provided, having a computer program stored thereon, the computer program being executed by a processor to implement the steps of the above method.

[0020] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, or a non-volatile computer readable storage medium carrying the computer program, the computer program being executed by a processor to implement the steps of the above method.

[0021] According to aspects of embodiments of the present disclosure, by determining the scheduling weight according to at least one of the execution progress of the at least one distributed inference task of each instance and the performance information of the model corresponding to each instance, dynamic adjustment of the scheduling weight can be achieved, and by distributing each instance into the at least one instance of the target model inference service according to the scheduling weight of each instance, scheduling based on the dynamically adjusted scheduling weight of each instance can be achieved, more balanced task distribution can be achieved, thereby facilitating improvement of high availability, response timeliness, and instance utilization efficiency of the model inference service.

[0022] Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0023] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the present disclosure and serve to explain the principles of the present disclosure.

[0024] Figure 1A flowchart illustrating a processing method for a target model inference service according to some embodiments of the present disclosure is shown.

[0025] Figure 2 A schematic diagram of the framework of a processing system for a target model inference service provided according to some embodiments of the present disclosure is shown.

[0026] Figure 3 A schematic diagram illustrating a processing flow of a processing system according to some embodiments of the present disclosure is shown.

[0027] Figure 4 A block diagram of a processing apparatus for a target model inference service is shown according to some embodiments of the present disclosure.

[0028] Figure 5 A block diagram of an electronic device according to some embodiments of the present disclosure is shown. Detailed Implementation

[0029] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0030] As used herein, the terms “comprising,” “including,” “having,” or variations thereof are open-ended and include one or more of the stated features, integrals, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integrals, elements, steps, components, functions, or groups thereof.

[0031] When an element is referred to as “connected,” “coupled,” “responding,” or a variation thereof relative to another element, it may be directly connected, coupled, or responding to another element, or there may be an intermediate element present.

[0032] Although the terms first, second, third, etc., may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Therefore, without departing from the teachings of the inventive concept, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.

[0033] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0034] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0035] As mentioned above, existing load balancing algorithms still suffer from problems such as insufficient availability, untimely response, and low instance utilization efficiency when applied to the distribution of inference tasks for large model inference services. The inventors have found that existing load balancing algorithms have some shortcomings when handling traffic from large model inference services: (1) They do not fully consider the special characteristics of large model inference, such as the long duration of a single inference task and the large amount of instance resource consumption. Simple load balancing based on the number of connections may lead to excessive load on individual instances. (2) They do not make full use of the contextual information available in large model inference services, such as model type, model performance, and task priority, and lack targeted scheduling strategies. (3) They lack fault tolerance handling for abnormal situations, such as instance failures and task interruptions, which may lead to request failures or increased latency.

[0036] In view of this, the present disclosure proposes a processing method for target model inference services, which can improve the high availability, timely response, and instance utilization efficiency of large model inference services, including but not limited to: (1) proposing an efficient scheduling algorithm based on the characteristics of large model inference tasks to achieve a more balanced task allocation. (2) utilizing the context information of large model inference tasks to determine the scheduling strategy and improve the responsiveness of inference tasks. (3) enhancing exception handling capabilities to improve the fault tolerance and stability of inference services.

[0037] In practical applications, the processing method for the target model inference service in this disclosure can be executed on the server side. For example, it can be executed in a server, specifically by a load balancing component / device deployed in the server. The server can be located in the cloud or locally, and can be a physical device or a virtual device, such as a virtual machine or container. For example, the server can be a node in a computing cluster used for the inference service, or it can be a node independent of the computer cluster. In some embodiments, the server can have wireless communication capabilities, wherein the wireless communication capabilities can be set in the server's chip (system) or other components, and the wireless communication capabilities refer to the ability to connect with other servers or terminal devices via wireless connection methods such as Wi-Fi and Bluetooth. In other embodiments, the server involved in this disclosure can also have the ability to communicate via a wired connection. Taking the server as an example, the server can obtain at least one inference task to be distributed for the target model inference service and execute the processing method of this disclosure to distribute the at least one inference task to be distributed to at least one instance corresponding to the target model inference service.

[0038] Figure 1 A flowchart illustrating a processing method for a target model inference service according to some embodiments of this disclosure is provided. The target model inference service corresponds to at least one instance, meaning that the target model inference service is provided by at least one instance. As mentioned above, an instance can be understood as a container that deploys a model (such as a large model). Figure 1 As shown, the processing method includes steps S11 to S13.

[0039] In step S11, at least one inference task to be distributed is obtained for the target model inference service.

[0040] Obtaining at least one undistributed inference task for the target model inference service can be understood as receiving at least one model inference request for the target model inference service. Each model inference request can carry one undistributed inference task, and at least one undistributed inference task can be obtained from the at least one model inference request. An undistributed inference task can be understood as an inference task to be distributed to at least one instance corresponding to the target model inference service.

[0041] Step S12: Under the condition of satisfying the preset conditions, for each instance in at least one instance, determine the scheduling weight of each instance based on at least one of the execution progress of at least one distributed inference task of each instance and the performance information of the model corresponding to each instance, wherein the scheduling weight represents the priority of each instance being scheduled in the at least one instance.

[0042] It should be noted that there is no strict execution order between obtaining the inference task to be distributed in step S11 and determining the scheduling weight of each instance in step S12. For example, the task to be distributed can be obtained first, and then the scheduling weight of each instance can be determined; or, the scheduling weight of each instance can be determined at the same time as obtaining the task to be distributed; or, the task to be distributed can be obtained periodically, and the scheduling weight of each instance can be determined periodically. This embodiment of the present disclosure does not limit this.

[0043] The preset conditions can be understood as the conditions that trigger the determination of the scheduling weight of each instance. For example, the preset conditions can be met by determining the scheduling weight after receiving a new inference task to be distributed or a new model inference request, or by determining the scheduling weight periodically, or by determining the scheduling weight when the number of at least one inference task to be distributed exceeds the threshold of the number of tasks to be distributed (which can be understood as the discovery of a large inference request traffic). This can be customized according to the actual situation, and this embodiment of the disclosure does not limit it.

[0044] Distributed inference tasks represent inference tasks that have been distributed to each instance. Specifically, they can be inference tasks that are being executed or waiting to be executed in each instance's task queue. Execution progress represents the execution status of the distributed inference tasks for each instance, specifically including the total number of distributed inference tasks for each instance (i.e., the total number of tasks in the instance's task queue) and the duration of each distributed inference task. The duration of any distributed inference task can be understood as the estimated time required for an instance to execute that distributed inference task. In practical applications, the task duration of the distributed inference task can be carried in the model inference request. In this case, the task duration of the distributed inference task can be obtained from the received model inference request. The duration of any distributed inference task can also be estimated using methods such as model prediction. This disclosure does not limit the estimation method for the task duration of any pending inference task. It should be understood that pending inference tasks and distributed inference tasks are only used to distinguish whether an inference task has been distributed. A task identifier can be used to mark whether a pending inference task and a distributed inference task are the same inference task.

[0045] Each instance's corresponding model can be understood as the model deployed in the container corresponding to each instance. When the target model inference service is provided by multiple instances, the model types corresponding to different instances can be the same, for example, all being ChatGPT models, or they can be different, for example, some instances using ChatGPT models and others using deepseek models. This disclosure does not impose any restrictions on this. The performance information of the instance's corresponding model can characterize the model's performance when performing inference tasks. For example, the model's performance information can include at least one of: first-to-first-token latency, throughput, and error rate. First-to-first-token latency refers to the time taken from when the user sends a request (i.e., the model receives complete input information) to when the model generates and returns the first output token (usually corresponding to the first character or word). Throughput refers to the total number of tokens that the model can process and output per unit of time. Error rate refers to the percentage of failed requests (number of failed tasks) of the model out of the total number of requests (total number of tasks). The error rate measures the proportion of inference requests that fail to execute. An inference request may fail to complete successfully and return the expected result for various reasons. In practical applications, the model information of the corresponding model of each instance can be monitored and collected by setting monitoring scripts in each instance. This embodiment of the disclosure does not limit the method of obtaining the model information of the corresponding model of each instance.

[0046] By determining the retrieval weight of each instance under the aforementioned preset conditions, the scheduling weight of each instance can be dynamically adjusted based on at least one of the execution progress of at least one distributed inference task of each instance and the performance information of the model corresponding to each instance. In some embodiments, the adjustment operation of determining the scheduling weight can be to increase, decrease, or remain unchanged; for example, if the total number of distributed inference tasks of an instance exceeds a total task threshold, the scheduling weight of that instance can be reduced; if the total number of distributed inference tasks of an instance does not exceed the total task threshold, the scheduling weight of that instance can be increased or remained unchanged; as another example, if the model corresponding to an instance has at least one of the following performance anomalies: first-word latency exceeds a specified latency threshold, throughput is lower than a specified throughput threshold, and error rate exceeds a specified error rate threshold, the current scheduling weight of that instance can be reduced; if the model corresponding to an instance does not have any of the above performance anomalies, the scheduling weight of that instance can be increased or remained unchanged, etc., and this disclosure embodiment does not limit this.

[0047] In one possible implementation, as described above, the execution progress status may include the task duration of each distributed inference task. In step S12, determining the scheduling weight of each instance based on at least one of the execution progress status of at least one distributed inference task for each instance and the performance information of the model corresponding to each instance may include:

[0048] Step S121: For each instance, determine a representative value representing the overall task duration level of at least one distributed inference task based on at least one task duration corresponding to at least one distributed inference task.

[0049] Step S122: Determine the scheduling weight of each instance based on the representative value, wherein the scheduling weight of each instance is negatively correlated with the representative value.

[0050] As mentioned above, the duration of any distributed inference task can be understood as the estimated time required for an instance to execute that distributed inference task. It should be understood that the longer the duration of an instance's distributed inference task, the longer the time required for the instance to complete the distributed inference task, which means that the instance will execute new inference tasks later, and thus the lower the scheduling weight of the instance can be.

[0051] In some embodiments, in step S121 above, for each instance, determining a representative value characterizing the overall task duration level of at least one distributed inference task based on at least one task duration corresponding to at least one distributed inference task includes:

[0052] Based on the relationship between the duration of at least one distributed inference task and one or more duration thresholds, determine the number of distributed inference tasks falling into each of multiple time intervals, wherein the multiple time intervals are divided by one or more duration thresholds; determine a representative value based on the number of distributed inference tasks falling into each time interval.

[0053] For example, the aforementioned representative value may be the number of distributed inference tasks for each instance that fall into at least one specified time interval among multiple time intervals; or it may be the average, median, etc. of the number of distributed inference tasks that fall into multiple specified time intervals among multiple time intervals, and this disclosure embodiment does not limit this.

[0054] It should be understood that users can set one or more duration thresholds. For example, a duration threshold of 5 minutes can be set, or multiple duration thresholds of 5 minutes and 2 minutes can be set, thereby dividing multiple time intervals, such as the intervals of 0 to 5 minutes and 5 minutes to infinity, or the intervals of 0 to 2 minutes, 2 minutes to 5 minutes, and 5 minutes to infinity. Based on the relationship between the task duration of each distributed inference task and the various duration thresholds, the number of distributed inference tasks falling into each of the multiple time intervals can be obtained. Furthermore, for example, the number of distributed inference tasks falling into a specified time interval (e.g., the time interval of 5 minutes to infinity) can be used as a representative value, or the number of distributed inference tasks falling into two specified time intervals (e.g., the time interval of 5 minutes to infinity and the time interval of 2 minutes to 5 minutes) can be used as a representative value, or the average of the number of distributed inference tasks falling into two specified time intervals (e.g., the time interval of 5 minutes to infinity and the time interval of 2 minutes to 5 minutes) can be used as a representative value. This disclosure does not limit the scope of the embodiments.

[0055] In some embodiments, when the representative value is the number of distributed inference tasks of each instance that fall into at least one specified time interval among multiple time intervals, the step S122 above, determining the scheduling weight of each instance based on the representative value, may further include: reducing the scheduling weight of the instance if the number of distributed inference tasks of any instance that fall into at least one specified time interval among multiple time intervals exceeds a specified task number threshold, or if the average, median, or other values ​​of the number of distributed inference tasks of at least one specified time interval among multiple time intervals exceed a specified threshold.

[0056] In some embodiments, taking the existence of a duration threshold as an example, the representative value can be the number of distributed inference tasks in each instance whose task duration exceeds the duration threshold; or, it can be the sum, maximum, average, median, etc., of the task durations of at least one distributed inference task in each instance, which is not limited in this embodiment. For example, in step S121 above, for each instance, determining the representative value representing the overall task duration level of at least one distributed inference task based on the task duration corresponding to at least one distributed inference task can include: determining whether there are distributed inference tasks in each instance whose task duration exceeds the duration threshold; if there are distributed inference tasks in any instance whose task duration exceeds the duration threshold, determining the number of distributed inference tasks in that instance whose task duration exceeds the duration threshold to obtain the representative value; or, it can also include: obtaining the representative value by calculating the sum, maximum, average, median, etc., of the task durations of at least one distributed inference task in each instance.

[0057] In some embodiments, where the representative value is the number of distributed inference tasks in at least one distributed inference task of each instance whose task duration exceeds a duration threshold, step S122 above, determining the scheduling weight of each instance based on the representative value, may include: reducing the scheduling weight of the instance if the number of distributed inference tasks in any instance whose task duration exceeds the duration threshold exceeds a specified task number threshold. It should be understood that the more distributed inference tasks whose task duration exceeds the duration threshold, the longer the time required for the instance to complete the distributed inference tasks, and therefore the lower the scheduling weight can be.

[0058] In some embodiments, when the representative value is the sum, maximum, average, median, etc. of the task durations of at least one distributed inference task for each instance, step S122 above, determining the scheduling weight of each instance based on the representative value, may further include: reducing the scheduling weight of an instance if one or more of the sum, maximum, average, and median of the task durations of at least one distributed inference task for any instance exceeds a specified threshold. It should be understood that if the sum, maximum, average, and median of the task durations of at least one distributed inference task for an instance are larger, it indicates that the instance requires a longer time to complete the distributed inference task, and therefore the scheduling weight can be lower.

[0059] In some embodiments, as described above, the execution progress status may further include the total number of at least one distributed inference task for each instance. In step S121 above, determining a representative value representing the overall task duration level of the at least one distributed inference task for each instance, based on the at least one task duration corresponding to the at least one distributed inference task, may include:

[0060] Step S1211: For each instance, determine whether the total number of at least one distributed inference task exceeds the total number of tasks threshold.

[0061] Step S1212: If the total number of tasks exceeds the threshold for the total number of tasks, determine a representative value representing the overall task duration level of at least one distributed inference task based on the task duration corresponding to at least one distributed inference task.

[0062] In some embodiments, if the total number of distributed inference tasks of an instance (i.e. the total number of tasks in the instance's task queue) does not exceed the total number of tasks threshold, the scheduling weight of the instance may remain unchanged. Of course, the final scheduling weight of the instance may be determined based on other judgment conditions, such as the performance information of the model corresponding to the instance and the resource usage of the instance. This disclosure does not limit this.

[0063] In practical applications, if the total number of at least one distributed inference task in an instance exceeds the total task threshold, it indicates that there are many inference tasks waiting to be executed and those currently being executed in that instance. In this case, if there are distributed inference tasks with long durations and a large number of such tasks in the instance, it indicates that the instance is in a head-blocking state, meaning that the instance has a heavy task load and is not suitable for distributing tasks to it. Therefore, a head-blocking detector can be introduced into each instance to detect in real time whether the task queue length (total number of tasks in the task queue) in each instance exceeds the total task threshold, and whether the number of tasks in the task queue whose duration exceeds the duration threshold exceeds the total task threshold. If both exceed the threshold, the instance is considered to be in a head-blocking state. For head-blocked instances, the scheduling weight of the instance can be temporarily reduced to the minimum to decrease the probability of new inference tasks being distributed to that instance.

[0064] Therefore, in step S1211 above, determining whether the total number of at least one distributed inference task exceeds the total number of tasks threshold can be understood as, for any instance, determining whether the total number of at least one distributed inference task (i.e., inference task in the task queue) of that instance exceeds the total number of tasks threshold; furthermore, in step S1212, if the total number of at least one distributed inference task (i.e., inference task in the task queue) of that instance exceeds the total number of tasks threshold, the above-mentioned determination of a representative value representing the overall task duration level of at least one distributed inference task based on the at least one task duration corresponding to at least one distributed inference task, and the above-mentioned determination of the scheduling weight of each instance based on the representative value.

[0065] In some embodiments, the following can be specifically performed: if the total number of at least one distributed inference task in the instance exceeds a total task threshold, determine whether there is a distributed inference task among the at least one distributed inference task in the instance whose task duration exceeds a duration threshold; and if there is a distributed inference task among the at least one distributed inference task in the instance whose task duration exceeds a duration threshold and the number of distributed inference tasks whose task duration exceeds a time duration threshold exceeds a specified task number threshold, adjust the scheduling weight of the instance to the lowest value. Of course, the scheduling weight of the instance can also be appropriately reduced, for example, it can be reduced to below a specified weight threshold, and this disclosure does not limit this.

[0066] In this embodiment of the disclosure, by first determining whether the total number of distributed inference tasks of an instance exceeds a threshold, and then combining the task duration to determine the scheduling weight of the instance, it is possible to effectively detect whether an instance is in a head-blocking state (i.e., the task load is too high). For instances in a head-blocking state, the scheduling weight of the instance is reduced, thereby reducing the probability that the head-blocking instance will be reassigned inference tasks, or reducing the number of inference tasks assigned to the head-blocking instance. This is conducive to achieving a more balanced task distribution and improving the high availability, response timeliness and instance utilization efficiency of the target model inference service.

[0067] It should be noted that the thresholds mentioned in the above embodiments of this disclosure, such as the total number of tasks threshold, the number of tasks threshold, the duration threshold, and the specified threshold, can all be configured based on experience or the development and debugging process, and this disclosure does not impose any restrictions on them.

[0068] As described above, the performance information of the model corresponding to the instance may include at least one of the following: first-word latency, throughput, and error rate. Therefore, in step S12, determining the scheduling weight of each instance based on the execution progress of at least one distributed inference task for each instance and at least one of the performance information of the model corresponding to each instance includes:

[0069] Step S123: Determine an evaluation value characterizing the performance level of the model corresponding to each instance based on at least one of the first-word latency, throughput, and error rate of the model corresponding to each instance, wherein the evaluation value is negatively correlated with the first-word latency and error rate and positively correlated with the throughput.

[0070] Step S124: Determine the scheduling weight of each instance based on the evaluation value, wherein the scheduling weight of each instance is positively correlated with the evaluation value.

[0071] In step S123, the evaluation value is negatively correlated with first-word latency and error rate, and positively correlated with throughput. This means that the lower the first-word latency and error rate, and the higher the throughput, the higher the evaluation value, and the higher the scheduling weight. Therefore, a pre-defined correspondence between the evaluation value and first-word latency, error rate, and throughput can be established. Based on this correspondence, the performance level evaluation value of the model corresponding to each instance can be determined according to at least one of the first-word latency, throughput, and error rate of the model corresponding to each instance.

[0072] In some embodiments, in step S124, the scheduling weight of each instance can be determined based on the relationship between the evaluation value and a specified evaluation threshold. For example, if the evaluation value is lower than the evaluation threshold, the scheduling weight of the instance is reduced; if the evaluation value is higher than the evaluation threshold, the scheduling weight of the instance is increased or remains unchanged. Alternatively, multiple evaluation value intervals can be preset, and the weight adjustment amount and adjustment operation (increase, decrease, or remain unchanged) corresponding to each evaluation value interval can be set. In this way, after obtaining the evaluation value representing the performance level of the model corresponding to each instance through step S123, the evaluation value interval in which the evaluation value falls is determined, and the scheduling weight of the instance is adjusted based on the weight adjustment amount and adjustment operation corresponding to the evaluation value interval in which the evaluation value falls. This embodiment of the present disclosure does not limit this.

[0073] In some embodiments, in step S124, the scheduling weight of an instance can be adjusted directly based on at least one of the first-word latency, throughput, and error rate of the model corresponding to each instance. Specifically, when any instance's corresponding model exhibits at least one performance anomaly, such as a first-word latency exceeding a specified latency threshold, throughput falling below a specified throughput threshold, or an error rate exceeding a specified error rate threshold, the scheduling weight of that instance can be reduced based on the at least one performance anomaly present in the model corresponding to that instance. It should be understood that different weight reduction amounts can be set for different performance anomalies, and thus the scheduling weight of the instance can be reduced based on the weight reduction amount set for the at least one performance anomaly that has occurred. The more performance anomalies that occur, the greater the reduction in scheduling weight. It should be understood that the aforementioned thresholds, namely the specified latency threshold, the specified throughput threshold, and the specified error rate threshold, can all be configured based on experience or according to the development and debugging process, etc., and this embodiment of the disclosure does not impose any limitations on this.

[0074] In some embodiments, the scheduling weight of each instance can be determined by at least one of the execution progress of at least one distributed inference task of each instance and the performance information of the model corresponding to each instance in the following manner.

[0075] First, for each instance, a representative value representing the overall task duration level of the at least one distributed inference task is determined based on at least one task duration corresponding to the at least one distributed inference task.

[0076] Then, based on at least one of the first-word latency, throughput, and error rate of the model corresponding to each instance, an evaluation value characterizing the performance level of the model corresponding to each instance is determined, wherein the evaluation value is negatively correlated with the first-word latency and error rate and positively correlated with the throughput.

[0077] Finally, a scheduling weight for each instance is determined based on the representative value and the evaluated value, wherein the scheduling weight of each instance is positively correlated with the evaluated value and negatively correlated with the representative value. For example, the average or weighted average of the representative value and the evaluated value can be used as the scheduling weight for each instance.

[0078] In some embodiments, the resource usage of each instance can also be obtained, and the scheduling weight of each instance can be determined based on the resource usage of each instance. Thus, in step S12 above, determining the scheduling weight of each instance based on at least one of the execution progress of at least one distributed inference task of each instance and the performance information of the model corresponding to each instance includes:

[0079] The scheduling weight of each instance is determined based on at least one of the execution progress of at least one distributed inference task for each instance, the performance information of the model corresponding to each instance, and the resource usage of each instance. The resource usage includes at least one of the following: CPU utilization, GPU utilization, memory usage, and disk usage.

[0080] The scheduling weight of each instance is determined based on its resource usage. This can include: for any instance, obtaining at least one of the following: CPU utilization, GPU utilization, memory usage, and disk usage; when the instance exhibits at least one of the following resource anomalies: CPU utilization exceeding a first utilization threshold, GPU utilization exceeding a second utilization threshold, memory usage exceeding a specified usage threshold, and disk usage exceeding a specified usage threshold, the scheduling weight of the instance is reduced based on this at least one resource anomaly. Different weight reduction amounts can be set for different resource anomalies, thus reducing the instance's scheduling weight based on the weight reduction amount set for each of the at least one resource anomaly. The more types of resource anomalies that occur, the greater the reduction in scheduling weight. It should be understood that the aforementioned thresholds, namely the first utilization threshold, the second utilization threshold, and the specified usage threshold, can all be configured based on experience or during the development and debugging process, etc., and this embodiment does not impose any limitations on this.

[0081] It should be noted that in step S12 of this embodiment, the scheduling weight of each instance can be adjusted based on at least one of the following: the execution progress of at least one distributed inference task for each instance, the performance information of the model corresponding to each instance, and the resource usage of each instance, thereby obtaining the final scheduling weight of each instance. For example, the scheduling weight of the instance can be adjusted comprehensively based on the above three criteria. For instance, based on the execution progress of the distributed inference tasks in the instance, the scheduling weight of the instance may remain unchanged; based on the performance information of the model, the scheduling weight of the instance may be reduced; and based on the resource usage, the scheduling weight of the instance may be reduced again, thereby obtaining the final scheduling weight of the instance. In some embodiments, different weights can also be set for different criteria, and the scheduling weights set for different criteria can be weighted to obtain the final scheduling weight of the instance.

[0082] In some embodiments, the running status of each instance can be monitored in real time. If an anomaly is detected (such as memory exhaustion, crash, etc.), the distribution of new inference tasks to that instance is immediately suspended. At the same time, the scheduling weight of that instance can be adjusted to the lowest level to reduce the probability of misscheduling the abnormal instance. The distribution of inference tasks to that instance is resumed after the instance recovers. Thus, the method may also include, for example, real-time monitoring of the running status of each instance, and if an anomaly is detected in the running status of any instance, suspending the distribution of inference tasks to the instance with the abnormal running status, and adjusting the scheduling weight of the instance with the abnormal running status to the lowest level (or possibly below a specified weight threshold) until the instance with the abnormal running status recovers. Then, based on the execution progress of the distributed inference tasks of the recovered instance, the model performance information, and the instance's resource usage, the scheduling weight of the recovered instance can be re-determined to facilitate the re-distribution of inference tasks to the recovered instance.

[0083] return Figure 1 In step S13, at least one inference task to be distributed is distributed to at least one instance according to the scheduling weight of at least one instance.

[0084] In some embodiments, in step S13, distributing at least one inference task to at least one instance according to the scheduling weight of at least one instance may include: sequentially distributing at least one inference task to at least one instance according to the scheduling priority of each instance as indicated by the scheduling weight of each instance, and the receiving time sequence of each inference task to be distributed (i.e., the time sequence in which the target model inference service receives the model inference task). It should be understood that the higher the priority indicated by the scheduling weight of an instance, the more preferentially the instance is distributed with inference tasks, and the earlier the receiving time sequence of the inference task to be distributed, the earlier the inference task to be distributed is distributed to the instance.

[0085] Considering that different inference tasks may arise in actual business scenarios, distributing inference tasks solely based on their reception timing may degrade the user experience. Therefore, in some embodiments, in step S13, distributing at least one inference task to the at least one instance according to the scheduling weight of at least one instance may include:

[0086] Step S131: Determine the distribution priority of each inference task to be distributed, wherein the distribution priority indicates the priority at which each inference task to be distributed is distributed;

[0087] Step S132: Based on the distribution priority of at least one inference task to be distributed and the scheduling weight of at least one instance, distribute at least one inference task to be distributed to at least one instance.

[0088] In step S131, the distribution priority of each inference task to be distributed can be set by the user who issued the model inference request. Thus, the distribution priority of each inference task to be distributed corresponding to each model inference request can be obtained from the model inference requests for the target model inference service.

[0089] Different business scenarios may generate inference tasks with different priorities and response requirements. Task priority represents the importance of each inference task. For example, inference tasks may be paid or free tasks. Paid tasks are more important than free tasks, so paid tasks have a higher priority and can be distributed to instances earlier. Response requirements indicate whether a model inference request (i.e., an inference task) requires a real-time response or reflect the timeliness of the response. For example, some inference tasks may require a real-time response (i.e., immediate output of model inference results), while others may not. For instance, if a user submits an inference task and is notified to check back in 24 hours, then simply providing the model inference results after 24 hours is sufficient. Tasks requiring real-time response should be distributed to instances earlier than those not requiring real-time response. It should be understood that task priority and task response requirements may be related, such as the higher the task response requirement of an inference task, the higher the task priority; or they may not be related, such as the task priority depending on whether the inference task to be distributed is a paid task, and the task response requirement depending on whether the inference task to be distributed requires a real-time response.

[0090] Therefore, in some embodiments, in step S131, the distribution priority of each inference task to be distributed can also be determined by the task priority and the task response requirements. Specifically, determining the distribution priority of each inference task to be distributed may include:

[0091] Obtain task information for each inference task to be distributed, wherein the task information includes the task priority and task response requirements for each inference task to be distributed, and the task priority represents the importance of each inference task to be distributed;

[0092] The distribution priority of each inference task is determined based on its task priority and task response requirements.

[0093] In some embodiments, an interactive interface for setting task information can be provided to the user. This interface may include options for setting task priority and task response requirements, allowing the user to configure task information. This task information can be carried in the model inference request sent to the target model inference service; therefore, the task information for the inference task to be distributed can be obtained from the model inference request. This disclosure does not limit the method of obtaining task information.

[0094] In some embodiments, determining the distribution priority of each inference task based on its task priority and task response requirement may include: determining the task weight value corresponding to the task priority and the response weight value corresponding to the task response requirement of each inference task based on a preset mapping relationship between task priority and task weight value, and a preset mapping relationship between task response requirement and response weight value; and determining the distribution priority of each inference task based on its corresponding task weight value and response weight value. Task priority and task weight value are positively correlated; the response weight value for tasks requiring real-time response is higher than that for tasks not requiring real-time response. For example, the sum, average, or weighted value of the task weight value and response weight value can be used as the distribution priority, and this embodiment does not limit this. Alternatively, a mapping relationship between task priority, task response requirement, and distribution priority can be directly set; for example, a higher task priority corresponds to a higher distribution priority, and a higher distribution priority for tasks requiring real-time response is higher than that for tasks not requiring real-time response. This mapping relationship can then be used to determine the distribution priority for any task priority and task response requirement, and this embodiment does not limit this.

[0095] In some embodiments, for step S132, based on the distribution priority of each inference task to be distributed and the scheduling weight of each instance, the higher the scheduling weight of an instance, the more priority the instance is given to the inference task, and the higher the distribution priority of the inference task to be distributed, the earlier the inference task to be distributed is distributed to the instance.

[0096] In some embodiments, when the target model inference service corresponds to multiple instances and there are multiple inference tasks to be distributed, step S132, based on the distribution priority of at least one inference task to be distributed and the scheduling weight of at least one instance, may include:

[0097] Step S1321: Based on the scheduling weights of multiple instances, determine at least one candidate instance from the multiple instances;

[0098] Step S1322: Based on the distribution priority of multiple inference tasks to be distributed, determine at least one target inference task from the multiple inference tasks to be distributed.

[0099] Step S1323: Distribute at least one target reasoning task to at least one candidate instance.

[0100] In step S1321, based on the scheduling weights of multiple instances, at least one candidate instance is determined from the multiple instances. This may include selecting at least one candidate instance from the multiple instances whose scheduling weight is greater than a specified weight threshold. The selected candidate instance may be an instance currently allowed to be scheduled, or an instance allowed to be assigned inference tasks. As mentioned above, instances may experience head-blocking (i.e., instance overload) or abnormal running status. Instances experiencing these conditions may be disallowed from scheduling, and their scheduling weight may be reduced to below the specified weight threshold, for example, to the lowest value below the specified weight threshold. Therefore, the candidate instance selected using the specified weight threshold may be an instance currently allowed to be assigned inference tasks.

[0101] In step S1322, based on the distribution priority of multiple pending inference tasks, at least one target inference task is determined from the multiple pending inference tasks. For example, this may include selecting at least one target inference task with the highest distribution priority from the multiple pending inference tasks. It should be understood that a higher distribution priority means the inference task should be distributed to the instance more preferentially. Therefore, the pending inference tasks can be sorted according to their distribution priority, and at least one target inference task with the highest distribution priority can be selected for priority distribution. The selected at least one target inference task may be a pending inference task that requires real-time response and has a high task priority. In this way, the response speed to high-priority inference tasks can be improved, enhancing the user experience.

[0102] In step S1323, after determining the target inference task and candidate instances, at least one target inference task can be directly distributed to at least one candidate instance. When there are multiple target inference tasks and multiple candidate instances, each target inference task can be distributed to a different candidate instance based on a strategy where the higher the scheduling weight of a candidate instance, the higher the priority of that candidate instance in receiving inference tasks, and the higher the distribution priority of a target inference task, the earlier that target inference task is distributed to a candidate instance.

[0103] As mentioned above, a single instance typically deploys one model within its container. When the target model inference service is provided by multiple instances, the types of models corresponding to different instances can be different. For example, some instances might use the chatGPT model, while others might use the deepseek model. Therefore, in some embodiments, when different types of models are deployed across multiple instances, step S1323, distributing at least one target inference task to the at least one candidate instance, may include:

[0104] For any target inference task, determine a target instance that matches the target inference task from at least one candidate instance, wherein the type of the model deployed in the target instance matches the type of the model required by the target inference task; distribute the target inference task to the target instance.

[0105] In some embodiments, where the target model inference service is provided by multiple instances and different types of models are deployed in the multiple instances, an option for selecting the model type can be provided in the user interface, allowing the user to select the model type required to perform the inference task. The model type can be represented by information such as the model name and the number of parameters used to distinguish different models. The model inference request can carry model information indicating the model type required to perform the inference task, thereby determining the target instance based on the model information of each target inference task. Therefore, determining the target instance matching the target inference task from at least one candidate instance can include:

[0106] For any target inference task, model information corresponding to the target inference task is obtained. This model information indicates the model type required to perform the target inference task. Based on the model information corresponding to the target inference task, a target instance matching the target inference task is determined from at least one candidate instance. The type of the model corresponding to the target instance is consistent with the model type indicated by the model information. It should be understood that the model information corresponding to any target inference task can be obtained from the model inference request corresponding to the target inference task, and this disclosure does not limit this aspect.

[0107] As described above, the model inference request for the target model inference service may carry data such as model information, task priority, task response requirements, and task duration. In some embodiments, after receiving any model inference request and obtaining such data from the model inference request, such data may be converted into a standardized format for easier application to the processing method of this disclosure embodiment.

[0108] According to the processing method of this disclosure, by determining the scheduling weight based on at least one of the execution progress of at least one distributed inference task of each instance and the performance information of the model corresponding to each instance, the scheduling weight can be dynamically adjusted. Then, each instance is distributed to at least one instance of the target model inference service according to the scheduling weight of each instance. This can achieve more balanced task distribution based on the dynamically adjusted scheduling weight of each instance, thereby improving the high availability, response timeliness and instance utilization efficiency of the model inference service.

[0109] Considering that the request traffic received by the model inference service fluctuates in actual situations, in order to cope with traffic fluctuations, in some embodiments, the method may further include:

[0110] Step S21: Obtain the inference task traffic and service performance data of the target model inference service. The inference task traffic represents the number of inference tasks to be distributed to the target model inference service received per unit time. The service performance data includes the average model performance and / or average resource usage corresponding to at least one instance.

[0111] Step S22: Adjust the number of instances corresponding to the target model inference service based on inference task traffic and service performance data.

[0112] As described above, by collecting performance information and / or resource usage of each instance corresponding to the target model inference service, the average performance information of at least one instance corresponding to the target model inference service can be calculated to obtain average model performance (including at least one of average first-word latency, average error rate, and average throughput), and / or the average resource usage of at least one instance corresponding to the target model inference service can be calculated to obtain average resource usage (including at least one of average CPU utilization, average GPU utilization, average memory usage, and average disk usage). The above-mentioned average model performance and average resource usage can characterize the current overall service performance of the target model inference service. Inference task traffic characterizes the number of inference tasks to be distributed to the target model inference service received per unit time, which can be understood as the number of model inference requests to the target model inference service received per unit time, such as requests per second. It should be understood that the embodiments of this disclosure do not limit the method of obtaining inference task traffic and service performance data.

[0113] It should be understood that if the inference task traffic is high and the service performance data is low, it means that the overall service performance of the target model inference service is no longer able to match the inference task traffic, and the number of instances corresponding to the target model inference service can be increased. Conversely, if the inference task traffic is low and the service performance data is high, it means that the overall service performance of the target model inference service is already excessive, and the number of instances corresponding to the target model inference service can be reduced. Therefore, in some embodiments, dynamically adjusting the number of instances corresponding to the target model inference service based on whether the inference task traffic and service performance data are compatible can improve instance utilization efficiency and reduce instance resource waste while enabling the target model inference service to cope with traffic fluctuations.

[0114] In some embodiments, the number of instances corresponding to the target model inference service can be adjusted by determining whether the inference task traffic and service performance data each exceed a certain threshold. This also allows the target model inference service to cope with traffic fluctuations while improving instance utilization efficiency and reducing instance resource waste. Specifically, in step S22, adjusting the number of instances corresponding to the target model inference service based on the inference task traffic and the service performance data includes:

[0115] If the inference task traffic exceeds the first traffic threshold, and / or the average resource usage exceeds the first resource threshold, and / or the average model performance is lower than the first performance threshold, determine the first number of new instances that need to be added, and start the first number of new instances to expand the number of instances corresponding to the target model inference service.

[0116] If the inference task traffic is lower than the second traffic threshold, and / or the average resource usage is lower than the second resource threshold, and / or the average model performance is higher than the second performance threshold, determine the number of second instances that need to be shut down, and shut down the existing instances of the second number of instances to reduce the number of instances in the model inference service.

[0117] If the inference task traffic is between the first traffic threshold and the second traffic threshold, the average resource usage is between the first resource threshold and the second resource threshold, and the average model performance is between the first performance threshold and the second performance threshold, the number of instances in the model inference service remains unchanged.

[0118] Determining the number of first instances to be added when inference task traffic exceeds a first traffic threshold, and / or average resource usage exceeds a first resource threshold, and / or average model performance falls below a first performance threshold, can be understood as determining the number of first instances to be added under at least one of the following conditions: inference task traffic exceeds a first traffic threshold, average resource usage exceeds a first resource threshold, and average model performance falls below a first performance threshold. In some embodiments, determining the number of first instances to be added may include: randomly generating a number of first instances within a certain range.

[0119] In some embodiments, determining the number of new instances to be added may include: if the inference task traffic exceeds a first traffic threshold, dividing the inference task traffic by the first traffic threshold and multiplying by the current number of instances to obtain a first candidate instance number; if the average resource usage exceeds a first resource threshold, dividing the average resource usage by the first resource threshold and multiplying by the current number of instances to obtain a second candidate instance number; if the average model performance is lower than a first performance threshold, dividing the average model performance by the first performance threshold and multiplying by the current number of instances to obtain a third candidate instance number; then, the number of new instances to be added can be determined based on the number of new instances, the number of new instances, and / or the number of new instances. For example, the maximum value among the number of new instances, and / or the number of new instances, and the number of new instances, and the number of new instances, can be selected as the number of new instances, or the average value of the number of new instances, and / or the number of new instances, and / or the number of new instances, and / or the number of new instances, can be calculated as the number of new instances. This disclosure does not limit this. In practical applications, after determining the number of new instances to be added, the number of new instances of the first instance number can be randomly started. For example, suppose the total number of requests per second (inference task traffic) for an inference service exceeds 1 million. A single instance cannot provide the inference service, and several or dozens of instances may be needed to meet the inference task traffic. In this case, several or dozens of new instances can be started.

[0120] Determining the number of second instances to be shut down when the inference task traffic is below the second traffic threshold, and / or the average resource usage is below the second resource threshold, and / or the average model performance is above the second performance threshold can be understood as determining the number of second instances to be shut down when at least one of the following conditions is met: the inference task traffic is below the second traffic threshold, the average resource usage is below the second resource threshold, and the average model performance is above the second performance threshold.

[0121] In some embodiments, determining the number of second instances to be shut down may include randomly generating the number of second instances within a certain range.

[0122] In other embodiments, a more suitable number of second instances can be determined based on inference task traffic and service performance data. Determining the number of second instances to be shut down may include: if the inference task traffic is lower than a second traffic threshold, dividing the inference task traffic by the second traffic threshold and multiplying by the current number of instances to obtain a fourth candidate instance number; if the average resource usage is lower than a second resource threshold, dividing the average resource usage by the second resource threshold and multiplying by the current number of instances to obtain a fifth candidate instance number; if the average model performance is higher than a second performance threshold, dividing the average model performance by the second performance threshold and multiplying by the current number of instances to obtain a sixth candidate instance number. Then, the number of second instances to be shut down can be determined based on the fourth, fifth, and / or sixth candidate instance numbers. For example, the minimum value among the fourth, fifth, and sixth candidate instance numbers can be selected as the second instance number, or the average value of the fourth, fifth, and / or sixth candidate instance numbers can be calculated as the second instance number. This disclosure does not limit this approach. In practical applications, after determining the number of second instances to be shut down, existing instances of that number can be randomly shut down. For example, suppose the inference service is currently provided by 5 instances, and the average CPU utilization of these 5 instances is less than 10%. In this case, it is believed that 5 instances are not needed to meet the user's inference service needs, and one or two instances can be randomly shut down.

[0123] It should be noted that the aforementioned thresholds, such as the first and second traffic thresholds, the first and second resource thresholds, and the first and second performance thresholds, can all be configured based on experience or during the development and debugging process, and this embodiment does not impose any restrictions on this. Furthermore, generally speaking, the first and second traffic thresholds, the first and second resource thresholds, and the first and second performance thresholds are different; however, in practice, the first and second traffic thresholds, the first and second resource thresholds, and the first and second performance thresholds can also be set to be the same, and this embodiment does not impose any restrictions on this either.

[0124] In some embodiments, steps S21 to S22 can be executed before steps S11 to S12, that is, the scaling of instances in the target model inference service can be performed first, and then the distribution of the inference task to be distributed can be performed. Of course, steps S21 to S22 can also be executed simultaneously with steps S11 to S12 or after them, and this embodiment of the present disclosure does not limit this.

[0125] According to embodiments of this disclosure, the number of instances corresponding to the target model inference service can be automatically adjusted based on inference task traffic and service performance data. This can effectively cope with traffic fluctuations while improving the high availability and instance utilization efficiency of the target model inference service and reducing instance resource waste.

[0126] Figure 2 A schematic diagram of the framework of a processing system for a target model inference service provided according to some embodiments of the present disclosure is shown. Figure 2 As shown, the system comprises: an application layer, a system module layer, and an inference service layer. The inference service layer is provided by multiple instances (Instance 1, Instance 2, Instance 3, Instance 4, and Instance 5).

[0127] The application layer is used to implement context configuration and security and permission configuration. Context configuration allows users to configure context information such as model information, task priority, task response requirements, and task duration. Security and permission configuration allows configuring authentication information for model inference requests. Authentication information determines whether a model inference request has permission to access the inference service. Therefore, when issuing a model inference request, authentication information can be included in the request. After receiving the model inference request, the system module layer can first extract the authentication information from the request, perform permission authentication and authorization, and then execute the request distribution (i.e., task distribution) operation after successful authentication. It should be understood that the security and permission configuration in the application layer can be automatically configured, or it can be written into the application layer code; this embodiment of the disclosure does not impose any limitations on this.

[0128] The system module layer may include: a context acquisition module, a traffic routing module, an automatic scaling module, and an observability module. The context acquisition module acquires and parses each received model inference request, and extracts at least some context information from the request, such as model information, task priority, task response requirements, and task duration. The traffic routing module intelligently routes and distributes model inference requests based on the context information within the request, as described in steps S11 to S13 of the above-described embodiments of this disclosure. The automatic scaling module automatically adjusts the number of instances based on inference task traffic and service performance data, as described in steps S21 to S22 of the above-described embodiments of this disclosure.

[0129] The observability module is used to monitor, log, and monitor traffic in real time for large model inference services, providing data support for the traffic routing and auto-scaling modules. Specifically, it can collect and monitor the performance information (such as first-word latency, error rate, throughput, etc.) and resource usage (such as CPU utilization, GPU utilization, memory usage, disk usage, etc.) of each instance's corresponding model. The observability module can also record critical business logs, including exception handling (i.e., exceptions that occur during instance operation and how they are handled) and decision information. Decision information serves as the basis for decisions from receiving each inference task to distributing it to an instance, including but not limited to the inference task's distribution priority, the task priority and task response requirements on which the distribution priority is based, and which instance the inference task is ultimately distributed to, and the scheduling weight of that instance. The observability module can also track and record the call chain of model inference requests (i.e., inference tasks to be distributed) between different modules. For example, the model inference request first enters the context acquisition module, then the traffic distribution module, and finally the nodes appearing in the entire chain where it is distributed to a specific instance.

[0130] Figure 3 A schematic diagram illustrating a processing flow of a processing system according to some embodiments of the present disclosure is shown. For example... Figure 3 As shown, the processing flow includes: the model inference request first arrives at the context acquisition module; the context acquisition module extracts key context information from the model inference request and sends it to the traffic routing module; the traffic routing module executes the processing methods of steps S11 to S13 of the above-described embodiments of this disclosure to distribute the model inference request to each instance in the inference service layer; the instances in the backend inference service layer process the model inference request (i.e., perform model inference) and return the inference result to the user. The observability module monitors the inference request traffic, the performance information of the model in the instance, and the resource usage of the instance in real time to provide data for the traffic routing module, provide guidance for the automatic scaling module, and also provide observable indicators for the user, such as recorded logs, monitored data, and recorded call chains. The automatic scaling module can start new instances or shut down unnecessary instances when it detects traffic peaks or resource bottlenecks, that is, execute the processing methods of steps S21 to S22 of the above-described embodiments of this disclosure.

[0131] According to the processing system and process flow of the above-described embodiments of this disclosure, the context acquisition module parses the model inference request to obtain the task to be inferred and context information such as task priority, task response requirements, and expected task duration related to the task to be inferred, providing a basis for intelligent routing decisions. During the processing of the target model inference service, the traffic routing module can generate targeted instance scheduling and task distribution strategies for different inference tasks based on the context information of the tasks to be distributed. This effectively improves the response capability of the target model inference service to inference tasks, especially for high-priority inference tasks. By determining the task distribution priority based on task priority and task response requirements, the response speed of high-priority and real-time tasks is improved. The traffic routing module detects whether an instance is in a head-blocking state based on the task duration and task queue length (i.e., the total number of distributed inference tasks). If a head-blocking state is detected, the scheduling weight of the instance is reduced, thereby reducing the probability of new inference tasks being distributed to blocked instances. The traffic routing module can also monitor the instance running status in real time, thereby stopping faulty instances in a timely manner and reducing the probability of inference tasks being distributed to faulty instances. The traffic routing module dynamically allocates inference tasks based on model performance information and resource usage. The auto-scaling module automatically scales up and down based on average model performance and average resource usage, dynamically adjusting the number of service instances to cope with traffic fluctuations and improve resource utilization efficiency. Observability model monitoring obtains model performance information and resource usage, providing data support for the traffic routing module's routing decisions.

[0132] Figure 4 A block diagram of a processing apparatus for a target model inference service is shown according to some embodiments of the present disclosure. Figure 4 As shown, the processing apparatus includes:

[0133] The acquisition module 401 is used to acquire at least one inference task to be distributed for the inference service of the target model;

[0134] The determining module 402 is configured to, under the condition of satisfying preset conditions, determine the scheduling weight of each instance for each of the at least one instances based on at least one of the execution progress of at least one distributed inference task of each instance and the performance information of the model corresponding to each instance, wherein the scheduling weight represents the priority of each instance being scheduled in the at least one instance.

[0135] The distribution module 403 is used to distribute the at least one inference task to be distributed to the at least one instance according to the scheduling weight of the at least one instance.

[0136] In some embodiments, the execution progress status includes at least the task duration of each distributed inference task. Determining the scheduling weight of each instance based on at least one of the execution progress status of at least one distributed inference task for each instance and the performance information of the model corresponding to each instance includes: for each instance, determining a representative value characterizing the overall task duration level of the at least one distributed inference task based on at least one task duration corresponding to the at least one distributed inference task; and determining the scheduling weight of each instance based on the representative value, wherein the scheduling weight of each instance is negatively correlated with the representative value.

[0137] In some embodiments, determining a representative value representing the overall task duration level of the at least one distributed inference task based on at least one task duration corresponding to the at least one distributed inference task for each instance includes: determining the number of distributed inference tasks falling into each of a plurality of time intervals based on the magnitude relationship between the task duration of the at least one distributed inference task and one or more duration thresholds, wherein the plurality of time intervals are divided by the one or more duration thresholds; and determining the representative value based on the number of distributed inference tasks falling into each time interval.

[0138] In some embodiments, the execution progress status further includes the total number of the at least one distributed inference task, and determining a representative value representing the overall task duration level of the at least one distributed inference task based on the duration of at least one task corresponding to the at least one distributed inference task includes: determining whether the total number of the at least one distributed inference task exceeds a total task threshold; if the total number of tasks exceeds the total task threshold, determining a representative value representing the overall task duration level of the at least one distributed inference task based on the duration of at least one task corresponding to the at least one distributed inference task.

[0139] In some embodiments, the performance information of the model includes at least one of first-word latency, throughput, and error rate. Determining the scheduling weight of each instance based on the execution progress of at least one distributed inference task for each instance and at least one of the performance information of the model corresponding to each instance includes: determining an evaluation value characterizing the performance level of the model corresponding to each instance based on at least one of the first-word latency, throughput, and error rate of the model corresponding to each instance, wherein the evaluation value is negatively correlated with the first-word latency and error rate and positively correlated with the throughput; and determining the scheduling weight of each instance based on the evaluation value, wherein the scheduling weight of each instance is positively correlated with the evaluation value.

[0140] In some embodiments, distributing the at least one inference task to be distributed to the at least one instance according to the scheduling weight of the at least one instance includes: determining the distribution priority of each inference task to be distributed, wherein the distribution priority indicates the priority at which each inference task to be distributed is distributed; and distributing the at least one inference task to be distributed to the at least one instance based on the distribution priority of the at least one inference task to be distributed and the scheduling weight of the at least one instance.

[0141] In some embodiments, determining the distribution priority of each inference task to be distributed includes: obtaining task information for each inference task to be distributed, wherein the task information includes the task priority and task response requirements of each inference task to be distributed, and the task priority characterizes the importance of each inference task to be distributed; and determining the distribution priority of each inference task based on the task priority and task response requirements of each inference task to be distributed.

[0142] In some embodiments, the target model inference service corresponds to multiple instances, and there are multiple inference tasks to be distributed. Distributing the at least one inference task to be distributed to the at least one instance based on the distribution priority of the at least one inference task to be distributed and the scheduling weight of the at least one instance includes: determining at least one candidate instance from the multiple instances based on the scheduling weight of the multiple instances; determining at least one target inference task from the multiple inference tasks to be distributed based on the distribution priority of the multiple inference tasks to be distributed; and distributing the at least one target inference task to the at least one candidate instance.

[0143] In some embodiments, different types of models are deployed in the plurality of instances, wherein distributing the at least one target inference task to the at least one candidate instance includes: for any target inference task, determining a target instance matching the target inference task from the at least one candidate instance, wherein the type of the model deployed in the target instance matches the type of the model required by the target inference task; and distributing the target inference task to the target instance.

[0144] In some embodiments, the apparatus further includes: an automatic scaling module, configured to acquire inference task traffic and service performance data of the target model inference service, wherein the inference task traffic represents the number of inference tasks to be distributed to the target model inference service received per unit time; the service performance data includes the average model performance and / or average resource usage corresponding to the at least one instance; and adjust the number of instances corresponding to the target model inference service based on the inference task traffic and the service performance data.

[0145] In some embodiments, adjusting the number of instances corresponding to the target model inference service based on the inference task traffic and the service performance data includes: determining the number of new instances to be added and starting the new instances of the first number of instances when the inference task traffic exceeds a first traffic threshold, and / or the average resource usage exceeds a first resource threshold, and / or the average model performance is lower than a first performance threshold; and determining the number of existing instances to be shut down when the inference task traffic is lower than a second traffic threshold, and / or the average resource usage is lower than a second resource threshold, and / or the average model performance is higher than a second performance threshold.

[0146] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to execute the processing method described in any of the foregoing embodiments. The specific implementation can be referred to the description of the processing method in the foregoing embodiments, and will not be repeated here.

[0147] This disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described processing method for a target model inference service.

[0148] This disclosure also provides a non-volatile computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described processing method for the target model inference service.

[0149] This disclosure also provides a computer program product, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above-described processing method for the target model inference service.

[0150] Figure 5 A block diagram of an electronic device 1900 according to some embodiments of the present disclosure is shown. For example, the electronic device 1900 may be provided as a server. (Refer to...) Figure 5 The apparatus 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the aforementioned processing methods.

[0151] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). Electronic device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM Mac OS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0152] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described processing method.

[0153] Computer-readable storage media can be tangible devices capable of holding and storing programs / instructions used by instruction execution devices. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0154] The computer program (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage medium in the respective computing / processing device.

[0155] The computer program (or computer program instructions) used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information of computer-readable program instructions. These electronic circuits can execute computer-readable program instructions to implement various aspects of this disclosure.

[0156] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0157] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0158] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0159] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0160] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A processing method for a target model inference service, characterized in that, The target model inference service corresponds to at least one instance, and the processing method includes: Obtain at least one inference task to be distributed for the inference service of the target model; Under the condition of satisfying the preset conditions, for each of the at least one instances, the scheduling weight of each instance is determined based on at least one of the execution progress of at least one distributed inference task of each instance and the performance information of the model corresponding to each instance, wherein the scheduling weight represents the priority of each instance being scheduled in the at least one instance. Based on the scheduling weight of the at least one instance, the at least one inference task to be distributed is distributed to the at least one instance.

2. The processing method according to claim 1, characterized in that, The execution progress status includes at least the task duration of each distributed inference task. Determining the scheduling weight of each instance based on at least one of the execution progress status of at least one distributed inference task for each instance and the performance information of the model corresponding to each instance includes: For each instance, a representative value representing the overall task duration level of the at least one distributed inference task is determined based on at least one task duration corresponding to the at least one distributed inference task; Based on the representative value, the scheduling weight of each instance is determined, wherein the scheduling weight of each instance is negatively correlated with the representative value.

3. The processing method according to claim 2, characterized in that, For each instance, determining a representative value representing the overall task duration level of the at least one distributed inference task based on at least one task duration corresponding to the at least one distributed inference task includes: Based on the relationship between the task duration of the at least one distributed inference task and one or more duration thresholds, determine the number of distributed inference tasks that fall into each of the multiple time intervals, wherein the multiple time intervals are divided by the one or more duration thresholds; The representative value is determined based on the number of distributed inference tasks falling into each time interval.

4. The processing method according to claim 2 or 3, characterized in that, The execution progress status also includes the total number of the at least one distributed inference task, and determining a representative value characterizing the overall task duration level of the at least one distributed inference task based on the duration of at least one task corresponding to the at least one distributed inference task includes: Determine whether the total number of the at least one distributed inference task exceeds the total number of tasks threshold; If the total number of tasks exceeds the total number of tasks threshold, a representative value representing the overall task duration level of the at least one distributed inference task is determined based on the task duration of at least one task corresponding to the at least one distributed inference task.

5. The processing method according to any one of claims 1-3, characterized in that, The model's performance information includes at least one of: first-word latency, throughput, and error rate. Determining the scheduling weight of each instance based on the execution progress of at least one distributed inference task for each instance and at least one of the model's performance information corresponding to each instance includes: Based on at least one of the first-word latency, throughput, and error rate of the model corresponding to each instance, an evaluation value characterizing the performance level of the model corresponding to each instance is determined, wherein the evaluation value is negatively correlated with the first-word latency and error rate and positively correlated with the throughput; Based on the evaluation value, a scheduling weight is determined for each instance, wherein the scheduling weight of each instance is positively correlated with the evaluation value.

6. The processing method according to any one of claims 1 to 3, characterized in that, The step of distributing the at least one inference task to be distributed to the at least one instance according to the scheduling weight of the at least one instance includes: Determine the distribution priority of each inference task to be distributed, wherein the distribution priority indicates the priority at which each inference task to be distributed is distributed; Based on the distribution priority of the at least one inference task to be distributed and the scheduling weight of the at least one instance, the at least one inference task to be distributed is distributed to the at least one instance.

7. The processing method according to claim 6, characterized in that, Determining the distribution priority of each inference task to be distributed includes: Obtain task information for each inference task to be distributed, wherein the task information includes the task priority and task response requirements of each inference task to be distributed, and the task priority represents the importance of each inference task to be distributed; The distribution priority of each inference task is determined based on the task priority and task response requirements of each inference task to be distributed.

8. The processing method according to claim 6, characterized in that, The target model inference service corresponds to multiple instances, and there are multiple inference tasks to be distributed. Distributing the at least one inference task to the at least one instance based on the distribution priority of the at least one inference task to be distributed and the scheduling weight of the at least one instance includes: Based on the scheduling weights of the multiple instances, at least one candidate instance is determined from the multiple instances; Based on the distribution priority of the plurality of reasoning tasks to be distributed, at least one target reasoning task is determined from the plurality of reasoning tasks to be distributed; The at least one target reasoning task is distributed to the at least one candidate instance.

9. The processing method according to claim 8, characterized in that, Different types of models are deployed in the multiple instances, wherein distributing the at least one target inference task to the at least one candidate instance includes: For any target inference task, a target instance matching the target inference task is determined from the at least one candidate instance, wherein the type of the model deployed in the target instance matches the type of model required by the target inference task; The target inference task is distributed to the target instance.

10. The processing method according to any one of claims 1 to 3, characterized in that, The method further includes: Obtain the inference task traffic and service performance data of the target model inference service. The inference task traffic represents the number of inference tasks to be distributed to the target model inference service received per unit time. The service performance data includes the average model performance and / or average resource usage corresponding to the at least one instance. The number of instances corresponding to the target model inference service is adjusted based on the inference task traffic and the service performance data.

11. The processing method according to claim 10, characterized in that, Adjusting the number of instances corresponding to the target model inference service based on the inference task traffic and the service performance data includes: If the inference task traffic exceeds a first traffic threshold, and / or the average resource usage exceeds a first resource threshold, and / or the average model performance is lower than a first performance threshold, determine the number of new instances required and launch the first number of new instances; and / or If the inference task traffic is lower than the second traffic threshold, and / or the average resource usage is lower than the second resource threshold, and / or the average model performance is higher than the second performance threshold, determine the number of second instances that need to be shut down, and shut down the existing instances of the second number of instances.

12. A processing apparatus for a target model inference service, characterized in that, The target model inference service corresponds to at least one instance, and the processing device includes: The acquisition module is used to acquire at least one inference task to be distributed for the inference service of the target model; The determining module is configured to, under the condition of satisfying preset conditions, determine the scheduling weight of each instance for each of the at least one instances based on at least one of the execution progress of at least one distributed inference task of each instance and the performance information of the model corresponding to each instance, wherein the scheduling weight represents the priority of scheduling each instance in the at least one instance. A distribution module is configured to distribute the at least one inference task to be distributed to the at least one instance based on the scheduling weight of the at least one instance.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 11.

14. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11.

Images