Task scheduling method and device for multi-model inference, equipment and storage medium

Abstract

The application relates to the field of data processing, and provides a task scheduling method and device for multi-model inference, computer equipment and a computer readable storage medium. The method divides a model library into an online model library stored in a memory and an offline model library stored in a disk, and determines all to-be-called models corresponding to each request task in a task request queue in advance based on a task model mapping table recording the mapping relationship between tasks and models. Then, the priority of each request task is determined based on the location of all to-be-called models corresponding to the request task, and the request task with all to-be-called models stored in the online model library is preferentially processed, so that the number of model loading and unloading is minimized, thereby greatly reducing the consumption of resources, further improving the response speed of the overall model and improving the model processing performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing and the field of financial technology, and in particular to a task scheduling method and device for multi-model inference, a computer device and a computer readable storage medium. BACKGROUND

[0002] With the development of technology, the application scenarios of multi-model inference are increasing. Multi-model inference refers to the execution of a task requiring multiple models for inference. In order to extract a large amount of information from a picture, inference needs to be performed during the extraction process, and multiple models need to be used simultaneously during the inference process. For example, to identify how many women are in a picture, the position and number of human faces need to be identified by a DNN model first, and then the gender of each human face needs to be identified by a gender recognition network. However, the resources of an edge server are very limited, and the memory space is small, which cannot store a large number of models, and only a part of the models can be loaded. When other models need to be used, the models need to be unloaded and reloaded. Frequent loading and unloading of models not only affects the processing performance of the models, but also causes waste of resources.

[0003] Therefore, how to improve the processing performance of the models while reducing resource consumption has become a technical problem to be solved at present. SUMMARY

[0004] The main purpose of the present application is to provide a task scheduling method, device and equipment for multi-model inference and a computer readable storage medium, which aims to improve the processing performance of the models while reducing resource consumption.

[0005] To achieve the above purpose, the present application provides a task scheduling method for multi-model inference, which comprises the following steps:

[0006] Obtain each request task in a task request queue, and determine all to-be-called models corresponding to each request task based on a task model mapping table;

[0007] Determine a first priority task in each request task based on all to-be-called models corresponding to each request task, wherein all to-be-called models corresponding to the first priority task are stored in an online model library, and the online model library is stored in a memory;

[0008] Call all to-be-called models corresponding to the first priority task in the online model library to process the first priority task, and process other tasks except the first priority task after the first priority task is processed, wherein at least one to-be-called model corresponding to the other tasks is stored in an offline model library, and the offline model library is stored in a disk.

[0009] Further, to achieve the above object, the present application provides a task scheduling device for multi-model inference, comprising:

[0010] A model calling determination module is configured to acquire each request task in the task request queue, and determine all to-be-called models corresponding to each request task based on a task model mapping table;

[0011] A priority task determination module is configured to determine a first priority task among the request tasks based on all to-be-called models corresponding to each request task, wherein all to-be-called models corresponding to the first priority task are stored in an online model library, and the online model library is stored in a memory;

[0012] A request queue processing module is configured to call all to-be-called models corresponding to the first priority task in the online model library, process the first priority task, and process other tasks except the first priority task after the first priority task is processed, wherein at least one to-be-called model corresponding to the other tasks is stored in an offline model library, and the offline model library is stored in a disk.

[0013] Further, to achieve the above object, the present application provides a computer device, comprising a processor, a memory, and a task scheduling program for multi-model inference stored in the memory and executable by the processor, wherein when the task scheduling program for multi-model inference is executed by the processor, the steps of the task scheduling method for multi-model inference are implemented.

[0014] Further, to achieve the above object, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a task scheduling program for multi-model inference, and when the task scheduling program for multi-model inference is executed by a processor, the steps of the task scheduling method for multi-model inference are implemented.

[0015] This application provides a task scheduling method for multi-model inference. The method obtains each request task in a task request queue and, based on a task model mapping table, determines all models to be invoked corresponding to each request task. Based on all models to be invoked corresponding to each request task, a first priority task is determined among the request tasks. All models to be invoked corresponding to the first priority task are stored in an online model library, which is stored in memory. All models to be invoked corresponding to the first priority task in the online model library are invoked to process the first priority task. After processing the first priority task, other tasks besides the first priority task are processed. At least one model to be invoked corresponding to each of the other tasks is stored in an offline model library, which is stored on disk. Through the above method, this application divides the model library into an online model library stored in memory and an offline model library stored on disk, and pre-determines all models to be invoked corresponding to each request task in the task request queue based on a task model mapping table that records the mapping relationship between tasks and models. Then, based on the location of all the models to be called corresponding to the request task, the priority of each request task is determined, and the request tasks for which all the models to be called are stored in the online model library are given priority processing to minimize the number of model loading and unloading, thereby significantly reducing resource consumption while further improving the overall model response speed and improving model processing performance. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the first embodiment of the task scheduling method for multi-model inference according to the present invention;

[0017] Figure 2 This is a flowchart illustrating the second embodiment of the task scheduling method for multi-model inference of the present invention;

[0018] Figure 3 This is a flowchart illustrating the third embodiment of the task scheduling method for multi-model inference of the present invention;

[0019] Figure 4 This is a schematic diagram of the task scheduling device for multi-model inference according to the present invention;

[0020] Figure 5 This is a schematic diagram of the hardware structure of the computer device involved in the embodiment of the present invention.

[0021] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] With reference to the drawings, the technical solutions in the embodiments of the present application will be clearly and completely described below, obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative work belong to the scope of protection of the present application.

[0023] The flowcharts shown in the drawings are only illustrative, not necessarily including all the contents and operations / steps, and not necessarily executed in the described order. For example, some operations / steps can also be decomposed, combined or partially merged, so the actual execution order can be changed according to the actual situation.

[0024] It should be understood that the terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification and the appended claims of the present application, unless otherwise clear from the context, the singular forms "a", "an" and "the" are intended to include the plural forms.

[0025] It should also be understood that the term "and / or" used in the specification and the appended claims of the present application means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0026] The task scheduling method for multi-model inference related to the embodiments of the present application is mainly applied to a computer device. The task scheduling generation device for multi-model inference can be a PC, a portable computer, a mobile terminal, and the like, which are devices with display and processing functions.

[0027] Some embodiments of the present application will be described in detail below with reference to the accompanying drawings. In the case of no conflict, the embodiments described below and the features in the embodiments can be combined with each other.

[0028] Referring to Figure 1 , Figure 1 The flowchart of the first embodiment of the task scheduling method for multi-model inference of the present application is shown in FIG. 1.

[0029] As Figure 1 shown, the embodiment of the present application provides a task scheduling method for multi-model inference, which includes steps S10 to S30.

[0030] In this embodiment, the task scheduling method includes the following steps:

[0031] Step S10, obtaining each request task in the task request queue, and determining all to-be-called models corresponding to each request task based on a task model mapping table;

[0032] At present, the resource of edge server is very limited, the memory space is small, and a large number of models cannot be stored, only part of the model can be loaded, and when other models are needed, the model needs to be unloaded and reloaded. Frequent loading and unloading of models not only affects the processing performance of the model, but also causes waste of resources.

[0033] In order to solve the above problems, the model library is divided into an online model library stored in the memory and an offline model library stored in the disk, and based on the task model mapping table recording the mapping relationship between the tasks and the models, the all to-be-called models corresponding to each request task in the task request queue are determined in advance. Then, based on the belonging position of the all to-be-called models corresponding to the request task, the priority of each request task is determined, and the request task whose all to-be-called models are stored in the online model library is preferentially processed, so as to minimize the number of model loading and unloading, thereby greatly reducing the consumption of resources, further improving the response speed of the overall model and improving the processing performance of the model.

[0034] Specifically, when the task request is received, the task request queue is generated based on each request task, and the waiting time of each request task is recorded. If the number of request tasks is N request , the waiting time of the first task is recorded as:

[0035]

[0036] Wherein, WaitMap is a key-value table, ReqIndex i is the index value of the i th request in the queue, and Waittime i is the waiting time of the i th request task in the task request queue.

[0037] Based on the task model mapping table, the all to-be-called models corresponding to each request task in the task request queue are determined.

[0038] Further, before the step S10, it further includes:

[0039] The all to-be-called models corresponding to each request task type are determined.

[0040] Based on the all to-be-called models corresponding to each request task type, the task model mapping table is generated.

[0041] In the embodiment, the all to-be-called models corresponding to each request task type are determined, and the model library is generated according to the frequently-called models of the request task. If the number of models in the model library is N model , the model library is represented as:

[0042]

[0043] wherein, M i for representing the i-th model.

[0044] Statistical analysis is performed on each request task type. If there are N task types, the task type library is represented as follows:

[0045]

[0046] wherein, TaskType i represents the i-th task type.

[0047] For example, the financial anti-fraud detection task type corresponds to a face detection model, an age recognition model, a gender recognition model, an expression recognition model, and a scene recognition model.

[0048] The number of women or men in a picture is identified, and the task type corresponds to a DNN model and a gender recognition model.

[0049] Each historical request task and its corresponding calling model are analyzed to obtain the mapping relationship between the request task and the corresponding calling model. Each request task needs to access one or more models, and the request task type and the corresponding calling model are in a one-to-one or one-to-many mapping relationship.

[0050] Suppose the i-th task TaskType i needs to use the sub-model library SubModelSet i , and the entire relationship library is represented as:

[0051]

[0052] wherein, RelationMap is a key-value table, SubModelSet i is a subset of ModelSet, i.e.

[0053] Step S20, based on the entire calling model corresponding to each request task, a first priority task is determined in each request task, wherein the entire calling model corresponding to the first priority task is stored in an online model library, and the online model library is stored in a memory;

[0054] In this embodiment, due to the limited memory control of the edge device, all models cannot be loaded, therefore the model library is divided into an online model library OnMemSet loaded in the memory and an offline model library OffMemSet existing in the disk. Based on the relationship mapping table (task model mapping table) of the request task type and the model, the location of the model involved in the request task is determined, and when all the to-be-called models corresponding to the request task are located in the online database, the request task is marked as a first priority task, so as to process the first priority task preferentially, thereby minimizing the loading and unloading times of the model.

[0055] In step S30, all to-be-called models corresponding to the first priority task in the online model library are called to process the first priority task, and after the first priority task is processed, other tasks except the first priority task are processed, wherein at least one to-be-called model corresponding to the other tasks is stored in the offline model library, and the offline model library is stored in the disk.

[0056] In this embodiment, first, all to-be-called models corresponding to the first priority task in the online model library are called to process the first priority task. After the first priority task is processed, it is determined that the other tasks are located in the to-be-called models in the online model library and the to-be-called models in the offline model library, the models other than the to-be-called models corresponding to the other tasks in the online model library are unloaded, the to-be-called models corresponding to the other tasks in the offline model library are loaded into the online model library, and then based on the updated online model library, all to-be-called models corresponding to the other tasks are processed.

[0057] Further, after the step S30, it further includes:

[0058] According to a preset period, the access frequency of each model in the online model library is counted, and each model identifier and the corresponding access frequency are associated and stored in an access frequency record table.

[0059] Based on the access frequency record table, the models in the online model library with an access frequency lower than a preset frequency threshold are unloaded.

[0060] In this embodiment, since the memory space of the edge device is limited, in order to minimize the loading and unloading times of the model, the online model with a relatively low access frequency in the online model library can be preferentially unloaded. Specifically, an access frequency record table is set, and the access frequencies of the models in the online model library are counted according to a preset period, such as five days or one week. The access frequency is the number of accesses of each model in the preset period. Then the identifier of each model and the corresponding access frequency are associated and stored in the access frequency record table. Then the access frequencies of all models in the online model library are counted according to the preset period. If the number of models in the online model library is N model , the access frequency record table is represented as:

[0061]

[0062] wherein Count i represents the cumulative access number of the i-th model.

[0063] Based on the access frequency record table, the models with an access frequency lower than a preset frequency threshold in the online model library are periodically unloaded, so that other models with a high subsequent loading frequency are loaded.

[0064] Further, before the step S20, the method further comprises:

[0065] Based on the calling frequency of each model, the models are sequentially filtered in a descending order of calling frequency, and a preset number of models are selected as high-frequency models.

[0066] The high-frequency models are stored in the memory to generate the online model library, and the other models except the high-frequency models are stored in the disk to generate the offline model library.

[0067] In this embodiment, after the model library is established, the calling frequency of each model in the model library is further counted. The high-frequency models are selected in a descending order of calling frequency. Specifically, the number of models that can be stored in the online model library can be determined according to the space size of the online model library. Then, based on the number of models that can be stored, a preset number of models are sequentially filtered in a descending order of calling frequency as high-frequency models.

[0068] The high-frequency models are stored in the online model library, and the other models except the high-frequency models are stored in the offline model library.

[0069] The embodiment provides a task scheduling method for multi-model inference. The method obtains each request task in a task request queue, and determines all to-be-called models corresponding to each request task based on a task model mapping table. The method determines a first priority task in each request task based on all to-be-called models corresponding to each request task, wherein all to-be-called models corresponding to the first priority task are stored in an online model library, and the online model library is stored in a memory. The method calls all to-be-called models corresponding to the first priority task in the online model library, processes the first priority task, and processes other tasks except the first priority task after the first priority task is processed, wherein at least one to-be-called model corresponding to the other tasks is stored in an offline model library, and the offline model library is stored in a disk. In the foregoing manner, the model library is divided into the online model library stored in the memory and the offline model library stored in the disk, and all to-be-called models corresponding to each request task in the task request queue are determined in advance based on the task model mapping table recording the mapping relationship between tasks and models. Then, the priority of each request task is determined based on the location of all to-be-called models corresponding to the request task, and the request task with all to-be-called models stored in the online model library is processed preferentially, so that the number of model loading and unloading is minimized, thereby greatly reducing the consumption of resources, further improving the response speed of the overall model, and improving the model processing performance.

[0070] Referring to Figure 2 , Figure 2 FIG. 2 is a flowchart of a second embodiment of the task scheduling method for multi-model inference.

[0071] Based on the foregoing Figure 1 embodiment, before the step S30, the method further includes:

[0072] In step S01, the waiting time of each request task in the task request queue is obtained, and it is determined whether a second priority task with a waiting time exceeding a preset time threshold exists in the task request queue.

[0073] In step S02, if the second priority task exists in the task request queue, a to-be-loaded model stored in the offline model library is determined in all to-be-called models corresponding to the second priority task.

[0074] In step S03, the to-be-loaded model is loaded into the online model library, and the second priority task is processed through all to-be-called models corresponding to the second priority task.

[0075] In this embodiment, in order to ensure the quality of the requested service and prevent a certain request task from waiting for too long, the request task with a long response time is processed preferentially.

[0076] Specifically, the waiting time of each request task in the task request queue is obtained, and it is determined whether there is a second priority task with a waiting time exceeding a preset time threshold in the task request queue; if there is no second priority task in the task request queue, step S30 is performed, that is, the first priority task is processed preferentially.

[0077] If there is a second priority task in the task request queue, the second priority task is processed preferentially. In all the to-be-called models corresponding to the second priority task, the to-be-loaded models stored in the offline model library are determined, and before the second priority task is processed, the to-be-loaded models corresponding to the second priority task and stored in the offline model library are loaded into the online model library.

[0078] Wherein, the priority of the request task R i is set as follows:

[0079]

[0080] 1. If the waiting time WaitSet[R i .ReqIndex] of the request task R i exceeds the preset time threshold Threshod, the request task is marked as a second priority task, and the priority corresponding to the second priority task is set to 3.

[0081] 2. If all the models of the sub-model library RelationSet[R i .TaskType] required by the task type R i .TaskType of the request task R i belong to the online model library, the request task is marked as a first priority task, and the priority corresponding to the first priority task is set to 2.

[0082] 3. The priority of other tasks in the task request queue except the first priority task and the second priority task is set to 1.

[0083] Wherein, the priority 3>2>1. That is, the second priority task is executed preferentially, then the first priority task is executed, and finally the other tasks are executed.

[0084] In specific embodiments, if there are multiple second priority tasks, the second priority task with a long waiting time or all the to-be-called models of which are located in the online model library is processed preferentially.

[0085] Further, the step S04 specifically comprises:

[0086] obtaining the number of unloadable models in the online model library, wherein the unloadable model is a model in the online model library other than all the to-be-called models corresponding to the first priority task;

[0087] when the number of to-be-loaded models of the to-be-loaded models does not exceed the number of unloadable models, unloading the unloadable models in the online model library based on the number of to-be-loaded models, and loading the to-be-loaded models into the online model library;

[0088] calling all the to-be-called models corresponding to the second priority task in the online model library to process the second priority task.

[0089] In the embodiment, before unloading the models in the online model library, the unloadable models other than all the to-be-called models corresponding to the first priority task in the online model library are obtained first, and the number of unloadable models is determined.

[0090] when the number of to-be-loaded models of the to-be-loaded models does not exceed the number of unloadable models, unloading the unloadable models in the online model library based on the number of to-be-loaded models, and loading the to-be-loaded models into the online model library, and calling all the to-be-called models corresponding to the second priority task in the online model library to process the second priority task. Thus, the second priority task is processed preferentially without affecting the processing of the first priority task, and the number of unloading and loading of the models is further reduced.

[0091] when the number of to-be-loaded models of the to-be-loaded models exceeds the number of unloadable models, the model with low access frequency is unloaded preferentially from all the to-be-called models corresponding to the first priority task.

[0092] It can be understood that the model corresponding to the first priority task is unloaded, and the first priority task needs to be updated. That is, the first priority task is reselected according to the updated online model library.

[0093] In the above manner, the embodiment preferentially minimizes the number of unloading and loading of the models in the online database based on the initial online database, and preferentially processes the first priority task and the second priority task in parallel based on the existing models in the online model library.

[0094] Referring to Figure 3 , Figure 3 is a flowchart of a third embodiment of the task scheduling method for multi-model inference.

[0095] Based on the aboveFigure 1 In the embodiment shown, the embodiment further comprises, before the step S30:

[0096] In the step S40, it is judged whether there is a second priority task in the task request queue based on the waiting time of each request task in the task request queue, wherein the waiting time of the second priority task exceeds a preset time threshold.

[0097] In the step S50, if the second priority task exists in the task request queue, the online model library is updated based on all the to-be-called models corresponding to the second priority task, so as to load all the to-be-called models corresponding to the second priority task into the online model library.

[0098] In the step S60, all the to-be-called models corresponding to the second priority task in the online model library are called to process the second priority task.

[0099] In the embodiment, in order to further improve the task scheduling efficiency, it is determined whether there is a second priority task in the task request queue before the first priority task is screened. If the second priority task exists in the task request queue, the online model library is updated based on all the to-be-called models corresponding to the second priority task, so as to load all the to-be-called models corresponding to the second priority task into the online model library; then all the to-be-called models corresponding to the second priority task in the online model library are called to process the second priority task.

[0100] In the above manner, in order to prevent the processing of the second priority task from causing the first priority task to change, the second priority task is processed first before the first priority task is screened, and the first priority task is screened according to the updated online model library.

[0101] Please refer to Figure 4 , Figure 4 is a functional module schematic diagram of a task scheduling device for multi-model inference provided by an embodiment of the present application.

[0102] As Figure 4 shown, the task scheduling device for multi-model inference 400 comprises:

[0103] The calling model determination module 10 is configured to acquire each request task in the task request queue and determine all the to-be-called models corresponding to each request task based on a task model mapping table.

[0104] The priority task determination module 20 is configured to determine a first priority task from the request tasks based on all the to-be-called models corresponding to the request tasks, wherein all the to-be-called models corresponding to the first priority task are stored in an online model library, and the online model library is stored in a memory.

[0105] The request queue processing module 30 is configured to call all the to-be-called models corresponding to the first priority task in the online model library, process the first priority task, and process other tasks except the first priority task after the first priority task is processed, wherein at least one to-be-called model corresponding to the other tasks is stored in an offline model library, and the offline model library is stored in a disk.

[0106] Further, the task scheduling apparatus 400 for multi-model inference further includes:

[0107] The second task determination module is configured to obtain the waiting time of each request task in the task request queue, and determine whether a second priority task with a waiting time exceeding a preset time threshold exists in the task request queue.

[0108] The loaded model determination module is configured to determine a to-be-loaded model stored in the offline model library from all the to-be-called models corresponding to the second priority task if the second priority task exists in the task request queue.

[0109] The second task processing module is configured to load the to-be-loaded model into the online model library, and process the second priority task through all the to-be-called models corresponding to the second priority task.

[0110] Further, the second task processing module specifically includes:

[0111] The unloaded model determination unit is configured to obtain the number of unloadable models in the online model library, wherein the unloadable models are models in the online model library except all the to-be-called models corresponding to the first priority task.

[0112] The loaded model loading unit is configured to, when the number of to-be-loaded models of the to-be-loaded model does not exceed the number of unloadable models, unload the unloadable models in the online model library based on the number of to-be-loaded models, and load the to-be-loaded model into the online model library.

[0113] The second task processing unit is configured to call all the to-be-called models corresponding to the second priority task in the online model library, and process the second priority task.

[0114] Further, the task scheduling apparatus 400 for multi-model inference further comprises:

[0115] a second task judging module configured to judge whether there is a second priority task in the task request queue based on the waiting time of each request task in the task request queue, wherein the waiting time of the second priority task exceeds a preset time threshold;

[0116] an online model updating module configured to, if there is the second priority task in the task request queue, update the online model library based on all the to-be-called models corresponding to the second priority task, so as to load all the to-be-called models corresponding to the second priority task into the online model library;

[0117] a task priority processing module configured to call all the to-be-called models corresponding to the second priority task in the online model library to process the second priority task.

[0118] Further, the task scheduling apparatus 400 for multi-model inference further comprises:

[0119] a mapping relationship analyzing module configured to count each historical request task and the calling model corresponding thereto, and determine each request task type and all the calling models corresponding thereto;

[0120] a mapping relationship generating module configured to generate the task model mapping table based on each request task type and all the calling models corresponding thereto.

[0121] Further, the task scheduling apparatus 400 for multi-model inference further comprises:

[0122] a model access statistical module configured to, according to a preset period, count the access frequency of each model in the online model library, and store each model identifier and the access frequency corresponding thereto in an access frequency record table;

[0123] an online model unloading module configured to, based on the access frequency record table, unload the model in the online model library whose access frequency is lower than a preset frequency threshold.

[0124] Further, the task scheduling apparatus 400 for multi-model inference further comprises:

[0125] a high-frequency model screening module configured to, based on the calling frequency of each model, screen a preset number of models from all the models in the order of calling frequency from high to low as high-frequency models;

[0126] The model library classification module is configured to store the high-frequency model in the memory, generate the online model library, and store other models except the high-frequency model in the disk, and generate the offline model library.

[0127] It should be noted that, for the convenience and brevity of description, the specific working processes of the above-described apparatus and modules can be clearly understood by those skilled in the art, and the corresponding processes in the foregoing method embodiments can be referred to, which will not be described herein.

[0128] The apparatus described above can be implemented in the form of a computer program, which can run on a computer device as shown in Figure 5 .

[0129] Please refer to Figure 5 , Figure 5 is a structural schematic block diagram of a computer device provided by an embodiment of the present application. The computer device can be a server.

[0130] Please refer to Figure 5 , the computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory can include a non-volatile storage medium and an internal memory.

[0131] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions which, when executed, can cause the processor to perform any one of the task scheduling methods for multi-model inference.

[0132] The processor is configured to provide computing and control capabilities to support the operation of the entire computer device.

[0133] The internal memory provides an environment for the execution of the computer program in the non-volatile storage medium, and the computer program, when executed by the processor, can cause the processor to perform any one of the task scheduling methods for multi-model inference.

[0134] The network interface is configured to perform network communication, such as sending assigned tasks. Those skilled in the art can understand that Figure 5 the structure shown in the figure, only the block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or less components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0135] It should be appreciated that the processor can be a central processing unit (CPU), the processor can also be other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc. Among them, the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0136] In one embodiment, the processor is configured to execute computer programs stored in the memory to implement the following steps:

[0137] Obtaining each request task in the task request queue, and determining all to-be-called models corresponding to each request task based on a task model mapping table;

[0138] Determining a first priority task in each request task based on all to-be-called models corresponding to each request task, wherein all to-be-called models corresponding to the first priority task are stored in an online model library, and the online model library is stored in a memory;

[0139] Calling all to-be-called models corresponding to the first priority task in the online model library, processing the first priority task, and processing other tasks except the first priority task after the first priority task is processed, wherein at least one to-be-called model corresponding to the other tasks is stored in an offline model library, and the offline model library is stored in a disk.

[0140] In one embodiment, the processor is configured to execute computer programs stored in the memory to implement the following steps:

[0141] Obtaining the waiting time of each request task in the task request queue, and determining whether a second priority task with a waiting time exceeding a preset time threshold exists in the task request queue;

[0142] If the second priority task exists in the task request queue, determining a to-be-loaded model stored in the offline model library in all to-be-called models corresponding to the second priority task;

[0143] Loading the to-be-loaded model into the online model library, and processing the second priority task through all to-be-called models corresponding to the second priority task.

[0144] In an embodiment, the processor is configured to execute a computer program stored in the memory to implement the following steps:

[0145] acquire a number of unloadable models in the online model library, wherein the unloadable models are models in the online model library other than all the to-be-called models corresponding to the first priority task;

[0146] when the number of to-be-loaded models does not exceed the number of unloadable models, unload the unloadable models in the online model library based on the number of to-be-loaded models, and load the to-be-loaded models into the online model library;

[0147] call all the to-be-called models corresponding to the second priority task in the online model library, and process the second priority task.

[0148] In an embodiment, the processor is configured to execute a computer program stored in the memory to implement the following steps:

[0149] determine whether there is a second priority task in the task request queue based on the waiting time of each request task in the task request queue, wherein the waiting time of the second priority task exceeds a preset time threshold;

[0150] if there is the second priority task in the task request queue, update the online model library based on all the to-be-called models corresponding to the second priority task, so as to load all the to-be-called models corresponding to the second priority task into the online model library;

[0151] call all the to-be-called models corresponding to the second priority task in the online model library, and process the second priority task.

[0152] In an embodiment, the processor is configured to execute a computer program stored in the memory to implement the following steps:

[0153] count each historical request task and its corresponding called model, and determine each request task type and its corresponding all called models;

[0154] generate the task model mapping table based on each request task type and its corresponding all called models.

[0155] In an embodiment, the processor is configured to execute a computer program stored in the memory to implement the following steps:

[0156] According to a preset period, access frequencies of each model in the online model library are counted, and each model identifier and the corresponding access frequency are stored in an access frequency record table;

[0157] Based on the access frequency record table, models in the online model library with an access frequency lower than a preset frequency threshold are unloaded.

[0158] In an embodiment, the processor is configured to run a computer program stored in the memory to implement the following steps:

[0159] Based on the calling frequencies of each model, a preset number of models are sequentially filtered from the models in descending order of calling frequency, as high-frequency models.

[0160] The high-frequency models are stored in the memory to generate the online model library, and other models except the high-frequency models are stored in the disk to generate the offline model library.

[0161] In an embodiment of the present application, a computer readable storage medium is also provided, which stores a computer program including program instructions. The processor executes the program instructions to implement any one of the task scheduling methods for multi-model inference provided in the embodiments of the present application.

[0162] The computer readable storage medium can be an internal storage unit of the computer device, such as a hard disk or a memory of the computer device. The computer readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc.

[0163] The above is merely specific embodiments of the present application, and the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements shall be encompassed in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A task scheduling method for multi-model inference, characterized in that, The task scheduling method comprises the following steps: acquiring each request task in a task request queue, and determining all to-be-called models corresponding to each request task based on a task model mapping table; determining a first priority task in each request task based on all to-be-called models corresponding to each request task, wherein all to-be-called models corresponding to the first priority task are stored in an online model library, the online model library is stored in a memory, the online model library is generated based on high-frequency models, and the high-frequency models are determined based on the calling frequency of each model; calling all to-be-called models corresponding to the first priority task in the online model library, processing the first priority task, and processing other tasks except the first priority task after the first priority task is processed, wherein at least one to-be-called model corresponding to the other tasks is stored in an offline model library, the offline model library is stored in a disk, and the offline model library is generated based on other models except the high-frequency models; wherein, before the calling all to-be-called models corresponding to the first priority task in the online model library, processing the first priority task, and processing other tasks except the first priority task after the first priority task is processed, the method further comprises: acquiring the waiting time of each request task in the task request queue, and determining whether a second priority task with a waiting time exceeding a preset time threshold exists in the task request queue; if the second priority task exists in the task request queue, determining a to-be-loaded model stored in the offline model library in all to-be-called models corresponding to the second priority task; acquiring the number of unloadable models in the online model library, wherein the unloadable model is a model in the online model library except all to-be-called models corresponding to the first priority task; when the number of to-be-loaded models of the to-be-loaded model does not exceed the number of unloadable models, unloading the unloadable model in the online model library based on the number of to-be-loaded models, and loading the to-be-loaded model into the online model library; calling all to-be-called models corresponding to the second priority task in the online model library, and processing the second priority task.

2. The task scheduling method for multi-model inference of claim 1, wherein, Before the acquiring each request task in a task request queue, and determining all to-be-called models corresponding to each request task based on a task model mapping table, the method further comprises: counting each historical request task and its corresponding calling model, and determining each request task type and its corresponding all calling models; generating the task model mapping table based on each request task type and its corresponding all calling models.

3. The task scheduling method for multi-model inference of claim 1, wherein, After the calling all to-be-called models corresponding to the first priority task in the online model library, processing the first priority task, and processing other tasks except the first priority task after the first priority task is processed, the method further comprises: According to a preset period, access frequencies of each model in the online model library are counted, and each model identifier and the corresponding access frequency are stored in an access frequency record table; Based on the access frequency record table, models in the online model library with an access frequency lower than a preset frequency threshold are unloaded.

4. The task scheduling method for multi-model inference of any one of claims 1-3, wherein, The method further includes, based on the total to-be-called models corresponding to each request task, before determining a first priority task in each request task, the method further includes: Based on the calling frequencies of each model, a preset number of models are sequentially filtered from each model in descending order of calling frequency as high-frequency models.

5. A task scheduling device for multi-model inference, characterized in that, The task scheduling device for multi-model inference includes: A calling model determination module is configured to obtain each request task in a task request queue and determine total to-be-called models corresponding to each request task based on a task model mapping table; A priority task determination module is configured to determine a first priority task in each request task based on the total to-be-called models corresponding to each request task, wherein the total to-be-called models corresponding to the first priority task are stored in an online model library, the online model library is stored in a memory, the online model library is generated based on high-frequency models, and the high-frequency models are determined based on the calling frequencies of each model; A request queue processing module is configured to call the total to-be-called models corresponding to the first priority task in the online model library, process the first priority task, and process other tasks except the first priority task after the first priority task is processed, wherein at least one to-be-called model corresponding to the other tasks is stored in an offline model library, the offline model library is stored in a disk, and the offline model library is generated based on other models except the high-frequency models; The task scheduling device for multi-model inference further includes: A second task determination module is configured to obtain waiting times of each request task in the task request queue and determine whether a second priority task with a waiting time exceeding a preset time threshold exists in the task request queue; A loaded model determination module is configured to determine a to-be-loaded model stored in the offline model library in the total to-be-called models corresponding to the second priority task if the second priority task exists in the task request queue; A second task processing module is configured to load the to-be-loaded model into the online model library and process the second priority task through the total to-be-called models corresponding to the second priority task. The second task processing module includes: An unloaded model determination unit is configured to obtain a number of unloadable models in the online model library, wherein the unloadable models are models in the online model library except the total to-be-called models corresponding to the first priority task; A loaded model loading unit is configured to, when the number of to-be-loaded models of the to-be-loaded model does not exceed the number of unloadable models, unload the unloadable models in the online model library based on the number of to-be-loaded models and load the to-be-loaded model into the online model library. A second task processing unit is configured to invoke all to-be-invoked models corresponding to the second priority task in the online model library, and process the second priority task.

6. A computer device, comprising: The computer device comprises a processor, a memory, and a task scheduler for multi-model inference stored on the memory and executable by the processor, wherein the task scheduler for multi-model inference, when executed by the processor, implements the steps of the task scheduling method for multi-model inference according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a task scheduler for multi-model inference, wherein the task scheduler for multi-model inference, when executed by the processor, implements the steps of the task scheduling method for multi-model inference according to any one of claims 1 to 4.

Images