Task scheduling method and device for multi-model inference, equipment and storage medium

Abstract

The invention relates to the field of data processing, and provides a task scheduling method and device for multi-model inference, computer equipment and a computer readable storage medium. According to the method, a model library is divided into an online model library stored in a memory and an offline model library stored in a disk; all to-be-called models corresponding to the request tasks in the task request queue are determined in advance based on a task model mapping table for recording the mapping relation between the tasks and the models; then, the priority of each request task is determined based on the positions of all the to-be-called models corresponding to the request tasks, and the request tasks, stored in the online model library, of all the to-be-called models are subjected to priority processing to minimize the number of times of model loading and unloading, so that the resource consumption is greatly reduced, and the efficiency is improved. The response speed of the whole model is further improved; and the model processing performance is improved.

Description

Technical Field

[0001] The present invention relates to the fields of data processing and financial technology, and in particular to a task scheduling method, device, computer equipment and computer-readable storage medium for multi-model inference. Background Art

[0002] With the development of science and technology, there are more and more scenarios for multi-model inference applications. Multi-model inference means that multiple models are needed to perform inference to perform a task. In order to extract a large amount of information from a picture, inference is required during the extraction process, and the inference process requires the use of multiple models at the same time. For example, to identify how many women are in a picture, you first need to use the DNN model to identify the location and number of faces, and then use the gender recognition network to identify the gender of each face. However, the resources of the edge server are very limited, the memory space is small, and a large number of models cannot be stored. Only some models can be loaded. When other models are needed, the model can only be unloaded and reloaded. Frequent loading and unloading of models not only affects the processing performance of the model, but also causes a waste of resources.

[0003] Therefore, how to improve model processing performance while reducing resource consumption has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The main purpose of the present invention is to provide a task scheduling method, device, equipment and computer-readable storage medium for multi-model inference, aiming to reduce resource consumption while improving model processing performance.

[0005] To achieve the above object, the present invention provides a task scheduling method for multi-model inference, the task scheduling method comprising the following steps:

[0006] Obtain each request task in the task request queue, and determine all to-be-called models corresponding to each request task based on the task model mapping table;

[0007] Based on all the to-be-called models corresponding to the respective request tasks, determining a first priority task in the respective request tasks, wherein all the to-be-called models corresponding to the first priority task are stored in an online model library, and the online model library is stored in a memory;

[0008] Call all the models to be called corresponding to the first priority task in the online model library, process the first priority task, and after processing the first priority task, process other tasks except the first priority task, wherein at least one model to be called corresponding to the other tasks is stored in the offline model library, and the offline model library is stored in the disk.

[0009] In addition, to achieve the above-mentioned purpose, the present invention also provides a task scheduling device for multi-model inference, and the task scheduling device for multi-model inference includes:

[0010] The calling model determination module is used to obtain each request task in the task request queue and determine all to-be-called models corresponding to each request task based on the task model mapping table;

[0011] A priority task determination module, used for determining a first priority task among each request task based on all to-be-called models corresponding to each request task, wherein all to-be-called models corresponding to the first priority task are stored in an online model library, and the online model library is stored in a memory;

[0012] A request queue processing module is used to call all the models to be called corresponding to the first priority task in the online model library, process the first priority task, and after processing the first priority task, process other tasks except the first priority task, wherein at least one model to be called corresponding to the other tasks is stored in the offline model library, and the offline model library is stored in the disk.

[0013] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a processor, a memory, and a task scheduling program for multi-model inference stored in the memory and executable by the processor, wherein when the task scheduling program for multi-model inference is executed by the processor, the steps of the task scheduling method for multi-model inference as described above are implemented.

[0014] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a task scheduling program for multi-model inference is stored, wherein when the task scheduling program for multi-model inference is executed by a processor, the steps of the task scheduling method for multi-model inference as described above are implemented.

[0015] The present application provides a task scheduling method for multi-model inference, the method obtaining each request task in a task request queue, and determining all models to be called corresponding to each request task based on a task model mapping table; determining a first priority task in each request task based on all models to be called corresponding to each request task, wherein all models to be called corresponding to the first priority task are stored in an online model library, and the online model library is stored in memory; calling all models to be called corresponding to the first priority task in the online model library, processing the first priority task, and after processing the first priority task, processing other tasks except the first priority task, wherein at least one model to be called corresponding to the other tasks is stored in an offline model library, and the offline model library is stored in a disk. In the above manner, the present application divides the model library into an online model library stored in memory and an offline model library stored in a disk, and predetermines all models to be called corresponding to each request task in the task request queue based on a task model mapping table that records the mapping relationship between tasks and models. Then, based on the location of all the models to be called corresponding to the request task, the priority of each request task is determined, and the request task in which all the models to be called are stored in the online model library is given priority to minimize the number of model loading and unloading times, thereby greatly reducing resource consumption while further improving the response speed of the overall model and improving model processing performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flowchart of a first embodiment of a task scheduling method for multi-model inference according to the present invention;

[0017] Figure 2 It is a flowchart of a second embodiment of a task scheduling method for multi-model inference according to the present invention;

[0018] Figure 3 It is a flowchart of a third embodiment of a task scheduling method for multi-model inference according to the present invention;

[0019] Figure 4 A schematic diagram of a task scheduling device for multi-model inference according to the present invention;

[0020] Figure 5 The figure is a schematic diagram of the hardware structure of a computer device involved in an embodiment of the present invention.

[0021] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0023] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0024] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0025] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0026] The task scheduling method for multi-model inference involved in the embodiment of the present invention is mainly applied to computer equipment. The task scheduling generation device for multi-model inference can be a PC, a portable computer, a mobile terminal, or other device with display and processing functions.

[0027] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0028] Reference Figure 1 , Figure 1 Schematic diagram of the flow of the first embodiment of the task scheduling method for multi-model inference of the present invention.

[0029] like Figure 1 As shown, an embodiment of the present invention provides a task scheduling method for multi-model inference, and the task scheduling method for multi-model inference includes steps S10 to S30.

[0030] In this embodiment, the task scheduling method includes the following steps:

[0031] Step S10, obtaining each request task in the task request queue, and determining all to-be-called models corresponding to each request task based on the task model mapping table;

[0032] At present, the resources of edge servers are very limited, and the memory space is small. It is impossible to store a large number of models. Only some models can be loaded. When other models are needed, the models can only be unloaded and reloaded. Frequent loading and unloading of models not only affects the processing performance of the model, but also causes a waste of resources.

[0033] In order to solve the above problems, this embodiment divides the model library into an online model library stored in the memory and an offline model library stored in the disk, and predetermines all the models to be called corresponding to each request task in the task request queue based on the task model mapping table that records the mapping relationship between tasks and models. Then, based on the location of all the models to be called corresponding to the request task, the priority of each request task is determined, and the request task whose all the models to be called are stored in the online model library is given priority to minimize the number of model loading and unloading, thereby greatly reducing resource consumption while further improving the response speed of the overall model and improving the model processing performance.

[0034] Specifically, when a task request is received, a task request queue is generated based on each requested task, and the waiting time of each requested task is recorded. If the number of requested tasks is N request , then the waiting time for the first task is recorded as:

[0035]

[0036] Among them, WaitMap is a key-value table, ReqIndex i is the index value of the i-th request in the queue, Waittime i is the waiting time of the i-th request task in the task request queue.

[0037] Based on the task model mapping table, all to-be-called models corresponding to each request task in the task request queue are determined.

[0038] Furthermore, before step S10, the method further includes:

[0039] Count each historical request task and its corresponding call model, and determine each request task type and all corresponding call models;

[0040] The task model mapping table is generated based on each requested task type and all corresponding call models.

[0041] In this embodiment, statistics are collected on each historical request task and its corresponding call model, and a model library is generated based on the models frequently called by the request tasks. model , then the model library is expressed as:

[0042]

[0043] Among them, M i Used to represent the i-th model.

[0044] Statistics are collected for each request task type. If there are N task types, the task type library is represented as follows:

[0045]

[0046] Among them, TaskType i Represents the i-th task type.

[0047] For example, the financial anti-fraud detection task type corresponds to the face detection model, age recognition model, gender recognition model, expression recognition model, and scene recognition model;

[0048] The task type is to identify the number of females or males in the image, corresponding to the DNN model and gender recognition model, etc.

[0049] Each historical request task and its corresponding call model are analyzed to obtain the mapping relationship between the request task and its corresponding model to be called. Each request task needs to access one or more models, and the request task type and the corresponding all call models are in a one-to-one or one-to-many mapping relationship.

[0050] Assume that the i-th task TaskType i The sub-model library needed is SubModelSet i , then the entire relational database is expressed as:

[0051]

[0052] Among them, RelationMap is a key-value table, SubModelSet i is a subset of ModelSet, that is

[0053] Step S20, based on all the to-be-called models corresponding to the respective request tasks, determining a first priority task in the respective request tasks, wherein all the to-be-called models corresponding to the first priority tasks are stored in an online model library, and the online model library is stored in a memory;

[0054] In this embodiment, since the memory control of the edge device is limited and all models cannot be loaded, the model library is divided into an online model library OnMemSet loaded in the memory and an offline model library OffMemSet stored in the disk. Based on the relationship mapping table between the requested task type and the model (task model mapping table), the location of the model involved in the requested task is determined, and when all the corresponding models to be called are located in the online database, the requested task is marked as a first-priority task to give priority to processing the first-priority task, thereby minimizing the number of times the model is loaded and unloaded.

[0055] Step S30, calling all the models to be called corresponding to the first priority task in the online model library, processing the first priority task, and after processing the first priority task, processing other tasks except the first priority task, wherein at least one model to be called corresponding to the other tasks is stored in the offline model library, and the offline model library is stored in the disk.

[0056] In this embodiment, all the to-be-called models corresponding to the first priority task in the online model library are first called to process the first priority task. After processing the first priority task, the to-be-called models of other tasks in the online model and the to-be-called models in the offline model library are determined, and the models other than the to-be-called models corresponding to the other tasks in the online model library are unloaded, and the to-be-called models corresponding to the other tasks in the offline model library are loaded into the online model library, and then the other tasks are processed based on all the to-be-called models corresponding to the other tasks in the updated online model library.

[0057] Furthermore, after step S30, the method further includes:

[0058] According to a preset period, the access frequency of each model in the online model library is counted, and each model identifier and its corresponding access frequency are associated and stored in an access frequency record table;

[0059] Based on the access frequency record table, models in the online model library whose access frequency is lower than a preset frequency threshold are uninstalled.

[0060] In this embodiment, since the memory space of the edge device is limited, in order to minimize the number of times the model is loaded and unloaded, the online models with relatively low access frequencies in the online model library can be unloaded first. Specifically, an access frequency record table is set, and the access frequency of each model in the online model library is counted according to a preset period, such as five days or a week. Among them, the access frequency is the number of times each model is accessed within a preset period. Then, each model identifier and its corresponding access frequency are associated and stored in the access frequency record table. Then, according to the preset period, the access frequency of all models in the online model is counted. If the number of models in the online model library is N model , then the access frequency record table is expressed as:

[0061]

[0062] Among them, Count i Represents the cumulative number of visits to the i-th model.

[0063] Based on the access frequency record table, models in the online model library whose access frequency is lower than a preset frequency threshold are regularly unloaded so as to subsequently load other models with high usage frequency.

[0064] Furthermore, before step S20, the method further includes:

[0065] Based on the calling frequency of each model, a preset number of models are selected in descending order of calling frequency as high-frequency models;

[0066] The high frequency model is stored in the memory to generate the online model library, and other models except the high frequency are stored in the disk to generate the offline model library.

[0067] In this embodiment, after the model library is established, the calling frequency of each model in the model library is further counted. And the high-frequency models are screened out in the order of calling frequency from high to low in each model. Specifically, the number of models that can be stored in the online model library can be determined according to the space size of the online model library. Then, based on the number of storable models, a preset number of models are sequentially screened out in the order of calling frequency from high to low in each model as high-frequency models.

[0068] The high frequency model is stored in an online model library, and other models except the high frequency are stored in an offline model library.

[0069] The present embodiment provides a task scheduling method for multi-model inference, the method obtains each request task in the task request queue, and determines all the models to be called corresponding to each request task based on the task model mapping table; based on all the models to be called corresponding to each request task, a first priority task is determined in each request task, wherein all the models to be called corresponding to the first priority task are stored in the online model library, and the online model library is stored in the memory; all the models to be called corresponding to the first priority task in the online model library are called, the first priority task is processed, and after processing the first priority task, other tasks except the first priority task are processed, wherein at least one model to be called corresponding to the other tasks is stored in the offline model library, and the offline model library is stored in the disk. Through the above method, the present application divides the model library into an online model library stored in the memory and an offline model library stored in the disk, and predetermines all the models to be called corresponding to each request task in the task request queue based on the task model mapping table that records the mapping relationship between tasks and models. Then, based on the location of all the models to be called corresponding to the request task, the priority of each request task is determined, and the request task in which all the models to be called are stored in the online model library is given priority to minimize the number of model loading and unloading times, thereby greatly reducing resource consumption while further improving the response speed of the overall model and improving model processing performance.

[0070] Reference Figure 2 , Figure 2 Schematic diagram of the flow chart of the second embodiment of the task scheduling method for multi-model inference of the present invention.

[0071] Based on the above Figure 1 In the embodiment shown, in this embodiment, before step S30, the following steps are further included:

[0072] Step S01, obtaining the waiting time of each request task in the task request queue, and determining whether there is a second priority task in the task request queue whose waiting time exceeds a preset time threshold;

[0073] Step S02, if there is a second priority task in the task request queue, determining a model to be loaded stored in the offline model library from all models to be called corresponding to the second priority task;

[0074] Step S03: load the model to be loaded into the online model library, and process the second priority task through all the models to be called corresponding to the second priority task.

[0075] In this embodiment, in order to ensure the service quality of the request and prevent a request task from waiting too long, request tasks with long response times are processed first.

[0076] Specifically, the waiting time of each request task in the task request queue is obtained, and it is determined whether there is a second priority task in the task request queue whose waiting time exceeds a preset time threshold; if there is no second priority task in the task request queue, step S30 is executed, that is, the first priority task is processed preferentially.

[0077] If there is a second priority task in the task request queue, the second priority task is processed first. The model to be loaded stored in the offline model library is determined among all the models to be called corresponding to the second priority task, and before processing the second priority task, the model to be loaded stored in the offline model library corresponding to the second priority task is first loaded into the online model library.

[0078] Among them, the request task R i The priority setting method is as follows:

[0079]

[0080] 1. If the requested task R i Waiting time WaitSet[R i .ReqIndex] exceeds the preset time threshold Threshod, then the request task is marked as a second priority task, and the priority corresponding to the second priority task is set to 3.

[0081] 2. If the requested task R i Task type R i .TaskType required sub-model library RelationSet[R i .TaskType] all models belong to the online model library, the requested task is marked as a first-priority task, and the priority corresponding to the first-priority task is set to 2.

[0082] 3. Set the priority of other tasks in the task request queue except the first priority task and the second priority task to 1.

[0083] Among them, priority 3≥2≥1. That is, the second priority task is executed first, then the first priority character, and finally other tasks.

[0084] In a specific embodiment, if there are multiple second priority tasks, priority is given to second priority tasks with long waiting time or whose models to be called are all located in the online model library.

[0085] Furthermore, the step S04 specifically includes:

[0086] Acquire the number of uninstallable models in the online model library, wherein the uninstallable models are models in the online model library except all to-be-called models corresponding to the first priority task;

[0087] When the number of models to be loaded of the models to be loaded does not exceed the number of unloadable models, in the online model library, the unloadable models are unloaded based on the number of models to be loaded, and the models to be loaded are loaded into the online model library;

[0088] All to-be-called models corresponding to the second priority task in the online model library are called to process the second priority task.

[0089] In this embodiment, before uninstalling the models in the online model library, firstly, the uninstallable models in the online model library except all the to-be-called models corresponding to the first priority task are obtained, and the number of uninstallable models is determined.

[0090] When the number of models to be loaded does not exceed the number of models that can be unloaded, in the online model library, the unloadable models are unloaded based on the number of models to be loaded, and the models to be loaded are loaded into the online model library, and all the models to be called corresponding to the second priority task in the online model library are called to process the second priority task. Thus, the second priority task is processed first without affecting the task processing of the first priority task, further reducing the number of times the model is unloaded and loaded.

[0091] When the number of the models to be loaded exceeds the number of the unloadable models, among all the models to be called corresponding to the first priority task, models with low access frequency are preferentially unloaded.

[0092] It is understandable that the model corresponding to the first priority task is uninstalled, and the first priority task needs to be updated, that is, the first priority task is re-screened according to the updated online model library.

[0093] Through the above-mentioned method, this embodiment gives priority to the initial online database, minimizes the number of times of model unloading and loading in the online database, and gives priority to parallel processing of the first priority tasks and the second priority tasks through the existing models in the online model library.

[0094] Reference Figure 3 , Figure 3 4 is a flow chart of a third embodiment of a task scheduling method for multi-model inference according to the present invention.

[0095] Based on the above Figure 1 In the embodiment shown, in this embodiment, before step S30, the following steps are further included:

[0096] Step S40, based on the waiting time of each request task in the task request queue, determining whether there is a second priority task in the task request queue, wherein the waiting time of the second priority task exceeds a preset time threshold;

[0097] Step S50, if the second priority task exists in the task request queue, then based on all the to-be-called models corresponding to the second priority task, the online model library is updated to load all the to-be-called models corresponding to the second priority task into the online model library;

[0098] Step S60: Call all the to-be-called models corresponding to the second priority task in the online model library to process the second priority task.

[0099] In this embodiment, in order to further improve the efficiency of task scheduling, before screening the first priority task, it is first determined whether there is a second priority task in the task request queue. If there is a second priority task in the task request queue, the online model library is updated based on all the models to be called corresponding to the second priority task, so as to load all the models to be called corresponding to the second priority task into the online model library; then all the models to be called corresponding to the second priority task in the online model library are called to process the second priority task.

[0100] In the above manner, in order to prevent the processing of the second priority task from causing changes in the first priority task, this embodiment preferentially processes the second priority task before filtering the first priority task, and filters out the first priority task based on the updated online model library.

[0101] See also Figure 4 , Figure 4 The present invention provides a functional module diagram of a task scheduling device for multi-model inference according to an embodiment of the present application.

[0102] like Figure 4 As shown, the task scheduling device 400 for multi-model inference includes:

[0103] The calling model determining module 10 is used to obtain each request task in the task request queue, and determine all to-be-called models corresponding to each request task based on the task model mapping table;

[0104] A priority task determination module 20, configured to determine a first priority task among the request tasks based on all to-be-called models corresponding to the request tasks, wherein all to-be-called models corresponding to the first priority task are stored in an online model library, and the online model library is stored in a memory;

[0105] The request queue processing module 30 is used to call all the models to be called corresponding to the first priority task in the online model library, process the first priority task, and after processing the first priority task, process other tasks except the first priority task, wherein at least one model to be called corresponding to the other tasks is stored in the offline model library, and the offline model library is stored in the disk.

[0106] Furthermore, the task scheduling device 400 for multi-model inference further includes:

[0107] A second task determination module is used to obtain the waiting time of each request task in the task request queue, and determine whether there is a second priority task in the task request queue whose waiting time exceeds a preset time threshold;

[0108] A loading model determination module, configured to determine a model to be loaded stored in the offline model library from among all the models to be called corresponding to the second priority task if the second priority task exists in the task request queue;

[0109] The second task processing module is used to load the model to be loaded into the online model library, and process the second priority task through all the models to be called corresponding to the second priority task.

[0110] Furthermore, the second task processing module specifically includes:

[0111] an uninstall model determination unit, configured to obtain the number of uninstallable models in the online model library, wherein the uninstallable models are models in the online model library except all to-be-called models corresponding to the first priority task;

[0112] A model loading unit, configured to, when the number of models to be loaded of the models to be loaded does not exceed the number of unloadable models, unload the unloadable models in the online model library based on the number of models to be loaded, and load the models to be loaded into the online model library;

[0113] The second task processing unit is used to call all the to-be-called models corresponding to the second priority task in the online model library, and process the second priority task.

[0114] Furthermore, the task scheduling device 400 for multi-model inference further includes:

[0115] A second task determination module, configured to determine whether there is a second priority task in the task request queue based on the waiting time of each request task in the task request queue, wherein the waiting time of the second priority task exceeds a preset time threshold;

[0116] An online model updating module, configured to update the online model library based on all to-be-called models corresponding to the second priority task if there is the second priority task in the task request queue, so as to load all to-be-called models corresponding to the second priority task into the online model library;

[0117] The task priority processing module is used to call all the to-be-called models corresponding to the second priority tasks in the online model library, and process the second priority tasks.

[0118] Furthermore, the task scheduling device 400 for multi-model inference further includes:

[0119] The mapping relationship analysis module is used to count each historical request task and its corresponding call model, and determine each request task type and all corresponding call models;

[0120] The mapping relationship generation module generates the task model mapping table based on each request task type and all corresponding call models.

[0121] Furthermore, the task scheduling device 400 for multi-model inference further includes:

[0122] A model access statistics module, used to count the access frequency of each model in the online model library according to a preset period, and associate each model identifier and its corresponding access frequency and store them in an access frequency record table;

[0123] The online model unloading module is used to unload the models in the online model library whose access frequency is lower than a preset frequency threshold based on the access frequency record table.

[0124] Furthermore, the task scheduling device 400 for multi-model inference further includes:

[0125] A high-frequency model screening module is used to screen out a preset number of models in descending order of calling frequency based on the calling frequency of each model as high-frequency models;

[0126] The model library classification module is used to store the high-frequency model in the memory to generate the online model library, and store other models except the high-frequency model in the disk to generate the offline model library.

[0127] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and each module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0128] The above-mentioned device can be implemented in the form of a computer program. Figure 5 Runs on the computer device shown.

[0129] See also Figure 5 , Figure 5 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device may be a server.

[0130] See also Figure 5 The computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0131] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any task scheduling method for multi-model inference.

[0132] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0133] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any task scheduling method for multi-model inference.

[0134] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0135] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0136] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0137] Obtain each request task in the task request queue, and determine all to-be-called models corresponding to each request task based on the task model mapping table;

[0138] Based on all the to-be-called models corresponding to the respective request tasks, determining a first priority task in the respective request tasks, wherein all the to-be-called models corresponding to the first priority task are stored in an online model library, and the online model library is stored in a memory;

[0139] Call all the models to be called corresponding to the first priority task in the online model library, process the first priority task, and after processing the first priority task, process other tasks except the first priority task, wherein at least one model to be called corresponding to the other tasks is stored in the offline model library, and the offline model library is stored in the disk.

[0140] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0141] Obtaining the waiting time of each request task in the task request queue, and determining whether there is a second priority task in the task request queue whose waiting time exceeds a preset time threshold;

[0142] If there is a task of the second priority in the task request queue, determining a model to be loaded stored in the offline model library from among all models to be called corresponding to the task of the second priority;

[0143] The model to be loaded is loaded into the online model library, and the second priority task is processed through all the models to be called corresponding to the second priority task.

[0144] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0145] Acquire the number of uninstallable models in the online model library, wherein the uninstallable models are models in the online model library except all to-be-called models corresponding to the first priority task;

[0146] When the number of models to be loaded of the models to be loaded does not exceed the number of unloadable models, in the online model library, the unloadable models are unloaded based on the number of models to be loaded, and the models to be loaded are loaded into the online model library;

[0147] All to-be-called models corresponding to the second priority task in the online model library are called to process the second priority task.

[0148] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0149] Based on the waiting time of each request task in the task request queue, determining whether there is a second priority task in the task request queue, wherein the waiting time of the second priority task exceeds a preset time threshold;

[0150] If there is a second priority task in the task request queue, then based on all to-be-called models corresponding to the second priority task, the online model library is updated to load all to-be-called models corresponding to the second priority task into the online model library;

[0151] All to-be-called models corresponding to the second priority task in the online model library are called to process the second priority task.

[0152] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0153] Count each historical request task and its corresponding call model, and determine each request task type and all corresponding call models;

[0154] The task model mapping table is generated based on each requested task type and all corresponding call models.

[0155] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0156] According to a preset period, the access frequency of each model in the online model library is counted, and each model identifier and its corresponding access frequency are associated and stored in an access frequency record table;

[0157] Based on the access frequency record table, models in the online model library whose access frequency is lower than a preset frequency threshold are uninstalled.

[0158] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0159] Based on the calling frequency of each model, a preset number of models are selected in descending order of calling frequency as high-frequency models;

[0160] The high frequency model is stored in the memory to generate the online model library, and other models except the high frequency are stored in the disk to generate the offline model library.

[0161] A computer-readable storage medium is also provided in an embodiment of the present application, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the processor executes the program instructions to implement any one of the task scheduling methods for multi-model inference provided in the embodiments of the present application.

[0162] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.

[0163] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A task scheduling method for multi-model inference, characterized in that: The task scheduling method comprises the following steps: Obtain each request task in the task request queue, and determine all to-be-called models corresponding to each request task based on the task model mapping table; Based on all the to-be-called models corresponding to the respective request tasks, determining a first priority task in the respective request tasks, wherein all the to-be-called models corresponding to the first priority task are stored in an online model library, and the online model library is stored in a memory; Call all the models to be called corresponding to the first priority task in the online model library, process the first priority task, and after processing the first priority task, process other tasks except the first priority task, wherein at least one model to be called corresponding to the other tasks is stored in the offline model library, and the offline model library is stored in the disk.

2. The task scheduling method for multi-model inference according to claim 1, characterized in that: The calling of all to-be-called models corresponding to the first priority task in the online model library, processing the first priority task, and after processing the first priority task and before processing other tasks except the first priority task, further includes: Obtaining the waiting time of each request task in the task request queue, and determining whether there is a second priority task in the task request queue whose waiting time exceeds a preset time threshold; If there is a task of the second priority in the task request queue, determining a model to be loaded stored in the offline model library from among all models to be called corresponding to the task of the second priority; The model to be loaded is loaded into the online model library, and the second priority task is processed through all the models to be called corresponding to the second priority task.

3. The task scheduling method for multi-model inference according to claim 2, characterized in that: The step of loading the model to be loaded into the online model library and processing the second priority task through all the models to be called corresponding to the second priority task specifically includes: Acquire the number of uninstallable models in the online model library, wherein the uninstallable models are models in the online model library except all to-be-called models corresponding to the first priority task; When the number of models to be loaded of the models to be loaded does not exceed the number of unloadable models, in the online model library, the unloadable models are unloaded based on the number of models to be loaded, and the models to be loaded are loaded into the online model library; All to-be-called models corresponding to the second priority task in the online model library are called to process the second priority task.

4. The task scheduling method for multi-model inference according to claim 1, characterized in that: The method further includes: before determining the first priority task in each request task based on all the to-be-called models corresponding to each request task; Based on the waiting time of each request task in the task request queue, determining whether there is a second priority task in the task request queue, wherein the waiting time of the second priority task exceeds a preset time threshold; If there is a second priority task in the task request queue, then based on all to-be-called models corresponding to the second priority task, the online model library is updated to load all to-be-called models corresponding to the second priority task into the online model library; All to-be-called models corresponding to the second priority task in the online model library are called to process the second priority task.

5. The task scheduling method for multi-model inference according to claim 1, characterized in that: The method of obtaining each request task in the task request queue and determining all to-be-called models corresponding to each request task based on the task model mapping table further includes: Count each historical request task and its corresponding call model, and determine each request task type and all corresponding call models; The task model mapping table is generated based on each requested task type and all corresponding call models.

6. The task scheduling method for multi-model inference according to claim 1, characterized in that: The calling of all to-be-called models corresponding to the first priority task in the online model library, processing the first priority task, and after processing the first priority task, processing other tasks except the first priority task, further includes: According to a preset period, the access frequency of each model in the online model library is counted, and each model identifier and its corresponding access frequency are associated and stored in an access frequency record table; Based on the access frequency record table, models in the online model library whose access frequency is lower than a preset frequency threshold are uninstalled.

7. The task scheduling method for multi-model inference according to any one of claims 1 to 6, characterized in that: The method further includes: before determining the first priority task in each request task based on all the to-be-called models corresponding to each request task; Based on the calling frequency of each model, a preset number of models are selected in descending order of calling frequency as high-frequency models; The high frequency model is stored in the memory to generate the online model library, and other models except the high frequency are stored in the disk to generate the offline model library.

8. A task scheduling device for multi-model inference, characterized in that: The task scheduling device for multi-model inference comprises: The calling model determination module is used to obtain each request task in the task request queue and determine all to-be-called models corresponding to each request task based on the task model mapping table; A priority task determination module, used for determining a first priority task among each request task based on all to-be-called models corresponding to each request task, wherein all to-be-called models corresponding to the first priority task are stored in an online model library, and the online model library is stored in a memory; A request queue processing module is used to call all the models to be called corresponding to the first priority task in the online model library, process the first priority task, and after processing the first priority task, process other tasks except the first priority task, wherein at least one model to be called corresponding to the other tasks is stored in the offline model library, and the offline model library is stored in the disk.

9. A computer device, characterized in that: The computer device includes a processor, a memory, and a task scheduling program for multi-model inference stored in the memory and executable by the processor, wherein when the task scheduling program for multi-model inference is executed by the processor, the steps of the task scheduling method for multi-model inference as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a task scheduling program for multi-model inference, wherein when the task scheduling program for multi-model inference is executed by a processor, the steps of the task scheduling method for multi-model inference as described in any one of claims 1 to 7 are implemented.

Images