Multiple deep learning model inference scheduling method and system based on RKNN platform

Abstract

The present invention proposes a new multi-deep learning model inference scheduling method and system based on the RKNN platform, the method comprising the following steps: S1, establishing a multi-deep learning model; S2, placing the tasks of the task set Ti of the model Mi into the task queue of the corresponding priority for waiting; S3, the scheduling engine searches the non-empty queues in the task queue in descending order of priority, selects the first task instance in the non-empty queue as the transmission task, and sends it to the NPU core; S4, calculates the load balance of the NPU core, and determines whether the load balance exceeds a preset threshold; S5, when the load balance exceeds the preset threshold, load balances the NPU core. The present invention improves the load balancing in the multi-task inference process, improves the system's continuous operation capability and energy efficiency resource utilization, avoids resource contention between tasks, reduces inference delay, and improves the system's response speed.

Description

Technical Field

[0001] The present invention relates to the field of computer data processing technology, and in particular to a multi-deep learning model reasoning scheduling method and system based on an RKNN platform. Background Art

[0002] Deep learning has been widely used in various fields, such as image recognition, speech processing, and natural language processing. However, because deep learning models are typically computationally complex and require strong hardware support, inference on edge computing devices, in particular, faces the dual challenges of computing resources and power consumption. When multiple deep learning tasks require inference on the same machine simultaneously, each task must wait for the previous one to complete before continuing, resulting in significant latency.

[0003] Currently, there are several key issues in scheduling multi-model inference tasks on the RKNN platform:

[0004] First, there's the issue of inference latency and response. In real-time applications, inference latency is extremely critical. Existing scheduling frameworks often experience significant inference latency under high load, making it difficult to guarantee real-time performance. Furthermore, while some scheduling frameworks consider task priorities, they often lack sufficient precision in determining the priorities of complex tasks.

[0005] Furthermore, deep learning inference tasks are computationally complex, especially when performing multi-task inference in parallel. This can lead to a rapid increase in device power consumption. While hardware acceleration can improve inference efficiency in most frameworks, it doesn't consider how to minimize energy consumption without compromising inference accuracy.

[0006] In view of this, it is necessary to propose a new multi-deep learning model inference scheduling method and system based on the RKNN platform. Summary of the Invention

[0007] The main purpose of the present invention is to provide a multi-deep learning model reasoning scheduling method and system based on the RKNN platform to solve the above technical problems.

[0008] To achieve the above objectives, the present invention provides a multi-deep learning model reasoning scheduling method based on the RKNN platform, comprising the following steps:

[0009] S1, establish multiple deep learning models, including: record the n deep learning models in the RKNN platform at time t as M={M1,…,Mi,…,Mn}, 1≤i≤n; record the deep learning model Mi as Mi={Ti, Li,pi}, where Li represents the criticality of Mi, high criticality and low criticality are represented by 1 and 0 respectively; Pi represents the priority of model Mi, which needs to be manually assigned by the user, and the preset priority levels are Q priorities from high to low; Ti is the task set of model Mi Ti={ ,.. ,.. }, n(Ti) is the number of tasks in the ready task set of model Mi, 1≤j≤n(Ti); task Expressed as ={ei,j ,Di,j ,Ti,j ,pi,j}; where ei,j represents the worst-case NPU inference task at maximum frequency The time required, Di,j represents the current task The time interval from the release start to the deadline, Ti,j represents the task The period, pi,j represents the task The priority of k NPU cores is recorded as V={ρ1, ρ2, …, ρk};

[0010] S2, determine the priority Pi of the inference task of model Mi, and assign the task of task set Ti of model Mi to Place it in the task queue que1~queQ of the corresponding priority and wait;

[0011] S3: The scheduling engine searches for a non-empty queue in the task queues que1 to queQ in descending order of priority, selects the first task instance in the non-empty queue as the transmission task, and sends it to the NPU core;

[0012] S4, computing NPU core Load balancing , determine the load balancing degree Whether it exceeds the preset threshold;

[0013] S5, in the load balancing When the preset threshold is exceeded, load balancing is performed on the NPU cores.

[0014] Preferably, the step S3 includes:

[0015] S31, the scheduling engine searches for a non-empty queue in the task queues que1 to queQ in descending order of priority, and selects the first task instance in the non-empty queue as the transmission task;

[0016] S32, check whether there is an idle NPU core, if so, distribute the selected task instance to the NPU core, otherwise distribute the selected task instance to the NPU core running the lowest priority task and with a relatively smaller load.

[0017] Preferably, the step of load balancing the NPU core in step S5 specifically includes one or more of the following methods: adjusting the scheduling strategy, suspending queue tasks, dynamically adjusting the task execution method, dynamically adjusting the hardware resource utilization, and dynamically adjusting the task accuracy requirements.

[0018] Preferably, in step S4, the formula , computing NPU core Load balancing ;in, is the core The theoretical load, is the core The actual load, , Core workload, is the number of cores of the NPU.

[0019] The present invention also provides a multi-deep learning model reasoning scheduling system based on the RKNN platform, comprising:

[0020] The multi-deep learning model establishment module is used to record the n deep learning models in the RKNN platform at time t as M={M1,…,Mi,…,Mn}, 1≤i≤n; record the deep learning model Mi as Mi={Ti, Li,pi}, where Li represents the criticality of Mi, high criticality and low criticality are represented by 1 and 0 respectively; Pi represents the priority of model Mi, which needs to be manually assigned by the user, and the preset priority levels are Q priorities from high to low; Ti is the task set of model Mi Ti={ ,.. ,.. }, n(Ti) is the number of tasks in the ready task set of model Mi, 1≤j≤n(Ti); task Expressed as ={ei,j ,Di,j ,Ti,j ,pi,j}; where ei,j represents the worst-case NPU inference task at maximum frequency The time required, Di,j represents the current task The time interval from the release start to the deadline, Ti,j represents the task The period, pi,j represents the task The priority of k NPU cores is recorded as V={ρ1, ρ2, …, ρk};

[0021] The task queue allocation module is used to determine the priority Pi of the inference task of model Mi and assign the task of task set Ti of model Mi to Place it in the task queue que1~queQ of the corresponding priority and wait;

[0022] The scheduling engine module is used to search for non-empty queues in the task queues que1 to queQ in descending order of priority, select the first task instance in the non-empty queue as the transmission task, and send it to the NPU core;

[0023] Load balancing monitor module, used to calculate NPU core Load balancing , determine the load balancing degree Whether it exceeds the preset threshold;

[0024] The load balancing module is used to balance the load When the preset threshold is exceeded, load balancing is performed on the NPU cores.

[0025] Preferably, the scheduling engine module is specifically used to: search for non-empty queues in the task queues que1~queQ in descending order of priority, and select the first task instance in the non-empty queue as the transmission task; find out whether there is an idle NPU core, and if so, distribute the selected task instance to the NPU core; otherwise, distribute the selected task instance to the NPU core running the lowest priority task and with a relatively smaller load.

[0026] Preferably, the step of load balancing the NPU core by the load balancing module specifically includes one or more of the following methods: adjusting the scheduling strategy, suspending queue tasks, dynamically adjusting the task execution method, dynamically adjusting the hardware resource utilization, and dynamically adjusting the task accuracy requirements.

[0027] Preferably, the load balancing monitor module is calculated by the formula , computing NPU core Load balancing ;in, is the core The theoretical load, is the core The actual load, , Core workload, is the number of cores of the NPU.

[0028] The present invention also provides a multi-deep learning model inference scheduling system based on the RKNN platform, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, the steps of the multi-deep learning model inference scheduling method based on the RKNN platform as described in any one of the above items are implemented.

[0029] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the multi-deep learning model inference scheduling method based on the RKNN platform as described in any one of the above items.

[0030] The present invention improves load balancing during multi-task reasoning, improves the system's sustainable operation capability and energy efficiency and resource utilization, avoids resource contention between tasks, reduces reasoning delay, and improves the system's response speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings are part of the present invention and are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but do not constitute an undue limitation of the present invention. Obviously, the drawings described below are only some embodiments. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without inventive effort. In the accompanying drawings:

[0032] Figure 1 2 is a flow chart of a method for inference scheduling of multiple deep learning models based on the RKNN platform in one embodiment of the present invention.

[0033] Figure 2 This is an example diagram of a task queue in one embodiment of the present invention.

[0034] Figure 3 This is a schematic diagram of the logical structure of a multi-deep learning model inference scheduling system based on the RKNN platform in one embodiment of the present invention.

[0035] Figure 4 Schematic diagram of the hardware structure for running the multi-deep learning model inference scheduling method based on the RKNN platform in one embodiment of the present invention.

[0036] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0037] The following is a clear and complete description of the technical problems solved by the embodiments of the present invention, the technical solutions adopted, and the technical effects achieved, in conjunction with the accompanying drawings and specific embodiments. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other equivalent or obviously modified embodiments obtained by ordinary technicians in this field without paying creative work fall within the scope of protection of the present invention. The embodiments of the present invention can be concretized in a variety of different ways as defined and covered in the claims.

[0038] It should be noted that in the following description, many specific details are given for ease of understanding, but it is obvious that the present invention can be implemented without these specific details.

[0039] It should be noted that, in the absence of clear limitations or conflicts, the various embodiments of the present invention and the technical features therein can be combined with each other to form a technical solution.

[0040] Please refer to Figure 1 , the present invention proposes a multi-deep learning model reasoning scheduling method based on the RKNN platform, comprising the steps of:

[0041] S1, establish multiple deep learning models, including: record the n deep learning models in the RKNN platform at time t as M={M1,…,Mi,…,Mn}, 1≤i≤n; record the deep learning model Mi as Mi={Ti, Li,pi}, where Li represents the criticality of Mi, high criticality and low criticality are represented by 1 and 0 respectively; Pi represents the priority of model Mi, which needs to be manually assigned by the user, and the preset priority levels are Q priorities from high to low; Ti is the task set of model Mi Ti={ ,.. ,.. }, n(Ti) is the number of tasks in the ready task set of model Mi, 1≤j≤n(Ti); task Expressed as ={ei,j ,Di,j ,Ti,j ,pi,j}; where ei,j represents the worst-case NPU inference task at maximum frequency The time required, Di,j represents the current task The time interval from the release start to the deadline, Ti,j represents the task The period, pi,j represents the task The priority of k NPU cores is recorded as V={ρ1, ρ2, …, ρk};

[0042] S2, determine the priority Pi of the inference task of model Mi, and assign the task of task set Ti of model Mi to Place it in the task queue que1~queQ of the corresponding priority and wait;

[0043] S3: The scheduling engine searches for a non-empty queue in the task queues que1 to queQ in descending order of priority, selects the first task instance in the non-empty queue as the transmission task, and sends it to the NPU core;

[0044] S4, computing NPU core Load balancing , determine the load balancing degree Whether it exceeds the preset threshold;

[0045] S5, in the load balancing When the preset threshold is exceeded, load balancing is performed on the NPU cores.

[0046] In step S1, assume that there are m NPU cores in the system and n deep learning models in the RKNN at time t. Assume that the set of these deep learning models is M={M1,…,Mi,…,Mn}, 1<=i<=n; model Mi can be expressed as Mi={Ti, Li,pi}, where Li represents the criticality of Mi, high criticality and low criticality are represented by 1 and 0 respectively, and the tasks of the high criticality model must meet the real-time constraints. Pi represents the priority of Mi, which needs to be manually assigned by the user. Ti is the task set of model Mi Ti={ ,.. ,.. }; n(Ti) is the number of tasks in the ready task set of model Mi, 1≤j≤n(Ti). Task Expressed as = {ei,j ,Di,j ,Ti,j ,pi,j}; where ei,j represents the worst-case inference time (WCET) required by the NPU to perform a task at maximum frequency, Di,j represents the interval from the release to the deadline of the current inference task, Ti,j represents the task period, and pi,j represents the task priority, which is calculated using the Priority Assignment Algorithm (LPA). ei,j and Di,j can be specified based on the specific application during system design. k NPU cores can be abstracted as V = {ρ1, ρ2, …, ρk}.

[0047] Preferably, the step S3 includes:

[0048] S31, the scheduling engine searches for a non-empty queue in the task queues que1 to queQ in descending order of priority, and selects the first task instance in the non-empty queue as the transmission task;

[0049] S32, check whether there is an idle NPU core, if so, distribute the selected task instance to the NPU core, otherwise distribute the selected task instance to the NPU core running the lowest priority task and with a relatively smaller load.

[0050] In a multi-task inference scheduling framework, the priority queue is a crucial component of the task scheduling process. Its primary function is to manage and store task instances to be inferred and assign them to the appropriate queue based on their priority, ensuring that the scheduling engine selects the appropriate inference task for execution. The priority queue not only optimizes the scheduling order of tasks and improves task execution efficiency, but also ensures that high-priority tasks are prioritized when system resources are limited, preventing low-priority tasks from blocking the system and meeting real-time requirements.

[0051] A priority queue is a specialized queue data structure in which tasks are sorted according to predefined priorities. In multi-task reasoning systems, priority is typically determined by factors such as the task's urgency, importance, and real-time requirements. When each task is submitted to the queue, it is inserted into the corresponding position based on its priority. High-priority tasks are placed at the front of the queue and are scheduled for execution first.

[0052] In order to achieve efficient task management, the priority queue is implemented using a red-black tree, which can perform insertion and deletion operations with logarithmic time complexity.

[0053] Please combine the specific Figure 2 , in this example, Q is taken as 8.

[0054] The task queue stores the task instances to be inferred. The scheduling principle is to give priority to scheduling tasks with higher priority and earlier arrival time for inference, such as Figure 2 As shown, the task queue has eight queues, que1 through que8. These queues are implemented using a priority and first-in-first-out (FIFO) approach. Priority means each queue has a fixed priority, ranging from 1 to 8. Queue 1 has the highest priority, indicating the highest priority in the queue. Queue 2 has a priority of 2, indicating the second highest priority, and so on. Priority ensures that model instances with higher priorities are scheduled first. FIFO means that model instances in the queue are prioritized based on the time they were inserted. FIFO ensures that stream instances with the same priority are sent first.

[0055] The priority is 1, which means the highest priority. has a priority of 8, which is the lowest priority, and so on. The priority ensures that task instances with higher priorities are reasoned first. and are the first and second task instances of the task set T2 of model M2, and are task instances of models M3 and M5, where and has a higher priority than and , and Will be in and The queue ensures that the tasks inserted earlier are scheduled first, such as the two instances of T3 and T5 have the same priority, but due to Compare Join earlier ,so Will be in The reasoning was obtained before, similarly Compare Prioritize reasoning.

[0056] Preferably, the step of load balancing the NPU core in step S5 specifically includes one or more of the following methods: adjusting the scheduling policy, temporarily suspending the fetching of tasks from the queue, dynamically adjusting the task execution method, dynamically adjusting the hardware resource utilization, and dynamically adjusting the task accuracy requirements. For example, without affecting the task results, power consumption can be reduced by reducing the calculation accuracy of the model (such as using 8-bit quantization), or the scheduling engine can be temporarily suspended from fetching tasks from the queue.

[0057] Preferably, in step S4, the formula , computing NPU core Load balancing ;in, is the core The theoretical load, is the core The actual load, , Core workload, is the number of cores of the NPU.

[0058] Generally speaking, you can set it based on experience. The load balancing is good when the value is between 1 and 1.8. At 1.8~2, the load balancing is still acceptable. The load balancing is considered to be poor at 2~4. >4 is considered load imbalance and unacceptable.

[0059] Specifically, during the whole process, the load balance monitor module monitors the hardware power consumption, logs, and the load of each NPU core (through the interface / sys / kernel / debug / rknpu / load provided by RKNN) and calculates the load situation. Load balancing is achieved through load imbalance. To measure, , where ρi is a core of NPU, and Depends on the core with the most unbalanced load among all cores. The closer the value is to 1, the more balanced the scheduling result is. Therefore, when the monitor detects that the load imbalance value exceeds the acceptable threshold, it can take measures to balance the load, adjust the scheduling policy, or temporarily suspend queue tasks. It can dynamically adjust the task execution mode, hardware resource utilization, and task accuracy requirements to reduce unnecessary power consumption and extend the device's service life.

[0060] In a multi-task inference scheduling framework, it is often necessary to simultaneously process inference tasks for multiple deep learning models. To improve the system's concurrency and performance, thread pool management is crucial. Thread pools effectively reuse threads, avoiding frequent thread creation and destruction, thereby reducing system overhead and improving task execution efficiency. Because a system may contain multiple thread pools, a sound thread pool management strategy is essential to ensure proper resource allocation and efficient task scheduling.

[0061] Different inference tasks may have different priorities. Thread pool resource allocation can be dynamically adjusted based on task priorities. For example, high-priority tasks can be allocated more thread pool resources, while low-priority tasks can wait or use fewer resources.

[0062] The thread pool size is dynamically adjusted based on the current task load. For compute-intensive tasks, more threads may be needed to increase parallelism; for memory-intensive tasks, the number of threads may need to be reduced to avoid excessive memory usage.

[0063] In a specific example, the multi-deep learning model inference scheduling based on the RKNN platform may include the following steps:

[0064] Step 1: Define the deep learning model that needs to be inferred in advance. If it is a non-RKNN model, it needs to be trained on a server or PC locally. Convert the trained model into an onnx model, and then further convert it into an rknn model that can be deployed on the RKNN platform. For example, the pytorch model can be converted to an onnx model through the torch.onnx.export interface, or directly load the corresponding model through the RKNN API function (load_caffe, load_tensorflow, load_tflite, load_onnx, load_pytorch), and then use the build interface to build the rknn model. Select the target version to be converted for the corresponding RK core board, and use export_onnx to convert it to the rknn model of the corresponding target board. Finally, push it to the edge RKNN device through the adb tool.

[0065] Step 2: Multi-threaded configuration: Install or update RKNPU2 on the development board, push rknn_server and librknnrt.so to the development board, start rknn_server, configure the rknpu environment, encapsulate and initialize the rknn model interface, initialize the rknn model as a list of rknn objects, encapsulate a parent class for controlling and pausing threads, the rknn thread pool class inherits the parent class, and loads each rknn model.

[0066] Step 3: If there is an inference task, calculate the priority of each rknn model task instance and add it to the corresponding priority queue.

[0067] Step 4: The scheduling engine traverses the priority queue in descending order of priority and selects the first task in the task queue with the highest priority.

[0068] Step 5: Check whether there is an idle NPU core. If so, distribute the selected task instance to the NPU core. Otherwise, select the NPU core that runs the lowest priority task and has a relatively smaller load. If the priority is not higher than the positive inference task, block and wait for a while before repeating this step.

[0069] Step 6: Start each thread in the thread pool to infer the captured input source through the rknn model.

[0070] Step 7: If all the tasks of the models are completed, the process ends and the occupied NPU cores and rknn models are released. Otherwise, return to step 3.

[0071] Please combine Figure 3 The present invention also provides a multi-deep learning model reasoning scheduling system based on the RKNN platform, comprising:

[0072] The multi-deep learning model establishment module is used to record the n deep learning models in the RKNN platform at time t as M={M1,…,Mi,…,Mn}, 1≤i≤n; record the deep learning model Mi as Mi={Ti, Li,pi}, where Li represents the criticality of Mi, high criticality and low criticality are represented by 1 and 0 respectively; Pi represents the priority of model Mi, which needs to be manually assigned by the user, and the preset priority levels are Q priorities from high to low; Ti is the task set of model Mi Ti={ ,.. ,.. }, n(Ti) is the number of tasks in the ready task set of model Mi, 1≤j≤n(Ti); task Expressed as ={ei,j ,Di,j ,Ti,j ,pi,j}; where ei,j represents the worst-case NPU inference task at maximum frequency The time required, Di,j represents the current task The time interval from the release start to the deadline, Ti,j represents the task The period, pi,j represents the task The priority of k NPU cores is recorded as V={ρ1, ρ2, …, ρk};

[0073] The task queue dispatcher module is used to determine the priority Pi of the inference task of the model Mi and assign the task of the task set Ti of the model Mi to the task Place it in the task queue que1~queQ of the corresponding priority and wait;

[0074] The scheduling engine module is used to search for non-empty queues in the task queues que1 to queQ in descending order of priority, select the first task instance in the non-empty queue as the transmission task, and send it to the NPU core;

[0075] Load balancing monitor module, used to calculate NPU core Load balancing , determine the load balancing degree Whether it exceeds the preset threshold;

[0076] The load balancing module is used to balance the load When the preset threshold is exceeded, load balancing is performed on the NPU cores.

[0077] Preferably, the scheduling engine module is specifically used to: search for non-empty queues in the task queues que1~queQ in descending order of priority, and select the first task instance in the non-empty queue as the transmission task; find out whether there is an idle NPU core, and if so, distribute the selected task instance to the NPU core; otherwise, distribute the selected task instance to the NPU core running the lowest priority task and with a relatively smaller load.

[0078] Preferably, the step of load balancing the NPU core by the load balancing module specifically includes one or more of the following methods: adjusting the scheduling strategy, suspending queue tasks, dynamically adjusting the task execution method, dynamically adjusting the hardware resource utilization, and dynamically adjusting the task accuracy requirements.

[0079] Preferably, the load balancing monitor module is calculated by the formula , computing NPU core Load balancing ;in, is the core The theoretical load, is the core The actual load, , Core workload, is the number of cores of the NPU.

[0080] The present invention also provides a multi-deep learning model inference scheduling system based on the RKNN platform, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, the steps of the multi-deep learning model inference scheduling method based on the RKNN platform as described in any one of the above items are implemented.

[0081] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the multi-deep learning model inference scheduling method based on the RKNN platform as described in any one of the above items.

[0082] The present invention improves load balancing during multi-task reasoning, improves the system's sustainable operation capability and energy efficiency and resource utilization, avoids resource contention between tasks, reduces reasoning delay, and improves the system's response speed.

[0083] Figure 4 FIG. 1 is a schematic diagram of the hardware structure for running a knowledge tracking method based on a two-state joint interaction mechanism provided by an embodiment of the present invention. Figure 4As shown, this embodiment / computer 6 includes: a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60, such as a program for executing a knowledge tracking method based on a two-state joint interaction mechanism. When the processor 60 executes the computer program 62, the steps of the aforementioned method embodiments are implemented. Alternatively, when the processor 60 executes the computer program 62, the functions of the modules / units in the aforementioned device embodiments are implemented.

[0084] For example, the computer program 62 may be divided into one or more modules / units, which are stored in the memory 61 and executed by the processor 60 to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program 62 in the computer 6.

[0085] The computer 6 can be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The computer 6 device can include, but is not limited to, a processor 60 and a memory 61. It can be understood by those skilled in the art that Figure 4 This is only an example of computer 6 and does not constitute a limitation on computer 6. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer 6 may also include input and output devices, network access devices, buses, etc.

[0086] The processor 60 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0087] The memory 61 can be an internal storage unit of the computer 6, such as the computer's hard drive or memory. Alternatively, the memory 61 can be an external storage device of the terminal device / computer 6, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. Furthermore, the memory 61 can include both the internal storage unit of the computer 6 and an external storage device. The memory 61 is used to store the computer program and other programs and data required by the terminal device. The memory 61 can also be used to temporarily store data that has been output or is about to be output.

[0088] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0089] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0090] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0091] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0092] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0093] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0094] If the integrated module / unit is implemented as a software functional unit and sold or used as a standalone product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention can implement all or part of the process steps in the above-mentioned method embodiments by using a computer program to instruct the relevant hardware. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content of the computer-readable medium can be appropriately increased or decreased based on the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, based on legislation and patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.

[0095] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A multi-deep learning model inference scheduling method based on the RKNN platform, characterized in that: Including steps: S1, establish multiple deep learning models, including: record the n deep learning models in the RKNN platform at time t as M={M1,…,Mi,…,Mn}, 1≤i≤n; record the deep learning model Mi as Mi={Ti, Li,pi}, where Li represents the criticality of Mi, high criticality and low criticality are represented by 1 and 0 respectively; Pi represents the priority of model Mi, which needs to be manually assigned by the user, and the preset priority levels are Q priorities from high to low; Ti is the task set of model Mi Ti={ ,.. ,.. }, n(Ti) is the number of tasks in the ready task set of model Mi, 1≤j≤n(Ti); task Expressed as ={ei,j ,Di,j ,Ti,j ,pi,j}; where ei,j represents the worst-case NPU inference task at maximum frequency The time required, Di,j represents the current task The time interval from the release start to the deadline, Ti,j represents the task The period, pi,j represents the task The priority of k NPU cores is recorded as V={ρ1, ρ2, …, ρk}; S2, determine the priority Pi of the inference task of model Mi, and assign the task of task set Ti of model Mi to Place it in the task queue que1~queQ of the corresponding priority and wait; S3: The scheduling engine searches for a non-empty queue in the task queues que1 to queQ in descending order of priority, selects the first task instance in the non-empty queue as the transmission task, and sends it to the NPU core; S4, computing NPU core Load balancing , determine the load balancing degree Whether it exceeds the preset threshold; S5, in the load balancing When the preset threshold is exceeded, load balancing is performed on the NPU cores.

2. The multi-deep learning model reasoning scheduling method based on the RKNN platform according to claim 1 is characterized in that: The step S3 comprises: S31, the scheduling engine searches for a non-empty queue in the task queues que1 to queQ in descending order of priority, and selects the first task instance in the non-empty queue as the transmission task; S32, check whether there is an idle NPU core, if so, distribute the selected task instance to the NPU core, otherwise distribute the selected task instance to the NPU core running the lowest priority task and with a relatively smaller load.

3. The multi-deep learning model reasoning scheduling method based on the RKNN platform according to claim 1 is characterized in that: The step of load balancing the NPU core in step S5 specifically includes one or more of the following methods: adjusting the scheduling strategy, suspending queue tasks, dynamically adjusting the task execution method, dynamically adjusting the hardware resource utilization, and dynamically adjusting the task accuracy requirements.

4. The multi-deep learning model reasoning scheduling method based on the RKNN platform according to claim 1 is characterized in that: In step S4, the formula , computing NPU core Load balancing ;in, is the core The theoretical load, is the core The actual load, , Core workload, is the number of cores of the NPU.

5. A multi-deep learning model inference scheduling system based on the RKNN platform, characterized by: include: The multi-deep learning model establishment module is used to record the n deep learning models in the RKNN platform at time t as M={M1,…,Mi,…,Mn}, 1≤i≤n; record the deep learning model Mi as Mi={Ti, Li,pi}, where Li represents the criticality of Mi, high criticality and low criticality are represented by 1 and 0 respectively; Pi represents the priority of model Mi, which needs to be manually assigned by the user, and the preset priority levels are Q priorities from high to low; Ti is the task set of model Mi Ti={ ,.. ,.. }, n(Ti) is the number of tasks in the ready task set of model Mi, 1≤j≤n(Ti); task Expressed as ={ei,j ,Di,j ,Ti,j ,pi,j}; where ei,j represents the worst-case NPU inference task at maximum frequency The time required, Di,j represents the current task The time interval from the release start to the deadline, Ti,j represents the task The period, pi,j represents the task The priority of k NPU cores is recorded as V={ρ1, ρ2, …, ρk}; The task queue allocation module is used to determine the priority Pi of the inference task of model Mi and assign the task of task set Ti of model Mi to Place it in the task queue que1~queQ of the corresponding priority and wait; The scheduling engine module is used to search for non-empty queues in the task queues que1 to queQ in descending order of priority, select the first task instance in the non-empty queue as the transmission task, and send it to the NPU core; Load balancing monitor module, used to calculate NPU core Load balancing , determine the load balancing degree Whether it exceeds the preset threshold; The load balancing module is used to balance the load When the preset threshold is exceeded, load balancing is performed on the NPU cores.

6. The multi-deep learning model inference scheduling system based on the RKNN platform according to claim 5 is characterized in that: The scheduling engine module is specifically used to: search for non-empty queues in the task queues que1~queQ in descending order of priority, and select the first task instance in the non-empty queue as the transmission task; check whether there is an idle NPU core, and if so, distribute the selected task instance to the NPU core; otherwise, distribute the selected task instance to the NPU core running the lowest priority task and with a relatively smaller load.

7. The multi-deep learning model inference scheduling system based on the RKNN platform according to claim 5 is characterized in that: The steps of the load balancing module for load balancing the NPU core specifically include one or more of the following methods: adjusting the scheduling strategy, suspending queue tasks, dynamically adjusting the task execution method, dynamically adjusting the hardware resource utilization, and dynamically adjusting the task accuracy requirements.

8. The multi-deep learning model inference scheduling system based on the RKNN platform according to claim 5 is characterized in that: The load balancing monitor module is calculated by the formula , computing NPU core Load balancing ;in, is the core The theoretical load, is the core The actual load, , Core workload, is the number of cores of the NPU.

9. A multi-deep learning model inference scheduling system based on the RKNN platform, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the multi-deep learning model reasoning scheduling method based on the RKNN platform are implemented as described in any one of claims 1 to 4.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the multi-deep learning model reasoning scheduling method based on the RKNN platform are implemented as described in any one of claims 1 to 4.

Images