RKNN platform-based multi-deep learning model reasoning scheduling method and system

Abstract

The invention provides a novel multi-deep learning model reasoning scheduling method and system based on an RKNN platform. The method comprises the following steps: S1, establishing a multi-deep learning model; s2, placing a task # imgabs0 # of a task set Ti of the model Mi into a task queue with a corresponding priority to wait; s3, the scheduling engine searches a non-empty queue in the task queues according to a priority descending order, selects a first task instance in the non-empty queue as a transmission task, and sends the transmission task to the NPU core; s4, calculating the load balance degree of the NPU core # imgabs1 #, and judging whether the load balance degree exceeds a preset threshold value or not; and S5, when the load balancing degree exceeds the preset threshold value, carrying out load balancing on the NPU core. According to the method, the load balance in the multi-task reasoning process is improved, the continuous operation capability, the energy efficiency and the resource utilization rate of the system are improved, the resource contention between the tasks is avoided, the reasoning delay is reduced, and the response speed of the system is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer data processing technology, and in particular to a multi-deep learning model reasoning scheduling method and system based on an RKNN platform. Background Art

[0002] Deep learning has been widely used in various fields, such as image recognition, speech processing, and natural language processing. However, since deep learning models are usually computationally complex and require strong hardware support, reasoning on edge computing devices faces the dual challenges of computing resources and power consumption. When multiple deep learning tasks need to be reasoned on the same machine at the same time, the task needs to wait for the previous task to complete before continuing reasoning, and the delay becomes particularly obvious.

[0003] At present, there are several key issues in scheduling multi-model reasoning tasks on the RKNN platform: The first is the problem of inference delay and response. In real-time application scenarios, inference delay is extremely strict. Existing scheduling frameworks often produce large inference delays under high load conditions, making it difficult to ensure real-time performance. In addition, although some scheduling frameworks consider the priority of tasks, the priority judgment of complex tasks is often not precise enough.

[0004] Furthermore, deep learning reasoning tasks are computationally complex, especially when performing multi-task parallel reasoning, which can cause the power consumption of devices to increase rapidly. In most frameworks, although hardware acceleration can improve reasoning efficiency, it does not consider how to minimize energy consumption without affecting reasoning accuracy.

[0005] In view of this, it is necessary to propose a new multi-deep learning model reasoning scheduling method and system based on the RKNN platform. Summary of the invention

[0006] The main purpose of the present invention is to provide a multi-deep learning model reasoning scheduling method and system based on the RKNN platform to solve the above technical problems.

[0007] To achieve the above object, the present invention provides a multi-deep learning model reasoning scheduling method based on the RKNN platform, comprising the steps of: S1, establish multiple deep learning models, including: record the n deep learning models in the RKNN platform at time t as M={M1,…,Mi,…,Mn}, 1≤i≤n; record the deep learning model Mi as Mi={Ti, Li,pi}, where Li represents the critical level of Mi, high criticality and low criticality are represented by 1 and 0 respectively; Pi represents the priority of model Mi, which needs to be manually assigned by the user, and the preset priority level is Q priorities from high to low; Ti is the task set of model Mi Ti= { ,.. ,,.. }, n(Ti) is the number of tasks in the ready task set of model Mi, 1≤j≤n(Ti); task Expressed as ={ei,j ,Di,j ,Ti,j ,pi,j}; where ei,j represents the worst case NPU inference task at maximum frequency The time required, Di,j represents the current task The time interval from the release start to the deadline, Ti,j represents the task The cycle of pi,j is represented by the task The priority of k NPU cores is recorded as V={ρ1, ρ2, …, ρk}; S2, determine the priority Pi of the inference task of model Mi, and assign the task of task set Ti of model Mi to Place it in the task queue que1~queQ of the corresponding priority and wait; S3, the scheduling engine searches for a non-empty queue in the task queues que1~queQ in descending order of priority, selects the first task instance in the non-empty queue as the transmission task, and sends it to the NPU core; S4, computing NPU core Load balancing , determine the load balancing degree Whether it exceeds the preset threshold; S5, at the load balancing level When the preset threshold is exceeded, load balancing is performed on the NPU cores.

[0008] Preferably, the step S3 comprises: S31, the scheduling engine searches for a non-empty queue in the task queues que1 to queQ in descending order of priority, and selects the first task instance in the non-empty queue as the transmission task; S32, find out whether there is an idle NPU core, if so, distribute the selected task instance to the NPU core, otherwise distribute the selected task instance to the NPU core running the lowest priority task and with a relatively smaller load.

[0009] Preferably, the step of load balancing the NPU core in step S5 specifically includes one or more of the following methods: adjusting the scheduling strategy, suspending queue tasks, dynamically adjusting the task execution mode, dynamically adjusting the hardware resource utilization, and dynamically adjusting the task accuracy requirement.

[0010] Preferably, in step S4, by formula , computing NPU core Load balancing ;in, Is the core The theoretical load, Is the core The actual load, , Core workload, is the number of cores of the NPU.

[0011] The present invention also provides a multi-deep learning model reasoning scheduling system based on the RKNN platform, comprising: The multi-deep learning model establishment module is used to record the n deep learning models in the RKNN platform at time t as M={M1,…,Mi,…,Mn}, 1≤i≤n; record the deep learning model Mi as Mi={Ti, Li,pi}, where Li represents the critical level of Mi, high criticality and low criticality are represented by 1 and 0 respectively; Pi represents the priority of model Mi, which needs to be manually assigned by the user, and the preset priority level is Q priorities from high to low; Ti is the task set of model Mi Ti= { ,.. ,,.. }, n(Ti) is the number of tasks in the ready task set of model Mi, 1≤j≤n(Ti); task Expressed as ={ei,j ,Di,j ,Ti,j ,pi,j}; where ei,j represents the worst case NPU inference task at maximum frequency The time required, Di,j represents the current task The time interval from the release start to the deadline, Ti,j represents the task The cycle of pi,j is represented by the task The priority of k NPU cores is recorded as V={ρ1, ρ2, …, ρk}; The task queue allocation module is used to determine the priority Pi of the inference task of model Mi and assign the task of task set Ti of model Mi to the task of Place it in the task queue que1~queQ of the corresponding priority and wait; The scheduling engine module is used to search for non-empty queues in the task queues que1~queQ in descending order of priority, select the first task instance in the non-empty queue as the transmission task, and send it to the NPU core; Load balancing monitor module, used to calculate NPU core Load balancing , determine the load balancing degree Whether it exceeds the preset threshold; The load balancing module is used to balance the load When the preset threshold is exceeded, load balancing is performed on the NPU cores.

[0012] Preferably, the scheduling engine module is specifically used to: search for non-empty queues in the task queues que1~queQ in descending order of priority, and select the first task instance in the non-empty queue as the transmission task; find out whether there is an idle NPU core, and if so, distribute the selected task instance to the NPU core; otherwise, distribute the selected task instance to the NPU core running the lowest priority task and with a relatively smaller load.

[0013] Preferably, the step of load balancing the NPU core by the load balancing module specifically includes one or more of the following methods: adjusting the scheduling strategy, suspending queue tasks, dynamically adjusting the task execution mode, dynamically adjusting the hardware resource utilization, and dynamically adjusting the task accuracy requirements.

[0014] Preferably, the load balancing monitor module is configured by the formula , computing NPU core Load balancing ;in, Is the core The theoretical load, Is the core The actual load, , Core workload, is the number of cores of the NPU.

[0015] The present invention also provides a multi-deep learning model reasoning scheduling system based on the RKNN platform, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the multi-deep learning model reasoning scheduling method based on the RKNN platform as described in any one of the above items are implemented.

[0016] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the multi-deep learning model reasoning scheduling method based on the RKNN platform as described in any of the above items.

[0017] The present invention improves load balancing in the multi-task reasoning process, improves the system's continuous operation capability and energy efficiency resource utilization, avoids resource contention between tasks, reduces reasoning delays, and improves the system's response speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are part of the present invention and are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but do not constitute an improper limitation of the present invention. Obviously, the drawings described below are only some embodiments. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the accompanying drawings: Figure 1 Schematic diagram of the process of a multi-deep learning model reasoning scheduling method based on the RKNN platform in one embodiment of the present invention.

[0019] Figure 2 This is an example diagram of a task queue in one embodiment of the present invention.

[0020] Figure 3 This is a schematic diagram of the logical structure of a multi-deep learning model reasoning scheduling system based on the RKNN platform in one embodiment of the present invention.

[0021] Figure 4 Schematic diagram of the hardware structure for running a multi-deep learning model reasoning scheduling method based on the RKNN platform in one embodiment of the present invention.

[0022] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0023] The following is a clear and complete description of the technical problems solved by the embodiments of the present invention, the technical solutions adopted, and the technical effects achieved in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other equivalent or obviously modified embodiments obtained by ordinary technicians in this field without creative work fall within the scope of protection of the present invention. The embodiments of the present invention can be concretized in a variety of different ways as defined and covered in the claims.

[0024] It should be noted that in the following description, many specific details are given for ease of understanding, but it is obvious that the present invention can be implemented without these specific details.

[0025] It should be noted that, in the absence of clear limitation or conflict, the various embodiments of the present invention and the technical features therein can be combined with each other to form a technical solution.

[0026] Please refer to Figure 1 The present invention proposes a multi-deep learning model reasoning scheduling method based on the RKNN platform, comprising the steps of: S1, establish multiple deep learning models, including: record the n deep learning models in the RKNN platform at time t as M={M1,…,Mi,…,Mn}, 1≤i≤n; record the deep learning model Mi as Mi={Ti, Li,pi}, where Li represents the critical level of Mi, high criticality and low criticality are represented by 1 and 0 respectively; Pi represents the priority of model Mi, which needs to be manually assigned by the user, and the preset priority level is Q priorities from high to low; Ti is the task set of model Mi Ti= { ,.. ,,.. }, n(Ti) is the number of tasks in the ready task set of model Mi, 1≤j≤n(Ti); task Expressed as ={ei,j ,Di,j ,Ti,j ,pi,j}; where ei,j represents the worst case NPU inference task at maximum frequency The time required, Di,j represents the current task The time interval from the release start to the deadline, Ti,j represents the task The cycle of pi,j is represented by the task The priority of k NPU cores is recorded as V={ρ1, ρ2, …, ρk}; S2, determine the priority Pi of the inference task of model Mi, and assign the task of task set Ti of model Mi to Place it in the task queue que1~queQ of the corresponding priority and wait; S3, the scheduling engine searches for a non-empty queue in the task queues que1~queQ in descending order of priority, selects the first task instance in the non-empty queue as the transmission task, and sends it to the NPU core; S4, computing NPU core Load balancing , determine the load balancing degree Whether it exceeds the preset threshold; S5, at the load balancing level When the preset threshold is exceeded, load balancing is performed on the NPU cores.

[0027] In step S1, assume that there are m NPU cores in the system, and there are n deep learning models in RKNN at time t. Assume that the set of these deep learning models is M={M1,…,Mi,…,Mn}, 1<=i<=n; model Mi can be expressed as Mi={Ti, Li,pi}, where Li represents the criticality of Mi, high criticality and low criticality are represented by 1 and 0 respectively, and the tasks of the high criticality model must meet the real-time constraints. Pi represents the priority of Mi, which needs to be manually assigned by the user. Ti is the task set of model Mi Ti= { ,.. ,,.. }; n(Ti) is the number of tasks in the ready task set of model Mi, 1≤j≤n(Ti). Task Expressed as ={ei,j ,Di,j ,Ti,j ,pi,j}; where ei,j represents the time required for the NPU to infer the task at the maximum frequency in the worst case (WCET), Di,j represents the time interval from the release to the deadline of the current inference task, Ti,j represents the task cycle, and pi,j represents the priority of the task, which is calculated by LPA [priority assignment algorithm]. ei,j and Di,j can be given according to the specific application during system design. k NPU cores can be abstracted as V={ρ1, ρ2, …, ρk}.

[0028] Preferably, the step S3 comprises: S31, the scheduling engine searches for a non-empty queue in the task queues que1 to queQ in descending order of priority, and selects the first task instance in the non-empty queue as the transmission task; S32, find out whether there is an idle NPU core, if so, distribute the selected task instance to the NPU core, otherwise distribute the selected task instance to the NPU core running the lowest priority task and with a relatively smaller load.

[0029] In the multi-task reasoning scheduling framework, the priority queue is a crucial component of the task scheduling process. Its main function is to manage and store the task instances to be reasoned, and assign them to the corresponding queues according to the priority of the task to ensure that the scheduling engine can select the appropriate reasoning task for execution. The priority queue can not only optimize the scheduling order of tasks and improve the efficiency of task execution, but also ensure that high-priority tasks are executed first when system resources are limited, avoid low-priority tasks blocking the system, and meet real-time requirements.

[0030] A priority queue is a special queue data structure in which tasks are sorted according to predefined priorities. In a multi-task reasoning system, priority is usually determined by factors such as the urgency, importance, and real-time requirements of the task. When each task is submitted to the queue, it will be inserted into the corresponding position according to its priority. High-priority tasks will be placed at the front of the queue and will be scheduled for execution first.

[0031] In order to achieve efficient task management, the priority queue is implemented using a red-black tree, which can perform insertion and deletion operations with logarithmic time complexity.

[0032] Please combine the specific Figure 2 , in this example, Q is taken as 8.

[0033] The task queue stores the task instances to be inferred. The scheduling principle is to give priority to scheduling tasks with higher priority and earlier arrival time for inference, such as Figure 2 As shown in the figure, a total of 8 queues are set for the task queue, namely que1~que8. The task queue is implemented in the form of priority and FIFO. The priority means that each queue has a fixed priority ranging from 1 to 8, where que 1 has the highest priority, indicating the highest priority in the queue, que 2 has a priority of 2, indicating the second highest priority, and so on. Priority ensures that model instances with higher priorities are scheduled first. FIFO means that model instances in the queue are prioritized according to the time of insertion. FIFO ensures that flow instances with the same priority are sent first.

[0034] The priority is 1, indicating the highest priority. has a priority of 8, which is the lowest priority, and so on. The priority ensures that task instances with higher priorities are reasoned first. and are the first and second task instances of task set T2 of model M2, and is the task instance of model M3 and M5, where and has a higher priority than and , and Will be in and The queue ensures that the tasks inserted earlier are scheduled first. For example, the two instances of T3 and T5 have the same priority, but due to Compare Join earlier ,so Will be in The reasoning was obtained before, similarly Compare Prioritize reasoning.

[0035] Preferably, the step of load balancing the NPU core in step S5 specifically includes one or more of the following methods: adjusting the scheduling strategy, suspending the fetching of queue tasks, dynamically adjusting the task execution mode, dynamically adjusting the hardware resource utilization, and dynamically adjusting the accuracy requirements of the task. For example, without affecting the task results, the power consumption can be reduced by reducing the calculation accuracy of the model (such as using 8-bit quantization), or the scheduling engine can be suspended from fetching tasks from the queue.

[0036] Preferably, in step S4, by formula , computing NPU core Load balancing ;in, Is the core The theoretical load, Is the core The actual load, , Core workload, is the number of cores of the NPU.

[0037] Generally speaking, it can be set based on experience. The value between 1 and 1.8 is considered to be well-balanced. At 1.8~2, the load balancing is still acceptable. The load balancing is considered poor at 2 to 4. >4: The load is unbalanced and unacceptable.

[0038] Specifically, during the whole process, the load balance monitor module monitors the hardware power consumption, logs, and load of each NPU core (through the interface / sys / kernel / debug / rknpu / load provided by RKNN) and calculates the load situation. Load balancing is achieved through load imbalance. To measure, , where ρi is a core of NPU, and The core with the most unbalanced load among all cores. The closer the value is to 1, the more balanced the scheduling result is. Therefore, when the monitor detects that the load imbalance value exceeds the acceptable threshold, it can take measures to balance the load or adjust the scheduling strategy or temporarily suspend queue tasks, dynamically adjust the task execution mode, hardware resource utilization, and task accuracy requirements, reduce unnecessary power consumption, and extend the use time of the device.

[0039] In a multi-task reasoning scheduling framework, it is usually necessary to process reasoning tasks of multiple deep learning models at the same time. In order to improve the concurrency and performance of the system, thread pool management is particularly important. The thread pool can effectively reuse threads and avoid frequent creation and destruction of threads, thereby reducing system overhead and improving the efficiency of task execution. Since there may be multiple thread pools in the system, a reasonable thread pool management strategy is required to ensure the reasonable allocation of resources and efficient scheduling of tasks.

[0040] Different reasoning tasks may have different priorities. The resource allocation of the thread pool can be dynamically adjusted according to the priority of the task. For example, high-priority tasks can be allocated more thread pool resources, while low-priority tasks can wait or use fewer resources.

[0041] The thread pool size is adjusted dynamically based on the current task load. For computationally intensive tasks, more threads may be needed to increase parallelism, while for memory intensive tasks, the number of threads may need to be reduced to avoid excessive memory usage.

[0042] In a specific example, the multi-deep learning model reasoning scheduling based on the RKNN platform may include the following steps: Step 1: Define the deep learning model that needs to be inferred in advance. If it is a non-RKNN model, it needs to be trained on the server or PC locally. Convert the trained model into an onnx model, and then further convert it into an rknn model that can be deployed on the RKNN platform. For example, the pytorch model can be converted to an onnx model through the torch.onnx.export interface, or directly load the corresponding model through the RKNN API function (load_caffe, load_tensorflow, load_tflite, load_onnx, load_pytorch), and then use the build interface to build the rknn model. Select the target version to be converted for the corresponding RK core board, and use export_onnx to convert it to the rknn model of the corresponding target board, and finally push it to the edge RKNN device through the adb tool.

[0043] Step 2: Multi-threaded configuration: Install or update RKNPU2 on the development board, push rknn_server and librknnrt.so to the development board, start rknn_server, configure the rknpu environment, encapsulate and initialize the rknn model interface, initialize the rknn model as a list of rknn objects, encapsulate a parent class for controlling and pausing threads, the rknn thread pool class inherits the parent class, and loads each rknn model.

[0044] Step 3: If there is an inference task, calculate the priority of each rknn model task instance and add it to the corresponding priority queue.

[0045] Step 4: The scheduling engine traverses the priority queue in descending order of priority and selects the first task in the task queue with the highest priority.

[0046] Step 5: Check whether there is an idle NPU core. If so, distribute the selected task instance to the NPU core. Otherwise, select the NPU core that runs the lowest priority task and has a relatively smaller load. If the priority is not higher than the task being inferred, block and wait for a while before repeating this step.

[0047] Step 6: Start each thread in the thread pool to infer the captured input source through the rknn model.

[0048] Step 7: If all the tasks of all models are completed, the process ends and the occupied NPU cores and rknn models are released, otherwise return to step 3.

[0049] Please combine Figure 3 The present invention also provides a multi-deep learning model reasoning scheduling system based on the RKNN platform, comprising: The multi-deep learning model establishment module is used to record the n deep learning models in the RKNN platform at time t as M={M1,…,Mi,…,Mn}, 1≤i≤n; record the deep learning model Mi as Mi={Ti, Li,pi}, where Li represents the critical level of Mi, high criticality and low criticality are represented by 1 and 0 respectively; Pi represents the priority of model Mi, which needs to be manually assigned by the user, and the preset priority level is Q priorities from high to low; Ti is the task set of model Mi Ti= { ,.. ,,.. }, n(Ti) is the number of tasks in the ready task set of model Mi, 1≤j≤n(Ti); task Expressed as ={ei,j ,Di,j ,Ti,j ,pi,j}; where ei,j represents the worst case NPU inference task at maximum frequency The time required, Di,j represents the current task The time interval from the release start to the deadline, Ti,j represents the task The cycle of pi,j is represented by the task The priority of k NPU cores is recorded as V={ρ1, ρ2, …, ρk}; The task queue dispatcher is used to determine the priority Pi of the inference task of model Mi and assign the task of task set Ti of model Mi to the task of Place it in the task queue que1~queQ of the corresponding priority and wait; The scheduling engine module is used to search for non-empty queues in the task queues que1~queQ in descending order of priority, select the first task instance in the non-empty queue as the transmission task, and send it to the NPU core; Load balancing monitor module, used to calculate NPU core Load balancing , determine the load balancing degree Whether it exceeds the preset threshold; The load balancing module is used to balance the load When the preset threshold is exceeded, load balancing is performed on the NPU cores.

[0050] Preferably, the scheduling engine module is specifically used to: search for non-empty queues in the task queues que1~queQ in descending order of priority, and select the first task instance in the non-empty queue as the transmission task; find out whether there is an idle NPU core, and if so, distribute the selected task instance to the NPU core; otherwise, distribute the selected task instance to the NPU core running the lowest priority task and with a relatively smaller load.

[0051] Preferably, the step of load balancing the NPU core by the load balancing module specifically includes one or more of the following methods: adjusting the scheduling strategy, suspending queue tasks, dynamically adjusting the task execution mode, dynamically adjusting the hardware resource utilization, and dynamically adjusting the task accuracy requirements.

[0052] Preferably, the load balancing monitor module is configured by the formula , computing NPU core Load balancing ;in, Is the core The theoretical load, Is the core The actual load, , Core workload, is the number of cores of the NPU.

[0053] The present invention also provides a multi-deep learning model reasoning scheduling system based on the RKNN platform, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the multi-deep learning model reasoning scheduling method based on the RKNN platform as described in any one of the above items are implemented.

[0054] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the multi-deep learning model reasoning scheduling method based on the RKNN platform as described in any of the above items.

[0055] The present invention improves load balancing in the multi-task reasoning process, improves the system's continuous operation capability and energy efficiency resource utilization, avoids resource contention between tasks, reduces reasoning delays, and improves the system's response speed.

[0056] Figure 4 FIG. 1 is a schematic diagram of the hardware structure for running a knowledge tracking method based on a two-state joint interaction mechanism provided by an embodiment of the present invention. Figure 4 As shown, the embodiment / computer 6 includes: a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60, such as a program for executing a knowledge tracking method based on a two-state joint interaction mechanism. When the processor 60 executes the computer program 62, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 60 executes the computer program 62, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0057] Exemplarily, the computer program 62 may be divided into one or more modules / units, which are stored in the memory 61 and executed by the processor 60 to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program 62 in the computer 6.

[0058] The computer 6 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The computer 6 may include, but is not limited to, a processor 60 and a memory 61. Those skilled in the art will appreciate that Figure 4 It is only an example of computer 6 and does not constitute a limitation of computer 6. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer 6 may also include input and output devices, network access devices, buses, etc.

[0059] The processor 60 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0060] The memory 61 may be an internal storage unit of the computer 6, such as a hard disk or memory of the computer 6. The memory 61 may also be an external storage device of the terminal device / computer 6, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device. Further, the memory 61 may also include both an internal storage unit of the computer 6 and an external storage device. The memory 61 is used to store the computer program and other programs and data required by the terminal device. The memory 61 may also be used to temporarily store data that has been output or is to be output.

[0061] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0062] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0063] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0064] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0065] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0066] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0067] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0068] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A multi-deep learning model reasoning scheduling method based on the RKNN platform, characterized in that: Includes steps: S1, establish multiple deep learning models, including: record the n deep learning models in the RKNN platform at time t as M={M1,…,Mi,…,Mn}, 1≤i≤n; record the deep learning model Mi as Mi={Ti, Li,pi}, where Li represents the critical level of Mi, high criticality and low criticality are represented by 1 and 0 respectively; Pi represents the priority of model Mi, which needs to be manually assigned by the user, and the preset priority level is Q priorities from high to low; Ti is the task set of model Mi Ti= { ,.. ,,.. }, n(Ti) is the number of tasks in the ready task set of model Mi, 1≤j≤n(Ti); task Expressed as ={ei,j ,Di,j ,Ti,j ,pi,j}; where ei,j represents the worst case NPU inference task at maximum frequency The time required, Di,j represents the current task The time interval from the release start to the deadline, Ti,j represents the task The cycle of pi,j is represented by the task The priority of k NPU cores is recorded as V={ρ1, ρ2, …, ρk}; S2, determine the priority Pi of the inference task of model Mi, and assign the task of task set Ti of model Mi to Place it in the task queue que1~queQ of the corresponding priority and wait; S3, the scheduling engine searches for a non-empty queue in the task queues que1~queQ in descending order of priority, selects the first task instance in the non-empty queue as the transmission task, and sends it to the NPU core; S4, computing NPU core Load balancing , determine the load balancing degree Whether it exceeds the preset threshold; S5, at the load balancing level When the preset threshold is exceeded, load balancing is performed on the NPU cores.

2. The multi-deep learning model reasoning scheduling method based on the RKNN platform according to claim 1 is characterized in that: The step S3 comprises: S31, the scheduling engine searches for a non-empty queue in the task queues que1 to queQ in descending order of priority, and selects the first task instance in the non-empty queue as the transmission task; S32, find out whether there is an idle NPU core, if so, distribute the selected task instance to the NPU core, otherwise distribute the selected task instance to the NPU core running the lowest priority task and with a relatively smaller load.

3. The multi-deep learning model reasoning scheduling method based on the RKNN platform according to claim 1 is characterized in that: The step of load balancing the NPU core in step S5 specifically includes one or more of the following methods: adjusting the scheduling strategy, suspending queue tasks, dynamically adjusting the task execution method, dynamically adjusting the hardware resource utilization, and dynamically adjusting the task accuracy requirements.

4. The multi-deep learning model reasoning scheduling method based on the RKNN platform according to claim 1 is characterized in that: In step S4, the formula , computing NPU core Load balancing ;in, Is the core The theoretical load, Is the core The actual load, , Core workload, is the number of cores of the NPU.

5. A multi-deep learning model reasoning scheduling system based on the RKNN platform, characterized in that: include: The multi-deep learning model establishment module is used to record the n deep learning models in the RKNN platform at time t as M={M1,…,Mi,…,Mn}, 1≤i≤n; record the deep learning model Mi as Mi={Ti, Li,pi}, where Li represents the critical level of Mi, high criticality and low criticality are represented by 1 and 0 respectively; Pi represents the priority of model Mi, which needs to be manually assigned by the user, and the preset priority level is Q priorities from high to low; Ti is the task set of model Mi Ti= { ,.. ,,.. }, n(Ti) is the number of tasks in the ready task set of model Mi, 1≤j≤n(Ti); task Expressed as ={ei,j ,Di,j ,Ti,j ,pi,j}; where ei,j represents the worst case NPU inference task at maximum frequency The time required, Di,j represents the current task The time interval from the release start to the deadline, Ti,j represents the task The cycle of pi,j is represented by the task The priority of k NPU cores is recorded as V={ρ1, ρ2, …, ρk}; The task queue allocation module is used to determine the priority Pi of the inference task of model Mi and assign the task of task set Ti of model Mi to the task of Place it in the task queue que1~queQ of the corresponding priority and wait; The scheduling engine module is used to search for non-empty queues in the task queues que1~queQ in descending order of priority, select the first task instance in the non-empty queue as the transmission task, and send it to the NPU core; Load balancing monitor module, used to calculate NPU core Load balancing , determine the load balancing degree Whether it exceeds the preset threshold; The load balancing module is used to balance the load When the preset threshold is exceeded, load balancing is performed on the NPU cores.

6. The multi-deep learning model reasoning scheduling system based on the RKNN platform according to claim 5 is characterized in that: The scheduling engine module is specifically used to: search for non-empty queues in the task queues que1~queQ in descending order of priority, and select the first task instance in the non-empty queue as the transmission task; find out whether there is an idle NPU core, and if so, distribute the selected task instance to the NPU core, otherwise distribute the selected task instance to the NPU core running the lowest priority task and with a relatively smaller load.

7. The multi-deep learning model reasoning scheduling system based on the RKNN platform according to claim 5 is characterized in that: The steps of the load balancing module to load balance the NPU core specifically include one or more of the following methods: adjusting the scheduling strategy, suspending queue tasks, dynamically adjusting the task execution mode, dynamically adjusting the hardware resource utilization, and dynamically adjusting the accuracy requirements of the task.

8. The multi-deep learning model reasoning scheduling system based on the RKNN platform according to claim 5 is characterized in that: The load balancing monitor module is calculated by the formula , computing NPU core Load balancing ;in, Is the core The theoretical load, Is the core The actual load, , Core workload, is the number of cores of the NPU.

9. A multi-deep learning model reasoning scheduling system based on the RKNN platform, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the multi-deep learning model reasoning scheduling method based on the RKNN platform are implemented as described in any one of claims 1 to 4.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the multi-deep learning model reasoning scheduling method based on the RKNN platform are implemented as described in any one of claims 1 to 4.

Images