NPU time division multiplexing real-time reasoning system and scheduling method

Abstract

The invention discloses an NPU time division multiplexing real-time reasoning system and a scheduling method, belongs to the technical field of edge intelligent computing, and solves the problem that the reasoning real-time performance of multiple reasoning tasks is difficult to guarantee by traditional edge intelligent computing service equipment and a reasoning scheduling method in the prior art. According to the method, an intelligent reasoning model is converted through a preparation state pre-divider in a preprocessing stage, namely, the intelligent reasoning model is divided into blocks with different granularities in combination with unit granularities, and operation attribute information of the model and the blocks of the model is obtained; in the execution stage, a task request called by a remote procedure is received through a runtime planner, then the optimal scheduling granularity is determined through nonlinear optimization, a corresponding task scheduling sequence is generated according to a low-segmentation NPU real-time scheduling algorithm, and model blocks needed by tasks are obtained through a runtime executor according to a job sequence and executed. According to the method, the real-time performance of time division multiplexing of NPU resources of the edge intelligent reasoning calculation task in a multi-task scene is effectively improved.

Description

Technical Field

[0001] The present invention relates to a real-time reasoning system and a scheduling method, and in particular to a real-time reasoning system and a scheduling method for NPU time-sharing multiplexing, belonging to the field of edge intelligent computing technology. Background Art

[0002] Edge intelligence is an emerging computing paradigm that deploys artificial intelligence technology directly to the source of data or its vicinity, providing fast-response intelligent computing services for many terminal devices and significantly reducing the latency of intelligent computing. However, edge computing devices are usually used in resource-constrained environments. Therefore, neural processing units (NPUs) have gradually become a popular choice in the field of edge intelligence due to their efficient intelligent processing capabilities and low energy consumption. However, edge intelligent computing service devices based on NPUs often need to process multiple time-sensitive task requests at the same time. For example, in application scenarios such as unmanned equipment and autonomous driving, the device needs to concurrently process multiple real-time model reasoning tasks such as target recognition, automatic obstacle avoidance, and path planning. Due to resource constraints in edge computing scenarios, the concurrent execution of the above real-time reasoning tasks usually needs to be completed on a single NPU, which poses a challenge to system resource management and scheduling capabilities.

[0003] The existing mainstream solutions solve the problem of real-time concurrent execution of model reasoning through technologies such as hardware virtualization and time-sharing multiplexing, but the application of the existing mainstream solutions on edge intelligent computing service devices is limited. This is because ① the hardware virtualization solution requires the support of specific hardware design, but the low-power reasoning NPU commonly used in edge intelligent computing service devices does not have hardware virtualization capabilities; ② time-sharing multiplexing can be used as a general resource scheduling technology to provide NPU with multi-task concurrency capabilities, however, the current NPU design can only execute model reasoning as a whole, and does not support switching to other tasks during the execution process. This results in the reasoning task granularity of the existing NPU time-sharing multiplexing solution being too coarse, and the real-time nature of the reasoning task cannot be guaranteed.

[0004] Prior art 1: Hardware virtualization technology divides the physical graphics processing unit (GPU) or NPU resources into multiple virtual devices, allowing multiple users or applications to share the same physical GPU resources and independently execute their own models. This technology not only significantly improves the utilization of GPU resources, but also helps to reduce costs and enhance system reliability. Mainstream virtualization technology implementations usually rely on specialized support at the hardware level. For example, NVIDIA's vGPU (virtual graphics processing unit) technology allows multiple virtual machines to bypass the host machine and directly connect to the physical GPU, thereby realizing the sharing of the physical GPU among multiple virtual machines. However, the above technology relies on the support of technologies such as virtualization (SR-IOV, Single Root I / O Virtualization) and access control services (ACS, Access Control Services), and is only available on some server-level or professional-level graphics cards of NVIDIA; Huawei's Atlas series NPU provides higher performance and lower latency by directly implementing the scheduling and management of virtual machines at the hardware level, and implements resource isolation by dividing physical NPU resources into multiple independent AI Cores and AI CPUs, but it relies on specific hardware design and is only applicable to specific NPU models. Therefore, the disadvantage of hardware virtualization technology is that it is difficult to apply to edge intelligent computing service scenarios. The existing hardware virtualization technology is constrained by the hardware design of NPU and GPU, and is only applicable to high-end training-pushing integrated processors. Low-power NPUs suitable for edge intelligent computing services are difficult to share hardware resources through hardware virtualization solutions.

[0005] Prior art 2: Time-sharing multiplexing technology divides the processing time of a GPU or NPU into discrete time slices. Each time slice allocates a portion of the processor's computing and memory resources to different applications. Specifically, the time-sharing multiplexing method usually places the arriving tasks in a queue managed by the scheduler and executes them in sequence according to the scheduling policy selected by the user. When the time slice allocated to a task ends or the task is completed, the scheduler will perform a context switch to load the state of the next task. In this way, time-sharing multiplexing technology can execute multiple tasks concurrently on a single processor, thereby improving resource utilization and scheduling flexibility. This technology can be applied to various types of processors including low-power NPUs, but it can usually only be scheduled after the model processing is completed, and it is difficult to switch tasks during the model processing process, resulting in the scheduling granularity of the existing time-sharing multiplexing NPU solution being too coarse, making it difficult to ensure real-time execution; at the same time, frequent context switching between different workloads will bring performance overhead and increase the delay in task execution, thereby reducing the overall efficiency of the system. When a low-power NPU performs reasoning, it can usually only schedule the execution of the next reasoning task after the model processing corresponding to a task is completed, resulting in tasks with longer running times that may cause other tasks in the queue to wait for a long time and miss the deadline. Therefore, the time-sharing multiplexing NPU solution is difficult to meet the real-time requirements of edge intelligence scenarios.

[0006] Prior art three, a patent document with a publication (announcement) number of CN112348172B discloses a GPU resource scheduling method, device, electronic device and storage medium, which are mainly used for the deployment and execution of large-scale neural network models. Its implementation method includes: first determining the service type of the target large-scale neural network model, and then obtaining the corresponding target preset model segmentation method according to the service type, dividing the large-scale neural network model into multiple sub-models, and finally loading the sub-model onto the corresponding GPU to complete the calculation; the model segmentation method can ensure that: each sub-model can independently complete its corresponding task part, and the output of the above task part can be directly combined into the final result. Through this method of automated segmentation and processing of neural networks, effective resource scheduling is achieved, avoiding the complex process of manually segmenting the model and aggregating the model block processing results; however, it neither considers the need to execute multiple models on the same device in the edge device, nor guarantees the real-time performance of the calculation, so it is not suitable for the application scenario of real-time reasoning of edge devices.

[0007] Prior art four, a patent document with publication (announcement) number CN114780240A discloses a workflow scheduling method, device and storage medium based on GPU time-sharing multiplexing. By dividing all GPU video memory resources in the cluster into multiple time segments and allocating and marking them on demand, efficient utilization of resources is ensured, and the problem of performance degradation caused by the inability to dynamically allocate GPU resources in the prior art is solved. Its innovation lies in: real-time monitoring of the utilization rate of GPU resources by workflows and dynamic allocation or recycling of resources, thereby improving the utilization efficiency of GPU resources; setting time-segment-based markings for GPU resources to avoid workflow failures caused by resource conflicts; however, the above method focuses on optimizing the dynamic allocation of video memory resources in a GPU cluster rather than the computing resource sharing of a single NPU. On the other hand, it does not consider the real-time scheduling of tasks, so it is not suitable for real-time reasoning application scenarios.

[0008] Prior art 5, the patent document with the publication (announcement) number CN114004730A discloses a multi-model parallel reasoning method for deep neural networks based on a graphics processor. The method constructs multiple reasoning engines according to the network model to be inferred, and binds the reasoning engine to the corresponding input address and the GPU stream object, then creates a reasoning group and binds it to the CPU thread object, and finally adds the reasoning engine to the corresponding reasoning group and constructs a reasoning manager to initiate multi-threaded operations. The above method realizes data sharing in the multi-model reasoning process, so that the parallel processing capability of the GPU can be more efficiently exerted, and the thread synchronization operation in the multi-model reasoning calculation can be effectively performed; but the above method is essentially a GPU parallel reasoning program flow. Although the efficiency of data sharing and thread synchronization is improved by introducing a multi-threaded reasoning grouping method, it is difficult to construct a real-time scheduling strategy for parallel reasoning threads. At the same time, the implementation details of the above method are highly bound to the GPU structure and processing flow, and are not suitable for NPU.

[0009] Prior art six, the patent document with the publication (announcement) number CN112348172B discloses a method for collaborative reasoning of deep neural networks based on an end-edge-cloud architecture. The method first collects the transmission bandwidth, data volume and reasoning energy consumption of the edge and cloud sides to the end side, and then divides the computing tasks in the model reasoning process into three parts based on the network environment, resource quotas and usage of the three parties of the end-edge-cloud through a convex optimization or reinforcement learning algorithm with latency / energy consumption as the target on the basis of the above information, and then sends the divided computing tasks to the corresponding end, edge and cloud to be executed in sequence and return the reasoning results. The above method can accelerate the reasoning speed of the end side through the cooperation of the end-edge-cloud, improve the real-time reasoning in business scenarios, and reduce the energy consumption of the resource end; however, the above method does not consider the situation that the resource-constrained edge device cannot respond to the task in time when processing multiple computing requests and generating resource competition, thus affecting the real-time performance. Therefore, it is impossible to maintain the real-time performance of the reasoning task under resource-constrained conditions.

[0010] In summary, most of the existing related methods are oriented towards GPU clusters or processor internal caches, and no targeted methods are proposed to ensure the real-time performance of multi-model reasoning tasks, so it is difficult to meet the requirements of real-time edge intelligent computing. The current mainstream solutions to the NPU multi-model execution problem, namely sharing NPU hardware resources and time-sharing NPU, have the following problems: (1) Sharing NPU hardware resources requires resource isolation at the hardware level through NPU virtualization technology, and allocating them to multiple reasoning tasks for parallel use, which can achieve the real-time performance of multiple tasks to a certain extent, but the cost and power consumption are high, so the solution of sharing NPU hardware resources cannot be used; (2) Although the existing NPU time-sharing multiplexing technology can be applied to low-power reasoning NPUs commonly used in edge intelligent scenarios, the existing technology schedules a single reasoning task as a whole. When there are multiple tasks, it is necessary to wait until the previous task is completed before executing the next reasoning task. At the same time, due to the coarse model granularity, other queued tasks may wait for a long time and miss the deadline, making it difficult to ensure the real-time performance of multiple reasoning tasks.

[0011] Therefore, a real-time inference system and scheduling method for NPU time-sharing multiplexing for real-time edge intelligent computing scenarios is needed. Summary of the invention

[0012] A brief overview of the present invention is provided below in order to provide a basic understanding of certain aspects of the present invention. It should be understood that this overview is not an exhaustive overview of the present invention. It is not intended to identify key or important parts of the present invention, nor is it intended to limit the scope of the present invention. Its purpose is merely to present certain concepts in a simplified form as a prelude to a more detailed description discussed later.

[0013] In view of this, in order to solve the problem that traditional edge intelligent computing service equipment and inference scheduling methods in the prior art are difficult to ensure the real-time reasoning of multiple inference tasks, the present invention provides a real-time inference system and scheduling method with NPU time-sharing multiplexing.

[0014] Technical solution 1 is as follows: a real-time inference system with NPU time-sharing multiplexing, including a heterogeneous computing system and a multi-model serial real-time inference controller;

[0015] The heterogeneous computing system includes an NPU module, a CPU module and a DDR memory, the NPU module is connected to the CPU module via a PCIe bus, and the CPU module is connected to the DDR memory;

[0016] The CPU module is used to control the NPU module to complete the intelligent computing process;

[0017] The DDR memory is used to cache model data and service data to be processed;

[0018] The multi-model serial real-time inference controller includes a ready-state pre-segmenter, a runtime planner, and a runtime executor connected in sequence. The ready-state pre-segmenter provides model blocks for the runtime executor, the runtime planner divides the inference request into task blocks, and the runtime executor calls the corresponding model blocks according to the task blocks until the entire model inference is completed.

[0019] Furthermore, the NPU module includes an on-chip cache, a matrix calculation unit, a vector calculation unit and a data handling unit. The data handling unit is connected to the on-chip cache to move data from the DDR into the on-chip cache. The cache is then connected to the matrix calculation unit and the vector calculation unit.

[0020] Furthermore, the prepared state pre-segmenter includes a model evaluation component and a model partitioning component, the model evaluation component is connected to the model partitioning component, the prepared state pre-segmenter is used to split the model in the preprocessing stage and place it into the model block warehouse, the runtime planner includes an RPC server, an inference task pool, an inference task scheduler and an inference queue connected in sequence, the runtime planner is used to receive the model inference calculation request task submitted by the terminal in the execution stage, and decompose the task and fill it into the inference queue containing the task blocks, the runtime executor is used to take out the tasks from the inference queue in sequence in the execution stage, and call the corresponding model block execution, the model partitioning component is connected to the inference task scheduler, and the inference task scheduler (and the inference queue therein) is connected to the runtime executor.

[0021] Technical solution 2 is as follows: A real-time reasoning scheduling method for NPU time-sharing multiplexing, used in a real-time reasoning system for NPU time-sharing multiplexing described in technical solution 1, comprising the following steps:

[0022] S1. Use the preparatory pre-segmenter to convert and segment the intelligent model, that is, convert the intelligent model into a format file executable by the NPU model, test and evaluate the execution time of each layer of the model, and split the intelligent model into model blocks of different granularities based on the unit granularity;

[0023] S2. The total execution time and task cycle of each intelligent model are discretized by the runtime planner, that is, the total execution time and task cycle of each intelligent model are discretized into integers to reduce the computational overhead of the scheduling algorithm. Before all tasks start, the scheduling granularity is determined by nonlinear optimization through the unit granularity and the user-given load threshold, and the scheduling granularity is used to solve the execution time and task cycle discretization values ​​of the intelligent model blocks to obtain the discretized execution time and task cycle used by the scheduling algorithm;

[0024] S3. Based on the discrete execution time and task cycle, a low-cut NPU real-time scheduling algorithm with a multiplexing mechanism is used to obtain the task scheduling sequence, and the runtime executor is used to execute according to the task scheduling sequence.

[0025] Furthermore, the S1 specifically includes the following steps:

[0026] S11. Obtain the intelligent model required for the task from the model warehouse;

[0027] S12. Use the model evaluation component to obtain the weight files of all intelligent models and convert the format of the intelligent model into an NPU operator graph file;

[0028] S13. Statistically measure the total execution time and each layer execution time of the operator graph files of all intelligent models based on the NPU runtime;

[0029] S14. According to the execution time of each layer of the intelligent model, unit granularity and other conditions, the model partitioning component is used to recursively split each intelligent model to obtain model blocks of different granularities;

[0030] In S12, the intelligent model is automatically quantized and integrated during the conversion process, and the intelligent model file executable by the NPU module is compiled and generated;

[0031] In S13, a time-consuming list τ is established, and the total execution time of the intelligent model of the i-th task is τ i , the execution time of the jth layer of the intelligent model is The task period of the corresponding type of task of the intelligent model is taken as the task period t of the intelligent model. i ;

[0032] In S14, the intelligent model or model block is divided into two parts with equal execution time granularity each time, with each layer as the unit, until further division will cause the execution time of the model block to be less than the unit granularity g.0 Or until the model block contains only one layer, all model blocks of different granularities generated by the division are stored in the model block warehouse.

[0033] Furthermore, in S2, the user task attributes are obtained, and the task execution time and task cycle are discretized. The specific method is: for task i, its execution time is τ i , the task cycle is t i , the execution time after discretization using the scheduling granularity g is ceil(τ i / g), the discretized task period is floor(t i / g), where ceil() and floor() functions represent rounding up and rounding down respectively. The task load after task i is discretized is U i (g), U i (g) = ceil(τ i / g) / floor(t i / g). For a system with I tasks, the total load is the sum of the loads of all tasks, that is,

[0034] Where I is the total number of tasks;

[0035] In order to reduce the scheduling overhead, that is, reduce the scheduling frequency and the increase in computation caused by model splitting, at least ensure that the total load does not exceed the preset threshold U max Under the premise of finding a larger scheduling granularity g. This problem is modeled as a nonlinear multi-objective constrained optimization problem, where the independent variable is the scheduling granularity g, and the optimization objectives are: 1. Minimize the number of schedulings required within the observation time, where the observation time is the least common multiple of the scheduling period of each task, and 2. Minimize the gaps generated by all model block scheduling on continuous time slices; the constraints are: 1. The total period of the discretized task does not exceed the preset load threshold U max ,2. The scheduling granularity g is in [g 0 ,τ min ] interval, where g 0 is the unit particle size, τ min is the minimum value of the task execution time among I tasks (where I is the total number of tasks). This optimization problem uses an optimization method to search for the optimal solution set and select the optimal scheduling granularity g from it. This scheduling granularity is used to discretize the execution time and task cycle of all tasks.

[0036] Furthermore, in S3, the low-cut NPU real-time scheduling algorithm uses the branch and bound method to generate a scheduling plan within the observation time. The algorithm starts from the initial state and repeatedly calculates the job that should be scheduled for the task state in the current time slice, thereby recursively pushing the task state to the next time slice. When preemption and other situations are detected during the generation process, the two options of executing the original job and executing the new job are recursively calculated, and the losses caused by the two options are calculated: 1. The loss of failing to execute the highest priority task, 2. The additional loss caused by model splitting. When the number of recursive generation times of the scheduling plan reaches the number of observation time slices given by the user, the algorithm terminates the generation, outputs the scheduling process corresponding to a series of choices with the smallest cumulative loss, and the block granularity of the corresponding intelligent model as the final result, and updates the inference queue. The above tasks include but are not limited to actual reasoning tasks such as target recognition and edge detection required by users, and the job is the specific execution status of the task at a specific point in time;

[0037] In the process of calculating the loss, the loss function is defined as Loss = cω + K. Where c is the number of task preemptions from the initial state to the current process; ω is the weight parameter defined by the user; K is the degree of violation of the principle of prioritizing the most urgent tasks in the process, defined as Among them, n is the number of time slices from the initial state to the current state, N is the total number of time slices, and k is the total number of time slices. n Indicates whether the scheduling of the nth time slice violates the principle of prioritizing the most urgent tasks. When the job executed in the nth time slice belongs to the most urgent task at this time, k n =0, otherwise, k n =1;

[0038] The scheduling branches are pruned according to the accumulated loss. The process is as follows: if multiple branches reach the same state at a certain moment, that is, the remaining execution time of tasks in different branches, the execution time of the most recent job in all branches, and the tasks to which the most recent job belongs in all branches are exactly the same, then only the branch with the lowest loss function value is retained among the branches with the same state. If there are multiple scheduling branches with the same state and the loss function values ​​are the same as the lowest, any one of them is retained.

[0039] The low-cut NPU real-time scheduling algorithm is compatible with non-periodic tasks by introducing a time slice reuse mechanism. After assigning tasks to each time slice, the low-cut NPU real-time scheduling algorithm generates the execution order of model blocks through two steps of merging and reuse, as follows:

[0040] 1. Merge the continuous time slices assigned to the same task, find the model block that is not longer than and closest to the length of the continuous time slice in the model block warehouse for allocation. If the idle time slot of the time slice is too large after allocation, find the model block that matches the idle time slot again for allocation;

[0041] 2. Without changing the execution order of model blocks, use the time slice reuse mechanism to advance the execution of model blocks to the larger value of the end time of the previous model block and the arrival time of the current task. Then merge the idle time slots generated by each time slice to generate continuous gaps, and use the continuous gaps to insert appropriate model blocks of sudden non-periodic tasks or other transactions into the inference queue for execution, thereby reducing the impact of sudden tasks on the real-time execution of periodic tasks.

[0042] The beneficial effects of the present invention are as follows:

[0043] 1. The present invention realizes the time-sharing reuse of NPU resources on edge intelligent computing service devices through fine-grained model segmentation and improved scheduling methods, thereby effectively improving the real-time performance of NPU reasoning in task scenarios with large differences in different task cycles;

[0044] 2. The present invention is a NPU time-sharing multiplexing real-time reasoning system, which is composed of hardware modules such as CPU, NPU, DDR memory, etc. It is also adapted to the scheduling method based on fine-grained model partitioning, and provides an acceleration mechanism that uses the NPU on-chip cache to pre-buffer the data to be processed for the next model partition reasoning. Through the NPU on-chip cache structure management, the data required for the next reasoning is prepared and buffered in advance, thereby achieving the effect of accelerated execution;

[0045] 3. The present invention proposes a real-time reasoning scheduling method for NPU time-sharing multiplexing. In the preparation stage, the model is divided into a series of blocks of different granularities through the preparation state pre-segmenter; before the task is executed, the scheduling granularity is determined according to the task attributes; when the task is executed, the corresponding reasoning job sequence is generated according to the real-time scheduling algorithm; finally, the model block of the corresponding granularity is called and executed according to the job sequence;

[0046] 4. The present invention proposes a scheme for recursively partitioning the model, which is used to divide the model into model blocks of different granularities and save them separately. On the one hand, the granularity of the model is refined; on the other hand, the model blocks of different granularities ensure that the subsequent scheduling algorithm can adopt a dynamic strategy. The recursive partitioning process of the model does not depend on the task attributes, and can dynamically adapt to the task scenario according to the changes in the task attributes.

[0047] 5. The present invention can select the optimal scheduling granularity, i.e., the scheduling time slice, according to the task execution time and cycle. The optimal granularity can be selected according to the discretization of the specific execution time, task cycle and total load of the model of all tasks;

[0048] 6. The present invention proposes a low-slicing NPU inference real-time scheduling algorithm, which achieves a trade-off between scheduling higher priority tasks as much as possible and avoiding splitting models as much as possible through the branch and bound method, thereby taking into account both the priority scheduling of urgent tasks and improving the execution efficiency of the algorithm on the NPU; at the same time, the time slice multiplexing mechanism is used to concentrate the free time slots generated by the model block execution process by executing the model blocks as early as possible, and using continuous gaps to insert the model blocks of non-periodic tasks into the inference queue for execution, thereby reducing the impact on the real-time performance of periodic tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0050] Figure 1 This is a system architecture diagram of a real-time reasoning system with NPU time-sharing multiplexing;

[0051] Figure 2 A system architecture diagram of an embodiment of a real-time reasoning system using NPU time-sharing multiplexing;

[0052] Figure 3 It is a schematic diagram of the structure of a heterogeneous computing system;

[0053] Figure 4 is a schematic diagram of a multi-model serial real-time inference controller;

[0054] Figure 5 A flowchart of a real-time reasoning scheduling method for NPU time-sharing multiplexing;

[0055] Figure 6 A flow chart of an embodiment of an algorithm for model segmentation and scheduling granularity acquisition;

[0056] Figure 7 The flowchart of an embodiment of a low-slicing NPU real-time scheduling algorithm.

[0057] Description of the drawings: 1. NPU module; 2. NPU module; 3. NPU module; 4. Ready state pre-segmenter; 5. Runtime planner; 6. Runtime executor. DETAILED DESCRIPTION

[0058] In order to make the technical solutions and advantages of the embodiments of the present invention more clearly understood, the exemplary embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than an exhaustive list of all the embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0059] Example 1: Reference Figure 1-Figure 3 Detailed description of this embodiment, a real-time inference system with NPU time-sharing multiplexing, a heterogeneous computing system as the hardware part and a multi-model serial real-time inference controller as the software part;

[0060] The heterogeneous computing system includes an NPU module 1, a CPU module 2 and a DDR memory 3, wherein the NPU module 1 is connected to the CPU module 2 via a PCIe bus, and the CPU module 2 is connected to the DDR memory 3;

[0061] The CPU module 2 is used to control the NPU module 1 to complete the intelligent computing process;

[0062] The DDR memory 3 is used to cache model data and service data to be processed;

[0063] The multi-model serial real-time inference controller includes a ready state pre-segmenter 4, a runtime planner 5, and a runtime executor 6 which are connected in sequence. The ready state pre-segmenter 4 provides model blocks for the runtime executor 6, the runtime planner 5 divides the inference request into task blocks, and the runtime executor 6 calls the corresponding model blocks according to the task blocks until the entire model inference is completed.

[0064] Furthermore, the NPU module 1 includes an on-chip cache, a matrix calculation unit, a vector calculation unit and a data handling unit. The data handling unit is connected to the on-chip cache to move data from the DDR into the on-chip cache. The cache is then connected to the matrix calculation unit and the vector calculation unit.

[0065] Specifically, refer to Figure 2 and Figure 3 The CPU module is used for scheduling algorithms and other CPU algorithms. The NPU module is used as a coprocessor for DNN model reasoning acceleration. The DDR memory is used to cache reasoning process data. The data handling unit handles data between the DDR memory and the on-chip cache.

[0066] Further, the prepared state pre-segmenter 4 includes a model evaluation component and a model partitioning component, the model evaluation component is connected to the model partitioning component, the prepared state pre-segmenter 4 is used to split the model in the preprocessing stage and place it into the model block warehouse, the runtime planner 5 includes an RPC server, an inference task pool, an inference task scheduler and an inference queue connected in sequence, the runtime planner 5 is used to receive the model inference calculation request task submitted by the terminal in the execution stage, and decompose the task and fill it into the inference queue containing the task blocks, the runtime executor 6 is used to take out the tasks from the inference queue in sequence in the execution stage, and call the corresponding model block execution, the model partitioning component is connected to the inference task scheduler, and the inference queue is connected to the runtime executor 6;

[0067] Specifically, refer to Figure 4 The pre-segmenter in the prepared state is used to segment the model in the preprocessing stage and place it into the model block warehouse, which includes a model evaluation component and a model partitioning component. The model evaluation component is used to convert the model format, obtain the model and the running time of each layer therein, and calculate the model segmentation granularity. The model partitioning component is used to divide the model into model blocks according to the segmentation granularity;

[0068] The runtime planner is used to receive the model inference calculation request task submitted by the terminal during the execution phase, and decompose the task according to the scheduling algorithm and fill it into the inference queue containing the task blocks. When the runtime planner is running, the RPC server is used to receive the task request initiated by the intelligent terminal through the RPC, and save the task in the inference task pool. The inference task scheduler is used to generate corresponding jobs according to the inference tasks in the inference task pool, adjust the job order and fill it into the inference queue. The inference queue is used to save the order of scheduling the NPU to execute the block inference job.

[0069] The runtime executor is used to take out the inference jobs from the inference queue in sequence, and then obtain the corresponding model blocks from the model block warehouse for calculation, and output the inference result after completing the calculation of the last model block of the task.

[0070] The runtime executor will use the NPU on-chip cache to pre-buffer the data to be processed for the next model block inference; when the runtime executor is executed, it will take the job from the head of the inference queue and obtain the corresponding model block from the model block warehouse for serial calculation until all the inference tasks in the inference queue are completed. 1. If the calculation result is an intermediate result of a certain inference task, when the on-chip cache capacity is sufficient, it is preferably temporarily stored in the on-chip cache instead of being moved to the DDR memory for use by subsequent tasks; when the on-chip cache is full, the on-chip cache is adjusted according to the queue order of the jobs in the inference queue, and the buffered data required by the jobs at the end of the queue are moved to the DDR memory. 2. If the calculation result is the final result, it is output through the corresponding interface.

[0071] Example 2: Reference Figure 4-Figure 6 The present embodiment is described in detail. A real-time reasoning scheduling method for NPU time-sharing multiplexing is used in a real-time reasoning system for NPU time-sharing multiplexing described in Embodiment 1. Specifically, the method includes the following steps:

[0072] S1. Use the pre-segmenter to convert and segment the intelligent model. First, convert the intelligent model into a format file that can be executed by the NPU. Then test and evaluate the execution time of each layer of the model. Finally, combine the unit granularity to split the model into model blocks of different granularities. If the calculation time of a model block of a certain granularity is equal on the CPU module and the NPU module, then define this calculation time as the unit granularity, denoted as g 0This granularity can be obtained based on testing of the target heterogeneous computing system.

[0073] S2. The total execution time and task cycle of each intelligent model are discretized by the runtime planner, that is, the total execution time and task cycle of each intelligent model are discretized into integers to reduce the computational overhead of the scheduling algorithm, specifically including: 1. By discretizing the execution time of the intelligent model, the scheduling time slice (generally the greatest common divisor of the execution time of I intelligent models) is not too small; 2. By discretizing the task cycle, the observation time of the scheduling algorithm (generally the least common multiple of I inferred task cycles) is not too large (where I is the total number of tasks);

[0074] Before all tasks start, first pass the unit granularity g 0 and user-given load threshold U max The scheduling granularity g, i.e. the length of the scheduling time slice, is determined through nonlinear optimization. The scheduling granularity is then used to discretize the execution time and task cycle of the intelligent model blocks, thereby obtaining the discretized execution time and task cycle used by the scheduling algorithm.

[0075] S3. Based on the discrete execution time and task cycle, a low-cut NPU real-time scheduling algorithm with a multiplexing mechanism is used to obtain the task scheduling sequence, and the runtime executor is used to execute according to the task scheduling sequence.

[0076] Furthermore, the S1 specifically includes the following steps:

[0077] S11. Obtain the intelligent model required for the task from the model warehouse;

[0078] S12. Use the model evaluation component to obtain the weight files of all intelligent models and convert the format of the intelligent model into an NPU operator graph file;

[0079] S13. Statistically measure the total execution time and each layer execution time of the operator graph files of all intelligent models based on the NPU runtime;

[0080] S14. According to the execution time of each layer of the intelligent model, unit granularity and other conditions, the model partitioning component is used to recursively split each intelligent model to obtain model blocks of different granularities;

[0081] In S12, the intelligent model is automatically quantized and integrated during the conversion process, and the intelligent model file executable by the NPU module is compiled and generated;

[0082] In S13, a time-consuming list τ is established, and the total execution time of the intelligent model of the i-th task (or the corresponding task execution time) is τ i , the execution time of the jth layer of the intelligent model is τ ji , the task period of the corresponding type of task of the intelligent model is taken as the task period t of the intelligent model i ;

[0083] In S14, the intelligent model or model block is divided into two parts with equal execution time granularity each time, until further division will cause the execution time of the model block to be less than the unit granularity g. 0 Or until the model block contains only one layer, all the model blocks of different granularities generated by the division are stored in the model block warehouse;

[0084] Specifically, the unit particle size g 0 It means that only when the execution time of the intelligent model is not less than g 0 In terms of the corresponding computing time, the model blocks can be executed on the NPU module to achieve acceleration compared to executing on the CPU module.

[0085] Furthermore, in S2, the user task attributes are obtained, and the task execution time and task cycle are discretized. The specific method is: for task i, its execution time is τ i , the task cycle is t i , the execution time after discretization using the scheduling granularity g is ceil(τ i / g), the discretized task period is floor(t i / g), where ceil() and floor() functions represent rounding up and rounding down respectively. The task load after task i is discretized is U i (g), U i (g) = ceil(τ i / g) / floor(t i / g). For a system with I tasks, the total load is the sum of the loads of all tasks, that is,

[0086] In order to reduce the scheduling overhead, that is, reduce the scheduling frequency and the increase in computation caused by model splitting, at least ensure that the total load does not exceed the preset threshold U max Under the premise of finding a larger scheduling granularity g. This problem is modeled as a nonlinear multi-objective constrained optimization problem, where the independent variable is the scheduling granularity g; the optimization objectives are: 1. Minimize the number of schedulings required within the observation time, where the observation time is the lowest common multiple of the scheduling period of each task, 2. Minimize the gaps generated by all model block scheduling on continuous time slices; the constraints are: 1. The total period of the discretized task does not exceed the preset load threshold U max ,2. The scheduling granularity is [g 0 ,τ min ] interval, where g 0is the unit particle size, τ min is the minimum execution time of I tasks. This optimization problem uses an optimization method to search for the optimal solution set and select the optimal scheduling granularity g from it. This scheduling granularity is used to discretize the execution time and task cycle of all tasks.

[0087] Furthermore, in S3, the low-slicing NPU real-time scheduling algorithm uses the branch and bound method to generate a scheduling plan within the observation time. The algorithm starts from the initial state and repeatedly calculates the job that should be scheduled for the task state in the current time slice, thereby recursively pushing the task state to the next time slice. When the generation process monitors that task preemption is about to occur, the two options of executing the original job and executing the new job are recursively calculated, and the losses caused by the two options are calculated respectively: 1. The loss of failing to execute the most urgent task, 2. The additional loss caused by model segmentation. When the number of recursive generation times of the scheduling plan reaches the number of observation time slices given by the user, the algorithm terminates the generation, outputs a series of scheduling processes corresponding to the choices with the smallest cumulative loss, and the block granularity of the corresponding intelligent model as the final result, and updates the inference queue. The above tasks include but are not limited to actual reasoning tasks such as target recognition and edge detection required by users, and the job is the specific execution of the task at a specific point in time;

[0088] In the process of calculating the loss, the loss function is defined as Loss = cω + K. Where c is the number of task preemptions from the initial state to the current process; ω is the weight parameter defined by the user; K is the degree of violation of the principle of prioritizing the most urgent tasks in the process, defined as Among them, n is the number of time slices from the initial state to the current state, N is the total number of time slices, and k is the total number of time slices. n Indicates whether the scheduling of the nth time slice violates the principle of prioritizing the most urgent tasks. When the job executed in the nth time slice belongs to the most urgent task at this time, k n =0, otherwise, k n =1;

[0089] Afterwards, the scheduling branches are pruned according to the accumulated losses. The process is as follows: if multiple branches reach the same state at a certain moment, that is, the remaining execution time of tasks in different branches, the execution time of the most recent job in all branches, and the tasks to which the most recent job belongs in all branches are exactly the same, then only the branch with the lowest loss function value is retained among the branches with the same state. If there are multiple scheduling branches with the same state and the loss function values ​​are the same as the lowest, any one of them is retained.

[0090] The low-cut NPU real-time scheduling algorithm is compatible with non-periodic tasks by introducing a time slice reuse mechanism. After assigning tasks to each time slice, the low-cut NPU real-time scheduling algorithm generates the execution order of model blocks through two steps of merging and reuse, as follows:

[0091] 1. Merge consecutive time slices assigned the same task, and then find a model block in the model block warehouse that is not greater than and closest to the time slice length for allocation. If the idle time slot of the allocated time slice is too large, find a model block that matches the idle time slot for allocation again;

[0092] 2. Without changing the execution order of model blocks, use the time slice reuse mechanism to advance the execution of model blocks to the larger value of the end time of the previous model block and the arrival time of the current task. Then merge the idle time slots generated by each time slice to generate continuous gaps, and use the continuous gaps to insert appropriate model blocks of sudden non-periodic tasks or other transactions into the inference queue for execution, thereby reducing the impact of sudden tasks on the real-time execution of periodic tasks.

[0093] refer to Figure 6 In this embodiment, the branch and bound method is used as an implementation method, and the initial state is modeled as the root node of the solution space tree. Then, a first-in-first-out (FIFO) queue is used to maintain all leaf nodes in the solution space tree, and their states are recursively pushed to the state of the next time slice in turn. The above FIFO queue is also called an active node queue;

[0094] Referring to Table 1, the data domain of the low-cut NPU real-time scheduling algorithm, to facilitate algorithm design, the active node queue may also include a sentinel node whose state time is the same as the state node but does not contain other information, so as to facilitate the determination of whether all state nodes have been recursively pushed to the next time slice. The state node may include a data domain;

[0095] Table 1 Data fields of low-cut NPU real-time scheduling algorithm

[0096]

[0097]

[0098] Based on the above data domain, the implementation method includes the following steps:

[0099] 1. Initialize the search tree:

[0100] Initialize the search time. If the lowest common multiple of each task cycle is less than the maximum search time T specified by the user, max , then the search time is set to the lowest common multiple of each task period;

[0101] Create a root node, initialize the data fields of a node such as the current state time, task execution status, job sequence, mandatory execution mark, and last executed task, and use it as the root node of the search tree;

[0102] Add the root node and a sentinel node with the same current state time as the root node to the active node queue;

[0103] 2. Process the nodes in the active node queue one by one, take a node from the head of the active node queue, and then process it according to the type of the node:

[0104] (1) If the node is a sentinel node:

[0105] a. If the state time reaches the search time, the search process ends and backtracking is performed to obtain the job scheduling sequence;

[0106] b. Otherwise, traverse the active node queue and compare the cumulative losses of all nodes with the same task execution table content. After that, only one node with the smallest cumulative loss is retained among all nodes with the same task execution table and counter, and prune the remaining nodes;

[0107] (2) If the node is a status node, then:

[0108] First, check the deadlines and join status of the tasks, including:

[0109] a. Check whether there are any tasks that have not been completed beyond the deadline. If so, prune the current node;

[0110] b. Determine whether there is a task whose cycle start time plus pre-processing time is equal to the current state time. If so, add the task to the task execution table;

[0111] (3) Then, depending on whether the mandatory execution flag is empty, select one of the following steps:

[0112] a. If the reference task is empty or the same as the task selected by the EDF algorithm, there will be no task preemption in the current time slice. At this time, if the mandatory execution mark is empty, the reference task and the task to be executed are set to the job selected by the EDF algorithm; if the mandatory execution mark is not empty, the task to be executed is set to be the same as the mandatory execution mark; in particular, if the minimum granularity block of some models is larger than a single time slice, multiple continuous time slices need to be allocated to this model block in the scheduling stage. For the above situation, when one of the following conditions is met, that is, 1) the sum of the execution time of the model block and the current value of the counter exceeds the preset threshold, 2) a more urgent task arrives within the execution time of the model block, then refer to b to create two child nodes for the current node to derive the two options of "starting to allocate these continuous time slices at the current time slice" and "scheduling other jobs at the current time slice" respectively, so as to evaluate the accumulated loss and prune in the subsequent time slice; otherwise, directly allocate continuous time slices that match the time required to execute the model block starting from the current time slice;

[0113] b. If the reference task is not empty and is different from the task selected by the EDF algorithm, task preemption will occur in the current time slice. At this time, two child nodes are created for the current node to derive the two choices of executing the original task and executing the new task respectively. The above child nodes inherit the state time, task execution table and accumulated loss of the current node, but clear the job sequence. Among them, the child node that executes the new task will set the forced execution mark to empty, the counter to zero, and the reference task and the job to be executed to the task selected by the EDF algorithm. If the forced execution mark of the child node that continues to execute the original task is empty, the forced execution mark is set to the value of the reference task. If the forced execution mark is not empty, it will not be changed. Afterwards, the child node that continues to execute the original task will set the task to be executed to be the same as the forced execution mark, the counter to be the same as the parent node, and the reference task to the task selected by the EDF algorithm;

[0114] (4) Finally, update the node status through the following steps. If a child node is generated, update the status of all child nodes:

[0115] a. Update the cumulative loss of the node. If the node is a newly generated child node for executing a new task, the loss caused by task preemption is included in the loss tuple. Afterwards, if the currently executed task is not the task selected by the EDF algorithm, the loss of not executing the most urgent task is included in the cumulative loss.

[0116] b. Update the remaining execution time and counter of the current task. If the remaining execution time of the current task becomes zero after the update, clear the current execution task from the task execution table, clear the reference task and the mandatory execution mark, and set the counter to zero. If the counter reaches the preset threshold, set the counter to zero, clear the reference task and the mandatory execution mark, and update the accumulated loss.

[0117] c. Place the currently executing task at the end of the task scheduling queue;

[0118] d. Increase the node's status time by one;

[0119] e. Place the current node at the end of the active node queue;

[0120] 3. Generate task scheduling sequence:

[0121] (1) Take the state node with the smallest cumulative loss in the active node queue and start backtracking until it reaches the root node. Then, according to the path from the root node to the state node, concatenate the task scheduling sequences of each node on the path to obtain the task scheduling sequence divided by time slices.

[0122] (2) Use the greedy method to convert the task scheduling sequence into a job scheduling sequence for executing model blocks, that is, treat the consecutive time slices assigned to the same task as a longer complete time slice, and arrange the largest possible model block for each time slice in turn without exceeding the maximum remaining time of the time slice, until no more can be arranged;

[0123] (3) Change the execution time of the model block to the execution time before it is discretized and advance the execution of the block as much as possible. Since this execution time is shorter than the execution time of the model block, the idle time generated by this process can be used to advance the execution of the model block as much as possible without changing the execution order, thereby generating as long as possible continuous idle time for the scheduling of burst tasks, thereby avoiding the impact of burst tasks on the execution of periodic tasks to a certain extent, and thus avoiding the impact on the real-time performance of periodic tasks;

[0124] (4) Place the generated job scheduling sequence into the inference queue.

[0125] Although the present invention has been described according to a limited number of embodiments, it will be apparent to those skilled in the art, with the benefit of the above description, that other embodiments may be envisioned within the scope of the invention thus described. In addition, it should be noted that the language used in this specification is selected primarily for readability and teaching purposes, rather than for explaining or defining the subject matter of the present invention. Therefore, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of the present invention is illustrative, not restrictive, with respect to the scope of the present invention, which is defined by the appended claims.

Claims

1. A real-time inference system with NPU time-sharing multiplexing, including a heterogeneous computing system and a multi-model serial real-time inference controller; The heterogeneous computing system comprises an NPU module (1), a CPU module (2) and a DDR memory (3), wherein the NPU (1) is connected to the CPU module (2) via a PCIe bus, and the CPU module (2) is connected to the DDR memory (3); The CPU module (2) is used to control the NPU module (1) to complete the intelligent computing process; The DDR memory (3) is used to cache model data and service data to be processed; The multi-model serial real-time reasoning controller comprises a ready state pre-segmenter (4), a runtime planner (5), and a runtime executor (6) which are connected in sequence. The ready state pre-segmenter (4) provides model blocks for the runtime executor (6), the runtime planner (5) divides the reasoning request into task blocks, and the runtime executor (6) calls the corresponding model block according to the task block until the entire model reasoning is completed.

2. According to claim 1, a real-time reasoning system with NPU time-sharing multiplexing, characterized in that: The NPU module includes an on-chip cache, a matrix calculation unit, a vector calculation unit and a data handling unit. The data handling unit is connected to the on-chip cache to move data from the DDR into the on-chip cache. The cache is then connected to the matrix calculation unit and the vector calculation unit.

3. A real-time reasoning system for NPU time-sharing multiplexing according to claim 2, characterized in that: The preparation state pre-segmenter (4) includes a model evaluation component and a model partitioning component, the model evaluation component is connected to the model partitioning component, the preparation state pre-segmenter (4) is used to partition the model in the preprocessing stage and place it into the model block warehouse, the runtime planner (5) includes an RPC server, an inference task pool, an inference task scheduler and an inference queue connected in sequence, the runtime planner (5) is used to receive the model inference calculation request task submitted by the terminal in the execution stage, and decompose the task and fill it into the inference queue containing the task blocks, the runtime executor (6) is used to take out the tasks from the inference queue in sequence in the execution stage, and call the corresponding model block for execution, the model partitioning component is connected to the inference task scheduler, and the inference queue is connected to the runtime executor (6).

4. A real-time reasoning scheduling method for NPU time-sharing multiplexing, characterized in that: A real-time reasoning system for NPU time-sharing multiplexing as claimed in any one of claims 1 to 3, comprising the following steps: S1. Use the preparatory pre-segmenter to convert and segment the intelligent model, that is, convert the intelligent model into a format file executable by the NPU model, test and evaluate the execution time of each layer of the model, and split the intelligent model into model blocks of different granularities based on the unit granularity; S2. The total execution time and task cycle of each intelligent model are discretized by the runtime planner, that is, the total execution time and task cycle of each intelligent model are discretized into integers to reduce the computational overhead of the scheduling algorithm. Before all tasks start, the scheduling granularity is determined by nonlinear optimization based on the unit granularity and the user-given load threshold, and the scheduling granularity is used to solve the execution time and task cycle discretization values ​​of the intelligent model blocks to obtain the discretized execution time and task cycle used by the scheduling algorithm; S3. Based on the discrete execution time and task cycle, a low-cut NPU real-time scheduling algorithm with a multiplexing mechanism is used to obtain the task scheduling sequence, and the runtime executor is used to execute according to the task scheduling sequence.

5. The real-time inference scheduling method for NPU time-division multiplexing according to claim 4 is characterized in that: The S1 specifically includes the following steps: S11. Obtain the intelligent model required for the task from the model warehouse; S12. Use the model evaluation component to obtain the weight files of all intelligent models and convert the format of the intelligent model into an NPU operator graph file; S13. Statistically measure the total execution time and each layer execution time of the operator graph files of all intelligent models based on the NPU runtime; S14. According to the execution time and unit granularity conditions of each layer of the intelligent model, each intelligent model is recursively split using the model partitioning component to obtain model blocks of different granularities; In S12, the intelligent model is automatically quantized and integrated during the conversion process, and the intelligent model file executable by the NPU module is compiled and generated; In S13, a time-consuming list τ is established, and the total execution time of the intelligent model of the i-th task is τ i , the execution time of the jth layer of the intelligent model is The task period of the corresponding type of task of the intelligent model is taken as the task period t of the intelligent model. i ; In S14, the intelligent model or model block is divided into two parts with equal execution time granularity each time, until further splitting will cause the execution time of the model block to be less than the unit granularity g0 or the model block contains only one layer, and all model blocks of different granularities generated by the division are stored in the model block warehouse.

6. The real-time inference scheduling method for NPU time-division multiplexing according to claim 5, characterized in that: In S2, the user task attributes are obtained, and the task execution time and task cycle are discretized. The specific method is: for task i, its execution time is τ i , the task cycle is t i , the execution time after discretization using the scheduling granularity g is ceil(τ i / g), the discretized task period is floor(t i / g), where ceil() and floor() functions represent rounding up and rounding down respectively. The task load after task i is discretized is U i (g), U i (g) = ceil(τ i / g) / floor(t i / g), for a system with I tasks, the total load is the sum of the loads of all tasks, that is, Where I is the total number of tasks; It will at least ensure that the total load does not exceed the preset threshold U max The problem of finding a larger scheduling granularity g under the premise of is modeled as a nonlinear objective constrained optimization problem, where the independent variable is the scheduling granularity g, and the optimization objectives are:

1. Minimize the number of schedulings required within the observation time, where the observation time is the least common multiple of the scheduling cycles of each task, and 2. Minimize the gaps generated by all model block scheduling on continuous time slices; the constraints are:

1. The total cycle of the discretized task does not exceed the preset load threshold U max , 2. The scheduling granularity g is in [g0,τ min ] interval, where g0 is the unit particle size, τ min The minimum execution time of a task in a task is obtained by using an optimization method to search for the optimal solution set, and the optimal scheduling granularity g is selected from it. This scheduling granularity is used to discretize the execution time and task cycle of all tasks.

7. The real-time inference scheduling method for NPU time-division multiplexing according to claim 6, characterized in that: In S3, the low-cut NPU real-time scheduling algorithm uses the branch and bound method to generate a scheduling plan within the observation time. It repeatedly calculates the task scheduling state of the task state in the time slice from the initial state, thereby recursively pushing the task state to the next time slice. When it is detected during the generation process that task preemption or scheduling of a larger model block is about to occur, the two options of executing the original job and executing the new job are recursively pushed, and the losses caused by the two options are calculated:

1. The loss of failing to execute the highest priority task; 2. Failure to avoid the loss of the segmented model. When the number of recursive generation of the scheduling plan reaches the number of observation time slices given by the user, the algorithm terminates the generation and outputs a series of corresponding scheduling processes with the smallest cumulative loss. The block granularity of the corresponding intelligent model is used as the final result to update the inference queue. The above tasks include but are not limited to the actual inference tasks of target recognition and edge detection required by the user, and the job is the specific execution of the task at a specific time point; In the process of calculating the loss, the loss function is defined as Loss = cω + K, where c is the number of task preemptions experienced from the initial state to the current process; ω is a user-defined weight parameter; K is the degree of violation of the principle of prioritizing the most urgent tasks in the process, defined as Among them, n is the number of time slices from the initial state to the current state, N is the total number of time slices, and k is the total number of time slices. n Indicates whether the scheduling of the nth time slice violates the principle of prioritizing the most urgent tasks. When the job executed in the nth time slice belongs to the most urgent task at this time, k n =0, otherwise, k n =1; The scheduling branches are pruned according to the accumulated loss. The process is as follows: if multiple branches reach the same state at a certain moment, that is, the remaining execution time of tasks in different branches, the execution time of the most recent job in all branches, and the tasks to which the most recent job belongs in all branches are exactly the same, then only the branch with the lowest loss function value is retained among the branches with the same state. If there are multiple scheduling branches with the same state and the loss function values ​​are the same as the lowest, any one of them is retained. The low-cut NPU real-time scheduling algorithm is compatible with non-periodic tasks by introducing a time slice reuse mechanism. After assigning tasks to each time slice, the low-cut NPU real-time scheduling algorithm generates the execution order of model blocks through two steps of merging and reuse, namely: (1) Merge consecutive time slices that are assigned the same task, find a model block in the model block warehouse that is not greater than and closest to the time slice length and assign it to the time slice. If the idle time slot of the allocated time slice is too large, find a model block that matches the idle time slot again for allocation. (2) Without changing the execution order of the model blocks, the time slice reuse mechanism is used to advance the execution of the model blocks to the larger value of the end time of the previous model block and the arrival time of the current task. The idle time slots generated by each time slice are then merged to generate continuous gaps. The continuous gaps are used to insert appropriate model blocks of sudden non-periodic tasks or other transactions into the inference queue for execution, thereby reducing the impact of sudden tasks on the real-time execution of periodic tasks.

Images