A multi-task scheduling method of a heterogeneous processor

Abstract

The application discloses a multi-task scheduling method of a heterogeneous processor and belongs to the computer field. In the case that the storage resource of an end side is insufficient, the user cannot use a single model to parallel infer different frame data, and the existing resources are reused as much as possible to improve the inference efficiency in the scene that the camera data / frame data needs to be processed in parallel. Meanwhile, different priorities are set for different inference tasks and different inference tasks of the same model, multiple models exist in cascade, and the multiple model cascades return in the middle, and the multi-task scheduling method is provided, which can effectively improve the efficiency of the end side inference task and significantly reduce the waste of the computing resource.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a multi-task scheduling method for heterogeneous processors. Background Technology

[0002] Heterogeneous processors are generally divided into a computation control unit (hereinafter referred to as the host program) and an acceleration computing unit (hereinafter referred to as the heterogeneous device, i.e., the edge chip). The host program is responsible for generating and submitting tasks, dispatching the task code and data to the edge chip for accelerated computation. If multiple tasks need accelerated computation, the host program can only submit these tasks sequentially, wait for their status to return, and then submit the next task, and so on.

[0003] Patent CN111258744A is a server-side heterogeneous computing task processing method. Because the server has sufficient memory resources, developers can spend more resources designing monitoring programs, load balancing programs, etc., to schedule heterogeneous computing tasks. However, this processing method has no advantage for edge environments with limited main memory or memory, parallel analysis of video frame data, model priority scheduling, non-return cascading, and return cascading.

[0004] At the same time, due to the frequent back-and-forth communication with the host computer, this method is very inefficient and also has the following drawbacks: 1) The same application can usually only process data serially and cannot process different data in parallel. That is, it cannot use a data-driven strategy to divide inference tasks, which greatly wastes computing resources and affects inference performance. 2) If users customize parallel programs to solve the problem in 1), they need to start multiple programs for parallel inference. However, at this time, multiple programs need to apply for independent host memory and device memory for their models, and existing resources cannot be reused, resulting in a huge waste of memory and video memory resources. This is extremely fatal on edge processors where memory resources are extremely precious. 3) Within the application, if there are subtasks that can be executed in parallel, they usually cannot be executed in parallel. 4) Multiple cascaded applications require manual code control by the user and cannot be automatically connected in sequence, nor can they enjoy the benefits of parallel programming; 5) Multiple cascaded applications with partial returns between them, i.e., program 1 runs some tasks first, then program 2 executes, and then returns to execute the remaining tasks in program 1. Such a situation cannot be supported by existing task scheduling methods. Summary of the Invention

[0005] The purpose of this invention is to provide a multi-task scheduling method for heterogeneous processors to solve the problems in the background art.

[0006] To address the aforementioned technical problems, this invention provides a multi-task scheduling method for heterogeneous processors, comprising: Step S1: Create a task parser in main memory to parse the Bin file generated by compiling the multi-level intermediate representation framework; Step S2: Based on the parsing information from the task parser in step S1, construct a Model object in main memory. This Model object stores all the tasks in the model. Step S3: Create an inference Stream object in main memory. This Stream object is bound to a Model object and is used to manage the specific inference matters of the model, including: Step S31: A Stream object is bound to a specific Model object. A Model object can be bound to multiple Stream objects. Step S32: In order to save memory and video memory, the Model object stores common information of all Tasks, and each Stream object can share the common information of Tasks; Step S33: In order to distinguish different App inference tasks of the same or different models, each Stream object allocates memory information for each Task and allocates a global globalTaskId; Step S4: Send a request to the kernel driver layer for each Task in the Stream object to request a global TaskID and Assembly memory to store the input and output Tensor data of the Task. Step S5: During task execution, the host program distributes Tasks that meet the conditions in the Stream object to the kernel driver layer, specifically including: Step S51, Serial Distribution: If each Task has data dependencies or logical dependencies, and they are related, then each Task is distributed to the next layer in sequence. Step S52, Parallel Distribution: If there are multiple Tasks without dependencies, they are distributed to the lower layers in parallel to achieve multi-task parallel scheduling and significantly improve task execution efficiency. Step S53: Set different priorities for each Stream object. If a certain Stream object has a higher priority, then the Tasks under that Stream object will be executed first. After all the Tasks of that Stream object are executed, the Tasks of other Stream objects will be executed. Step S54: In the case of multiple Stream objects being cascaded, the Tasks in the preceding Stream object are executed first, then the Tasks in the following Stream objects are executed, and so on. In step S55, when multiple Stream objects are cascaded and returned, the first Task in the preceding Stream object is executed first, then the Task in the following Stream object is executed, and then the subsequent Task of the preceding Stream object is returned. Step S6: After receiving the task submitted by the user interface runtime layer, the kernel driver layer records the information of each task and sends it to the firmware. Step S7: After receiving the Task, the firmware parses the Task information and registers the information. The subtask unit Block in the Task is the basic unit of firmware processing. It is dispatched to an available Kernel scheduler for distribution. Step S8: The Kernel scheduler, having information on available processors, dispatches the received Block tasks to heterogeneous edge processors for computation. Step S9: After the calculation is completed, return the Block task execution status to the firmware; Step S10: The firmware returns the status to the kernel driver layer, and after the kernel driver layer records it, it returns the status to the user interface runtime layer. Step S11: After receiving the status, the user interface runtime layer modifies the Task status. If the Task is completed normally, it notifies its consumer task and repeats steps S4 to S11 until the end. Step 12: If the Task is in a failed state, then the entire reasoning task ends.

[0007] The Bin file is generated by compiling the ONNX file of the specific model using the executable program of the multi-level intermediate representation framework. It stores all the tasks of the entire model. Each Task contains several sub-task units (Blocks), and each sub-task unit (Block) contains several minimum task units (Ops). The minimum task units (Ops) are Conv / ReLU / Avgpool / Fc / Maxpool / Resize operators in deep learning. These Ops map the Task, Block, and Op numbers, quantity, offset address in the file, register value, and required number of computation cores.

[0008] In step S2, the Model object is used to manage the common information of all Tasks, which is beneficial for recording the overall information of the task to be inferred and for scheduling the upper-level inference tasks.

[0009] In step S3, a Stream object is created and bound to the Model object to be inferred. All Stream objects bound to the same Model can use the common information of the Task in the Model, including localTaskId, binary Kernel functions, dynamic link libraries, input and output Shape information, and input and output offset information.

[0010] Since each Stream object performs inference on different data, it needs to apply for a global TaskID for its own and all Tasks to ensure that each Task received by the kernel driver layer is unique and easy to manage. At the same time, if a user uses the same model to start multiple different inference tasks, or uses different models to start different inference tasks, the global TaskID ensures that the data and results of different inference tasks are not confused and that there is a one-to-one correspondence.

[0011] Step S5 provides additional support for serial, parallel, inference priority, cascading, and cascading with internal return conditions, including: In the usual case, the Conv, ReLU, and AvgPool operators are executed sequentially. In the parallel scenario, after task T0 is completed, tasks T1, T2, and T3 can be executed in parallel because their dependent parent tasks have been satisfied. Inference priority: For multiple Stream objects of the same model, different data are used for inference. Different priorities are set for different Stream objects. When they are distributed to the kernel driver layer KDL, they are sorted according to their priorities. Tasks in high-priority Stream objects are dispatched to the firmware for further processing first. After processing, tasks in low-priority Stream objects are dispatched. If two Stream objects, stream1 and stream2, are cascaded, the user can call the cascade interface with stream1 and stream2 as input parameters. The tasks in stream1 will be executed first, followed by the tasks in stream2, with synchronization enabled by default. If parallel execution is possible during the execution of the two streams, the user can enjoy the benefits of parallel operation. In cases of cascading with internal returns, refer to the case of model cascading and use the cascading return interface.

[0012] In step S6, several applications at the upper layer submit different inference tasks, which can all be received by the kernel driver layer KDL, and recorded and distributed in a unified manner. The relationship between the kernel driver layer KDL and the upper layer applications is similar to 1:N. At the same time, if there are multiple heterogeneous processors on the edge, only one kernel driver layer KDL is needed for expansion processing.

[0013] In step S7, the firmware parses out the Block task units from all received Tasks, registers the information, and distributes them to the Kernel-level scheduler.

[0014] In step S8, the Kernel-level scheduler obtains the usage status of heterogeneous processors in real time. When it receives a Block task unit, it finds that a certain Core is idle, distributes the Block to it, and registers the Core's occupancy. If all Cores are busy, it waits for a certain clock cycle, then checks the Core usage status again and dispatches tasks.

[0015] In steps S9 to S11, the Kernel-level scheduler receives the execution status of the Block task unit. If it is completed, the information is sent to the firmware, KDL, and Unified Intermediate Representation (UIR) in sequence. If several Ops in the Block fail to be calculated, the Block fails, the information is returned to the firmware, the Task fails, and the information is returned to the Unified Intermediate Representation (UIR), and the App inference program fails.

[0016] This invention provides a multi-task scheduling method for heterogeneous processors, which improves the efficiency of task scheduling, avoids unbalanced computing load, thereby reducing the waste of computing resources and improving resource utilization efficiency. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the multi-task scheduling method for heterogeneous processors provided by the present invention.

[0018] Figure 2 The software task sequence diagram provided for this invention.

[0019] Figure 3 The relationship diagram of Model, Stream, and Task provided by this invention.

[0020] Figure 4 This invention provides a task relationship graph in deep learning.

[0021] Figure 5 The model cascade diagram in deep learning provided by this invention.

[0022] Figure 6 This is a model cascade return diagram in deep learning provided by the present invention.

[0023] Figure 7 This is the App-Task-Block-Op relationship diagram provided by the present invention. Detailed Implementation

[0024] The following detailed description, in conjunction with the accompanying drawings and specific embodiments, provides a more detailed explanation of the multi-task scheduling method for heterogeneous processors proposed in this invention. The advantages and features of this invention will become clearer from the following description. It should be noted that the accompanying drawings are all in a very simplified form and use non-precise proportions, and are only used to facilitate and clarify the illustration of the embodiments of this invention.

[0025] This invention is based on the following basic concepts: CUDA (Compute Unified Device Architecture) is a unified programming language provided by NVIDIA for its GPU chip products, and it is currently the mainstream language worldwide.

[0026] CUDA does not have the concept of Model; instead, it provides the concept of Stream to organize operator / task computations.

[0027] Stream, meaning queue, is a software concept and an important one in CUDA. It's a first-in, first-out (FIFO) structure, meaning that if two tasks are in the queue, the task at the front of the queue will be executed before the task at the back. A Stream is the unit of inference in CUDA computation. A Stream can typically be considered as a specific inference task, such as face detection, face recognition, or face analysis.

[0028] A Stream can contain a series of operators, such as Conv→Relu→AvgPool→Resize→... These operators must be placed into the Stream in sequence and executed sequentially.

[0029] In deep learning algorithms, the Conv operator is used to extract features from the data and plays a central role in the algorithm. Each Conv operator has weight / Bias information, which are the most important components of a deep learning model and constitute all the information stored in a deep learning model file.

[0030] When deploying deep learning model inference tasks using CUDA, in common scenarios such as parallel processing of camera data / video frame data, it is necessary to use the same inference method for single and double frame data. This requires repeatedly and mechanically constructing multiple Streams and putting all operators into these Streams in sequence. This will cause all Conv's Weight / Bias information to be copied multiple times, wasting a lot of host memory and on-chip memory during computation (if there are N identical inference programs, then (N-1) copies of Weight / Bias resources are wasted).

[0031] At the same time, users need to manually write programs to process different data in parallel streams to achieve the effect of parallel processing.

[0032] At the same time, operators in a Stream, if they can run independently, cannot be executed in parallel and must be executed sequentially, even if these operators have no data or logical dependencies.

[0033] If multiple inference tasks are assigned to different streams and are cascaded, the next stream can only begin computation after the previous stream has been completed and synchronized. There is no possibility of parallel processing in between.

[0034] CUDA Streams does not support multi-task cascading and internal return scenarios.

[0035] This invention provides a multi-task scheduling method for heterogeneous processors, the process of which is as follows: Figure 1 As shown, it includes the following steps: Step S1: Create a task parser in main memory to parse MLIR Bin files.

[0036] This MLIR Bin file is generated by compiling a model's ONNX file using the MLIR (Multi-Level Intermediate Representation) executable program, which converts a common model into a model file that can run on heterogeneous chips. It stores all the tasks of the entire model. Each task contains several blocks (sub-task units), and each block contains several ops (minimum task units). These ops store information such as the task, block, and op numbers, their offset addresses in the file, register values, and the number of required computation cores. This task hierarchy can be found in [reference needed]. Figure 7 Furthermore, all Tasks are established with parent-child dependencies based on the producer and consumer relationship, thus reflecting the information of a directed acyclic graph (DAG).

[0037] In this context, the producer is the parent task of a given task, and the child task can only begin to be computed after all parent tasks have been completed; the consumer is the child task of a given task.

[0038] Step S2: Create an inference Model object in main memory. Based on the parsing information of the task parser in step S1, construct the Model object, which stores all the tasks in the model. Step S3: Create an inference Stream object in main memory. This Stream object is bound to a Model object and used to manage the specific inference matters of the model. The relationship between Model, Stream, and Task can be found in [reference needed]. Figure 3 As shown; Step S31: A Stream object is bound to a specific Model object. A Model can be bound to multiple Stream objects. Step S32: In order to save memory and video memory, the Model object stores information about all Tasks. Each Stream object can share common information about Tasks, such as localTaskId, dynamic link libraries, and binary executable program information of the running functions. Step S33: In order to distinguish different App inference tasks of the same or different models, each Stream object allocates memory information for each Task and allocates a global globalTaskId; Step S4: Send a request to the kernel driver layer for each Task in the Stream object to request a global TaskID and Assembly memory to store the input and output Tensor data of the Task. Step S5: During task execution, the host program distributes Tasks that meet the conditions in the Stream object to the kernel driver layer, specifically including: Step S51, Serial Distribution: If each Task has data dependencies or logical dependencies, and they are related, then each Task is distributed to the next layer in sequence. Step S52, Parallel Distribution: If multiple Tasks have no dependencies, they can be distributed to lower layers in parallel to achieve multi-task parallel scheduling, significantly improving task execution efficiency. (See also...) Figure 4 As shown, T1, T2, and T3 are independent of each other and can be distributed in parallel; Step S53: Different priorities can be set for each Stream object. If a certain Stream has a higher priority, the Tasks under that Stream will be executed first. After all the Tasks of that Stream are executed, the Tasks of other Streams will be executed. Step S54: In the case of multiple Stream objects cascading, the Tasks in the preceding Stream are executed first, followed by the Tasks in the following Stream, and so on. Figure 5 As shown; Step S55: In the case of multiple Stream objects being cascaded and returned, the tasks in the preceding Stream are executed first, then the tasks in the following Stream are executed, and then the subsequent tasks in the preceding Stream are returned, and so on. Figure 6 As shown; Step S6: After receiving the task submitted by the user interface runtime layer, the kernel driver layer records the information of each task and sends it to the firmware. Step S7: After receiving the Task, the firmware parses the Task information and registers the information. The subtask unit Block in the Task is the basic unit of firmware processing. It is dispatched to an available Kernel scheduler for distribution. Step S8: The Kernel scheduler, having information on available processors, dispatches the received Block tasks to heterogeneous edge processors for computation. Step S9: After the calculation is completed, return the Block task execution status to the firmware; Step S10: The firmware returns the status to the kernel driver layer, and after the kernel driver layer records it, it returns the status to the user interface runtime layer. Step S11: After receiving the status, the user interface runtime layer modifies the Task status. If the Task is completed normally, it notifies its consumer task and repeats steps S4 to S11 until the end. Step 12: If the Task is in a failed state, the entire inference task ends.

[0039] The embodiments are further explained below: Inference tasks can specify the required number of cores, i.e., how many processors are needed to run a particular task. The Unified Intermediate Representation (UIR) receives this value, configures it in the Task, and then passes it to the KDL, which in turn passes it to the firmware. Once the firmware knows the required number of cores, it groups the currently available cores at an appropriate granularity (e.g., multiples of 2, 4, or 8). The computing cores execute tasks in groups, thus achieving rapid task acquisition and fine-grained management of computing resources.

[0040] The implementation plan is as follows: 1. After parsing the model information, a Model object is used to record the model information. The Model object retains key deep learning information such as Weights / Bias used by each Task, and maps this information to Task. This data can be shared by all Stream objects derived from the same Model. 2. Based on the Model object, create a Stream object according to user requirements, and call the interface "aiStreamstream; aiStreamCreate(stream, model)" to bind the two together; 3. For each Task in the Stream object, allocate Assembly memory. This information is recorded in the Stream object and mapped to the Task. Also, allocate a global GlobalTaskId for all Tasks. IV. Users can set the priority of a Stream object. Once set, all Tasks under that Stream object will have that priority. If there are multiple Stream objects, you need to call the interface to set the cascading relationship. For example, to cascade Stream1, Stream2, Stream3, and Stream4, you can call the interface "aiCascade(Stream1, Stream2); aiCascade(Stream2,Stream3); aiCascade(Stream3, Stream4);" to set their cascading relationship. The "aiCascade" interface adds an Event object after the last Task in the queue represented by the first parameter, pointing to the first Task in the queue represented by the second parameter, thus establishing a sequential connection. The Event object is a lightweight object structure used to represent sequential connections.

[0041] If there are multiple Streams and there are cases where the process returns midway, the interface needs to be called to set the cascading relationship. For example, if Stream1 and Stream2 are cascaded, the first 3 tasks in Stream1 will be executed first, then the tasks in Stream2 will be executed. After the tasks in Stream2 are executed, the process will return to execute the 4th task in Stream1 until Stream1 finishes its computation. This requirement can be achieved by calling the overloaded interface "aiCascade(Stream1, Stream2, 3)". The “aiCascade” interface is a sufficient interface with the same name as the interface above, but the parameter list and parameter meanings are different. This interface will insert an Event object after the third task in the first queue, and point the Event object to the first task represented by the second parameter. Then, after the last task represented by the second parameter, insert another Event object and point it to the fourth task represented by the first parameter, thus establishing the connection relationship.

[0042] 5. Provide data to the Stream, call the Stream execution interface, and the Stream will submit its contained Tasks to the global task pool; 6. After receiving all Tasks submitted by Streams, the global task pool sorts the Tasks according to their priority. 7. The global thread pool analyzes the dependencies of all sorted Tasks. If it finds that a Task has no dependency, it dispatches all Tasks to KDL and waits for its return status. 8. KDL receives the task, records the information, and dispatches it to the firmware; 9. The firmware receives the Task, parses out the Block, and dispatches it to the Kernel scheduler. 10. The kernel scheduler records the load status of all edge processors, dispatches them to idle processors, and waits for the status to be returned. 11. Upon receiving the return status, return sequentially in the order described above. If all blocks of a Task complete successfully, the Task status returns a unified intermediate representation UIR sequentially, and the Task is run again in the order described in steps 7 to 11 above. If a block in a task fails to compute, all tasks in the stream to which that task belongs will fail, the global task pool will remove them all, and an error will be reported to the terminal.

[0043] Software tasks are performed as follows Figure 2 The order shown: 1) The module flow is sequentially from App application -> UIR -> KDL -> Firmware -> Kernel-level scheduler -> processor, and is also decomposed step by step according to task level: 2) The workflow flows sequentially from App application -> Stream management of a specific reasoning task -> Task -> Block -> Op, with the two complementing each other and clearly defined. This scheduling method is completely different from the approach described in the background content.

[0044] The above description is merely a description of preferred embodiments of the present invention and is not intended to limit the scope of the present invention in any way. Any changes or modifications made by those skilled in the art based on the above disclosure shall fall within the protection scope of the claims.

Claims

1. A multi-task scheduling method for heterogeneous processors, characterized in that, Includes the following steps: Step S1: Create a task parser in main memory to parse the Bin file generated by compiling the multi-level intermediate representation framework; Step S2: Based on the parsing information from the task parser in step S1, construct a Model object in main memory. This Model object stores all the tasks in the model. Step S3: Create an inference Stream object in main memory. This Stream object is bound to a Model object and is used to manage the specific inference matters of the model, including: Step S31: A Stream object is bound to a specific Model object. A Model object can be bound to multiple Stream objects. Step S32: In order to save memory and video memory, the Model object stores common information of all Tasks, and each Stream object can share the common information of Tasks; Step S33: In order to distinguish different App inference tasks of the same or different models, each Stream object allocates memory information for each Task and allocates a global globalTaskId; Step S4: Send a request to the kernel driver layer for each Task in the Stream object to request a global TaskID and Assembly memory to store the input and output Tensor data of the Task. Step S5: During task execution, the host program distributes Tasks that meet the conditions in the Stream object to the kernel driver layer, specifically including: Step S51, Serial Distribution: If each Task has data dependencies or logical dependencies, and they are related, then each Task is distributed to the next layer in sequence. Step S52, Parallel Distribution: If there are multiple Tasks without dependencies, they are distributed to the lower layers in parallel to achieve multi-task parallel scheduling and significantly improve task execution efficiency. Step S53: Set different priorities for each Stream object. If a certain Stream object has a higher priority, then the Tasks under that Stream object will be executed first. After all the Tasks of that Stream object are executed, the Tasks of other Stream objects will be executed. Step S54: In the case of multiple Stream objects being cascaded, the Tasks in the preceding Stream object are executed first, then the Tasks in the following Stream objects are executed, and so on. In step S55, when multiple Stream objects are cascaded and returned, the first Task in the preceding Stream object is executed first, then the Task in the following Stream object is executed, and then the subsequent Task of the preceding Stream object is returned. Step S6: After receiving the task submitted by the user interface runtime layer, the kernel driver layer records the information of each task and sends it to the firmware. Step S7: After receiving the Task, the firmware parses the Task information and registers the information. The subtask unit Block in the Task is the basic unit of firmware processing. It is dispatched to an available Kernel scheduler for distribution. Step S8: The Kernel scheduler, having information on available processors, dispatches the received Block tasks to heterogeneous edge processors for computation. Step S9: After the calculation is completed, return the Block task execution status to the firmware; Step S10: The firmware returns the status to the kernel driver layer, and after the kernel driver layer records it, it returns the status to the user interface runtime layer. Step S11: After receiving the status, the user interface runtime layer modifies the Task status. If the Task is completed normally, it notifies its consumer task and repeats steps S4 to S11 until the end. Step 12: If the Task is in a failed state, then the entire reasoning task ends.

2. The multi-task scheduling method for heterogeneous processors as described in claim 1, characterized in that, The Bin file is generated by compiling the ONNX file of the specific model using the executable program of the multi-level intermediate representation framework. It stores all the tasks of the entire model. Each Task contains several sub-task units (Blocks), and each sub-task unit (Block) contains several minimum task units (Ops). The minimum task units (Ops) are Conv / ReLU / Avgpool / Fc / Maxpool / Resize operators in deep learning. These Ops map the Task, Block, and Op numbers, quantity, offset address in the file, register value, and required number of computation cores.

3. The multi-task scheduling method for heterogeneous processors as described in claim 2, characterized in that, In step S2, the Model object is used to manage the common information of all Tasks, which is beneficial for recording the overall information of the task to be inferred and for scheduling the upper-level inference tasks.

4. The multi-task scheduling method for heterogeneous processors as described in claim 2, characterized in that, In step S3, a Stream object is created and bound to the Model object to be inferred. All Stream objects bound to the same Model can use the common information of the Task in the Model, including localTaskId, binary Kernel functions, dynamic link libraries, input and output Shape information, and input and output offset information.

5. The multi-task scheduling method for heterogeneous processors as described in claim 4, characterized in that, Since each Stream object performs inference on different data, it needs to apply for a global TaskID for its own and all Tasks to ensure that each Task received by the kernel driver layer is unique and easy to manage. At the same time, if a user uses the same model to start multiple different inference tasks, or uses different models to start different inference tasks, the global TaskID ensures that the data and results of different inference tasks are not confused and that there is a one-to-one correspondence.

6. The multi-task scheduling method for heterogeneous processors as described in claim 5, characterized in that, Step S5 provides additional support for serial, parallel, inference priority, cascading, and cascading with internal return conditions, including: In the usual case, the Conv, ReLU, and AvgPool operators are executed sequentially. In the parallel scenario, after task T0 is completed, tasks T1, T2, and T3 can be executed in parallel because their dependent parent tasks have been satisfied. Inference priority: For multiple Stream objects of the same model, different data are used for inference. Different priorities are set for different Stream objects. When they are distributed to the kernel driver layer KDL, they are sorted according to their priorities. Tasks in high-priority Stream objects are dispatched to the firmware for further processing first. After processing, tasks in low-priority Stream objects are dispatched. If two Stream objects, stream1 and stream2, are cascaded, the user can call the cascade interface with stream1 and stream2 as input parameters. The tasks in stream1 will be executed first, followed by the tasks in stream2, with synchronization enabled by default. If parallel execution is possible during the execution of the two streams, the user can enjoy the benefits of parallel operation. In cases of cascading with internal returns, refer to the case of model cascading and use the cascading return interface.

7. The multi-task scheduling method for heterogeneous processors as described in claim 6, characterized in that, In step S6, several applications at the upper layer submit different inference tasks, which can all be received by the kernel driver layer KDL, and recorded and distributed in a unified manner. The relationship between the kernel driver layer KDL and the upper layer applications is similar to 1:N. At the same time, if there are multiple heterogeneous processors on the edge, only one kernel driver layer KDL is needed for expansion processing.

8. The multi-task scheduling method for heterogeneous processors as described in claim 7, characterized in that, In step S7, the firmware parses out the Block task units from all received Tasks, registers the information, and distributes them to the Kernel-level scheduler.

9. The multi-task scheduling method for heterogeneous processors as described in claim 8, characterized in that, In step S8, the Kernel-level scheduler obtains the usage status of heterogeneous processors in real time. When it receives a Block task unit, it finds that a certain Core is idle, distributes the Block to it, and registers the Core's occupancy. If all Cores are busy, it waits for a certain clock cycle, then checks the Core usage status again and dispatches tasks.

10. The multi-task scheduling method for heterogeneous processors as described in claim 9, characterized in that, In steps S9 to S11, the Kernel-level scheduler receives the execution status of the Block task unit. If it is completed, the information is sequentially protected to the firmware, KDL, and Unified Intermediate Representation (UIR). If several Ops in a Block fail to compute, then the Block fails, the information is returned to the firmware, the Task fails, and the information is returned to the Unified Intermediate Representation (UIR), indicating that the App's inference program has failed.

Images