Method and apparatus for real-time task scheduling for non-preemptive systems

Abstract

A method for scheduling real-time (RT) tasks includes receiving a task request defined by at least one RT task, wherein the at least one real-time (RT) task is to be queued based on an entry event trigger; obtaining a completion time for the at least one RT task based on an execution deadline of the at least one RT task, an execution deadline of a next RT task, and a maximum execution time of execution of the next RT task; creating a bandwidth reservation task having a deadline defined by the completion time; inserting the at least one real-time task with the bandwidth reservation task into an RT wait queue based on a deadline of each of the at least one RT task and the bandwidth reservation task according to an early deadline first (EDF) criterion; and scheduling unscheduled tasks based on an available time of the RT wait queue according to an EDF-based scheduling.

Description

Technical Field

[0001] This disclosure relates to a method and apparatus for real-time task scheduling in a non-preemptive system. Specifically, this disclosure relates to providing scheduling of mixed task sets in a non-preemptive system through bandwidth reservation. Background Technology

[0002] In any computing system, the operating system (OS) provides the interface between the user and the computer hardware. An operating system is software that performs all the basic tasks, such as file management, memory management, process management, handling input and output devices, interrupts, and controlling various peripheral devices. Currently, based on type, operating systems can be defined as 1) General Purpose Operating System (GPOS) and 2) Real-Time Operating System (RTOS).

[0003] Scheduling is a method of allocating tasks to resources to perform the work. Specifically, scheduling is based on various algorithms, such as Completely Fair Scheduler (CFS), Round-Robin Scheduler (RR), and deadline schedulers. Tasks are scheduled based on adaptive scheduling mechanisms / algorithms. Tasks can be virtual computing elements, such as threads, processes, or data streams, which are sequentially scheduled onto hardware resources, such as processors, network links, or expansion cards.

[0004] Tasks can be further divided into real-time (RT) tasks and non-real-time (NRT) tasks. For example, RT tasks are tasks with a strict timeline. Examples include tasks from any real-time system (such as air traffic control systems, network multimedia systems, command and control systems, etc.). NRT tasks are tasks without time constraints.

[0005] The scheduler performs all scheduling activities to keep all computer resources busy and allow multiple users to efficiently share system resources or achieve a target quality of service. The scheduler is an integral part of a computing system that provides multitasking.

[0006] Furthermore, preemptive systems such as RTOS can support both RT and NRT tasks. Therefore, task scheduling is based on the system and operating system. Currently, RTOS typically employs preemptive scheduling algorithms. Preemptive scheduling algorithms are priority-based, where the scheduler can preempt a lower-priority running process at any time when a higher-priority process enters the ready state. This process is called context switching. For example, RTOS often uses hardware that supports preemption, such as CPUs. In general, preemption is one of the fundamental mechanisms that facilitates the implementation of scheduling concepts.

[0007] However, non-preemptive systems such as GPOS only support NRT tasks. GPOS typically employs a non-preemptive scheduling algorithm. The design of a non-preemptive algorithm ensures that once a process enters the running state, it cannot be preempted until it completes its allocated time. For example, most computing units such as GPUs / NPUs / DSPs are non-preemptive systems. These non-preemptive systems do not support scheduling for hard real-time applications with deadline constraints. In other words, a running task in a non-preemptive system (NPU / GPU) cannot be interrupted by a higher-priority task. Therefore, RT scheduling does not exist in non-preemptive systems. Consequently, existing NPU / GPU systems cannot support RT multitasking environments.

[0008] Therefore, GPOS-based computing units such as GPUs / NPUs / DSPs do not support scheduling like RTOSs, such as deadline scheduling. Non-preemptive scheduling of computing units such as GPUs / NPUs / DSPs is assigned to specific processes. A process that keeps a computing unit busy for a GPU / NPU / DSP will release the computing unit by switching contexts or terminating. In other words, non-preemptive scheduling occurs when a process voluntarily enters a waiting state or terminates. Non-preemptive systems can be used on a variety of different hardware platforms because they do not require dedicated hardware (e.g., timers) like preemptive scheduling.

[0009] More specifically, as part of a non-preemptive system, GPOS-based systems, such as NPU drivers, involve interaction between NPU hardware and software and use fixed-priority scheduling to schedule task requests. This also only provides fair scheduling or acceleration for specific applications. Solutions like fixed-priority queues can only prioritize, cannot be reused, and have low NPU utilization. Summary of the Invention

[0010] Technical issues

[0011] Accordingly, all NPU requests are scheduled in order of arrival time, with the earliest arriving requests scheduled first in first-in, first-out (FIFO). Regarding hybrid task scheduling, soft real-time tasks can be scheduled upon arrival, causing RT tasks to miss their deadlines. Figure 1 This illustrates a problem in existing scheduling methods. For example, consider a scheduler that implements the FIFO technique. Let's consider a time T, for the first RT task T... B The request arrives. Therefore, when the scheduler NPU is idle for time T, task T... B Scheduled ΔT BTime. Furthermore, let's consider a time-of-time (RT) task like task A in job 3, where A3 has an arrival time T3 and a deadline D3. However, task A3 is not scheduled and tends to miss its deadline D3. One possible reason is that task T... B The long runtime of non-preemptive systems is a significant issue. One problem arising from this is that any sufficiently long-running time-limited (RT) task can cause another RT task to miss its deadline. Furthermore, solutions like fixed-priority queues only prioritize tasks, are not reusable, and have low NPU utilization.

[0012] In summary, current non-preemptive systems lack the concept of deadlines required to support real-time applications. Similarly, real-time applications with strict timing requirements cannot be guaranteed. Current schedulers are limited by hardware support for preemption and cannot interrupt running tasks by scheduling higher-priority tasks. In scenarios with multiple applications running, even lower-priority applications may be scheduled, causing higher-priority applications to miss their timelines.

[0013] In summary, current non-preemptive systems cannot guarantee high real-time priority for applications. Fixed-priority schemes suffer from low utilization and inability to handle requests with dynamic arrival times.

[0014] Solution to the problem

[0015] This summary is provided to introduce, in a simplified form, some concepts that will be further described in the detailed description of the invention. This summary is not intended to identify key or essential inventive concepts, nor is it intended to define the scope of the invention.

[0016] According to embodiments of this disclosure, a method is provided for scheduling real-time (RT) tasks by an electronic device including a scheduler. The method includes: receiving, by the scheduler, a task request defined by at least one RT task from an operating system (OS) runtime, wherein the at least one RT task is queued based on an enqueue event trigger; obtaining, by the scheduler, an output time of at least one RT task based on at least one of an execution deadline of the at least one RT task, an execution deadline of the next RT task following the at least one RT task, and a maximum execution time associated with the execution of the next RT task following the at least one RT task; creating, by the scheduler, a bandwidth reservation task having a deadline defined by the output time for the timed execution of the next RT task following the at least one RT task, the bandwidth reservation task being non-executable and defined by an empty execution time; inserting, by the scheduler, at least one real-time task along with the bandwidth reservation task into an RT waiting queue based on the deadlines of each of the at least one RT task and the bandwidth reservation task, according to an Early Deadline First (EDF) criterion, to enable EDF-based scheduling; and scheduling, by the scheduler, unscheduled tasks based on the available time of the RT waiting queue according to the EDF-based scheduling.

[0017] According to embodiments of this disclosure, an electronic device for scheduling real-time (RT) tasks is provided. The electronic device includes a scheduler configured to: receive task requests defined by at least one RT task from an operating system (OS) runtime, wherein the at least one RT task is to be queued based on an enqueue event trigger; obtain the output time of the at least one RT task based on at least one of the execution deadline of the at least one RT task, the execution deadline of the next RT task following the at least one RT task, and a maximum execution time associated with the execution of the next RT task following the at least one RT task; create a bandwidth reservation task having a deadline defined by the output time for the timed execution of the next RT task following the at least one RT task, the bandwidth reservation task being non-executable and defined by an empty execution time; and insert the at least one RT task, together with the bandwidth reservation task, into an RT waiting queue based on the deadlines of each of the at least one RT task and the bandwidth reservation task, according to an Early Deadline First (EDF) criterion, to enable EDF-based scheduling; and schedule unscheduled tasks based on the available time of the RT waiting queue according to the EDF-based scheduling.

[0018] To further illustrate the advantages and features of the invention, the invention will be described in more detail with reference to specific embodiments shown in the accompanying drawings. It should be understood that these drawings depict only typical embodiments of the invention and should not be considered as limiting its scope. The invention will be described and explained with additional features and details in conjunction with the drawings.

[0019] Beneficial effects of the invention

[0020] Embodiments of this disclosure provide a Reserved Bandwidth Server (RBS) that implements the concept of reserved tasks to provide an a priori guarantee for hard real-time tasks. This also provides a method for reserving bandwidth based on the yield time of hard real-time tasks rather than the budget, allowing it to operate without any preemption support.

[0021] The embodiments of this disclosure prioritize hard real-time applications and leverage prior guarantees to serve them predictably. Dynamic arrival times are handled efficiently and reliably using an EDF with RBS through event (interrupt)-driven scheduling. Priority inversion is used to run soft real-time and RT tasks in slack time to achieve high throughput and improve NPU utilization. Attached Figure Description

[0022] These and other features, aspects, and advantages of the invention will become better understood when the following detailed description is read with reference to the accompanying drawings, in which the same characters denote the same parts, wherein:

[0023] Figure 1 This illustrates the current level of development.

[0024] Figure 2 The method steps according to embodiments of this disclosure are shown;

[0025] Figure 3 An embodiment according to this disclosure is shown. Figure 2 The implementation of the method steps;

[0026] Figure 4 The generation of reserved tasks according to embodiments of this disclosure is illustrated;

[0027] Figure 5A and Figure 5B A priority inversion mechanism according to an embodiment of this disclosure is illustrated;

[0028] Figure 6 This illustrates a working scenario according to an embodiment of the present disclosure;

[0029] Figure 7 An embodiment according to this disclosure is shown. Figure 6 Another work scenario that followed;

[0030] Figure 8 An embodiment according to this disclosure is shown. Figure 7 Another work scenario that followed;

[0031] Figure 9 An embodiment according to this disclosure is shown. Figure 2 The implementation of the method steps;

[0032] Figure 10A and Figure 10B An embodiment according to this disclosure is shown. Figure 8 Another work scenario that followed;

[0033] Figure 11A and Figure 11B An embodiment according to this disclosure is shown. Figure 10A and Figure 10B Another work scenario that followed;

[0034] Figure 12A and Figure 12B An embodiment according to this disclosure is shown. Figure 11A and Figure 11B Another work scenario that followed;

[0035] Figure 13A and Figure 13B An embodiment according to this disclosure is shown. Figure 12A and Figure 12B Another work scenario that followed;

[0036] Figure 14A and Figure 14B An embodiment according to this disclosure is shown, in Figure 13A and Figure 13B Another work scenario that followed;

[0037] Figure 15 Example operations according to embodiments of this disclosure are shown;

[0038] Figure 16 An embodiment according to this disclosure is shown. Figure 2 The implementation method of the steps;

[0039] Figure 17 An embodiment according to this disclosure is shown. Figure 2 Another implementation of the method steps;

[0040] Figure 18A and Figure 18B An embodiment according to this disclosure is shown. Figure 2 Another implementation of the method steps;

[0041] Figure 19 Another system architecture according to an embodiment of this disclosure is shown;

[0042] Figure 20 Another exemplary implementation according to embodiments of this disclosure is shown; and

[0043] According to embodiments of this disclosure, Figure 21 , Figure 22A , Figure 22B , Figure 23 , Figure 24A , Figure 24B , Figure 24C , Figure 25A , Figure 25B and Figure 25C Various exemplary use cases based on this mechanism are shown.

[0044] Furthermore, those skilled in the art will understand that the elements in the accompanying drawings are shown for simplicity and may not necessarily be drawn to scale. For example, the flowcharts illustrate the method according to the most significant steps involved to aid in understanding various aspects of the invention. Additionally, regarding the construction of the apparatus, one or more components of the apparatus may have already been represented by conventional symbols in the drawings, and the drawings may only show those specific details relevant to understanding embodiments of the invention, so as not to obscure details that would be readily understood by those of ordinary skill in the art who would benefit from the description herein. Detailed Implementation

[0045] First, it should be understood that although illustrative implementations of embodiments of this disclosure are shown below, the invention can be implemented using any number of techniques, whether currently known or existing. This disclosure should not be limited in any way to the exemplary embodiments, drawings, and techniques shown below, including the exemplary designs and implementations shown and described herein, but modifications can be made within the full scope of the appended claims and their equivalents.

[0046] The term "some" as used herein is defined as "no," "one," "more than one," or "all." Therefore, the terms "no," "one," "more than one," "more than one, but not all," or "all" all fall under the definition of "some." The term "some embodiments" can refer to no embodiments, one embodiment, several embodiments, or all embodiments. Therefore, the term "some embodiments" is defined as meaning "no embodiments, one embodiment, more than one embodiment, or all embodiments."

[0047] The terminology and structure used herein are intended to describe, teach, and elucidate certain embodiments and their specific features and elements, and not to limit, constrain, or narrow the spirit and scope of the claims or their equivalents.

[0048] More specifically, any terms used herein, such as, but not limited to, “including,” “contains,” “having,” “comprising,” and their grammatical variations, do not specify exact limitations or constraints, and certainly do not preclude the possible addition of one or more features or elements unless otherwise stated, and furthermore, they shall not be used to preclude the possible removal of one or more listed features and elements unless otherwise stated in the restrictive language “must include” or “requires inclusion.”

[0049] Regardless of whether a feature or element is restricted to use only once, it may still be referred to as "one or more features," "one or more elements," "at least one feature," or "at least one element." Furthermore, the use of the terms "one or more" or "at least one" does not preclude the absence of that feature or element, unless otherwise specified by restrictive language such as "one or more are required" or "one or more elements are required."

[0050] Unless otherwise defined, all terms used herein, especially any technical and / or scientific terms, may be considered to have the same meaning as commonly understood by one of ordinary skill in the art.

[0051] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0052] Figure 2 Method steps according to embodiments of this disclosure are shown.

[0053] In such Figure 2 The embodiments shown in this disclosure relate to a real-time task scheduling method for a non-preemptive system.

[0054] Method 200 includes receiving (step 201) a task request defined by at least one real-time (RT) task from an operating system (OS) runtime, wherein the at least one RT task is queued based on an enqueue event trigger. Based on this, the at least one RT task is extracted as one or more attributes of arrival time, execution deadline, worst-case execution time (WCET), period, and output time (step 203). For example, the output time is calculated based on one or more of the execution deadline of the at least one RT task, the execution deadline of the next RT task following the at least one RT task, and the maximum execution time associated with the execution of the next RT task following the at least one RT task.

[0055] Subsequently, the bandwidth reservation module creates (step 205) a bandwidth reservation task with a deadline defined by the output time, for the timed execution of the next RT task following at least one RT task. The bandwidth reservation task is non-executable and defined by an empty execution time. The bandwidth reservation task is associated with at least one RT task in the RT waiting queue and is defined by one or more of the following: the same arrival time as at least one RT task, an empty execution time, and a deadline corresponding to the sum of the arrival time of at least one RT task, the execution deadline of at least one RT task, and the output time.

[0056] In another implementation, upon receiving an enqueue event, the bandwidth reservation module creates another bandwidth reservation task by updating the bandwidth reservation task associated with a previous subtask within the same RT task. It is understood that a given RT task with n subtasks (n >= 1) may only reserve a single, further updated reservation task. In this case, the unscheduled task may be referred to as a different application task. The other bandwidth reservation task is calculated by determining another output time for the next at least one RT task and using that other output time to update the deadline of the bandwidth reservation task, thus defining the deadline of the other bandwidth reservation task.

[0057] Then, method 200 inserts at least one real-time task along with a bandwidth reservation task into the RT waiting queue (step 207) based on the early deadline priority (EDF) criterion and the deadline of each of at least one real-time task and a bandwidth reservation task, for use by an event-based task scheduler to enable EDF-based scheduling based on enqueue and dequeue events. Furthermore, the method includes scheduling (step 209) unscheduled tasks by the event-based task scheduler based on the available time in the RT waiting queue, according to the EDF-based scheduling.

[0058] In a further embodiment, at least one real-time task, along with a bandwidth reservation task, is inserted into the RT wait queue for EDF-based scheduling. The insertion of at least one real-time task also enables the event-based task scheduler to schedule at least one RT task based on the execution deadline of at least one task in the RT wait queue. Subsequently, another unscheduled RT / NRT enqueue request can be dynamically received for scheduling the unscheduled task, which may include another real-time (RT) task or non-real-time (NRT) task from the OS runtime. The unscheduled task request arrives at a time corresponding to one or more of the reserved tasks, expired bandwidth reservation tasks, and updated bandwidth reservation tasks in the wait queue. In this embodiment, the RT or NRT task arriving at a time corresponding to a bandwidth reservation task in the RT wait queue corresponds to an executable task not present in the RT wait queue, thus corresponding to an unscheduled task.

[0059] Subsequently, for unscheduled tasks, priority inversion is performed within the available time of the RT wait queue. If the execution deadline of another RT task has elapsed before the deadline associated with the bandwidth reservation task, priority inversion is performed by scheduling another unscheduled RT task within the available time of the RT wait queue. Based on running an EDF on the Real-Time Wait Queue (RT-WQ) at this event, the available time of the RT wait queue is defined by the relaxation time of the bandwidth reservation task returned by the EDF, which provides the available time based on the selected task, reflecting either a) an unscheduled task or b) a reserved task. In other words, the available time of the RT wait queue is defined by the arrival time of the unscheduled task request triggered by the enqueue event or the current time in the case of the dequeue event. The arrival of the task is calculated based on: a) the relaxation time of the bandwidth reservation task; and b) the deadline associated with the unscheduled task. Unscheduled tasks are returned by running an EDF on the RT-WQ at this event, which provides the available time based on the selected task, reflecting either a) an unscheduled real-time task or b) a reserved task.

[0060] In another embodiment, method 200 receives from the OS runtime a task request defined by at least one non-real-time (NRT) task, wherein the at least one NRT task is queued based on an enqueue event trigger, and the at least one NRT task is inserted into an NRT waiting queue based on the arrival time of the NRT task or the worst-case execution time of the NRT task to schedule the execution of the NRT task.

[0061] In another embodiment, the task scheduler receives an RT waiting queue and an NRT waiting queue from a reserved bandwidth module. The task scheduler can be defined as a real-time task scheduler / hybrid task scheduler for a non-preemptive system, and is referred to hereinafter as the task scheduler. Tasks within the RT waiting queue are then scheduled according to EDF scheduling. Scheduling of the RT waiting queue also includes applying priority inversion to schedule one or more schedulable tasks within the available time of the RT waiting queue, wherein the available time is determined by the current time and triggered by an event corresponding to one of the enqueue or dequeue events. The available time is defined by the presence of bandwidth-reserved tasks within the RT waiting queue. Scheduleable tasks correspond to unscheduled tasks, and the RT waiting queue is a priority-based waiting queue with task priorities defined by an earlier deadline according to EDF scheduling criteria. Subsequently, at least one scheduled RT task or one or more schedulable tasks associated with the available time are submitted to the processing driver for processing. In another embodiment, optionally, if at least one scheduled RT task is determined to be unschedulable based on the deadline associated with at least one scheduled RT task, at least one scheduled RT task is canceled.

[0062] Subsequently, tasks in the NRT waiting queue are scheduled based on their sequence, and at least one scheduled NRT task is submitted to the processing driver for processing. In a further embodiment, optionally, if at least one scheduled NRT task is determined to be unschedulable based on the corresponding maximum tardiness time associated with the NRT task, at least one scheduled NRT task is cancelled.

[0063] In another embodiment, task submission based on the RT waiting queue includes applying EDF scheduling criteria to the RT waiting queue to identify the current tasks within it. Subsequently, the current tasks in the RT waiting queue are identified as at least one RT task. Based on scheduling feasibility, at least one RT task is submitted to the processing driver. This feasibility is defined at least by meeting the execution deadline of at least one RT task.

[0064] In another implementation, task submission based on the RT wait queue involves applying EDF scheduling criteria to the RT wait queue to define the current tasks within it. The current tasks in the RT wait queue are then identified as bandwidth-reserved tasks. Based on this, as part of priority inversion, schedulable RT tasks or schedulable NRT tasks are submitted for execution by the processing driver within the available time of the RT wait queue. If a schedulable task has an execution deadline before the output time associated with a bandwidth-reserved task has elapsed, and in the absence of schedulable RT and NRT tasks, an NIL task is submitted within the available time of the RT wait queue.

[0065] In another embodiment, scheduling of tasks within the NRT waiting queue includes scheduling at least one NRT task based on a non-EDF-based scheduling criterion associated with the NRT waiting queue. The non-EDF-based scheduling is defined by one or more of the following: Shortest Job First (SJF) or FIFO, and submitting at least one scheduled NRT task to a processing driver for processing.

[0066] In a further implementation, at least the submitted RT and / or NRT tasks are processed by the processing driver via the processor hardware unit. Then, an interrupt based on task completion is received from the processor hardware unit, and an interrupt handler forming part of the processing driver is invoked. Based on the interrupt handler invocation, the processing driver raises a task dequeue trigger to the task scheduler. The trigger signal corresponds to the completion of the submitted task and triggers the event-based task scheduler to select one or more of the following for submission: a bandwidth-reserved task in the RT waiting queue, an unscheduled RT task, an unscheduled NRT task, the next RT task in the RT waiting queue, and the next NRT task in the NRT waiting queue.

[0067] Method 200 further includes a task analyzer, which forms part of a task scheduler, determining the actual completion time for each task. The completion time corresponds to the difference between the planned completion time and the finish time of the task. The completion time of each task is then recorded by the task analyzer, and one or more parameters of the task scheduler are optimized to further schedule RT and NRT tasks based on the recorded completion times.

[0068] Figure 3 Examples of embodiments according to this disclosure are shown as follows. Figure 2 The detailed implementation of method 200 explained herein, and correspondingly relates to methods for real-time scheduling in non-preemptive systems. As an example, method 300 in… Figure 16 It is executed on the computing system 1600 shown.

[0069] In block 301, task initialization is performed. Initially, as... Figure 16 As shown, an enqueue event is received from the NN runtime 1603 of the computing system 1600. For example, an enqueue event can be triggered based on the occurrence of any event. Based on this, as... Figure 16As shown, the NPU driver 1601 processes task requests that will be enqueued from the task requests received from the NN runtime 1603. To do this, the NPU driver 1601 invokes the "task enqueue" function of the scheduler kernel interface to provide the task to the scheduler (or reserved bandwidth server (RBS)) 1605. As described above, block 301 corresponds to... Figure 2 Step 201.

[0070] In step 303, assess whether the task at hand is an RT task. If so, the control flow proceeds to step 305.

[0071] In step 305, corresponding to steps 203 and 205, the computing system 1600 extracts attributes of at least one RT task. Specifically, the extraction of attributes may include, for example, arrival time, execution deadline, worst-case execution time (WCET), period, and output time. (Already referred to...) Figure 4 Step 305 has been described in detail; thereafter refer to step 305 and the subsequent steps. Figure 3 The description.

[0072] Figure 4 This section provides example scenarios illustrating various attributes of a task. The task's arrival time can be determined by subtask A. j The 401 error is used to describe the task and is defined as the time when the task arrives. The execution deadline or task deadline can be determined by subtask A. j The 403 parameter is used to describe this, and is defined as the time that an RT task must complete. Furthermore, the worst-case execution time (WCET) can be defined as the maximum time a task takes to fully execute on a computational unit. Therefore, it can be seen that for subtask A... j WCET is 2 ms. Inter-job time can be defined as the arrival interval between consecutive tasks. For example, subtask A... j Sub-task A j The time between jobs +1 is 7 ms. As another example, the output time can be defined as the difference between the (j+1)th worst-case scheduling time and the jth deadline. Therefore, the offset of the maximum schedulable time of the next task relative to the current task's deadline is provided by the following formula:

[0073] arrival j + deadline j + yieldTime j = MaxSchedulingTime j+1 --------(1)

[0074] The process of extracting and calculating various attributes described above corresponds to Figure 2 The box 203.

[0075] Continuing with step 305, a reservation task is created, whose deadline is equal to the sum of the task's deadline and its output time (D). resv = D + Y). As an example, without departing from the scope of this disclosure, the reservation task is alternatively referred to as a bandwidth reservation task in the description. The bandwidth reservation task can be configured by the RBS (or the bandwidth reservation module) 1605. For each RT task, the RBS 1605 additionally inserts a reservation task with the following attributes: Deadline resv = Deadline j + YieldTime j This allows for time isolation and ensures the schedulability of RT tasks. Bandwidth-reserved tasks are non-executable tasks defined by empty execution time and act as barriers with deadlines calculated from the output time.

[0076] D resv = D1 + Y1 ------------------(2)

[0077] For example, from Figure 4 In the middle, reserved task D resv It is calculated as 6 + 5 = 11 ms. Among them, the deadline is 6 ms and the output time (Y1) is 5 ms.

[0078] right Figures 5A to 8 The description further elaborates on step 305. Thereafter, regarding step 307 and subsequent stages, the process restarts. Figure 3 The description.

[0079] In another example, consider task A with 'N' subtasks, where N = 3, also as will be discussed later. Figure 6 As shown in the diagram. Therefore, task A has 3 subtasks (N=3). Aij can be a sequence relative to task A as shown below, and includes the following subtasks:

[0080] (A 11 A 12 A 13 ... (A) i1 A i2 A i3 ...

[0081] The period can be defined as the different arrival intervals between the same subtasks. The period is 32 ms, i.e., the difference between A21-A11, A22-A12, etc., or in other words, A... (i+1)j - A iThe difference between them.

[0082] Therefore, if N=1,

[0083] Then Y = P - W

[0084] Otherwise Y = D i(j+1) - D ij - W

[0085] in,

[0086] P = period

[0087] Yj - Production Time

[0088] D j+1 - Deadline for the next assignment

[0089] W j+1 - Next assignment for WCET

[0090] D j - Deadline for the next assignment

[0091] Figure 5A Visualize the output time of tasks (RT / NRT) to illustrate that output (availability) time is also periodic. Figure 16 The RBS 1605 mentioned above uses reserved tasks to reflect this timing in its logic.

[0092] In a given m RT(τ) rt" ) and n" (τ) nrt In a system with mixed NRT tasks, the i-th (τ) subtask with j subtasks can be considered. rt (Periodic RT task). For example, Artificial Intelligence (AI) Voice (AIS) could be such that the i-th task has 3 sub-tasks (j = 0, 1, 2). Therefore, as... Figure 5A As shown, the production (availability) time is also periodic. RBS 1605 uses reserved tasks to reflect this logical time.

[0093] Figure 5B An example diagram of hybrid task set scheduling using RBS according to an embodiment of this disclosure is shown. It can be seen that when T=T1, A j Arrival time is D1, production time is Y1 [task attribute]. Then, RBS 1605 will have a deadline of D1. resv The reserved task is inserted into RT-WQ, D resv = D1 + Y1 and These reserved tasks are not executable and act as obstacles. If any task has a deadline of D... kIf a task arrives within the interval (T1, T2], the RBS scheduler schedules the task (D). k < D resv Tasks can be scheduled. For example, task B2 can be scheduled by the RBS, but task B1 cannot. By reserving tasks, the need for preemption and reserving bandwidth for upcoming tasks can be eliminated. For each RT task, the RBS inserts only one reserved task, and the (j+1)th task will update the deadline of the reserved task added by the j-th task, so only one advance reservation is added at a time.

[0094] Figure 6 An example diagram illustrating output time calculation according to an embodiment of this disclosure is shown. Here, the output time, reflecting the "available" time required by the RBS, is calculated. Tasks have attributes such as WCET, period, and deadline, which are known to a real-time scheduler like the EDF algorithm. The output time of the RBS can be calculated using the following formula:

[0095] Y j = D j+1 - W j+1 - D j ---------(3)

[0096] Among them, Y j - Production time

[0097] D j+1 - Deadline for the next assignment

[0098] W j+1 - Next assignment for WCET

[0099] D j - Deadline for the next assignment

[0100] refer to Figure 6 The AIS task consists of three periodic subtasks, each with a relative deadline of 2 ms for a 1 ms WCET, and a period of 32 milliseconds. For example,

[0101] J1 arrives at sequences 8, 40, 72, 104...

[0102] J2 arrives at sequences 5, 47, 79, 111...

[0103] J3 arrived at sequences 22, 54, 86, 118...

[0104] Therefore, the output time from Equation 3 can be calculated, and thus, as shown in Table 1, the absolute output times at points 16, 23, 41, 48, 55, and 73 (second to last column) can be calculated based on this.

[0105] Table 1

[0106]

[0107] The timestamp reflects the time of production / availability. (Already...) Figures 7 to 8 The upcoming description details the RBS1605 insertion of reserved tasks with deadlines.

[0108] Figures 7-8 Various example diagrams are shown illustrating implementations of an RBS-based scheduler according to embodiments of this disclosure.

[0109] Figure 7 In an example scenario, at T=T1, task A1 inserts reserved task A1resv into the waiting queue with a deadline of T2.

[0110] Figure 8 Consider an example scenario where task A2 arrives at T=T2. Therefore, at T=T2, task A2 updates the reserved task A1resv to A2resv in the waiting queue, with a deadline of T3.

[0111] Refer back Figure 3 Regarding step 307, this scenario corresponds to the post-reservation task creation phase. Once a reserved task is created, RBS 1605 inserts or adds at least one real-time task along with the bandwidth reserved task into the RT waiting queue based on the Early Deadline First (EDF) criterion, according to the deadlines of at least one real-time task and each of the bandwidth reserved task, for use by the event-based task scheduler to enable EDF-based scheduling based on enqueue and dequeue events. Specifically, if the task has a previous reserved task, its deadline is updated. RBS 1605 maintains that only one reserved task exists for a task (a task including all subtasks). The task and the reserved task are then inserted into their respective RT waiting queues. The current step 307 corresponds to Figure 2 Step 207.

[0112] In a further implementation, at block 309, if it is determined in block 303 that a task is an NRT task, then it is directly added to the NRT-WQ. No reserved tasks are added for NRT tasks. At least one NRT task is inserted into the NRT waiting queue to schedule the execution of NRT tasks based on the arrival time of the NRT task or the worst-case execution time of the NRT task.

[0113] Therefore, once tasks have been added to their respective queues, control is transferred via step 311 to step 313, which can be referred to as "scheduling jobs". According to embodiments of this disclosure, computing system 1600 uses a multi-(2) level hierarchical scheduler for non-preemptive computing units, as will be discussed later. Figure 9 As shown in the diagram. In step 313, scheduler 1605 has two priority work queues: 1. Real-time wait queue (RT-WQ); 2. Non-real-time wait queue (NRT-WQ). These queues will hold all pending tasks, and the scheduler will select a suitable task to submit to the computing unit for execution.

[0114] Therefore, as in Figure 9 as well as Figure 17 , Figure 18A and Figure 18B Further reference in the description, in block 313, scheduler 1605 schedules jobs. Scheduler 1605 finds the optimal task through hierarchical logic. This can be an RT task, an NRT task, or even "none". "None" indicates that the pending task does not meet the output (available) time, so the process ends. Since the scheduler is event-driven, in this case, the task will be selected when an NPU interrupt occurs. However, if a "schedulable" job is found, "Submit Task" submits the task back to the NPU driver. The NPU driver then runs the job. The interrupt will mark the job's completion (Task Dequeue interface). Therefore, if a schedulable job is found, the process moves to block 315 to execute the submitted task. Furthermore, if the RT-WQ is found to be empty, the scheduler will move to perform hierarchical scheduling, such as... Figure 9 As shown.

[0115] In summary, inserting at least one real-time task along with a bandwidth-reserved task into the RT waiting queue for EDF-based scheduling corresponds to... Figure 3 And will Figure 9 The description provides a detailed explanation of the hierarchical scheduling performed.

[0116] Figure 9 The first and second level scheduling mechanisms of the scheduler 1605 according to an embodiment of this disclosure are shown. The second level scheduling process 1000 is executed by the scheduler 1605 of the computing system 1600. As described above, in block 313, if RT-WQ is found to be empty, the scheduler will switch to executing NRT-WQ. In principle, the two-level scheduling can be executed as follows.

[0117] Priority access or first-level scheduling:

[0118] The waiting queue is processed in the following order according to priority.

[0119] PriorityRT Priority NRT [Waiting Queue]

[0120] Second-level or second-level scheduling:

[0121] The scheduler applies a "scheduling scheme".

[0122] For RT-WQ: Apply EDFF or

[0123] NRT-WQ: First apply Shortest Job First / FIFO

[0124] In the operation, at block 1003, scheduler 1605 determines whether the RT-WQ is empty. If the RT-WQ is not empty, it means that the first-level scheduling's "pass-through" takes effect. Scheduler 1605 applies (step 1005) an EDF to the RT-WQ to find a suitable task. It further determines whether the selected task has a reserved task (step 1009). If the selected task is determined to be an RT task, it further determines at block 1013 whether the RT task is schedulable. If the task is schedulable, i.e., if its deadline can be met, then at block 1015, the task is submitted or run. Otherwise, if the retrieved task is a reserved task, it means that bandwidth is reserved for a higher-priority task. During this "available / idle time," the scheduler attempts to submit other schedulable tasks as long as it does not affect other deadlines, so priority inversion occurs (step 1011), corresponding to Figure 2 Step 209. Scheduler 1605 thus provides optimal utilization (in the case of priority inversion) and therefore submits or runs another task (in step 1015).

[0125] Now, if the RT-WQ is empty but the NRT-WQ is not empty (step 1007), then the second pass representing the second level of scheduling begins to take effect in step 1007. Scheduler 1605 uses the NRT-WQ and applies different strategies (shortest job first / FIFO) to find schedulable jobs and submit them to the NPU driver. If both are empty or no best schedulable task is returned, the state of scheduler 1605 is set to idle.

[0126] In summary, steps 1003 and 1007 refer to priority passing or first-level scheduling, while Figure 9 The remaining steps, such as 1005, 1009, 1011, 1013, and 1015, represent the second-level scheduling.

[0127] Figures 10A-14B Various example diagrams are shown of an implementation based on scheduler 1605 according to embodiments of this disclosure.

[0128] Figure 10A and Figure 10B Consider an example scenario where task B arrives between T=T2 and T=T3. Describe the current level of development. Figure 10A As shown, task B is scheduled immediately upon arrival. However, according to... Figure 10B In the embodiment of this disclosure shown, task B is "bare" by A2resv and therefore not scheduled immediately. As discussed later, B is scheduled after a delay to allow task A2 to occur first.

[0129] Figure 11A and Figure 11B This involves an example scenario where task A3 arrives at time T=T3. Describe the current level of development. Figure 11A As shown, task A3 cannot run until task B completes. However, according to... Figure 11B In the embodiment of the present invention shown, when T=T3, task A3 updates the reserved task A2resv in the waiting queue to A3resv, and the deadline of the reserved task is extended until T1 appears as the next cycle.

[0130] Figure 12A and Figure 12B This involves another example scenario, such as describing the current level of development. Figure 12A As shown, task A3 cannot run until task B completes. However, according to... Figure 12B In the embodiment of this disclosure shown, task B is scheduled after A3.

[0131] Figure 13A and Figure 13B This involves another example scenario, such as describing the current level of development. Figure 13A As shown, task B is complete, and A3 is now scheduled. However, A3 has passed its deadline. However, according to... Figure 13B In the embodiment of this disclosure shown, task B is scheduled after A3. Therefore, both A3 and B are scheduled based on their deadlines.

[0132] Figure 14A and Figure 14B This involves another example scenario, such as describing the current level of development. Figure 14A As shown, any scheduled task (RT) with a sufficiently long runtime can cause other scheduled tasks to miss their deadlines. Task A3 (Job A, Job 3) was not scheduled and missed its deadline D3. However, according to... Figure 14B In the embodiment of this disclosure shown, scheduler 1605 uses RBS to insert reserved tasks with deadlines. Tasks are selected from the waiting queue using EDF. Therefore, both A3 and B are scheduled based on their last available time.

[0133] Figure 15 An example diagram of an event sequence according to an embodiment of the present disclosure is shown. In the example diagram, the arrival of tasks T1, T2, and T3 can be considered in the order of T1, T2, and T3. The operations according to this diagram can be referred to as the sequence of steps (a) to (f).

[0134] Initially, in step (a), scheduler 1605 is idle.

[0135] Then in step (b), task 1, i.e., T1 (deadline: D1), is enqueued [RT-WQ with D1 & D1resv (=D1+Y1).

[0136] Subsequently, in step (c), D1 (task T1) is scheduled by EDF, i.e., D1 < D2. resv < D3 < D2 < D2 resv <D3 resv .

[0137] In step (d), RT-WQ is given a deadline of D1. resv The reserved task (from task T1).

[0138] In step (e), task T2 (deadline: D2) is enqueued.

[0139] In step (f), D2 & D2resv (D2+Y2) are enqueued into RT-WQ. [RT-WQ has D1resv, D2, D2resv].

[0140] In step (g), since D1resv is minimum, EDF returns D1resv and keeps the scheduler running.

[0141] Now in step (h), task T3 (deadline: D3) is enqueued. D3 & D3resv are enqueued to RT-WQ[D1resv is updated to D3resv (D3+Y3).

[0142] In step (i), D3 (task 3) is enqueued by EDF.

[0143] Therefore, in step (j), task T2 is enqueued.

[0144] In summary, the execution order of tasks is T1, then T3, then T2. ​​Therefore, the reserved tasks provided by the RBS mechanism ensure that prior guarantees are maintained for multiple RT tasks in a non-preemptive system. RBS handles dynamic task arrival and outperforms current developments—budget-based bandwidth reservation, which fails for non-preemptive systems. Furthermore, RBS uses the concept of output time and associates reserved tasks with a deadline Dresv. Dresv ensures that tasks with later deadlines are not scheduled. Therefore, it provides a guarantee for task A. j +1 reserved bandwidth, such as Figure 4 , Figure 5A and Figure 5B As shown. Furthermore, by applying, as Figure 5A and Figure 5B The priority inversion shown allows the scheduler (RBS) to achieve maximum utilization without compromising schedulability.

[0145] Figure 16 An architecture of a computing system according to an embodiment of this disclosure is illustrated. In one embodiment, the computing system 1600 may include major components including an H / W layer 1607, a driver 1609, an NN runtime 1603, and an application layer 1611. In one embodiment, the H / W layer 1607 includes NPU hardware 1613 and an image quality / DSP 1615. The NPU hardware 1613 is a non-preemptive computing processor for AI / ML workloads, and the image quality / DSP 1615 is used for multimedia pipelines. Furthermore, the driver 1609 may include an NPU driver 1601, a scheduler (or RBS) 1605, and a media driver 1617. In this embodiment, various software stacks utilize hardware. This layer schedules task requests. A scheduler (or RBS) 1605 is added to control the scheduling logic to support real-time clients. The scheduler is configured to execute the above... Figure 2 and Figure 3 The method described in the document. Furthermore, the NN runtime 1603 is configured for a middleware framework that exposes platform-specific NN APIs and connects to the driver layer. Additionally, the application layer 1611 includes a collection of real-time / non-real-time clients that wish to use NPU capabilities to process their workloads.

[0146] Figure 17Various components of a scheduler (or RBS) 1605 according to embodiments of this disclosure are illustrated. Scheduler 1605 may include interface 1701, a reserved bandwidth server 1703, a task scheduler 1705, and a task analyzer 1707. In one embodiment, interface 1701 is a kernel interface with an NPU driver to support request scheduling via the RT scheduler. As described above, the reserved bandwidth server (RBS) 1703 creates reserved tasks according to steps 201 to 207, the reserved tasks having deadlines calculated based on task attributes. Furthermore, the reserved tasks are used for RT tasks in the RT work queue. Task scheduler 1705 provides, as... Figure 9 The first and second level scheduling explained in the document, combined with, for example, Figure 2 The priority inversion mechanism described in step 209. Task analyzer 1707 analyzes the actual processing time (complete-start) and provides feedback to the scheduler.

[0147] Figure 18A and Figure 18B A detailed implementation of the computing system and task dequeueing process according to embodiments of the present disclosure is shown.

[0148] Figure 18A Involving depiction Figure 16 and Figure 17 The detailed design of the control flow under the implementation method is described below. In one embodiment, the scheduler 1801 is event-driven based on an enqueue event of an NPU task request, such as that shown in step 001, or a dequeue ISR event received from an NPU hardware interrupt. As a further example, in step 001, the NPU task input is a request with attributes (output time) from the NN runtime.

[0149] Subsequently, in step 002, the NPU driver 1811 processes the request as follows. In step 002a, the NPU driver 1811 enqueues the task request using the scheduler interface. In step 002(b), if the task is real-time, the RBS creates a reserved task with a deadline (D+Y) using the output time from the task attributes. This reserved task is not executable and is reserved only for upcoming tasks. The output time (Y) indicates how much bandwidth is available for other application tasks. At any given time instance, only one reserved task exists for a "real-time task". If a task has already been reserved from a previous request, its deadline is updated with a new deadline to reflect the new bandwidth reservation. In step 002c, the scheduler inserts the task with the corresponding deadline and the reserved task into the appropriate (RT / NRT) work queue.

[0150] Steps 001 and 002 correspond to the reference. Figure 2Steps 201 to 207 are performed to create an RT / NRT work queue so that scheduler 1801 can invoke a "scheduled job".

[0151] Subsequently, in step 003, if scheduler 1801 is idle, scheduler 1801 uses Early Deadline First (EDF) to select the next schedulable task. If a reserved task that cannot be executed has the earliest deadline, scheduler 1801 applies priority inversion to find other schedulable tasks; otherwise, it selects the selected task for scheduling and submits it to the driver for execution. If a task is selected, the scheduler state changes to busy. If no task is selected, it waits for the next enqueuing event. Subsequently, in step 003a, the NPU analyzer begins accounting from the current time. Then, in step 003b, scheduler 1801 submits the task to NPU driver 1811 to run it. Subsequently, in step 003c, NPU driver 1811 submits the task to NPU hardware 1809 to execute / run the job.

[0152] Step 003 corresponds to Figure 2 See step 209 for reference.

[0153] In step 004, upon task completion, the NPU hardware 1809 triggers an interrupt, and the NPU ISR is invoked. More specifically, at least the submitted RT and / or NRT tasks are processed by the NPU driver 1811 via the NPU hardware 1809. When a task completion-based interrupt is received from the NPU hardware 1809, the interrupt handler or NPU ISR that forms part of the NPU driver 1811 is invoked.

[0154] In step 004a, based on an interrupt handler or NPU ISR call, the NPU driver 1811 calls a dequeue task to update the status of the scheduler 1801. The dequeue trigger signals the completion of the submitted task and triggers the event-based scheduler 1801 to select the next task for submission. The next task can be one or more of the following:

[0155] Bandwidth reservation tasks in the RT waiting queue;

[0156] Unscheduled RT tasks;

[0157] Unscheduled NRT tasks;

[0158] The next RT task in the RT waiting queue; and

[0159] The NRT is waiting for the next NRT task or NIL task in the NRT queue.

[0160] In an implementation that is part of step 004a, the NPU driver 1811 calls the "task dequeue" interface upon receiving an interrupt to signal the completion of the submitted task.

[0161] In step 004b, the scheduler 1801 is set to idle. The task analyzer 1807 completes the recording and analysis (dequeue-schedule), i.e., the actual processing time. The task analyzer 1807 determines the actual completion time of each task, which corresponds to the difference between the task's scheduling time and its end time. The task analyzer 1807 records the completion time of each task.

[0162] Because scheduler 1801 is event-driven, the NPU ISR processor relays completion notifications by calling the "task dequeue" interface. Task analyzer 1807 dynamically determines the "actual" completion time of each task. It stores historical data on the "completion" times of the last N (most recent) tasks. This dynamic data can be fed back to the scheduler to "optimize" and "enhance" task attributes, leading to better runtime decisions. Alternatively, it can be used by the system to adjust parameters. If there are pending requests in the RT / NRT work queue, the scheduler is invoked again to schedule the job. Therefore, task dequeue and task enqueue are the two interfaces used by scheduler 1801 to schedule tasks.

[0163] In step 004c, the task analyzer 1807 sends updated task parameters, such as (WCET), and optimizes the task scheduler 1805.

[0164] Figure 18B It shows the basis Figure 18A The operation interactions of steps 004a to 004c describe the operation of scheduler 1801.

[0165] Step 1902 corresponds to step 004a, where the NPU driver 1811 calls the "task dequeue" interface upon receiving an interrupt to signal the completion of the submitted task. Because the scheduler 1801 is event-driven, the NPU ISR processor relays completion notifications by calling the "task dequeue" interface.

[0166] Step 1904 corresponds to step 004b, where the task analyzer 1807 in scheduler 1801 dynamically determines the "actual" completion time of each task. It stores historical data on the "completion" times of the last N (most recent) tasks. This dynamic data can be fed back to scheduler 1801 to "optimize" and "enhance" task attributes, thereby making better runtime decisions, or to adjust parameters.

[0167] Step 1906 corresponds to steps 004b and 004c. If there are pending requests in the RT / NRT work queue, "Schedule Job" will be called again. In summary, task dequeueing and task enqueueing are the two interfaces used by the RBS scheduler 1803 to schedule tasks.

[0168] Figure 19 A representative architecture 2400 is shown, which provides the tools and development environment described herein for the technical implementation of servers, controllers, and nodes in a mesh network via computing devices. Figure 19 This is merely a non-limiting example, and it should be understood that many other architectures can be implemented to facilitate the functionality described herein. This architecture can be used in, for example... Figure 19 It executes on the hardware of a computer machine, which includes a processor, memory, and various dedicated hardware components.

[0169] Architecture 2400 may include an operating system, libraries, frameworks, or middleware. An operating system can manage hardware resources and provide common services. An operating system may include, for example, a kernel, services, and drivers that define hardware interface layers. Drivers may be responsible for controlling or interfacing with the underlying hardware. For example, depending on the hardware configuration, drivers may include display drivers, camera drivers, Bluetooth® drivers, flash memory drivers, serial communication drivers (e.g., Universal Serial Bus (USB) drivers), Wi-Fi drivers, audio drivers, power management drivers, and so on.

[0170] The hardware interface layer includes libraries, which may include system libraries such as file systems (e.g., the C standard library) that provide functions such as memory allocation, string manipulation, and mathematical functions. Additionally, libraries may include API libraries, such as audiovisual media libraries (e.g., multimedia databases supporting the presentation and manipulation of various media formats such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG), database libraries (e.g., SQLite providing various relational database functionalities), and network libraries (e.g., WebKit providing web browsing functionality), etc.

[0171] Middleware can provide higher-level public infrastructure, such as various graphical user interface (GUI) functions, advanced resource management, advanced location services, and so on. Middleware can provide various additional APIs that can be utilized by applications or other software components / modules, some of which may be specific to a particular operating system or platform.

[0172] As used in this disclosure, the term "module" can refer to a unit that includes one or any combination of hardware, software, and firmware. For example, a module can be used interchangeably with a unit, logic, logic block, component, or circuit. A module can be the smallest unit or part of performing one or more specific functions. A module can be formed mechanically or electronically. For example, the modules disclosed herein can include at least one of known or to be developed ASIC (Application-Specific Integrated Circuit) chips, FPGA (Field-Programmable Gate Array), and programmable logic devices.

[0173] Furthermore, architecture 2400 depicts a collection of mechanisms based on audio / video processing devices and ML / NLP-based mechanisms according to embodiments of this disclosure. The user interface defined as input and interaction 2401 refers to the overall input. It may include one or more of the following: touchscreen, microphone, camera, etc. A first hardware module (e.g., ML-specific H / W) 2402 depicts dedicated hardware for ML / NLP-based mechanisms. In the example, the first hardware module 2402 includes one or more of a neural processor, FPGA, DSP, GPU, etc.

[0174] The second hardware module (e.g., H / W) 2412 describes dedicated hardware for performing data segmentation and transmission. The ML / NLP-based framework and API 2404 corresponds to the hardware interface layer for performing ML / NLP logic 2406 based on the underlying hardware. In the example, the framework can be one or more of the following: TensorFlow, Cafe, NLTK, GenSim, ARM Compute, etc. The simulation framework and API 2414 can include one or more of the following: audio kernel, audio toolkit, Unity, Unreal, etc.

[0175] Database 2408 describes a pre-trained database. ML / NLP logic 2406 can access database 2408 remotely via the cloud. In other examples, based on usage statistics, database 2408 may reside partly in the cloud and partly on the device.

[0176] Another database, 2418, references this storage. Database 2418 can be accessed remotely via the cloud. In other examples, based on usage statistics, database 2418 may reside partly in the cloud and partly on the device.

[0177] A display module 2405 is provided for displaying audio output and triggering further practical operations. The display module 2405 can function as a display / touchscreen, monitor, speaker, projection screen, etc.

[0178] General-purpose hardware and driver module 2403 corresponds to Figure 20The computer system 2500 mentioned in the text, and the drivers for general-purpose hardware units and special-purpose units (2402, 2412) are instantiated.

[0179] In this example, the ML mechanism underlying this 2400 architecture can be remotely accessible and cloud-based, allowing for remote access via a network connection. Audio / video processing devices can be configured for remote access to the NLP / ML modules, and the simulation modules can include skeletal components such as microphones, cameras, screens / monitors, and speakers.

[0180] Furthermore, at least one of the multiple modules of the mesh network can be implemented using AI based on ML / NLP logic 2406. AI-related functions can be executed using non-volatile memory, volatile memory, and a processor constituting the first hardware module 2402 (i.e., dedicated hardware for the ML / NLP-based mechanism). The processor can include one or more processors. Here, the one or more processors can be general-purpose processors (such as central processing unit (CPU), application processor (AP), etc.), graphics-specific processing units (such as graphics processing unit (GPU), vision processing unit (VPU)), and / or AI-specific processors (such as neural processing unit (NPU)). The aforementioned processors collectively correspond to... Figure 20 The processor is 2502.

[0181] One or more processors control the processing of input data based on predefined operating rules or artificial intelligence (AI) models stored in non-volatile and volatile memory. These predefined operating rules or AI models are provided through training or learning.

[0182] Here, "provided through learning" means that predefined operating rules or AI models with desired characteristics are generated by applying learning logic / techniques to multiple learning data sets. "Acquired through training" means that predefined operating rules or AI models configured to perform desired characteristics (or purposes) are obtained by training a basic AI model with multiple training data sets using training techniques. This learning can be performed within the device (i.e., architecture 2400 or computer system 2500) that performs AI according to the embodiment, and / or can be implemented via a separate server / system.

[0183] AI models can consist of multiple neural network layers. Each layer has multiple weight values, and neural network layer operations are performed by computation between the results of the previous layer and the operations of the multiple weights. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q-networks.

[0184] ML / NLP Logic 2406 is a method for training a predetermined target device (e.g., a robot) using multiple learning datasets to enable, allow, or control the target device to make determinations or predictions. Examples of learning techniques include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0185] Figure 20 Further exemplary implementations of embodiments of the present invention are illustrated, along with another typical hardware configuration of architecture 2400 in the form of a computer system 2500. The computer system 2500 may include a set of instructions that can be executed to cause the computer system 2500 to perform any one or more of the disclosed methods. The computer system 2500 may operate as a standalone device or may be connected to other computer systems or peripheral devices, for example, via a network.

[0186] In a networked deployment, computer system 2500 can operate as a server, or as a client user computer in a server-client user network environment, or as a peer-to-peer (or distributed) computer system in a peer-to-peer (or distributed) network environment. Computer system 2500 can also be implemented as various devices or combined across various devices, such as personal computers (PCs), tablet PCs, personal digital assistants (PDAs), mobile devices, handheld computers, laptop computers, desktop computers, communication devices, wireless telephones, landline telephones, network devices, network routers, switches or bridges, or any other machine capable of executing a set of instructions (sequential or otherwise) specifying the actions to be taken by that machine. Furthermore, although a single computer system 2500 is shown, the term "system" should also be considered to include any collection of systems or subsystems that individually or jointly execute one or more sets of instructions to perform one or more computer functions.

[0187] Computer system 2500 may include processor 2502, such as a central processing unit (CPU), graphics processing unit (GPU), or both. Processor 2502 can be a component in a variety of systems. For example, processor 2502 may be part of a standard personal computer or workstation. Processor 2502 may be one or more general-purpose processors, digital signal processors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), servers, networks, digital circuits, analog circuits, combinations thereof, or other devices now known or later developed for analyzing and processing data. Processor 2502 may implement software programs, such as manually generated code (i.e., programmed).

[0188] Computer system 2500 may include memory 2504, such as memory 2504 that can communicate via bus 2508. Memory 2504 may include, but is not limited to, computer-readable storage media, such as various types of volatile and non-volatile storage media, including but not limited to random access memory, read-only memory, programmable read-only memory, electrically programmable read-only memory, electrically erasable read-only memory, flash memory, magnetic tape or disk, optical media, etc. In one example, memory 2504 includes a cache or random access memory for processor 2502. In alternative examples, memory 2504 is separate from processor 2502, such as processor cache memory, system memory, or other memory. Memory 2504 may be an external storage device or database for storing data. Memory 2504 is used to store instructions executable by processor 2502. The functions, actions, or tasks shown or described in the figures can be performed by a programmed processor 2502 for executing instructions stored in memory 2504. Functions, actions, or tasks are independent of a specific type of instruction set, storage medium, processor, or processing strategy, and can be executed individually or in combination by software, hardware, integrated circuits, firmware, microcode, etc. Similarly, processing strategies can include multiprocessing, multitasking, parallel processing, etc.

[0189] As shown in the figure, computer system 2500 may or may not further include a display 2510, such as a liquid crystal display (LCD), an organic light-emitting diode (OLED), a flat panel display, a solid-state display, a cathode ray tube (CRT), a projector, a printer, or other display devices now known or later developed for outputting specific information. Display 2510 may serve as an interface for a user to view the functions of processor 2502, or specifically as an interface to software stored in memory 2504 or drive unit 2516.

[0190] Furthermore, computer system 2500 may include an input device 2512 configured to allow a user to interact with any component of computer system 2500. Computer system 2500 may also include a disk or optical drive unit 2516. Disk drive unit 2516 may include a computer-readable medium 2522 in which one or more sets of instructions 2524, such as software, may be embedded. Furthermore, the instructions 2524 may specifically implement one or more of the described methods or logic. In a particular example, during execution by computer system 2500, the instructions 2524 may reside wholly or at least partially within memory 2504 or processor 2502.

[0191] This invention envisions a computer-readable medium comprising instructions 2524, or receiving and executing instructions 2524 in response to a propagation signal, such that a device connected to a network 2526 can transmit voice, video, audio, images, or any other data through the network 2526. Furthermore, instructions 2524 can be sent or received on the network 2526 via a communication port or communication interface 2520 or using a bus 2508. The communication port or communication interface 2520 may be part of a processor 2502 or may be a separate component. The communication interface 2520 may be created in software or may be a physical connection in hardware. The communication interface 2520 may be configured to connect to the network 2526, external media, a display 2510, or any other component or combination thereof in a computer system 2500. The connection to the network 2526 may be a physical connection, such as a wired Ethernet connection, or may be established wirelessly as discussed later. Similarly, additional connections to other components of the computer system 2500 may be physical or wirelessly established. The network 2526 may also be directly connected to the bus 2508.

[0192] Network 2526 may include a wired network, a wireless network, an Ethernet AVB network, or a combination thereof. The wireless network may be a cellular telephone network, 802.11, 802.16, 802.20, 802.1Q, or WiMax network. Furthermore, network 826 may be a public network such as the Internet, a private network such as an intranet, or a combination thereof, and may utilize various network protocols currently available or developed in the future, including but not limited to TCP / IP-based network protocols. The system is not limited to operating using any particular standard and protocol. For example, standards for transmission on the Internet and other packet-switched networks (e.g., TCP / IP, UDP / IP, HTML, and HTTP) may be used.

[0193] AI Voice is a real-time application requiring strict deadline guarantees. Due to the limitations of AI Voice (using an NPU), inference is performed at approximately 8ms intervals; otherwise, audio lag / skipping would jeopardize system operation. The RBS scheduler uses these parameters and ensures prior guarantees so that all jobs complete successfully on time. RBS can schedule other tasks within the output time while maintaining these guarantees.

[0194] Figure 21 Example applications of AI decoders and decoder forms based on this topic are shown.

[0195] The AI ​​decoder described in this topic provides an AI-based 4K streaming solution to overcome network bandwidth limitations. This also corresponds to image pre / post-processing compression techniques based on deep learning for standard codec interoperability. In another example, the AI ​​decoder uses an NPU to render 2K to 4K upscale on devices such as TVs.

[0196] The amplification pipeline consists of multiple hardware components for processing (a pipeline defined by a video decoder, preprocessing, NPU, post-processing, and display), executed for a single frame output / display. In operation, for example at 30 fps, all components need to complete their execution within strict timing constraints. The image is divided into multiple tiles and needs to be completed within, for example, 32 milliseconds based on 30 fps. The scheduler needs to ensure the NPU completes the job at the precise deadline. No other tasks are expected to cause missed deadlines, resulting in incorrect / delayed video frames. The RBS based on this topic can schedule AID and other hybrid tasks to improve NPU utilization.

[0197] Figure 22A and Figure 22B An example application of an AI enhancer (AIE) for images based on this topic is shown.

[0198] Figure 22A This involves AI Enhancer (AIE) applications that act as NRT applications. As part of SISR (Single Image Super-Resolution), the display quality of the images may be lower than that of television resolutions (4K / 8K) (2K / 4K). Therefore, the images always need to be processed before considering H / W scalers and image quality engines. As part of the single image super-resolution implementation on NPU devices, simultaneous operation with AIS is achieved by applying 250x250 size slices four times to thumbnail images of various sizes. While non-real-time (NRT) applications do not have strict deadlines, they can also be scheduled as real-time running tasks, according to this topic.

[0199] Figure 22B Representations for hybrid task scheduling include, for example: Figure 22AThe diagram illustrates the scheduling of RT applications (e.g., AIS) and NRT applications (AIE). In AI-enabled voice enhancement, scheduling logic may be urgently needed, where NPU utilization can be improved by giving other clients (i.e., tasks) reasonable changes to prevent starvation. The RBS scheduler in this topic employs, for example... Figure 22B The architecture shown demonstrates optimal scheduling in this hybrid task scheduling.

[0200] Figure 23 This paper demonstrates scene classification through real-time application based on this topic, thereby improving the image quality performance of DLSR through the development of an image quality classification network. More specifically, a flexible DLSR image engine is applied to scene classification features. Scene classification is performed by resolution / source / image quality via AI-based scene classification.

[0201] Application scenario classification is a real-time application that can be scheduled alongside other AI applications with strict deadlines. For scenario classification and other tasks, the RBS scheduler provides optimal solutions by offering dynamically changing scheduling decisions.

[0202] Figure 24A , Figure 24B and Figure 24C An example environment illustrating a mixed scenario of scheduling real-time and non-real-time applications is presented. Given this mixed scenario, a fixed-priority scheduling scheme cannot provide an optimal solution.

[0203] As can be observed, fixed-priority scheduling schemes do not provide the optimal solution. The RBS scheduler can dynamically handle task arrivals and schedule tasks in the correct order.

[0204] Figure 24A This involves the scheduling of the RT task AIS, which includes three subtasks.

[0205] Figure 24B This involves the simultaneous scheduling of two RT applications in RT-WQ. Task 1 corresponds to AIS, while Task 2 corresponds to another RT application, AID (33 FPS). Figure 21 As discussed in [the document]. Therefore, Figure 23 This refers to a hybrid scheduling scenario.

[0206] Figure 24C Another hybrid scheduling scenario is involved, which involves the simultaneous scheduling of two RT applications in RT-WQ. Task 1 corresponds to the first RT application AIS, while Task 2 corresponds to another RT application that corresponds to the scenario classification.

[0207] Output time and reserved tasks provide guarantees and enable efficient scheduling of all RT / NRT mixed cases. RBS provides a method for handling such varying scheduling requirements by utilizing the concept of reserved tasks that reflect output time.

[0208] Figure 25A , Figure 25B and Figure 25C An example environment of a system with only NRT tasks is shown.

[0209] Figure 25A This involves another hybrid scheduling scenario, namely, the simultaneous scheduling of a real-time application as an AIS and an NRT application as an AIE. Therefore, for real-time tasks, scheduler 1801 can effectively handle the schedulability issues of non-preemptive systems like the NPU.

[0210] Figure 25B This involves scheduling of NRT applications (corresponding to AI coaching at 10 FPS). Figure 25C This involves scheduling for NRT applications (corresponding to 4FPS face detection applications). In this scenario, scheduler 1801 outputs the same as the default NPU driver, i.e., FIFO.

[0211] The scheduling design of this topic can handle a variety of possible hybrid UX scenarios, where some models / tasks, such as AIS, run in the background. Furthermore, non-real-time tasks can still be executed without starvation.

[0212] While specific language has been used to describe this disclosure, it is not intended to impose any limitations. It will be apparent to those skilled in the art that various working modifications can be made to the method to realize the inventive concepts taught herein.

[0213] The accompanying drawings and the foregoing description provide examples of embodiments. Those skilled in the art will understand that one or more of the described elements can be well combined into a single functional element. Alternatively, certain elements can be divided into multiple functional elements. Elements in one embodiment can be added to another embodiment. For example, the order of the processes described herein can be changed and is not limited to the manner described herein.

[0214] Furthermore, the actions in any flowchart do not need to be performed in the order shown; nor is it necessary to execute all actions. Similarly, actions that do not depend on other actions can be executed in parallel with other actions. The scope of the embodiments is by no means limited to these specific examples. Many variations are possible, such as differences in structure, size, and materials used, whether explicitly stated in the specification. The scope of the embodiments is at least as broad as that given by the appended claims.

[0215] The benefits, other advantages, and solutions to problems have been described above with reference to specific embodiments. However, the benefits, advantages, solutions to problems, and any components that may lead to or make more significant any benefit, advantage, or solution should not be construed as key, essential, or necessary features or components of any or all claims.

Claims

1. A method for scheduling real-time (RT) tasks by an electronic device including a scheduler, the method comprising: The scheduler receives task requests defined by at least one RT task from the operating system (OS) runtime, wherein the at least one RT task is queued based on an enqueue event trigger. The scheduler obtains the output time of the at least one RT task based on at least one of the execution deadline of the at least one RT task, the execution deadline of the next RT task after the at least one RT task, and the maximum execution time associated with the execution of the next RT task after the at least one RT task. The scheduler creates a bandwidth reservation task with a deadline defined by the output time for the timed execution of the next RT task after the at least one RT task, the bandwidth reservation task being non-executable and defined by an empty execution time. The scheduler inserts the at least one RT task along with the bandwidth reservation task into the RT waiting queue based on the early deadline priority (EDF) criterion and the deadline of each of the at least one RT task and the bandwidth reservation task, in order to enable EDF-based scheduling. as well as The scheduler schedules unscheduled tasks based on EDF-based scheduling and the available time of the RT waiting queue.

2. The method of claim 1, wherein, The scheduler inserts the at least one RT task along with the bandwidth reservation task into the RT waiting queue, enabling the scheduler to: Identify the RT waiting queue and the non-real-time NRT waiting queue from the reserved bandwidth server of the scheduler; Based on the confirmed existence of tasks in the RT waiting queue, first-level scheduling is performed by prioritizing the RT waiting queue over the NRT waiting queue; Execution includes one or more of the following second-level schedules: The at least one RT task is scheduled according to the execution deadline of at least one task in the RT waiting queue; The OS runtime receives an unscheduled task request for scheduling an unscheduled task, including another RT task or NRT task, the unscheduled task request arriving at a time corresponding to one or more of the bandwidth reserved task, expired bandwidth reserved task, or updated bandwidth reserved task in the waiting queue; or During the available time of the RT waiting queue, priority inversion is performed by at least one of the following: If the execution deadline of another RT task has elapsed before the deadline associated with the bandwidth reservation task, the other RT task is scheduled during the available time of the RT waiting queue, the available time of which is defined by the relaxation time of the bandwidth reservation task. and If the execution deadline of the NRT task has elapsed before the deadline associated with the bandwidth reservation task, the NRT task is scheduled during the available time of the RT waiting queue. The available time of the RT waiting queue is defined by the arrival time of unscheduled task requests triggered by enqueue or dequeue events, and is calculated based on the slack time of the bandwidth reserved task and the deadline associated with the unscheduled task.

3. The method according to claim 2, further comprising: The scheduler receives task requests defined by at least one non-real-time NRT task from the OS runtime, wherein the at least one NRT task is queued based on an enqueue event trigger. as well as Based on the arrival time of the NRT task or the worst-case execution time of the NRT task, the at least one NRT task is inserted into the NRT waiting queue for scheduling the execution of the NRT task.

4. The method of claim 1, wherein, The bandwidth reservation task associated with at least one RT task in the RT waiting queue is a non-executable task including one or more of the following: Same arrival time as the at least one RT task; Empty execution time; or The deadline corresponding to the sum of the arrival time of the at least one RT task, the execution deadline of the at least one RT task, and the output time.

5. The method according to claim 1, further comprising: The scheduler receives task requests from the OS runtime as the next at least one RT task to be queued based on an enqueue event trigger. as well as Another bandwidth reservation task is created by updating a bandwidth reservation task associated with at least one previous RT task, the other bandwidth reservation task being calculated based on: Determine the next output time for at least one RT task; and Update the deadline of the bandwidth reservation task with the other output time to define the deadline of the other bandwidth reservation task.

6. The method of claim 2, wherein, The RT task or NRT task that arrives at the time corresponding to the bandwidth reserved task in the RT waiting queue corresponds to an executable task that does not exist in the RT waiting queue, and corresponds to an unscheduled task.

7. The method according to claim 3, further comprising: The scheduler identifies the RT waiting queue and the NRT waiting queue from the scheduler's reserved bandwidth server; The scheduler schedules one or more schedulable tasks within the available time of the RT waiting queue by applying priority inversion, and schedules tasks within the RT waiting queue according to EDF scheduling, wherein the available time is determined by the current time and triggered by an event corresponding to one of the enqueue or dequeue events, the available time is defined by the appearance of the bandwidth reserved task within the RT waiting queue, the schedulable task corresponds to the unscheduled task, and the RT waiting queue is a priority-based waiting queue with task priorities defined by an earlier deadline according to EDF scheduling; The scheduler submits at least one scheduled RT task or one or more schedulable tasks to the processing driver that are associated with the available time for processing. When the at least one scheduled RT task is determined to be unschedulable based on the deadline associated with the at least one scheduled RT task, the scheduler cancels the at least one scheduled RT task. The scheduler processes tasks in the NRT waiting queue based on the sequence in the NRT waiting queue and based on at least one scheduled NRT task submitted to the processing driver. as well as Optionally, when the at least one scheduled NRT task is determined to be unschedulable based on the corresponding maximum latency associated with the NRT task, the scheduler cancels the at least one scheduled NRT task.

8. The method according to claim 7, wherein, Submitting at least one scheduled RT task includes: Apply EDF scheduling to the RT waiting queue to determine the current task within the RT waiting queue; Identify the current task in the RT waiting queue as the at least one RT task; and The at least one RT task is submitted to the processing driver based on scheduling feasibility, which is defined at least by compliance with the execution deadline of the at least one RT task.

9. The method according to claim 7, wherein, Submitting at least one scheduled RT task includes: Apply EDF scheduling to the RT waiting queue to determine the current task within the RT waiting queue; The current task in the RT waiting queue is identified as the bandwidth reserved task; Submit a schedulable RT task or a schedulable NRT task to be executed by the processing driver within the available time of the RT waiting queue, the schedulable task having an execution deadline before the elapsed output time associated with the bandwidth reservation task; and In the absence of schedulable RT and NRT tasks, submit the NIL task within the available time of the RT waiting queue.

10. The method according to claim 7, wherein, Scheduling tasks within the NRT waiting queue includes: The at least one NRT task is scheduled based on a non-EDF-based scheduling criterion associated with the NRT waiting queue, wherein the non-EDF-based scheduling is defined by one or more of Shortest Job First and FIFO; and The at least one scheduled NRT task is submitted to the processing driver for processing.

11. The method of claim 7, further comprising: The processing driver processes at least the submitted RT or NRT tasks via the processor hardware unit; Receives task-completed interrupts from the processor hardware unit and invokes the interrupt handler that forms part of the processing driver; as well as Based on the invoked interrupt handler, the processing driver issues a task dequeue trigger to the task scheduler. The trigger signaling corresponds to the completion of the submitted RT or NRT task and triggers the event-based task scheduler to select one or more of the following as the next task to be submitted: The bandwidth reservation task in the RT waiting queue; Unscheduled RT tasks; Unscheduled NRT tasks The next RT task in the RT waiting queue; The next NRT task in the NRT waiting queue; or NIL tasks determined by the task scheduler.

12. The method of claim 11, further comprising: The actual completion time of each task is determined by a task analyzer that forms part of the task scheduler; the actual completion time corresponds to the difference between the scheduling time of the task and the end time of the task. The task analyzer records the actual completion time of each task. as well as Based on the recorded actual completion time, one or more parameters of the task scheduler are optimized to further schedule RT and NRT tasks.

13. An electronic device for scheduling real-time (RT) tasks, the electronic device comprising: The scheduler is configured as follows: Receive task requests defined by at least one RT task from the operating system (OS) runtime, wherein the at least one RT task is queued based on an enqueue event trigger; The output time of the at least one RT task is obtained based on at least one of the execution deadline of the at least one RT task, the execution deadline of the next RT task following the at least one RT task, and the maximum execution time associated with the execution of the next RT task following the at least one RT task. Create a bandwidth reservation task with a deadline defined by the output time for the timed execution of the next RT task following the at least one RT task, wherein the bandwidth reservation task is non-executable and defined by an empty execution time, and According to the Early Deadline Priority (EDF) criterion, based on the deadlines of each of the at least one RT task and the bandwidth reservation task, the at least one RT task, together with the bandwidth reservation task, is inserted into the RT waiting queue to enable EDF-based scheduling. Unscheduled tasks are scheduled based on the available time of the RT waiting queue, according to the EDF-based scheduling.

14. The electronic device according to claim 13, wherein, The electronic device is also configured to operate according to the method of any one of claims 2 to 12.

Images