A heterogeneous accelerator task scheduling system

Abstract

The application provides a heterogeneous accelerator task scheduling system, which comprises a general virtual machine, a real-time virtual machine, a virtual machine monitor and a heterogeneous accelerator; the real-time virtual machine is used for declaring consumption timing information in a consumer deadline schedule; the general virtual machine is used for writing an accelerator task descriptor into a shared submission queue; an accelerator control plane supervisor is used for retrieving the consumption timing information by using a data flow identification to search the consumer deadline schedule; an effective deadline is determined based on an end-to-end deadline, an estimated delay time and an estimated processing time; a consumer-aware scheduler is used for reordering the accelerator tasks based on a consumer criticality level, the effective deadline and a task submission time, and delivering the accelerator tasks to the heterogeneous accelerator for execution. By using the above-mentioned heterogeneous accelerator task scheduling system, the problem of task scheduling failure and safety accidents caused thereby in the existing heterogeneous hybrid virtualization platform is solved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a heterogeneous accelerator task scheduling system. Background Technology

[0002] With the application of domestically produced CPUs and NPUs in fields such as autonomous driving, industrial robots, and humanoid robots, more and more systems are adopting multi-core hybrid deployment architectures. On a single hardware chip, two types of isolated virtual machines run in parallel through a Type-1 hypervisor. One type is a general-purpose virtual machine running a general-purpose operating system, responsible for initiating AI inference tasks such as image recognition and state detection, and sending computation requests to heterogeneous accelerators such as NPUs, GPUs, and DSPs through front-end drivers. The other type is a real-time virtual machine running a hard real-time operating system, internally running safety-critical containers, and relying on the inference output results from the accelerators to complete time-constrained control tasks such as servo motor control and equipment safety monitoring. The two types of virtual machines achieve strong isolation in address space, task scheduling, and hardware resources through a virtualization isolation mechanism. Inference data streams rely on shared memory and virtual interrupts to complete cross-domain transmission, forming an end-to-end business link where the general-purpose virtual machine submits inference and the real-time virtual machine consumes the inference results.

[0003] In a hybrid virtualization architecture, heterogeneous accelerators have limited computing power. When multiple AI inference tasks compete for hardware resources concurrently, the accelerator scheduling logic directly determines the timing compliance of security-critical services. Currently, the industry has developed various technical solutions focusing on NPU internal time-sharing scheduling, virtual NPU computing power partitioning, semi-virtualized device passthrough, single-core EDF deadline scheduling, and cross-virtual machine data communication.

[0004] However, existing scheduling schemes can only solve local scheduling problems within a single virtual machine domain, accelerator hardware, or CPU resources. They cannot establish cross-virtualization boundary timing links for general virtual machine inference submission and real-time virtual machine secure container consumption. They also cannot use the trusted end-to-end deadline on the consumer side as the basis for accelerator task sorting. In high-security real-time scenarios of heterogeneous hybrid virtualization platforms, they suffer from multiple technical shortcomings such as untrusted priorities, security link timeouts, compromised isolation, and abuse of resource preemption, leading to task scheduling failures and security incidents. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide a heterogeneous accelerator task scheduling system to overcome at least one of the above-mentioned defects.

[0006] In a first aspect, embodiments of this application provide a heterogeneous accelerator task scheduling system, which includes a general virtual machine, a real-time virtual machine, a virtual machine monitor, and a heterogeneous accelerator. The real-time virtual machine includes a safety-critical container, the general virtual machine includes an accelerator front-end driver, and the virtual machine monitor includes an accelerator control plane regulator. Real-time virtual machines are used to declare the consumption timing information of data streams in the consumer deadline schedule through a controlled channel provided by the virtual machine monitor when safety-critical containers subscribe to data streams output from heterogeneous accelerators. A general-purpose virtual machine used to write accelerator task descriptors to a shared commit queue via an accelerator front-end driver; The accelerator control plane regulator is used to extract data stream identifiers from accelerator task descriptors in a shared submission queue, retrieve consumer deadline schedules using data stream identifiers, and obtain consumption timing information of data streams, including end-to-end deadlines, estimated latency, and estimated processing time. The effective deadline for accelerator tasks is determined based on the end-to-end deadline, estimated delay time, and estimated processing time. Based on the consumer's criticality level, effective deadline, and task submission time, the accelerator tasks in the task queue to be submitted are reordered, and then the accelerator tasks are sequentially distributed to the heterogeneous accelerators for execution in the reordered order.

[0007] In an optional implementation, the accelerator control plane regulator determines the effective deadline for the accelerator task through the following process: determining the task completion deadline based on the sum of the task acceptance time and the end-to-end deadline; and subtracting the estimated delay time and the estimated processing time from the task completion deadline in sequence to obtain the effective deadline.

[0008] In an alternative implementation, the consumer deadline schedule allows only real-time virtual machine writes and only accelerator control plane regulator reads.

[0009] In an optional implementation, the accelerator control plane regulator is also configured to perform the following process: if no data stream identifier is found in the consumer deadline schedule, the accelerator task is placed after a designated accelerator task, which is an accelerator task for which the corresponding data stream identifier is found in the consumer deadline schedule and has not yet exceeded its valid deadline.

[0010] In an optional implementation, the consumer-aware sorter reorders the accelerator tasks in the task queue by performing the following process: determining the priority level according to three levels: consumer criticality level, effective deadline, and task submission time, and reordering the accelerator tasks in the task queue based on the priority determination results.

[0011] In an optional implementation, the accelerator control plane regulator is further configured to apply at least one of the following constraints: criticality quota constraint, consistency verification constraint, and deadline lower bound constraint. The criticality quota constraint means that each consumer or data stream, based on its consumer criticality level, receives fewer service times or accumulated service time for accelerator tasks that take precedence over those with lower consumer criticality levels than a preset quota. The consistency verification constraint constrains the criticality of the accelerator task corresponding to the first target data stream, which is a data stream for which a consumer has declared a high-urgency time-series constraint but has not yet retrieved the results. The high-urgency time-series constraint means that the consumer's criticality level is greater than a set level threshold, the effective deadline is less than a first set time threshold, and the end-to-end deadline is less than a second set time threshold. The deadline lower bound constraint means that the end-to-end deadline is greater than a minimum valid threshold, which is determined based on the consumer control cycle.

[0012] In an optional implementation, the accelerator control plane regulator is also used to perform the following processing: when a consumer subscribes to a second target data stream and no accelerator task corresponding to the second target data stream arrives within the consumer's control period, a virtual interrupt representing the lack of upstream data is injected into the real-time virtual machine so that the consumer can continue to execute according to a preset fallback strategy.

[0013] In an optional implementation, the accelerator control plane regulator is further configured to perform the following processes: reuse the contents of the first target native field in the accelerator task descriptor as a data stream identifier, or use the data in the reserved extended field of the accelerator task descriptor as a data stream identifier field, or construct a data stream identifier from multiple second target native fields in the accelerator task descriptor according to preset rules.

[0014] In an optional implementation, the accelerator control plane regulator is also configured to perform the following processing: validate the accelerator task descriptor and refuse to write the accelerator task descriptor to the shared commit queue if the validation fails.

[0015] In an optional implementation, the heterogeneous accelerator is used to perform the following process: upon completion of inference computation, a hardware completion interrupt is sent to the accelerator control plane regulator, so that the accelerator control plane regulator updates the accelerator shadow state and sends a result-ready notification to the corresponding consumer.

[0016] The embodiments of this application bring the following beneficial effects: This application provides a heterogeneous accelerator task scheduling system, including a general-purpose virtual machine, a real-time virtual machine, a virtual machine monitor, and multiple heterogeneous accelerators. The real-time virtual machine includes a safety-critical container, the general-purpose virtual machine includes an accelerator front-end driver, and the virtual machine monitor includes an accelerator control plane regulator. The real-time virtual machine is used to declare the consumption sequence information of the data stream in the consumer deadline schedule through a controlled channel provided by the virtual machine monitor when the safety-critical container subscribes to the data stream output by the heterogeneous accelerator. The general-purpose virtual machine is used to write accelerator task descriptors to a shared commit queue through the accelerator front-end driver. The accelerator control plane regulator... The surface supervisor extracts data stream identifiers from the accelerator task descriptors in the shared submission queue, retrieves the consumer deadline schedule using the data stream identifiers, and obtains the consumption timing information of the data stream, including end-to-end deadline, estimated delay time, and estimated processing time. Based on the end-to-end deadline, estimated delay time, and processing time, it determines the effective deadline of the accelerator task. Based on the consumer criticality level, effective deadline, and task submission time, it reorders the accelerator tasks in the task queue to be submitted and distributes the accelerator tasks to the heterogeneous accelerators for execution in the reordered order.

[0017] This application can isolate and store real-time domain consumption timing information through controlled channels, obtain reliable end-to-end timing while ensuring virtual machine isolation, and solve the priority distortion problem. At the same time, it calculates the effective deadline based on the timing parameters of data flow matching, and performs multi-level task rearrangement based on the effective deadline, avoiding disordered computing power preemption and eliminating the timing asynchrony between general virtual machines and real-time virtual machines. Compared with the heterogeneous accelerator task scheduling methods in the prior art, it solves the problem of task scheduling failure and security incidents in the existing heterogeneous hybrid virtualization platform.

[0018] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A schematic diagram of the heterogeneous accelerator task scheduling system provided in an embodiment of this application is shown; Figure 2A flowchart of the first-stage heterogeneous accelerator task scheduling steps provided in an embodiment of this application is shown; Figure 3 A flowchart of the second-stage heterogeneous accelerator task scheduling steps provided in an embodiment of this application is shown; Figure 4 A schematic diagram of the timing sequence of accelerator tasks provided in an embodiment of this application is shown; Figure 5 This diagram illustrates the task execution process of the prior art and the embodiments of this application under the same task deployment; Figure 6 A schematic diagram of the heterogeneous accelerator task scheduling device provided in an embodiment of this application is shown; Figure 7 A schematic diagram of the structure of the electronic device provided in the embodiments of this application is shown. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.

[0022] Please see Figure 1 , Figure 1 This is a schematic diagram of the structure of a heterogeneous accelerator task scheduling system provided in an embodiment of this application. Figure 1 As shown, the heterogeneous accelerator task scheduling system provided in this application embodiment includes a real-time virtual machine 100, a general-purpose virtual machine 200, a virtual machine monitor 300, a heterogeneous accelerator 400, a consumer deadline schedule 500, and a shared submission queue 600.

[0023] The real-time virtual machine 100 includes a safety-critical container 110, the general-purpose virtual machine 200 includes an accelerator front-end driver 210, the virtual machine monitor 300 includes an accelerator control plane supervisor 310 and a queue of tasks to be submitted 320, and the heterogeneous accelerator 400 includes multiple ones.

[0024] Real-time Virtual Machine 100 is an isolated virtual machine running on a Type-1 bare-metal hypervisor, equipped with a hard real-time operating system (RTOS). It achieves strong isolation from general-purpose virtual machines in terms of CPU, memory, and address space through virtualization mechanisms. Real-time Virtual Machine 100 deploys a kernel-level safety-critical container 110, which handles time-constrained safety-critical tasks such as servo control, safety monitoring, and emergency shutdown of industrial robots, intelligent driving systems, inspection equipment, and humanoid robots. It possesses hard real-time characteristics such as deterministic microsecond or millisecond-level scheduling cycles, controllable interrupt latency, and fault isolation, making it a trusted high-security domain within the system. Real-time Virtual Machine 100 also has a built-in end-to-end deadline declaration agent, acting as the sole entity with write permissions to the Consumer Deadline Schedule (DC-Table).

[0025] Real-time virtual machine 100 is used to declare the consumption timing information of the data stream in the consumer deadline schedule through a controlled channel provided by virtual machine monitor 300 when the safety-critical container 110 subscribes to the data stream output by the heterogeneous accelerator.

[0026] The general-purpose virtual machine 200 is an isolated guest virtual machine running on a Type-1 bare metal virtual machine monitor and equipped with a general-purpose operating system (such as Linux). It is separated from the real-time virtual machine 100 by a virtualization hardware isolation mechanism, with independent CPU and memory address spaces, and belongs to the system's low-security trust domain. The general-purpose virtual machine deploys an accelerator front-end driver 210. The accelerator front-end driver 210 is responsible for generating accelerator tasks (such as AI inference tasks such as image detection and state recognition), assembling accelerator task descriptors carrying data stream identifier stream_id and writing them to the shared submission queue 600. It is the sole submitter of accelerator tasks and does not know or have permission to read the end-to-end timing parameters of the safety-critical container 110 inside the real-time virtual machine 100. It cannot be used to execute hard real-time safety-critical business such as servo control and safety emergency stop.

[0027] The Virtual Machine Monitor 300 is a Type-1 bare-metal virtualization management program running on the hardware privilege level (EL2) of a domestically produced heterogeneous SoC chip. It is deployed directly on the hardware and does not rely on a general operating system. The Virtual Machine Monitor 300 logically isolates and divides the hardware resources of a single chip, and carries general-purpose virtual machines and real-time virtual machines in parallel, completing the virtualization isolation and management of CPU, memory, interrupts, and I / O devices. At the same time, it maintains a consumer deadline time table (DC-Table) in an independent memory space, provides a one-way controlled channel for the real-time virtual machine 100 to write timing declarations, and only grants read permissions to the accelerator control plane regulator 310, realizing secure backpropagation of end-to-end deadlines across virtual machines. It also coordinates global management logic such as accelerator control path verification, cross-virtual machine virtual interrupts, result notification forwarding, and fault detection and degradation. It is the security isolation hub and cross-domain scheduling data intermediary of the entire hybrid virtualization platform.

[0028] The accelerator control plane regulator 310 extracts data stream identifiers from the accelerator task descriptors in the shared submission queue 600, retrieves the consumer deadline schedule using the data stream identifiers, and obtains the consumption timing information of the data streams, including end-to-end deadlines, estimated latency, and estimated processing time. Based on the end-to-end deadlines, estimated latency, and estimated processing time, it determines the effective deadline for the accelerator tasks. The accelerator control plane regulator 310 can be regulated by a hypervisor, or it can be hosted by a Device Service Virtual Machine (DVM) for heterogeneous accelerators, with the hypervisor only providing isolation and forwarding.

[0029] The accelerator control plane supervisor 310 is also used to reorder the accelerator tasks in the task queue 320 based on the consumer criticality level, effective deadline, and task submission time, and then distribute the accelerator tasks to the heterogeneous accelerators 400 for execution in the reordered order.

[0030] The heterogeneous accelerator 400 is a dedicated hardware computing unit integrated into the CPU and NPU heterogeneous SoC hardware platform, independent of the general-purpose CPU core. It includes types such as NPU, GPU, DSP, VPU, and FPGA, specifically designed to handle AI inference-intensive computing tasks such as image recognition, object detection, and state analysis, thus offloading CPU computing power. Within this Type-1 virtualization platform, it is uniformly managed by the virtual machine monitor 300. It receives accelerator task descriptors from the general-purpose virtual machine 200 to execute accelerator operations. After the operation is completed, the output result data stream is supplied to the security-critical container 110 within the real-time virtual machine 100 for consumption. It serves as the hardware computing carrier connecting the inference submission business of the general-purpose virtual machine 200 with the security control business of the real-time virtual machine 100. The heterogeneous accelerator is also known as a physical heterogeneous accelerator or a physical accelerator.

[0031] The following reference Figure 2 and Figure 3 The task scheduling process of heterogeneous accelerators is explained in detail.

[0032] The heterogeneous accelerator task scheduling method in this application is generally divided into two stages: the first stage is for the consumer (a safety-critical container in a real-time virtual machine) to establish and maintain an end-to-end deadline declaration; the second stage is for the accelerator side (accelerator control plane regulator) to reverse-calculate the effective deadline based on the declaration and reorder and submit the accelerator tasks.

[0033] In the first phase, safety-critical containers within real-time virtual machines establish and maintain claim records indexed by data flow identifiers to the consumer deadline schedule (DC-Table) via a controlled channel provided by the virtual machine monitor.

[0034] Figure 2 A flowchart of the first-stage heterogeneous accelerator task scheduling steps provided in the embodiments of this application is shown, as follows: Figure 2 As shown, the first-stage heterogeneous accelerator task scheduling steps include: Step S101: Assign a unique and fixed data stream identifier to each data stream.

[0035] First, a unique stream identifier (stream_id) is defined for each data stream output by the heterogeneous accelerator. For example, during application deployment, a trusted system configuration tool or accelerator control plane supervisor can assign a unique stream identifier (stream_id) to each logical data stream based on the application's data dependency graph. The data stream configuration includes at least the stream_id, the data stream producer, the data stream consumer, the virtual machines allowed to submit the data stream task, and the real-time virtual machines or containers allowed to declare the consumer attributes of the data stream. The data stream configuration results are loaded into the accelerator front-end driver of the general virtual machine, the deadline declaration proxy of the real-time virtual machine, and the data stream registry maintained by the accelerator control plane supervisor, respectively.

[0036] This data stream identifier maintains consistency between the accelerator task descriptor submitted by the general virtual machine and the subscription declaration of the real-time virtual machine, and is used to associate accelerator tasks with their consumers. Specifically, the general virtual machine is responsible for carrying the assigned stream_id in the accelerator task descriptor; the real-time virtual machine is responsible for declaring the consumer deadline, consumer criticality level, and other consumption attributes for the same stream_id.

[0037] A data stream can generate multiple accelerator task instances periodically or through event triggering. Each accelerator task within this data stream carries the same stream_id, but has a different data instance sequence number (instance_seq), task identifier (task_id), task acceptance time, and effective deadline. The accelerator control plane supervisor associates the accelerator task with one or more consumer claims using the stream_id and selects the dominant consumer for this accelerator task.

[0038] In step S102, the safety-critical container subscription accelerator output data stream within the real-time virtual machine is written to the consumer's deadline schedule via a controlled unidirectional channel to record timing statements.

[0039] Consumers can refer to safety-critical containers or real-time tasks within a real-time virtual machine that actually subscribe to and use a data stream output by an accelerator. A consumer represents a data dependency relationship, but it does not mean that the safety-critical container must be blocked waiting for the accelerator's results. The safety-critical container still operates according to its fixed cycle and phase. When the result of the current cycle does not arrive in time, it can use the valid result from the previous cycle according to a preset safety policy, or perform conservative safety actions such as slowing down or stopping.

[0040] When a safety-critical container within a real-time virtual machine subscribes to a data stream output from a heterogeneous accelerator, it writes a record indexed by the data stream identifier stream_id to the consumer deadline schedule DC-Table through a controlled channel provided by the virtual machine monitor. In other words, it declares the consumption timing information of the data stream in the consumer deadline schedule.

[0041] Consumption timing information is a set of parameters used to characterize the real-time timing constraints on the consumption side of the real-time domain. It is the data flow consumption constraint parameter declared by the real-time virtual machine to the DC-Table. It uses the data flow identifier stream_id as a unique index and is a reliable basis for the accelerator control plane regulator to calculate the effective deadline of the task and determine the task priority.

[0042] In step S103, the security-critical container inside the real-time virtual machine completes the writing of all declaration fields through the end-to-end deadline declaration proxy.

[0043] The consumption timing information includes at least the following items: stream identifier (stream_id), consumer identifier (consumer_id) of the container consuming the data stream, end-to-end deadline, consumer criticality level, consumer control period, estimated delay time, estimated processing time, claim validity period, and claim version number.

[0044] The end-to-end deadline can be defined as the maximum allowable relative time limit from when the accelerator task can be submitted until the task execution result is transmitted and mapped and can be read and used by the safety-critical container, denoted as D_e2e.

[0045] The consumer criticality level is a business security classification identifier declared by the real-time virtual machine to the consumer's deadline schedule. It characterizes the degree of danger of the data stream consumption business and is the primary criterion for accelerator task sequencing, with higher-level inference tasks scheduled first. The higher the consumer criticality level, the higher the scheduling priority of the corresponding accelerator task. This is used to separate high-security control businesses that may cause personal injury or equipment damage from auxiliary businesses that are only used for logging, screen previewing, and data archiving and do not pose an immediate security risk. The consumer criticality level is denoted as: criticality.

[0046] The criticality level of a consumer is not determined automatically by the safety-critical container at runtime, but rather configured by system designers during the deployment phase based on the safety importance of the function, the consequences of functional failure, hazard analysis results, control link security integrity level, deadline requirements, and post-failure security policies. For example: if the data stream consumption service (data stream and consumer) provides personnel intrusion detection results to the robotic arm emergency stop container, the consumer criticality level is A; if the data stream consumption service provides workpiece pose results to the trajectory correction container, the consumer criticality level is B; if the data stream consumption service provides product quality statistics results to the backend analysis container, the consumer criticality level is C. Specifically, consumer criticality level A indicates safety criticality, and exceeding the timeout may pose a safety risk to personnel or equipment; consumer criticality level B indicates task criticality, and exceeding the timeout may affect control quality or task success rate; consumer criticality level C indicates a normal function, allowing for delays, degradation, or frame drops.

[0047] Consumer criticality levels can be set at the consumer, container, or "consumer-data flow" relationship, rather than being uniformly set at the real-time virtual machine level. Therefore, containers with three different consumer criticality levels (A, B, and C) can coexist within the same real-time virtual machine.

[0048] The consumer control period is a fixed, recurring scheduling period for security control services running in safety-critical containers within a real-time virtual machine. It is determined by the container's own hard real-time scheduling logic and serves as the baseline time window for the safety-critical container to process accelerator inference results and output device control commands. This parameter describes the job cycle or expected data update cycle of the consumer task, helps to limit the reasonable lower limit of the end-to-end deadline, supports the accelerator control plane regulator in performing lower limit clamping of the deadline to prevent abuse, and reflects the container's required frequency of obtaining inference results. The consumer control period is denoted as period_hint; for example, the consumer control period can be 4ms.

[0049] The estimated latency refers to the estimated transmission and mapping latency of the task execution result from the heterogeneous accelerator to the real-time virtual machine, that is, the estimated maximum latency of transmission and mapping to the real-time virtual machine memory, denoted as: t_transfer_est.

[0050] Estimated processing time refers to the estimated processing time of the safety-critical container of the real-time virtual machine for the task execution result. That is, after the inference result data is delivered to the memory of the real-time virtual machine, the worst estimated time required for the safety-critical container to parse the AI ​​inference data and execute safety control logic (collision detection, emergency stop, balance adjustment, etc.) is denoted as: t_rtos_proc_est.

[0051] The validity period of the declaration represents the expiration time of the validity of this data stream timing declaration record. From this time, the virtual machine monitor determines that the consumer-side timing information corresponding to the stream_id is invalid. When the accelerator control plane supervisor looks up the DC-Table, it will identify that there is no valid consumer declaration. The corresponding accelerator task will be downgraded to the lowest priority scheduling. This is used to automatically clean up expired and no longer used subscription timing records to prevent invalid and outdated data from interfering with the accelerator task sorting logic. It is denoted as: valid_until.

[0052] The declaration version number is a monotonically increasing sequence number in the DC-Table. It is automatically incremented when the data flow timing declaration is updated, so that the accelerator control plane monitor can identify whether the record has been updated and ensure that the latest consumption timing parameters are read. It is denoted as: seq.

[0053] Step S104: Apply read / write permission isolation constraints to the consumer's deadline schedule.

[0054] The consumer deadline schedule only allows real-time virtual machine writes and only allows accelerator control plane regulators to read.

[0055] Set a unique write subject and a unique read subject for the DC-Table. Only real-time virtual machines can write consumption timing information to the DC-Table through the controlled unidirectional channel provided by the virtual machine monitor. Only the accelerator control plane regulator has the right to read the consumption timing information stored in the DC-Table and use it for accelerator task sorting.

[0056] General-purpose virtual machines do not have read and write permissions to the DC-Table, and cannot obtain various timing parameters inside the real-time virtual machine, thus ensuring that the security isolation boundary between virtual machines is not compromised.

[0057] In the second phase, the accelerator control plane supervisor receives accelerator tasks submitted by the general virtual machine, retrieves consumption timing information from the DC-Table, calculates the effective deadline based on the consumption timing information, and reorders the competing tasks accordingly before submitting them to the heterogeneous accelerator.

[0058] Figure 3 A flowchart illustrating the second-stage heterogeneous accelerator task scheduling steps provided in an embodiment of this application is shown, as follows: Figure 2 As shown, the second-stage heterogeneous accelerator task scheduling steps include: Step S201: The accelerator front-end driver generates an accelerator task descriptor and writes the accelerator task descriptor into the shared commit queue.

[0059] The accelerator front-end driver of the general virtual machine writes accelerator task descriptors to the shared submission queue. The accelerator task descriptors carry the data stream identifier stream_id.

[0060] The implementation methods for data stream identifiers include: reusing the content of the first target native field in the accelerator task descriptor as the data stream identifier, or using the data in the reserved extended field of the accelerator task descriptor as the data stream identifier field, or constructing the data stream identifier from multiple second target native fields in the accelerator task descriptor according to preset rules.

[0061] Specifically, the first approach is to select the first target native field in the accelerator task descriptor as the data stream identifier field. The first target native field can refer to the existing session identifier field, context identifier field, or channel identifier field in the accelerator task descriptor, so as to agree that the semantics of the first target native field is the data stream identifier.

[0062] The second approach is to select a field from the reserved or extended fields of the accelerator task descriptor as the data stream identifier field.

[0063] The third approach is to select the second target native field in the accelerator task descriptor as the data stream identifier field. Multiple second target native fields can refer to the existing general virtual machine identifier field, model identifier field, and output channel field in the accelerator task descriptor. The triples corresponding to the above three fields are exported according to predetermined rules to form the data stream identifier.

[0064] In one embodiment, if it is inconvenient to expand the accelerator task descriptor, the accelerator front-end driver of the general virtual machine and the accelerator control plane supervisor can agree on a mapping table between the submission sequence number and the data stream identifier. When submitting the accelerator task descriptor, the accelerator front-end driver writes the mapping table into the metadata area of ​​the shared submission queue.

[0065] In step S202, the accelerator control plane monitor uses the data stream identifier to look up the consumer deadline schedule and obtain consumption time sequence information.

[0066] When verifying accelerator task descriptors, the accelerator control plane regulator can verify the address, length, access permissions, and critical quotas of the accelerator task descriptors.

[0067] If the verification passes, the accelerator control plane supervisor reads the trusted monotonic clock, records the task admission time t_admit, assigns a monotonically increasing admission sequence number admission_seq to the accelerator task, and writes the accelerator task descriptor into the task queue to be submitted (also known as the accelerator shadow queue). If the verification fails, the accelerator task descriptor is refused to be written into the task queue to be submitted, and an error status is returned to the general virtual machine.

[0068] The accelerator control plane regulator only accepts stream_ids that have been registered and passed producer, consumer, and access permission verification. Using the data stream identifier carried by the verified accelerator task descriptor, the accelerator control plane regulator performs a reverse lookup in the DC-Table to obtain the end-to-end deadline and estimated latency of the data stream.

[0069] In one embodiment, the accelerator control surface regulator 310 includes a consumer perception sequencer, which is a functional module within the accelerator control surface regulator. The two are related as an overall module and its internal sub-module.

[0070] The accelerator control plane regulator is used to read the shared submission queue, verify the accelerator task descriptor, record the trusted task acceptance time and acceptance sequence number, query the data stream registry, call the consumer-aware sorter, maintain the queue of tasks to be submitted, trigger the controlled doorbell, receive hardware completion interrupts, update shadow states, and send result-ready notifications to consumers.

[0071] The consumer-aware sorter is used to query the DC-Table based on stream_id, handle single-consumer or multi-consumer associations, determine the dominant consumer, calculate the effective deadline, check critical priority service quotas, generate composite sort keys, and reorder pending tasks in the pending task queue.

[0072] For example, the accelerator control plane regulator calls the consumer-aware sorter to calculate the effective deadline and composite sorting key. The consumer-aware sorter updates the task order in the task queue to be submitted. The accelerator control plane regulator triggers the controlled doorbell based on the task sorting result in the task queue to be submitted and submits the corresponding task to the heterogeneous accelerator.

[0073] Step S203: Determine the effective deadline based on the end-to-end deadline and the estimated delay time.

[0074] If the corresponding data stream identifier is found in the consumer deadline schedule, the task completion deadline is determined based on the sum of the task acceptance time and the end-to-end deadline. The effective deadline is obtained by subtracting the estimated delay time and the estimated processing time from the task completion deadline.

[0075] The effective deadline is denoted as D_eff, and the formula for calculating the effective deadline is: .

[0076] If the corresponding data stream identifier is not found in the consumer deadline schedule (i.e., an accelerator task descriptor has been submitted but there is no consumer declaration, it is an orphan accelerator task or the declaration has expired), it will be processed according to the preset configuration mode, which includes security mode and compatibility mode.

[0077] In safe mode, the accelerator task is reduced to the lowest priority so that it can be served after the designated accelerator task, which is the accelerator task whose corresponding data stream identifier is retrieved in the consumer deadline schedule and has not yet exceeded the valid deadline.

[0078] In this context, reducing the accelerator task to the lowest priority means marking the accelerator task as an orphan accelerator task, setting the consumer criticality level to empty, and placing the accelerator task into the best-effort service category in the task queue to be submitted. The best-effort service category means that the task will be served after all other tasks with valid consumer claims and that have not yet exceeded their deadline risk.

[0079] In compatibility mode, orphan accelerator tasks can be sorted according to the clamped submitter deadline, the task acceptance time recorded by the accelerator control plane monitor, or the admission_seq, and services are provided to orphan accelerator tasks in the sorted order. In compatibility mode, the consumer criticality level is set to empty, does not occupy the consumer criticality quota, and does not allow skipping Class A or Class B tasks with approaching deadlines based solely on submitter self-reporting prompts. To prevent orphan accelerator tasks from permanent starvation, the counter control plane monitor configures a minimum service share, maximum waiting time, or aging mechanism for the best-effort service category.

[0080] Step S204: Reorder the accelerator tasks and submit them to the heterogeneous accelerator.

[0081] The accelerator control plane supervisor rearranges the set of tasks that have passed verification and are ready to be submitted to the heterogeneous accelerator stored in the task queue according to the composite sorting rules, and submits them to the heterogeneous accelerator through the controlled doorbell in the reordered order.

[0082] The composite sorting rule refers to determining priority level by three tiers: consumer criticality, effective deadline, and task submission time. Based on the priority determination results, accelerator tasks in the task queue (accelerator shadow queue) to be submitted are reordered. Among them, the task submission time is a runtime attribute of each specific accelerator task instance and is trustedly recorded by the accelerator control plane regulator.

[0083] For example: First, sort by consumer criticality level (e.g., consumer criticality level A takes precedence over consumer criticality level B, and consumer criticality level B takes precedence over consumer criticality level C); then, within the same consumer criticality level, sort by valid deadline, with the earliest valid deadline being processed first; if both consumer criticality level and valid deadline are the same, sort by task submission time (or admission_seq), with the accelerator task submitted earlier being ranked higher.

[0084] When a new accelerator task with a higher consumer criticality level or an earlier effective deadline arrives, accelerator tasks that have not yet been submitted to the heterogeneous accelerator are reordered. Furthermore, accelerator tasks already submitted to the heterogeneous accelerator and those currently executing are not interrupted; only the subsequent submission order is affected.

[0085] When the heterogeneous accelerator completes its inference operation, a hardware completion interrupt is sent to the accelerator control plane regulator, so that the accelerator control plane regulator updates the accelerator shadow state and sends a result ready notification to the corresponding consumer.

[0086] Among them, the accelerator shadow state is a set of physical accelerator image state data tables maintained by the accelerator control plane supervisor on the software side. It is used to record the life cycle of physical accelerators and tasks, and fully synchronize the real-time operation of the underlying hardware accelerators. The real hardware state only exists inside physical hardware such as NPU and GPU. The accelerator control plane supervisor cannot directly read the high-frequency state of hardware registers, so a copy is established at the software layer for full-process management, scheduling, and fault detection.

[0087] For example, accelerator shadow status can be used to determine whether a physical accelerator is currently busy; record the complete lifecycle of a task from acceptance, waiting, submission, running to completion; support consumer-perceived sorting; correlate hardware completion interruptions with specific tasks, data streams, and consumers; count the number of accelerator tasks to be submitted and the depth of the hardware queue; detect orphan accelerator tasks, tasks that have not been completed for a long time, and consumption chain breaks; support critical quota management and anti-abuse detection; and support the correspondence between result completion and consumption confirmation.

[0088] The pending task queue is a trusted software queue maintained by the accelerator control plane regulator, located between the shared submission queue and the physical accelerator hardware queue. The pending task queue receives tasks that have passed address, length, access permission, and critical quota checks, controlling the order in which accelerator tasks actually enter the physical accelerator hardware queue; it also saves the accelerator task state for matching after a hardware completion interrupt. This embodiment only reorders tasks in the pending task queue that have not yet been submitted to the heterogeneous accelerator; it does not interrupt accelerator tasks already executed or in a non-reorderable state in the heterogeneous accelerator (physical accelerator).

[0089] Step S205: Apply constraints to the virtual machine.

[0090] To prevent the real-time virtual machine side (or compromised container) from monopolizing the heterogeneous accelerator for an extended period by declaring a very small end-to-end deadline after a switch of trust source, the accelerator control plane regulator imposes at least one of the following constraints: critical quota constraint, consistency verification constraint, and deadline lower bound constraint.

[0091] The criticality quota constraint means that each consumer or data stream, based on its criticality level, can only receive a limited number of service visits or accumulated service time from accelerator tasks that prioritize tasks with lower criticality levels (lower criticality level refers to a consumer whose criticality level is lower than the criticality level of the accelerator task itself). This limit cannot exceed a preset quota. Once the criticality quota is exhausted, subsequent accelerator tasks under that data stream will no longer receive criticality upgrades, but will still participate in task ranking based on the effective internal deadline, acceptance sequence number, and best-effort service rules. The criticality quota includes visit quota, time quota, and service quota.

[0092] A critical boost refers to an accelerator task receiving a critical boost when it matches a valid, unexpired consumer claim in the DC-Table, identifies a valid leading consumer, and that consumer has not exhausted its remaining critical boost credit and is not under penalty restrictions. Specifically, a valid, unexpired consumer claim means that a record in the DC-Table is found using the stream_id and is within the valid_until period, and is not an orphan accelerator task; identifying a valid leading consumer means that the selected leading consumer has a consumer criticality level of A, B, or C; the consumer's remaining critical boost credit has not been exhausted, meaning that the remaining boost credit (promotion_credit) corresponding to the stream_id has not been exhausted, and the number of times / time / service credit within a unit time window has not exceeded the limit; and not being under penalty restrictions means that the consumer is not within a penalty period. If there are a large number of abnormal misuse counts and penalties are triggered, the critical boost qualification will be temporarily revoked.

[0093] The remaining upgrade quota is a critical upgrade quota maintained independently for each consumer. It is used to limit the frequency with which high-level consumers seize the computing power of low-level tasks within a unit of time. When a task is given priority scheduling based on the criticality level of the leading consumer, the upgrade quota of the corresponding account will be deducted. After the quota is exhausted, the task corresponding to that consumer will no longer enjoy critical upgrade privileges.

[0094] The penalty constraint state is identified by `penalty_state`, which is the penalty status identifier for consumers. For consumers who frequently declare timing constraints but fail to retrieve inference results within their own control cycle, the accelerator control plane regulator marks them as being in a penalty state. Consumers in a penalty state will no longer trigger critical boosts for their corresponding data stream tasks; they will only participate in basic ranking based on their valid deadlines. Simultaneously, the critical boost recovery speed will be reduced until the penalty period ends and the consumer behavior returns to normal, at which point the penalty will be lifted.

[0095] The abnormal abuse count is a count of abnormal consumer defaults. It is used to count the number of times a consumer fails to retrieve the inference results in a timely manner within its own control cycle after declaring time constraints. When the count exceeds the threshold, the corresponding consumer enters a penalty state, temporarily cancels the key promotion qualification, and restricts priority scheduling rights.

[0096] For example: The frequency quota refers to the number of tasks allowed to receive a critical boost within a unit time window. If the preset frequency quota (e.g., 10 times) is exceeded, all accelerator tasks in that data stream will no longer receive critical boosts based on the consumer's criticality level, but will instead be sorted according to the task's own valid deadline. The time quota refers to the cumulative accelerator execution time obtained within a unit time window. The service quota refers to the cumulative priority service allowance obtained within a unit time window.

[0097] Consistency verification constraints refer to constraints imposed on the criticality of the accelerator task corresponding to the first target data stream.

[0098] Specifically, the accelerator control plane regulator counts the number of times a data stream's task completion event is retrieved by the corresponding consumer in a timely manner. If the data stream is one in which the consumer has declared a high-urgency time constraint but has not retrieved the result in a timely manner (i.e., the declared information and the actual consumption behavior are consistently inconsistent), then it is called the first target data stream.

[0099] If the consumer criticality level of the consumer corresponding to the first target data stream is A, the accelerator control plane regulator triggers a security anomaly notification, and the real-time virtual machine enters the predefined security degradation strategy. The security management module in the real-time virtual machine performs link checks, resets, or fail-safe processing, and the criticality level of the A-class consumer remains unchanged.

[0100] If the consumer criticality level of the consumer corresponding to the first target data stream is B or C, the accelerator tasks associated with that consumer can be restricted from gaining additional ranking advantages in the queue of tasks to be promoted based on their consumer criticality level (i.e., their critical priority service eligibility is restricted; critical priority service eligibility refers to whether a task can gain additional ranking advantages based on its consumer criticality level, which can be restricted at runtime). After being restricted, the consumer's static consumer criticality level remains unchanged; however, its subsequent accelerator tasks will temporarily no longer use the consumer criticality level as a priority ranking dimension, but will instead participate in the submission queue ranking based on the effective deadline, acceptance sequence number, and basic service rules. This can be mitigated by reducing the priority service quota recovery rate, extending the quota recovery period, increasing the anomaly count, or, during the penalty period, participating in the basic ranking only based on the effective deadline.

[0101] Among them, the high urgency time constraint refers to a consumer's criticality level being greater than a set level threshold (e.g., the set level threshold is C), the effective deadline being less than a first set time threshold, and the end-to-end deadline being less than a second set time threshold. Timely retrieval means that the consumer obtains and processes the task execution result within its own control cycle time limit; failure to retrieve in a timely manner means that the consumer's control cycle window has ended, but the task output has not been read, and the task execution result has expired and is invalid.

[0102] A lower bound constraint on deadlines refers to a constraint that the end-to-end deadline is greater than a minimum valid threshold. This lower bound constraint prevents declared unachievable deadlines from disrupting the ordering. The minimum valid threshold is determined based on the consumer's control period.

[0103] Step S206: Detect whether the consumption chain is abnormal. If it is determined that the consumption chain is broken, perform downgrade processing on the data stream task.

[0104] For orphan accelerator tasks (those that have submitted an accelerator task descriptor but have no consumer declaration), they are reduced to the lowest priority and do not occupy critical quotas.

[0105] For consumers lacking data (consumers subscribed to a data stream, but no accelerator tasks arrive within the consumer's control window), when a consumer subscribes to a second target data stream and no corresponding accelerator task arrives within the consumer's control window, a virtual interrupt representing a lack of upstream data is injected into the real-time virtual machine. This allows the consumer to continue execution according to a pre-defined fallback strategy. For example, the accelerator control plane regulator injects a virtual interrupt indicating "upstream data shortage" into the real-time virtual machine, allowing the consumer to continue execution according to a pre-defined fallback strategy that either uses the results from the previous cycle or performs a conservative safety action, without blocking the consumer.

[0106] When there are no valid records for the consumer's deadline schedule, the system reverts to the existing task sorting method based on deadline prompts.

[0107] In one embodiment, to enable the accelerator control plane regulator to look up the DC-Table and reorder accelerator tasks along the control path, the accelerator's interaction logic can be divided into a control path and a data path. Control interactions such as task submission, scheduling, and hardware completion interruption are all processed by the accelerator control plane regulator, allowing it to perform DC-Table lookups and task reordering within the control flow. Meanwhile, business data such as models, images, and inference results undergo hardware address permission verification only through the Memory Management Unit (IOMMU) or System Memory Management Unit (SMMU), and are directly transported by the Address Translation (DMA) engine without being parsed and forwarded by the accelerator control plane regulator software. This supports the consumer-aware scheduling mechanism while minimizing data transmission latency.

[0108] For example, input tensors, output tensors, and model weights of neural networks in heterogeneous accelerators are mapped via restricted addresses using the SMMU / IOMMU. The accelerator's DMA engine then moves these data from the system's shared DDR main memory to the NPU's on-chip cache / device-specific memory for inference computation. The output tensors generated during inference are then moved back to the system's shared DDR memory via DMA for use by real-time virtual machine consumers. The entire data path involves only hardware address and permission verification; the accelerator control plane supervisor does not intercept or parse tensors or model weights, achieving near-pass-through transmission.

[0109] The submission operations of accelerator task descriptors, doorbell trigger signals, and hardware completion interrupts all belong to the accelerator control path and are all uniformly managed by the accelerator control plane supervisor. The accelerator control plane supervisor must read all the information carried by the above control events, verify their legality, access permissions, and critical quotas, and complete the corresponding scheduling processing such as table lookup and reordering, accelerator shadow state updates, and result notification distribution. Control events are not allowed to bypass the accelerator control plane supervisor and be directly sent to physical accelerators or other virtual machines.

[0110] In one embodiment, the end-to-end deadline, estimated delay time, and estimated processing time can be obtained offline during deployment using an end-to-end latency analysis tool.

[0111] Worst-case latency analysis is performed on the end-to-end link to obtain upper bounds for result transmission and mapping latency and real-time virtual machine container processing time. The upper bound for result transmission and mapping latency is used as the estimated latency, and the upper bound for real-time virtual machine container processing time is used as the estimated processing time. The end-to-end link can refer to the link between accelerator task submission, accelerator computation, result transmission and mapping, and real-time virtual machine container processing.

[0112] Meanwhile, end-to-end deadlines can be determined based on the deadline constraints of the safety-critical operations to which the consumer container belongs. In addition to the offline calibration mentioned above, end-to-end deadlines can also be derived from the consumer's criticality level and control cycle, or converted from the absolute deadline of the operation and the safety execution budget.

[0113] In one embodiment, in scenarios where the container set and data flow remain fixed (no new, deleted, or switched business flows will be added after the device goes online), static binding during deployment can be adopted.

[0114] For example, the binding relationship is pre-defined by platform configuration tools and offline calibration tools, that is, the complete set of consumption timing parameters corresponding to each data stream identifier is pre-defined. During the startup phase, the Hypervisor writes all static mapping relationships to the DC-Table in batches. The safety-critical containers of the real-time virtual machine no longer perform dynamic subscription, dynamic declaration, or dynamic update of the DC-Table. The records in the DC-Table remain unchanged throughout the entire operation, and there are no runtime table writing operations such as adding, modifying, expiring, or auto-incrementing versions.

[0115] The following reference Figure 4 Let's introduce the sequencing order of accelerator tasks.

[0116] Figure 4 A schematic diagram illustrating the timing sequence of accelerator tasks provided in an embodiment of this application is shown, as follows: Figure 4 As shown, the safety-critical container within the real-time virtual machine pre-executes step S101 to declare the urgent data flow D and register the end-to-end deadline and high consumer criticality level corresponding to this data flow in the DC-Table. Through step S201, the general virtual machine synchronously submits two accelerator tasks: task A (accelerator task) and task B. Accelerator task A is bound to non-urgent data flow C (stream=classify) and belongs to non-urgent business; accelerator task B is bound to urgent data flow D for consumption by the aforementioned safety-critical container. The accelerator control plane monitor executes step S202, reverse-checks the DC-Table according to the data flow identifier, and then, through step S203, reverse-calculates the effective deadlines of the two accelerator tasks, determining that the effective deadline of accelerator task B is much earlier than the effective deadline of accelerator task A. The accelerator control plane monitor executes step S204, reorders the queue of tasks to be submitted, and prioritizes submitting accelerator task B to the heterogeneous accelerator. After accelerator task B completes its computation, the heterogeneous accelerator sends a hardware completion interrupt for task B to the accelerator control plane supervisor. The accelerator control plane supervisor then pushes a result readiness notification for task B to the safety-critical container of the corresponding real-time virtual machine, and finally submits accelerator task A to the heterogeneous accelerator for execution.

[0117] Throughout the process, the safety-critical containers of the real-time virtual machine maintain their original 4ms runtime and execution rhythm, and will not be blocked while waiting for the accelerator inference results.

[0118] The following example, using the human-machine collaboration safety of a vision-guided multi-motor collaborative assembly station, illustrates the scheduling process of accelerator tasks.

[0119] The platform is a domestically produced CPU+NPU heterogeneous SoC (the example uses Rockchip RK3588S), including 8 ARM Cortex-A series CPU cores and 2 NPU cores. The Type-1 virtual machine monitor is implemented in a memory-safe language, runs on ARMv8EL2, and concurrently hosts a general-purpose virtual machine (general-purpose operating system plus object detection) and a real-time virtual machine (real-time operating system plus kernel-level secure container, controlling multiple servo motors to coordinate their movements via EtherCAT bus). The task deployment includes the main body, accelerator tasks, data flow, and urgency, as shown in the table below:

[0120] The following reference Figure 5 This section describes the task execution of existing solutions and embodiments of this application under the task deployment corresponding to the table above.

[0121] Figure 5 The diagram illustrates the task execution process of the prior art and the embodiments of this application under the same task deployment, as shown below. Figure 5 As shown, the execution process under the existing technology is as follows: The general virtual machine submits two inference tasks, classification and intrusion. The accelerator front-end driver indicates that the end-to-end deadlines for both are close (the general virtual machine's visual side cannot know the approximately 4ms consumer control cycle of the safety-critical container). When the NPU is congested, classification is executed first (approximately 30ms), while intrusion is executed after classification, taking approximately 60ms to complete. The safety-critical container requires the intrusion result at the approximately 4ms control cycle boundary, causing a timeout. Multiple motors that are working collaboratively do not receive the intrusion signal in time and continue cutting, posing a risk of worker injury. However, even if classification is delayed by 30ms, it will not affect the MES (Manufacturing Execution System) records.

[0122] It should be noted that the main body includes two general virtual machines - AI. These two general virtual machines - AI are the same general virtual machine, but this general virtual machine instance executes two different AI inference tasks at the same time; and physically there is only one real virtual machine. The two lines of "real virtual machine" are a split display of two independent safety-critical containers within the same virtual machine.

[0123] The execution process of this application embodiment is as follows: First, the safety-critical container of the real-time virtual machine subscribes to stream=intrusion and declares to the DC-Table via a controlled channel that the end-to-end deadline is approximately 4ms, the consumer criticality level is urgent, and the consumer control cycle is approximately 4ms; if the consumer (MES side) declares this, then stream=classify, the end-to-end deadline is approximately 150ms, and the consumer criticality level is non-urgent. Second, the general virtual machine submits two accelerator tasks, each with a corresponding data flow identifier in its task descriptor; after verification, the accelerator control plane supervisor looks up the DC-Table and calculates that the effective deadline of accelerator task B bound to the data flow identifier intrusion is much earlier than the effective deadline of accelerator task A bound to the data flow identifier classify, and the consumer criticality level is higher. Third, the accelerator control plane supervisor reorders the queue of tasks to be submitted, first submitting accelerator task B bound to the data flow identifier intrusion to the heterogeneous accelerator NPU, and after the task is completed, sending a result ready notification to the safety-critical container, and then submitting accelerator task A bound to the data flow identifier classify. Fourth, the safety-critical container retrieves the execution result of accelerator task B, bound to the data stream identifier "intrusion," within its approximately 4ms cycle, triggering emergency stop or deceleration if necessary; both the multi-motor synchronous control container and the safety-critical container strictly execute according to their respective cycles, without waiting for the NPU throughout the process. Fifth, accelerator task A, bound to the data stream identifier "classify," completes slightly later, with no impact on MES records.

[0124] The heterogeneous accelerator task scheduling system provided in this application has the following technical effects: First, the accelerator task execution order matches the urgency of real business: In the case of NPU congestion in heterogeneous accelerators, high-priority AI inference tasks used by real-time safety-critical containers will be scheduled and executed first, and will not be squeezed out by low-urgency tasks recorded by MES, thus preventing safety accidents such as equipment injury caused by safety control link timeout.

[0125] Second, it eliminates the need to rely on untrusted general virtual machines to fill in priority: By entrusting deadlines and critical declaration permissions to a trusted real-time virtual machine domain, and relying on a time-series record table that can be written to unidirectionally by only real-time virtual machines and whose permissions are uniformly managed by the Hypervisor, the consumer-side constraint parameters are transmitted, thus avoiding the vulnerability of general virtual machines maliciously forging deadlines to seize accelerator resources.

[0126] Third, the security isolation boundary of the virtual machine is fully preserved: general virtual machines have no right to read or write the DC-Table and cannot obtain sensitive timing information such as the real-time internal control cycle and latency of the virtual machine. The virtualization isolation mechanism is not compromised and the system security level is not reduced.

[0127] Fourth, the original task timing logic on the real-time side will not be changed: the various security-critical containers in the real-time virtual machine will maintain their original running cycle and scheduling phase, and only the NPU task queuing order will be adjusted, which will not cause timing interference to the original real-time control business.

[0128] Fifth, multiple anti-abuse mechanisms and smooth degradation ensure strong system robustness: Through key quota constraints, deadline lower bound constraints, and consistency verification constraints, the system prevents the real-time side from abusing high-priority exclusive accelerators; abnormal scenarios such as no consumer declaration or empty data table can automatically degrade to traditional scheduling logic, and the performance will not be inferior to existing solutions.

[0129] Based on the same inventive concept, this application also provides a heterogeneous accelerator task scheduling method corresponding to the heterogeneous accelerator task scheduling system. Since the principle of solving the problem by the method in this application is similar to the heterogeneous accelerator task scheduling system described above in this application, the implementation of the method can refer to the implementation of the system, and the repeated parts will not be described again.

[0130] Please see Figure 6 , Figure 6 This is a schematic diagram of a heterogeneous accelerator task scheduling device provided in an embodiment of this application. Figure 6 As shown, the heterogeneous accelerator task scheduling method includes: Step S301: When a safety-critical container subscribes to the data stream output by a heterogeneous accelerator, it declares the consumption timing information of the data stream in the consumer deadline schedule through the controlled channel provided by the virtual machine monitor. Step S302: Write the accelerator task descriptor to the shared commit queue through the accelerator front-end driver; Step S303: Extract the data stream identifier from the accelerator task descriptor of the shared submission queue, retrieve the consumer deadline schedule using the data stream identifier, and obtain the consumption time sequence information of the data stream. Step S304: Determine the effective deadline for the accelerator task based on the task acceptance time, end-to-end deadline, estimated delay time, and estimated processing time. Step S305: Based on the consumer's criticality level, effective deadline, and task submission time, the accelerator tasks in the task queue to be submitted are reordered, and the accelerator tasks are sequentially sent to the heterogeneous accelerators for execution in the reordered order.

[0131] The consumption timing information includes end-to-end deadline, estimated delay time, and estimated processing time.

[0132] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 7 As shown, the electronic device 700 includes a processor 710, a memory 720, and a bus 730.

[0133] The memory 720 stores machine-readable instructions executable by the processor 710. When the electronic device 700 is running, the processor 710 communicates with the memory 720 via the bus 730. When the machine-readable instructions are executed by the processor 710, they can perform the operations described above. Figure 4 The steps of the heterogeneous accelerator task scheduling method in the method embodiment shown are described in detail in the method embodiment, and will not be repeated here.

[0134] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described actions. Figure 4 The steps of the heterogeneous accelerator task scheduling method in the method embodiment shown are described in detail in the method embodiment, and will not be repeated here.

[0135] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0136] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0137] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0138] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0139] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0140] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A heterogeneous accelerator task scheduling system, characterized in that, The heterogeneous accelerator task scheduling system includes a general virtual machine, a real-time virtual machine, a virtual machine monitor, and a heterogeneous accelerator. The real-time virtual machine includes a safety-critical container, the general virtual machine includes an accelerator front-end driver, and the virtual machine monitor includes an accelerator control plane regulator. The real-time virtual machine is used to declare the consumption timing information of the data stream in the consumer deadline schedule through the controlled channel provided by the virtual machine monitor when the safety-critical container subscribes to the data stream output by the heterogeneous accelerator. The general virtual machine is used to write accelerator task descriptors into a shared commit queue through the accelerator front-end driver. The accelerator control plane supervisor is used to extract data stream identifiers from the accelerator task descriptors of the shared submission queue, retrieve the consumer deadline schedule using the data stream identifiers, and obtain the consumption timing information of the data streams. The consumption timing information includes end-to-end deadline, estimated delay time, and estimated processing time. Based on the end-to-end deadline, the estimated delay time, and the estimated processing time, the effective deadline for the accelerator task is determined. Based on the consumer's criticality level, the effective deadline, and the task submission time, the accelerator tasks in the task queue to be submitted are reordered, and the accelerator tasks are sequentially sent to the heterogeneous accelerators for execution in the reordered order.

2. The heterogeneous accelerator task scheduling system according to claim 1, characterized in that, The accelerator control plane monitor determines the valid deadline for the accelerator mission through the following process: The task completion deadline is determined based on the sum of the task acceptance time and the end-to-end deadline. The effective deadline is obtained by subtracting the estimated delay time and the estimated processing time from the task completion deadline in sequence.

3. The heterogeneous accelerator task scheduling system according to claim 1, characterized in that, The consumer deadline schedule is only allowed to be written to by the real-time virtual machine and only allowed to be read by the accelerator control plane regulator.

4. The heterogeneous accelerator task scheduling system according to claim 1, characterized in that, The accelerator control surface monitor is also used to perform the following processes: If the data stream identifier is not found in the consumer deadline schedule, the accelerator task will be placed after the designated accelerator task. The designated accelerator task refers to the accelerator task for which the corresponding data stream identifier is found in the consumer deadline schedule and has not yet exceeded the valid deadline.

5. The heterogeneous accelerator task scheduling system according to claim 1, characterized in that, The consumer-perceived sorter reorders the accelerator tasks in the queue of tasks to be submitted by performing the following process: Based on the consumer criticality level, the effective deadline, the task submission time, and the three levels of priority are determined sequentially, and the accelerator tasks in the task queue to be submitted are reordered based on the priority determination results.

6. The heterogeneous accelerator task scheduling system according to claim 1, characterized in that, The accelerator control surface regulator is also used to apply at least one of the following constraints: critical quota constraints, consistency verification constraints, and deadline lower bound constraints; The critical quota constraint means that each consumer or data stream, based on its consumer criticality level, receives fewer service times or cumulative service time from accelerator tasks that take priority over those with lower consumer criticality levels than a preset quota. The consistency verification constraint refers to the constraint on the criticality of the accelerator task corresponding to the first target data stream. The first target data stream is the data stream in which the consumer declares a high-urgency time-series constraint but does not retrieve the result in time. The high-urgency time-series constraint means that the consumer's criticality level is greater than a set level threshold, the effective deadline is less than a first set time threshold, and the end-to-end deadline is less than a second set time threshold. The lower bound constraint of the deadline means that the end-to-end deadline is greater than the minimum valid threshold, which is determined based on the consumer control cycle.

7. The heterogeneous accelerator task scheduling system according to claim 1, characterized in that, The accelerator control surface monitor is also used to perform the following processes: When a consumer subscribes to a second target data stream and no accelerator task corresponding to the second target data stream arrives within the consumer's control period, a virtual interrupt representing the lack of upstream data is injected into the real-time virtual machine so that the consumer can continue execution according to a preset fallback strategy.

8. The heterogeneous accelerator task scheduling system according to claim 1, characterized in that, The accelerator control surface regulator is also used to perform the following processes: The content of the first target native field in the accelerator task descriptor can be reused as the data stream identifier, or the data in the reserved extended field of the accelerator task descriptor can be used as the data stream identifier field, or the data stream identifier can be constructed from multiple second target native fields in the accelerator task descriptor according to preset rules.

9. The heterogeneous accelerator task scheduling system according to claim 1, characterized in that, The accelerator control surface monitor is also used to perform the following processes: The accelerator task descriptor is validated, and if the validation fails, the accelerator task descriptor is refused to be written to the shared commit queue.

10. The heterogeneous accelerator task scheduling system according to claim 1, characterized in that, The heterogeneous accelerator is used to perform the following processes: Upon completion of inference operations, a hardware completion interrupt is sent to the accelerator control plane regulator, causing the accelerator control plane regulator to update the accelerator shadow state and send a result-ready notification to the corresponding consumer.

Images