A centralized management and control system for computing resources and an AI computer

The centralized management and control system for computing resources solves the problem of insufficient CPU-NPU collaborative scheduling in AI PCs, realizes dynamic weight balancing and cross-terminal computing power sharing, and improves the overall computing power utilization and response capability of AI PCs.

CN123114759APending Publication Date: 2026-11-17GUOGUANG ELECTRONICS INFORMATION TECH
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202611325985.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-29
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing AI computing power scheduling technologies lack a global perspective CPU-NPU collaborative scheduling mechanism, leading to resource waste and performance bottlenecks. Static allocation also results in resource waste and performance bottlenecks. Furthermore, there is a lack of task-level reverse overflow capability, a disconnect between the application layer and the hardware layer, and a lack of cross-terminal computing power sharing capability.

Method used

A centralized scheduling and management system for computing resources is provided, including a scheduling client and a server. Through computing power monitoring driver, task interception and routing mechanism, computing power balance scheduler and NPU hardware abstraction layer, dynamic weight balance scheduling between CPU and NPU is realized, and tasks are migrated back to CPU for execution when NPU is overloaded. Cross-terminal scheduling is supported.

Benefits of technology

It achieves effective collaboration between the CPU and NPU, dynamically adjusts the computing power allocation, avoids resource waste and system response delay, supports task-level reverse migration and cross-terminal computing power sharing, and improves the overall computing power utilization and system response capability of AI PC.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123114759A_ABST
    Figure CN123114759A_ABST
Patent Text Reader

Abstract

This invention discloses a centralized scheduling and management system for computing resources and an AI PC, relating to the field of heterogeneous computing and neural network processor resource scheduling technology in artificial intelligence. The system includes: an AI PC terminal configured with a CPU and an NPU; a scheduling client running on the AI ​​PC terminal, comprising a computing power monitoring driver, a task interception and routing mechanism, a computing power balancing scheduler, and an NPU hardware abstraction layer. The computing power balancing scheduler executes a dynamic weight balancing algorithm to determine the scheduling strategy of tasks between the CPU and NPU and supports reverse hot migration; and a scheduling server running on a remote server, comprising a global scheduling engine for scheduling tasks to other idle AI PC terminals when local computing power is insufficient. This invention also provides an AI PC. This invention achieves collaborative scheduling of the CPU and NPU and cross-terminal computing power sharing, improving the computing power utilization efficiency of the AI ​​PC.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence (AI) heterogeneous computing and neural network processor (NPU) resource scheduling technology, and in particular to a centralized scheduling and control system for computing resources and an AI computer. Background Technology

[0002] With the rapid development of artificial intelligence technology, traditional general-purpose computing devices are evolving towards heterogeneous computing architectures. In recent years, AI PCs have gradually become the mainstream computing platform. In addition to traditional central processing units (CPUs) and graphics processing units (GPUs), these devices also integrate neural network processors specifically designed to accelerate AI inference tasks. As a hardware accelerator designed specifically for AI inference operations such as matrix multiplication and convolution, the NPU can provide high TOPS (Tera Operations Per Second) computing power with low power consumption, and has become the core computing resource for edge AI inference.

[0003] From the perspective of technological evolution, AI computing power scheduling has mainly gone through the following stages of development:

[0004] The first phase was the era of single accelerators (approximately 2018 to 2021). AI inference primarily relied on discrete GPUs (such as the NVIDIA GTX / RTX series), with the operating system assigning tasks to the GPU for execution via graphics card drivers. During this phase, edge devices did not yet integrate dedicated NPU hardware, and there was no need for coordinated scheduling between the CPU and NPU.

[0005] The second phase was the NPU hardware introduction period (approximately 2021 to 2024). Intel (Meteor Lake NPU, 11.5 TOPS), Apple (Neural Engine, 18-38 TOPS), and Qualcomm (Hexagon NPU, 45 TOPS) successively integrated NPUs into consumer processors. Simultaneously, inference frameworks such as OpenVINO, DirectML, and CoreML emerged at the software level, allowing developers to manually select the execution device (CPU / GPU / NPU) via APIs. However, the device selection in these solutions is static; that is, once the developer specifies the execution device when loading the model, the entire model's inference task is fixed to that device and cannot be dynamically switched based on system load during runtime.

[0006] The third phase is the exploration period of heterogeneous scheduling (2024 to present). Many manufacturers and research institutions have begun to experiment with dynamic scheduling schemes. For example, Honor proposed an edge-side model splitting scheme (GPU responsible for the Transformer backbone, NPU responsible for CNN feature extraction); Intel publicly released its XPU strategy (speech → CPU, translation → iGPU, CV → NPU); and the academic community's Agent.xpu implemented operator-level preemptive scheduling on the Intel Core Ultra platform. However, all of the above schemes are limited to single-machine scenarios and only support unidirectional offloading from the CPU to the accelerator.

[0007] Furthermore, in October 2025, NVIDIA filed a patent application (publication number CN121957794A) involving task reverse offloading, which describes how the GPU can reverse-offload some subtasks from a received task, such as product summation and reduction, back to the CPU for execution via a shared memory queue. This scheme aims to solve the problem that the GPU is less efficient than the CPU when performing certain bandwidth-intensive reduction operations, and belongs to the operator-level optimization strategy within a single task.

[0008] In summary, existing AI computing power scheduling solutions have at least the following technical defects and shortcomings.

[0009] Defect 1: Lack of a global perspective CPU-NPU collaborative scheduling mechanism.

[0010] Existing CPU schedulers focus on thread priority and frequency adjustment, while NPU drivers only manage their own task queues. The lack of effective communication and coordination mechanisms between the two prevents computing resources from working together effectively. In typical scenarios, when the CPU is fully loaded with data preprocessing, the NPU may be idle while waiting for input data; conversely, when the NPU is fully loaded with inference, the CPU may not be able to respond to user interaction requests in a timely manner, causing system response delays.

[0011] Defect 2: Static allocation leads to resource waste and performance bottlenecks.

[0012] Traditional solutions allocate a fixed amount of computing power to the NPU once an AI task is detected. However, different AI models (such as large language models and small image classification models) have significantly different dependencies on the CPU and NPU. The fixed allocation strategy cannot dynamically adjust the computing power ratio between the CPU and NPU according to real-time load characteristics, often resulting in one being overloaded while the other is idle, failing to fully utilize the overall computing power of the AI ​​PC.

[0013] Defect 3: Lack of task-level reverse overflow capability.

[0014] Existing technical solutions (all mainstream solutions such as OpenVINO, DirectML, and CoreML) only support unidirectional offloading from the CPU to the accelerator (GPU / NPU). When the NPU is overloaded, tasks cannot be migrated back to idle CPUs for execution, causing the NPU to become a system bottleneck—the NPU queue accumulates while the CPU is in a low-load state.

[0015] Defect 4: There is a disconnect between the application layer and the hardware layer.

[0016] In the current software ecosystem, applications send inference instructions directly to specific hardware, lacking awareness of the overall computing power status of the system. When multiple AI applications run simultaneously (such as simultaneously performing video background blurring, document AI summarization, and local LLM inference), each application competes for NPU resources, while CPU resources may be idled and wasted. Existing technologies lack a centralized management system to uniformly take over all AI computing power requests and perform intelligent arbitration and routing.

[0017] Defect 5: Lack of cross-terminal computing power sharing capability.

[0018] When the computing power of a single AI PC is insufficient, existing solutions can only queue up or degrade services, and cannot overflow the tasks of overloaded terminals to other idle AI PCs within the organization for execution. This results in a large number of terminals wasting computing power during idle periods, and an efficient computing network cannot be formed. Summary of the Invention

[0019] To overcome the above-mentioned technical problems, the present invention provides a centralized scheduling and management system for computing resources and an AI computer.

[0020] The technical solution adopted by this invention to solve its technical problem is as follows: This invention provides a centralized scheduling and management system for computing resources, including a scheduling client, a scheduling server, and at least one AI computer terminal. The AI ​​computer terminal is equipped with a central processing unit (CPU) and a neural network processor (NPU). The scheduling client runs on the AI ​​computer terminal and includes a computing power monitoring driver, a task interception and routing mechanism, a computing power balancing scheduler, and an NPU hardware abstraction layer (HAL). The computing power monitoring driver is used to collect the utilization data of the CPU and the NPU according to a preset time granularity. The task interception and routing mechanism is used to intercept AI inference tasks initiated on the AI ​​computer terminal. The computing power balancing scheduler is used to execute an enhanced dynamic weight balancing algorithm (eDWB) based on the real-time data collected by the computing power monitoring driver to determine the scheduling strategy of the AI ​​inference task between the CPU and the NPU. The scheduling strategy includes allocating the task to the CPU or the NPU for execution, and migrating non-critical AI inference tasks back from the NPU to the CPU for execution when the overall load of the NPU exceeds a preset threshold. The non-critical AI inference tasks are determined according to task priority tags and / or QoS levels. Among them, responsive interactive tasks are determined to be critical tasks, and background batch processing tasks are determined to be non-critical tasks. The NPU hardware abstraction layer is used to shield the interface differences of NPUs from different manufacturers. The scheduling server runs on a remote server and includes a global scheduling engine, which is used to schedule tasks to other idle AI computer terminals for execution when the local computing power of the AI ​​computer terminal is insufficient.

[0021] Preferably, the AI ​​computer terminal may further include a GPU and other accelerators.

[0022] Preferably, the computing power monitoring driver runs in kernel mode and collects CPU / NPU utilization, temperature and power consumption data at a granularity of 100ms.

[0023] Preferably, the task interception and routing mechanism includes a framework adaptation module, a user-mode library interception module, and a kernel-mode observation module, which work together in a layered manner and complete task construction and forwarding only on user-mode paths with inference semantics.

[0024] The framework adaptation module, acting as the main interception path, runs in user space and is used to take over tasks at the device selection and inference submission interfaces of the AI ​​inference framework. Specifically, it intercepts calls to standard AI inference APIs (such as the ONNX Runtime session execution interface and the OpenVINO inference request interface) through custom ExecutionProvider (the hardware execution backend abstract interface of ONNX Runtime, which allows for custom inference backends), inference plugin proxies, or Delegate (a proxy mechanism in the inference framework used to offload part of the computation graph to dedicated hardware execution). Based on the current session state, it constructs a task descriptor (ai_task_descriptor) containing task type, model features, and hardware preferences. The task descriptor is then passed to the scheduling center, which makes a decision on routing between the local CPU, local NPU, local GPU, hybrid execution, or remote nodes, and backfills the execution results to the caller according to the original API conventions.

[0025] The user-space library interception module, as an auxiliary interception path, runs in user space and is used to intercept processes that have not loaded the framework adaptation module but still call the standard AI inference library in a dynamic linking manner. It hooks the exported functions of the standard AI inference library through user-space API hooks (such as Detours on Windows, LD_PRELOAD on Linux, and DYLD_INSERT_LIBRARIES on macOS) to construct the task descriptor at the same inference semantic level and pass it to the scheduling center, without rewriting the task at the GPU / NPU microkernel submission level.

[0026] The kernel-mode observation module runs in kernel mode and is used to discover, audit, and impose quality-of-service constraints on raw hardware access that bypasses the standard AI inference API. Specifically, it can monitor system calls or driver-level access to GPU / NPU devices through eBPF (an extended Berkeley package filter in the Linux kernel that can be used for kernel-mode observation), kprobe, or operating system device I / O observation mechanisms. This is used for identifying unknown accelerator access, recording audit logs, and rate limiting or quota control for low-priority tasks. The kernel-mode observation module does not parse the complete model semantics, does not construct task descriptors for cross-device or cross-terminal forwarding, and does not directly redirect system calls or device I / O requests to scheduling execution paths.

[0027] Therefore, forwardable AI tasks are intercepted and submitted for scheduling only by the framework adaptation module and / or the user-mode library interception module at the user-mode inference semantic layer; the kernel mode only provides observation and governance capabilities, thereby achieving the correct separation of task interception, scheduling decision and result backfilling at the capability boundary.

[0028] Preferably, the dynamic weight balancing algorithm considers the following six factors when making scheduling decisions:

[0029] (1) The weighted utilization difference between the CPU and the NPU;

[0030] (2) The temperature status of the CPU and the NPU, when the temperature exceeds the safety threshold, restricts the allocation of new tasks to the corresponding hardware;

[0031] (3) Power consumption constraint: In battery mode, prioritize routing tasks to NPUs with higher energy efficiency;

[0032] (4) Task priority label: Real-time interactive tasks cannot be migrated, and background tasks are given priority in being transferred;

[0033] (5) Scene context: Inject different hardware weights according to different user scenarios such as video conferencing, gaming, and office work;

[0034] (6) Oscillation suppression: Automatically increase the inertia threshold during continuous reverse migration to avoid tasks frequently migrating back and forth between the CPU and NPU.

[0035] The decision result is one of the following five strategies: local CPU execution, local NPU execution, hybrid execution (operator-level splitting), remote overflow, or queuing.

[0036] Preferably, the scheduling client further includes: an NPU virtualization module, used to uniformly abstract multiple computing power units including at least the CPU and the NPU, and provide a unified API access interface to the upper-layer scheduling module, enabling the computing power balance scheduler to access different computing power units through the same API; and a QoS resource isolation module, used to configure three levels of QoS for different AI inference tasks, the three levels of QoS including: a guaranteed resource reservation level, a basic resource plus borrowable resource level, and a level that only uses remaining resources.

[0037] Preferably, the NPU hardware abstraction layer supports at least one of the following vendors' NPUs: Intel NPU, Qualcomm NPU, AMD NPU, and Apple Neural Engine. The NPU hardware abstraction layer is also used to automatically map the target operator to an equivalent operator combination or downgrade it to execution on the CPU when the target operator is not supported on the current NPU.

[0038] Preferably, the global scheduling engine uses multi-factor weighted matching for target node selection, whereby factors include: load rate, NPU operator compatibility, network latency, energy consumption, security level, task type, and historical success rate. The scheduling server also includes a computing resource registration and discovery module, employing a consistent hash ring to manage the computing resources of each AI computer terminal, achieving O(log N) node lookup. The scheduling server uses a master-slave architecture and the Raft protocol for failover, with a recovery time objective (RTO) of less than 5 seconds.

[0039] Preferably, the AI ​​computer terminal further includes a terminal autonomy module, used to automatically switch to a pure local scheduling mode when the AI ​​computer terminal disconnects from the scheduling server, and automatically reconnect to the scheduling server after the network is restored. If the target terminal crashes during execution, the scheduling server detects the loss of heartbeat and reschedules the task to other candidate nodes before the task deadline has passed; if the timeout has passed, it notifies the source terminal to degrade to local CPU execution.

[0040] Preferably, the AI ​​computer terminal further includes: a unified memory management module, which utilizes the unified memory architecture (UMA) of the AI ​​computer terminal to achieve physical memory sharing between the CPU and the NPU through page table mapping, thereby eliminating cross-device DMA copying; and a bandwidth awareness module, which monitors the memory bandwidth utilization of the NPU and the CPU and triggers rate limiting when the threshold is exceeded.

[0041] Preferably, the AI ​​computer terminal further includes a scene perception engine for identifying the current user scenario, which includes at least one of video conferencing, gaming, office work, development, media playback, and idle time; the scene perception engine is also used to preset differentiated hardware weights and overflow strategies for each scenario, and adjust scheduling parameters after confirmation by anti-jitter when the scenario changes.

[0042] When the computing power of a single AI PC cannot meet the current task requirements, the system implements cross-terminal overflow scheduling through the scheduling server. The specific process is as follows:

[0043] Step S1: When the source terminal detects that the overall load exceeds the preset threshold through the computing power balance scheduler, it sends an overflow request to the scheduling server, carrying task specification information, including estimated computing power, required NPU computing power, required operator type, maximum tolerable latency, and data sensitivity flag.

[0044] Step S2: The global scheduling engine of the scheduling server performs two rounds of matching: the first round is hard constraint filtering, which selects candidate nodes with load rate lower than the preset value, NPU operator compatibility, network round-trip latency less than the task deadline, security level meeting the requirements and sensitive data not leaving the domain; the second round is comprehensive scoring, which calculates the comprehensive score according to the weights of 40% load rate, 25% network latency, 15% energy consumption, 10% historical success rate, and 10% NPU operator matching degree, and selects the optimal target node.

[0045] Step S3: The scheduling server distributes the task to the target terminal, carrying the model parameters and input data. Preferably, the model parameters and input data are compressed using the Snappy or Zstd algorithm before transmission.

[0046] Step S4: The target terminal loads the model in the sandbox and executes NPU inference.

[0047] Step S5: The target terminal sends the execution result back to the source terminal via the scheduling server.

[0048] The present invention also provides an AI computer, which is applied to the above-mentioned centralized scheduling and management system for computing resources, and serves as a terminal node in the at least one AI computer terminal.

[0049] The AI ​​computer includes a CPU, an NPU, a memory, and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0050] Step 1: Collect the utilization data of the CPU and NPU in real time through the kernel-mode computing power monitoring driver;

[0051] Step 2: Intercept AI inference tasks through task interception and routing mechanisms;

[0052] Step 3: Execute a dynamic weight balancing algorithm based on the usage data to determine the scheduling strategy for the AI ​​inference task. The scheduling strategy includes allocating the task to the CPU or the NPU for execution.

[0053] Step 4: When the overall load of the NPU is detected to exceed a preset threshold, non-critical AI inference tasks are migrated back from the NPU to the CPU for execution.

[0054] Preferably, when the processor executes the computer program, it further performs the following steps:

[0055] Step 5: Construct a heterogeneous execution graph. During the model loading phase, the execution graph is constructed offline, and hardware affinity annotation (CPU / GPU / NPU) is performed on each operator. At runtime, it supports reactive tasks preempting the NPU computing power of proactive tasks and backfilling the small operators of proactive tasks in the gaps between the operator chains of the reactive tasks.

[0056] Preferably, when the processor executes the computer program, it further performs the following steps:

[0057] Step Six: Utilize the Unified Memory Architecture (UMA) of the AI ​​computer to achieve physical memory sharing between the CPU and the NPU through page table mapping, eliminating cross-device DMA copying; monitor the memory bandwidth utilization of the NPU and the CPU, and trigger rate limiting when it exceeds the threshold.

[0058] Preferably, when the processor executes the computer program, it further performs the following steps:

[0059] Step 7: Identify the current user scenario, which includes at least one of video conferencing, gaming, office work, development, media playback, and idle time; preset differentiated hardware weights and overflow strategies for each scenario, and adjust scheduling parameters after confirmation by anti-jitter when the scenario changes.

[0060] Preferably, when the processor executes the computer program, it further performs the following steps:

[0061] Step 8: Configure three QoS levels for different AI inference tasks. The three QoS levels include: a guaranteed resource reservation level, a basic resource plus available resource level, and a level that uses only remaining resources level. Compared with existing technologies, this invention has the following advantages:

[0062] 1. By setting up a computing power monitoring driver and a computing power balancing scheduler on the AI ​​computer terminal, this invention achieves effective communication and coordination between the CPU scheduler and the NPU driver. The computing power balancing scheduler automatically calculates the imbalance between the CPU and NPU and dynamically adjusts the computing power allocation ratio to ensure that the CPU and NPU work together, avoiding the problems of wasted computing resources and system response delays caused by lack of coordination in the prior art.

[0063] 2. The dynamic weight balancing algorithm of this invention comprehensively considers six dimensions of factors: utilization rate, temperature, power consumption, priority, scenario, and oscillation suppression. It dynamically allocates computing power based on actual load characteristics rather than a fixed strategy. Compared with the static allocation method in the prior art, which fixes the allocation to the NPU once an AI task is detected, this invention can dynamically adjust the computing power ratio between the CPU and NPU according to real-time load characteristics, effectively avoiding the problem of one side being overloaded while the other side is idle, and maximizing the utilization of the overall computing power of the AI ​​PC.

[0064] 3. Existing technologies only support unidirectional offloading from the CPU to the accelerator, and cannot migrate tasks back to the CPU when the NPU is overloaded. The computing power balancing scheduler of this invention actively migrates non-critical AI inference tasks from the NPU to the CPU for execution when it detects that the overall NPU load exceeds a preset threshold, achieving task-level reverse hot migration. Compared with existing patents that only involve operator-level subtask reverse distribution, this invention achieves the overall migration of complete AI inference tasks under system-level load driving, completely solving the problem of the NPU becoming a system bottleneck.

[0065] 4. This invention takes over AI inference requests through a layered task interception mechanism: The user-space framework adaptation module is the main path (intercepting tasks at the device selection and inference submission interface of the standard AI inference framework through a custom ExecutionProvider, inference plugin proxy, or delegate), while the user-space library interception module is the auxiliary path (intercepting processes that do not load the framework adaptation module but still call the standard AI inference library via dynamic linking by hooking their exported functions at the same semantic level). The intercepted calls construct task descriptors in user space and submit them to the scheduling center, which then arbitrates and routes them to the local CPU, local NPU, local GPU, hybrid execution, or remote nodes, and backfills the results according to the original API conventions. The kernel-space observation module only discovers, audits, and provides quality of service constraints for behaviors that bypass the standard inference framework and access accelerators via vendor SDKs or device I / O paths; it does not parse the complete model semantics, nor does it redirect system calls to scheduling execution paths. Therefore, forwardable AI tasks are uniformly intercepted only at the user-space inference semantic layer. For applications that have already integrated with the framework's adaptation path, they can be included in the scheduler without modifying the application code; for applications that have not loaded the adaptation module and use statically linked private runtimes, transparency is constrained by injection and compatibility scope.

[0066] 5. This invention achieves orderly allocation of resources when multiple AI applications are running concurrently through QoS three-level resource isolation: The Guaranteed level reserves 20% of the total NPU computing power (and at least 2 CPU cores), which cannot be preempted; the BurstShare level allocates 10% as a base, and can borrow up to 40% additionally during idle periods (totaling no more than 50%), which cannot encroach on the Guaranteed reservation; the BestEffort level only uses the remaining computing power and can be interrupted at any time. Arbitration is conducted according to QoS level, preemption priority within the same level, urgency of deadline, and arrival order, thereby reducing interaction stuttering caused by multiple applications competing for NPU.

[0067] 6. This invention employs a consistent hash ring management node in its scheduling server, and a global scheduling engine performs multi-factor weighted matching. By connecting each AI PC as a computing power node to the global network, it achieves automatic overflow of tasks from overloaded terminals to idle terminals. When the computing power of a single AI PC is insufficient, the system automatically schedules tasks to other idle AI PCs within the organization, aggregating the computing power of a large number of terminals during idle periods into a highly efficient computing power network. Compared to existing solutions that can only perform single-machine scheduling, this invention achieves a leap from single-machine scheduling to cross-terminal sharing capabilities.

[0068] 7. The terminal autonomy module of this invention automatically switches to a pure local eDWB scheduling mode when the terminal disconnects from the scheduling server, and seamlessly reconnects after the network recovers, realizing a dual-mode working mode of "shared computing power when connected to the network, and local autonomy when disconnected". At the same time, the scheduling server adopts a primary-backup architecture and Raft protocol to achieve failover. When the target terminal crashes during task execution, it automatically reschedules, ensuring the high availability and robustness of the system.

[0069] 8. This invention shields the interface differences between NPUs from Intel, Qualcomm, AMD, and Apple through an NPU hardware abstraction layer, supporting operator-level compatibility mapping and graceful degradation. Unsupported operators are automatically mapped to equivalent combinations or fall back to the CPU. Compared to single-vendor SDK solutions, this invention achieves seamless cross-platform and cross-brand deployment, significantly reducing the adaptation costs of AIPC computing power scheduling solutions for different NPU hardware.

[0070] 9. The heterogeneous execution graph and operator-level preemptive scheduling mechanism of the present invention constructs the execution graph offline during the model loading stage and performs hardware affinity annotation on each operator. During runtime, it supports reactive tasks preempting the NPU computing power of proactive tasks and backfills the small operators of proactive tasks in the gaps between reactive task operator chains, realizing the spatiotemporal reuse of NPU computing power at the microsecond / millisecond level. According to the test, in the AI ​​PC running scenario described in Example 1, the SoC utilization rate can be improved by 1.6 to 6.8 times. Attached Figure Description

[0071] Figure 1 This is a schematic diagram of the overall architecture of the centralized scheduling and management system for computing resources of this invention.

[0072] Figure 2 This is the core scheduling flowchart for AI task processing in this invention.

[0073] Figure 3 This is a flowchart of the cross-terminal overflow scheduling of the present invention. Detailed Implementation

[0074] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0075] like Figure 1 As shown, the present invention adopts a two-layer architecture of centralized management and terminal autonomy. The system includes, from bottom to top, a hardware base layer, a scheduling client, a scheduling server, and a security and governance layer that runs through the entire system. The four layers work together to realize a complete scheduling link from single-machine CPU-NPU computing power balancing to cross-terminal computing power sharing.

[0076] The hardware foundation layer includes at least one AI computer terminal, which is equipped with a central processing unit (CPU), a graphics processing unit (GPU), and a neural network processing unit (NPU), and may also include other accelerators such as FPGAs / ASICs. Modern AIPCs, such as the Intel Core Ultra series (integrated Intel AI Boost NPU), AMD Ryzen AI series, Qualcomm Snapdragon X series (integrated Hexagon NPU), and Apple M series (integrated Neural Engine), all feature this heterogeneous hardware configuration.

[0077] Preferably, the NPU is a neural network processor specifically designed to accelerate AI inference tasks, with its computing power measured in TOPS, and is specifically hardware-optimized for AI inference operations such as matrix multiplication and convolution.

[0078] The scheduling client is an intermediate scheduling layer that runs on the AI ​​computer terminal. It consists of two parts: kernel mode and user mode. The kernel mode includes a computing power monitoring driver, a task interception and routing mechanism, a computing power balancing scheduler, and an NPU hardware abstraction layer. The user mode includes a server communication module, a local task manager, and an AI framework compatibility layer.

[0079] The computing power monitoring driver runs in kernel mode and is used to collect CPU and NPU utilization data at preset time granularities. Preferably, the computing power monitoring driver collects CPU utilization, NPU utilization, core temperature, and power consumption data at 100ms granularities. The computing power monitoring driver provides the collected real-time indicator data to the scheduling center via an ioctl interface.

[0080] The task interception and routing mechanism is used to intercept AI inference tasks initiated on the AI ​​computer terminal.

[0081] Preferably, the task interception and routing mechanism includes a framework adaptation module, a user-mode library interception module, and a kernel-mode observation module. The three modules work together in a layered manner, and task construction and forwarding are only completed on user-mode paths with inference semantics.

[0082] The framework adaptation module, acting as the main interception path, runs in user space and is used to take over tasks at the device selection and inference submission interfaces of the AI ​​inference framework. Specifically, it intercepts calls to standard AI inference APIs (such as the ONNX Runtime session execution interface and the OpenVINO inference request interface) through custom ExecutionProviders, inference plugin proxies, or delegates. Based on the current session state, it constructs a task descriptor (ai_task_descriptor) containing task type, model features, and hardware preferences. The task descriptor is then passed to the scheduling center, which makes a decision on routing between the local CPU, local NPU, local GPU, hybrid execution, or remote nodes. Finally, the execution result is backfilled to the caller according to the original API convention.

[0083] The user-space library interception module, as an auxiliary interception path, runs in user space and is used to intercept processes that have not loaded the framework adaptation module but still call the standard AI inference library in a dynamic linking manner. It hooks the exported functions of the standard AI inference library through user-space API hooks (such as Detours on Windows, LD_PRELOAD on Linux, and DYLD_INSERT_LIBRARIES on macOS), constructs the task descriptor at the same inference semantic level, and passes it to the scheduling center, without rewriting the task at the GPU / NPU microkernel submission level.

[0084] The kernel-mode observation module runs in kernel mode and is used to discover, audit, and constrain quality of service (QoS) access to raw hardware that bypasses the standard AI inference API. Specifically, it can monitor system calls or driver-level access to GPU / NPU devices through eBPF, kprobe, or operating system device I / O observation mechanisms. This is used for identifying unknown accelerator access, recording audit logs, and rate limiting or quota control for low-priority tasks. The kernel-mode observation module does not parse the complete model semantics, does not construct separate task descriptors for cross-device or cross-terminal forwarding, and does not directly redirect system calls or device I / O requests to scheduling execution paths.

[0085] Therefore, forwardable AI tasks are intercepted and submitted for scheduling only by the framework adaptation module and / or the user-mode library interception module at the user-mode inference semantic layer; the kernel mode only provides observation and governance capabilities, thereby achieving the correct separation of task interception, scheduling decision and result backfilling at the capability boundary.

[0086] Preferably, the task descriptor (ai_task_descriptor) includes the following fields:

[0087] Task identifiers: task_id (globally unique task ID), parent_task_id (parent task ID), intercept_source (interception source: framework adaptation / library interception).

[0088] Application source: source_pid (initiating process PID), source_app (application identifier), source_node_id (source endpoint ID).

[0089] Scheduling semantics: task_type (task type, including inference, CV, NLP, ASR, TTS, LLM, multimodal, batch offline, etc.), priority (priority, 0 is the highest), qos_class (QoS level: Guaranteed / BurstShare / BestEffort), create_ts_ns (creation time), deadline_ns (absolute deadline (ns), specified by the application, configured by the user, or assigned by the system default; 0 = unlimited), max_latency_ms (maximum tolerable latency (ms), specified by the application, configured by the user, or assigned a default value according to the QoS level), scene_id (user scene).

[0090] Model and framework features: model_framework (AI framework, including ONNX Runtime, OpenVINO, DirectML, TensorRT, CoreML, TFLite, Llama.cpp, vLLM, etc.), model_name, model_hash (SHA-256, used for caching and remote hits), model_size_bytes, estimated_flops_m (estimated computational cost (MFLOPs), obtained by shape inference accumulation during ONNX graph loading), npu_ops_coverage (NPU operator coverage), required_ops_bitmap (required operator bitmap (256 bits), generated by traversing operator nodes during ONNX graph loading, aligned with the NPUHAL operator support table).

[0091] Input and output tensors: input_count / output_count, name, dtype (data type), shape (shape), nbytes (number of bytes), buffer_kind (buffer type: process virtual address / cross-device shared handle / framework native tensor handle / embedded small data).

[0092] Hardware preferences and constraints: preferred_device (preferred device: CPU / GPU / NPU / hybrid), forbidden_device_mask (disabled device mask), min_device_cap_tops (minimum computing power requirement (TOPS, INT8 caliber), calculated based on the total computational cost of the model and expected latency), require_unified_memory (whether UMA zero-copy is required).

[0093] Security and remote policies: is_sensitive (sensitivity flag (0 / 1), set to 1 when security_level is 2 or 3), allow_remote (allow cross-terminal overflow), allow_migrate (allow local hot migration), allow_preempt (allow preemption), security_level (security level (0-3), assigned according to data sensitivity classification criteria), data_domain (data domain label, used for same-domain matching constraints).

[0094] The execution backfill conventions are: completion_mode (synchronous / asynchronous), completion_token (asynchronous completion token), result_status (execution status), exec_device_actual (actual execution device), and exec_latency_ns (actual execution time).

[0095] The computing power balancing scheduler is used to execute a dynamic weight balancing algorithm based on real-time data collected by the computing power monitoring driver, and to determine the scheduling strategy for the AI ​​inference task between the CPU and the NPU. The scheduling strategy includes assigning the task to the CPU or the NPU for execution, and migrating non-critical AI inference tasks back from the NPU to the CPU for execution when the overall load of the NPU exceeds a preset threshold. The non-critical AI inference task is determined based on task priority tags and / or QoS levels.

[0096] Preferably, the dynamic weight balancing algorithm considers the following six factors when making scheduling decisions:

[0097] (1) Weighted difference in utilization between CPU and NPU: Calculate the weighted difference between CPU utilization and NPU utilization as the core indicator for judging whether computing power is balanced;

[0098] (2) Temperature status of the CPU and the NPU: When the temperature exceeds the safety threshold, the allocation of new tasks to the corresponding hardware is restricted to avoid overheating that could lead to performance degradation or hardware damage;

[0099] (3) Power consumption constraint: When the system is running in battery mode, tasks are routed to the NPU with higher energy efficiency to extend the device's battery life;

[0100] (4) Task priority tags: Real-time interactive tasks (such as video background blurring) are marked as non-transferable, and background batch processing tasks (such as document summarization and photo classification) are marked as priority transfer;

[0101] (5) Scene context: Different hardware weights are injected according to different user scenarios such as video conferencing, gaming, and office work to achieve differentiated scheduling based on scene awareness;

[0102] (6) Oscillation suppression: Automatically increase the inertia threshold during continuous reverse migration to avoid performance jitter caused by frequent back-and-forth migration of tasks between the CPU and NPU.

[0103] The decision result is one of the following five strategies: local CPU execution, local NPU execution, hybrid execution (operator-level splitting), remote overflow, or queuing.

[0104] The NPU hardware abstraction layer is used to shield the interface differences between NPUs from different manufacturers. Preferably, the NPU hardware abstraction layer supports at least one of the following manufacturers' NPUs: Intel NPU, Qualcomm NPU, AMD NPU, and Apple NeuralEngine. The NPU hardware abstraction layer is also used to automatically map the target operator to an equivalent operator combination or downgrade it to the CPU for execution when the target operator is not supported on the current NPU.

[0105] Preferably, the AI ​​computer terminal further includes a unified memory management module, which utilizes the unified memory architecture (UMA) of the AI ​​computer terminal to achieve physical memory sharing between the CPU and the NPU through page table mapping, eliminating cross-device DMA copying. In traditional solutions, DMA copying overhead is 12-40μs, while this invention reduces this overhead to 0.

[0106] Preferably, the AI ​​computer terminal further includes a bandwidth sensing module for monitoring the memory bandwidth utilization of the NPU and the CPU, and triggering rate limiting when the utilization exceeds a threshold. The bandwidth threshold is preferably 80% of the memory bandwidth utilization, and the rate limiting includes at least one of reducing the NPU's operating frequency, limiting batch size, or suspending memory access requests for non-critical tasks.

[0107] Preferably, the AI ​​computer terminal further includes a scene perception engine for identifying the current user scene, which includes at least one of video conferencing, gaming, office work, development, media playback, and idle time; the scene perception engine is also used to preset differentiated hardware weights and overflow strategies for each scene, and adjust scheduling parameters after anti-jitter confirmation (preferably 2 seconds) when the scene changes.

[0108] Preferably, the AI ​​computer terminal further includes an NPU virtualization module, which is used to uniformly abstract various computing power units such as CPU, GPU, and NPU, and provide a unified API access interface to the upper-layer scheduling module. The NPU virtualization module encapsulates the device discovery, session creation, memory allocation, task submission and synchronization operations of different computing power units into a unified API interface through a unified computing power abstraction layer, so that the computing power balance scheduler can access computing power units of different manufacturers and types through the same API call without being aware of the specific interface differences of the underlying hardware.

[0109] Preferably, the AI ​​computer terminal further includes a QoS resource isolation module for configuring three QoS levels for different AI inference tasks. The three QoS levels include: Guaranteed level (guaranteed resource reservation level), configured for system-level AI functions, reserving 20% ​​of the total NPU computing power, having the highest priority and not being preempted; BurstShare level (basic resources plus borrowable resources level), configured for user AI applications, with a basic allocation of 10% computing power, and up to 40% can be borrowed additionally when the NPU is idle; and BestEffort level (only use remaining resources level), configured for background analysis tasks, utilizing only the remaining computing power released by other tasks, and can be interrupted at any time.

[0110] Preferably, the QoS resource isolation module configures three QoS levels for different AI inference tasks, including:

[0111] (I) Definition of Level 3 QoS

[0112] Guaranteed Level (Guaranteed Resource Reservation Level): System-level AI functions (such as Windows Studio Effects background blur) are configured at this level. 20% of the total NPU computing power is reserved specifically for tasks at this level, which cannot be preempted, ensuring deterministic responses to real-time interactive tasks.

[0113] BurstShare Tier (Base Resources Plus Borrowable Resources Tier): This tier is configured for user AI applications (such as Copilot document summarization). It allocates 10% of the total NPU computing power as base, and allows borrowing up to 40% additionally when the NPU is idle (not exceeding 50% of the total NPU computing power for a single application), without encroaching on the reserved resources of the Guaranteed tier.

[0114] BestEffort Tier (Use Remaining Resources Only): Background analytics tasks (such as file indexing AI analysis) are configured to this tier. Only the remaining computing power released by the Guaranteed and BurstShare tiers is utilized, and can be interrupted or suspended at any time.

[0115] (ii) Arbitration Rules

[0116] QoS arbitration is conducted in the following lexicographical order:

[0117] QoS levels: Guaranteed > BurstShare > BestEffort, with higher-level tasks receiving priority computing resources;

[0118] Priority of preemption within the same QoS level: the lower the preempt_priority value, the higher the priority;

[0119] Deadline urgency: the closer the deadline_ns, the higher the priority (only when deadline_ns > 0 is it included in the sorting).

[0120] Arrival order: When the above conditions are the same, the task that arrives first has priority (FIFO).

[0121] (III) Resource Allocation Process

[0122] The following allocation process is executed every 100ms:

[0123] Step 1: Lock the reserved quota of the Guaranteed level (20% of the total NPU computing power) to ensure that it is not encroached upon;

[0124] Step 2: Allocate a base quota (10% of the total NPU computing power) to each task in the BurstShare level.

[0125] Step 3: Check the current idle NPU computing power. If there is any remaining power, borrow it out in order of priority according to the BurstShare level tasks. No single application may borrow more than 40% (total not exceeding 50% of the total NPU computing power).

[0126] Step 4: Allocate all remaining NPU computing power after the above allocation to BestEffort level tasks;

[0127] Step 5: If the NPU computing power is insufficient, preemption is determined according to the arbitration rules: the BestEffort task is preempted by the Guaranteed or BurstShare task, or is migrated back to the CPU or overflowed to a remote node by the eDWB algorithm.

[0128] (iv) Preemptive Recovery Strategy

[0129] Preemption only occurs at operator boundaries or time slice boundaries and does not affect the internal state of the currently executing operator.

[0130] When a BurstShare task is preempted by a Guaranteed task, the borrowed free quota (burst portion) should be reclaimed first, while preserving the basic 10% quota as much as possible to avoid frequent context switching;

[0131] Once a BestEffort task is preempted, the task status is suspended, and execution will resume in subsequent cycles according to the arbitration Rank order.

[0132] The server-side communication module is used to communicate with the scheduling server. It uses the gRPC+QUIC protocol to implement functions such as registration, heartbeat reporting, indicator reporting, and task distribution.

[0133] The local task manager is used to manage the local task queue, perform priority arbitration, and execute decisions.

[0134] The AI ​​framework compatibility layer is designed for transparent compatibility with more than 7 mainstream AI inference frameworks, including ONNX Runtime, OpenVINO, and DirectML, ensuring that AI applications can be integrated into this system without modifying their code.

[0135] The scheduling server is a centralized management layer running on a remote server, including a global scheduling engine, used to schedule tasks to other idle AI computer terminals when the local computing power of the AI ​​computer terminal is insufficient. Preferably, the scheduling server is deployed as an independent service process and can run on an independent physical server, virtual machine, or containerized environment. The scheduling server provides computing power service discovery and registration functions through a fixed port (preferably 16000). Preferably, the determination conditions for insufficient local computing power of the AI ​​computer terminal include at least one of the following: (1) the core temperature of the CPU or the NPU exceeds a preset temperature threshold (preferably 90℃); (2) the QoS resource quota is exhausted; (3) the utilization rate of the CPU and / or the NPU exceeds a preset utilization rate threshold (preferably 70%). When one of the above conditions is met, the computing power balancing scheduler initiates an overflow request to the scheduling server.

[0136] Preferably, the global scheduling engine uses multi-factor weighted matching to select target nodes, and the factors include: load rate, NPU operator compatibility, network latency, energy consumption, security level, task type, and historical success rate.

[0137] The scheduling server also includes a computing resource registration and discovery module, which uses a consistent hashing ring to manage the computing resources of each AI computer terminal, achieving O(log N) node lookup. When each AI computer terminal registers with the scheduling server, it generates a globally unique node identifier (node_id). The generation rule for the node identifier is as follows:

[0138] node_id = {region}-{nodetype}-{SHA256(cluster_id || device_id)[:8]}

[0139] Wherein, region is the region identifier (e.g., cn-sh, beijing, us-west, etc.), nodetype is the node type (e.g., AI_PC, SERVER, EDGE, etc.), cluster_id is the cluster identifier, device_id is the unique device identifier (e.g., hardware UUID or MAC address), || represents string concatenation, and SHA256(cluster_id || device_id)[:8] represents performing a SHA256 hash operation on the concatenated string and taking the first 8 hexadecimal characters. The scheduling server uses the node identifier as the node key value of the consistent hash ring to map each AI computer terminal to the hash ring. When it is necessary to select a target node for a task, the task ID or model hash in the task descriptor is used as the key value to search on the hash ring, searching for the nearest node clockwise, achieving a node location efficiency of O(log N). During registration, each AI computer terminal simultaneously reports node capability description information, including hardware configuration (CPU / GPU / NPU model and computing power), current load status, list and version of installed AI frameworks, supported task types and NPU operator coverage, energy consumption strategy and power status, security level, and historical scheduling statistics. The scheduling server stores the above information in the node registry for hard constraint filtering and comprehensive scoring of the global scheduling engine.

[0140] The scheduling server also includes a computing power sharing network management module (node ​​lifecycle management, topology maintenance, and failover) and a fault tolerance and high availability module (master-slave architecture + Raft protocol, recovery time target RTO < 5 seconds).

[0141] The security and governance layer spans the hardware foundation layer, the scheduling client, and the scheduling server, and includes:

[0142] Transmission encryption module: Secure communication is achieved using TLS 1.3 and mTLS two-way authentication;

[0143] Data isolation module: Implements multi-task isolation using process-level sandboxes and cgroup resource limits;

[0144] Task audit module: The audit log is tamper-proof by using full-link tracing and hash chain to record the complete life cycle of each task (CREATED→SCHEDULED→DISPATCHED→EXECUTING→COMPLETED / FAILED).

[0145] Model protection module: used to restrict sensitive models from executing outside the domain in cross-terminal scheduling based on data sensitivity marking, and to protect model intellectual property rights by combining parameter encryption and inference watermarking.

[0146] like Figure 2 As shown, the AI ​​task processing flow of this invention is divided into four stages.

[0147] Phase 1: Task Generation and Interception

[0148] When a task is intercepted by the framework adaptation module or the user-space library interception module, the system fills in the fields of the task descriptor according to the current session state, including task type, source application, QoS level, framework type, model hash, estimated computational load, required operator bitmap, input and output tensor description, hardware preference, security constraints, etc., and then passes the task descriptor to the scheduling center.

[0149] The model hash in the task descriptor is used for model cache management and cross-terminal scheduling hits: when executing locally, the scheduling client looks up the local model cache repository based on the model hash; when a remote overflow occurs, the scheduling server determines whether the target terminal has cached the model based on the model hash. If it has been cached, only the input data is transmitted; if it has not been cached, the model parameters are transmitted together or downgraded.

[0150] Phase 2: Scheduling Decisions

[0151] The scheduling center queries local CPU / NPU real-time metrics, obtains 100ms granular usage, temperature, and power consumption data collected in kernel mode via ioctl, and executes an enhanced dynamic weight balancing algorithm.

[0152] The enhanced dynamic weight balancing algorithm makes decisions based on the following six factors:

[0153] (1) Weighted difference in CPU and NPU utilization: Calculate the weighted difference Δ between CPU utilization and NPU utilization. When Δ exceeds the inertia threshold, a scheduling decision is triggered.

[0154] (2) CPU and NPU temperature status: When the core temperature of the CPU or NPU exceeds the safety threshold, the allocation of new tasks to the corresponding hardware is restricted, and tasks are migrated to hardware with lower temperature first.

[0155] (3) Power consumption constraint: When the system is running in battery mode, tasks are routed to the NPU with higher energy efficiency to extend the device's battery life;

[0156] (4) Task priority tags: Real-time interactive tasks (such as video background blurring, with a latency requirement of <10ms) are marked as non-transferable to ensure user experience; background batch processing tasks (such as document summarization, photo classification, with a latency requirement of >100ms) are marked as priority to be transferred.

[0157] (5) Scene context: Different hardware weights are injected according to different user scenarios such as video conferencing, gaming, and office work to achieve differentiated scheduling based on scene awareness;

[0158] (6) Oscillation suppression: Automatically increase the inertia threshold during continuous reverse migration to avoid performance jitter caused by frequent back-and-forth migration of tasks between the CPU and NPU.

[0159] Preferably, the dynamic weight balancing algorithm makes scheduling decisions according to the following quantification method.

[0160] (a) Smoothing of usage rate

[0161] CPU and NPU utilization data were collected at 100ms intervals, and the collected values ​​were smoothed using an exponentially weighted moving average (EWMA). The smoothing window consisted of 5 sampling points, and the smoothing coefficient λ = 0.4.

[0162]

[0163] Where U(t) is the current sampled value, This represents the smoothed CPU utilization value. Let the smoothed CPU utilization be U. cpu NPU utilization rate is U npu The values ​​range from [0, 100].

[0164] (ii) Temperature and pressure coefficients and weighted utilization rate

[0165] Define a temperature-pressure coefficient ϕ(T) to quantify the impact of temperature on hardware scheduling priority:

[0166]

[0167] Among them, the safe temperature threshold T safe Preferably 70℃, with a maximum temperature threshold T maxPreferably 90℃. When the temperature is below T safe When the temperature-pressure coefficient is 0, temperature does not affect the utilization rate assessment; when the temperature is between T... safe With T max Between these two values, the temperature-pressure coefficient increases linearly with increasing temperature; when the temperature exceeds T... max At this time, the corresponding hardware enters an overload state.

[0168] The weighted utilization rate U' is the smoothed utilization rate multiplied by the temperature and pressure coefficient:

[0169]

[0170] (III) Hardware weight and overall load

[0171] Hardware weights α and β are defined to measure the relative importance of the CPU and NPU in comprehensive load computing. Preferably, the hardware weights are allocated inversely proportional to the nominal computing power.

[0172]

[0173] Where C cpu and C npu These are the equivalent INT8 computing power (TOPS) of the CPU and NPU, respectively; when there is no nominal computing power value, α=β=0.5.

[0174] Define a comprehensive load stress index P to determine the overall load status of the current system:

[0175]

[0176] in and As scene weights, they are dynamically adjusted by the scene-aware engine based on the current user scene. (Default scene) = =1; In video conferencing scenarios, take... =1.2, to strengthen the load weight on the NPU side; in game scenarios, take =1.3, to strengthen the CPU-side load weight.

[0177] The threshold division of the comprehensive load pressure index P is as follows:

[0178] When P < 30, the overall system load is determined to be low, and scheduling is not triggered (NO_OP).

[0179] When 30≤P≤85, the system is determined to be within the locally adjustable range, and local CPU-NPU balanced scheduling is executed.

[0180] When P > 85, the overall system load is determined to be too high, triggering cross-terminal overflow or load reduction.

[0181] Preferably, the preset threshold is 85.

[0182] (iv) Imbalance and Inertia Threshold

[0183] The imbalance Δ is defined as the absolute difference between the CPU-weighted utilization and the NPU-weighted utilization:

[0184]

[0185] Set the inert threshold Δ th This is used to avoid frequent task migration between the CPU and NPU. The initial value of the inertia threshold is preferably 15. When three consecutive migrations occur in opposite directions, an oscillation suppression mechanism is triggered, and the inertia threshold is dynamically adjusted according to the following rules:

[0186]

[0187] That is, the inertia threshold increases by 50% each time, but does not exceed the upper limit of 40.

[0188] When Δ≥Δ th When a significant load imbalance is detected between the CPU and NPU, a migration direction determination is triggered:

[0189] If U cpu ′−U npu ′≥Δ th If the CPU load is determined to be significantly higher than the NPU load, forward unloading from the CPU to the NPU is executed.

[0190] If U npu ′−U cpu ′≥Δ th If the load on the NPU side is determined to be significantly higher than that on the CPU side, a reverse migration from the NPU to the CPU is executed.

[0191] (v) NPU overall load and reverse backflight triggering conditions

[0192] Define the NPU overall load L npu Used to accurately determine the true load status on the NPU side:

[0193]

[0194] in Normalized length of the NPU task queue:

[0195]

[0196] q npu q represents the current NPU task queue depth. max This is the upper limit of the queue depth, preferably 16.

[0197] The reverse transferback trigger condition is: L npu ≥Θ npu Preferred Θ npu =85, and CPU-side weighted utilization U cpu If the value is less than 70, it means the CPU still has a receiving margin. If the CPU has no receiving margin, then a cross-terminal overflow is triggered instead of a local migration.

[0198] (vi) Quantitative assessment of non-critical tasks

[0199] Define a transferable fraction S for task i. i This is used to quantify the priority of the task being suitable for migration from the NPU to the CPU for execution:

[0200]

[0201] The symbols are defined as follows:

[0202] Q i For QoS mobility: Guaranteed level is 0, BurstShare level is 1, and BestEffort level is 2;

[0203] Π i Priority transferability: p i This represents the task priority value, with 0 being the highest; p max This is the highest priority value;

[0204] D i For time delay leniency: when there is a deadline constraint, When there is no deadline constraint, D i =1;

[0205] M i For migration permission: set to 1 if allow_migrate=1, otherwise deny the migration directly;

[0206] R i Real-time / Interactive Penalty: When the task type is a real-time task (such as ASR real-time, CV background blurring) or the maximum tolerable latency does not exceed 30ms, R... i =1; otherwise R i =0.

[0207] The recommended weights are a=0.40, b=0.25, c=0.20, and d is gating (forced S when Mi=0). i =−∞), e=0.35.

[0208] The hard rules for determining non-critical tasks are as follows:

[0209] (1) QoS hard constraint: Tasks with qos_class=Guaranteed are prohibited from backtracking, and non-critical tasks are deemed false;

[0210] (2) Priority hard constraint: priority≤p_crit (recommended p) crit Tasks with value 1) are prohibited from being relocated; non-critical checks are considered false.

[0211] (3) Soft decision: Not Guaranteed and allow_migrate=1 and S i ≥S th (Recommended S) th Tasks with a value of 1.2 are classified as non-critical tasks and are allowed to be migrated back.

[0212] In summary, the Boolean decision for non-critical tasks is:

[0213] That is: the QoS level is BurstShare or BestEffort, the priority is lower than the critical priority threshold, and migration is allowed; real-time interactive tasks are excluded by default.

[0214] The rescheduling execution order is as follows: All reschedulable tasks in the NPU queue are executed in sequence according to S. i Sort in descending order and migrate to high-priority tasks in turn, until U. npu ′≤Θ npu -δ (δ=10, used for hysteresis debouncing), or CPU-side U cpu Until ≥70.

[0215] (vii) Initial assignment of new tasks

[0216] For a newly arrived task i, select the optimal execution device from the local set of available devices D⊆{CPU,NPU}:

[0217]

[0218] in:

[0219] U d The current weighted utilization rate of device d;

[0220] npu_ops_coverage i The NPU operator coverage for task i (0-100);

[0221] cost d Let be the energy consumption cost coefficient of equipment d;

[0222] γ1=0.6, γ2=0.25, and γ3=0.15 are the recommendation weights.

[0223] The constraints include:

[0224] Follow the preferred_device and forbidden_device_mask in the task descriptor;

[0225] When the QoS level is Guaranteed and the NPU reserved quota is not full, priority will be given to allocating to the NPU;

[0226] Reduce the energy cost of the NPU in battery mode. NPU This is to improve NPU allocation preference.

[0227] The decision outcome is one of the following five strategies: local CPU execution, local NPU execution, hybrid execution (operator-level splitting), remote overflow, or queuing. Hybrid execution (operator-level splitting) refers to splitting the computation graph of an AI inference model into multiple subgraph partitions according to operator granularity and hardware affinity of each operator. These subgraphs are then executed in parallel or pipelined on different hardware devices (CPU, NPU, GPU), and the execution results of each subgraph are merged and returned. Specifically, this includes the following steps:

[0228] (1) Model loading stage: parse the ONNX model computation graph, perform operator fusion on operators that can be fused such as Conv+BN+ReLU, and split the fused operators into independently schedulable cores; query the operator support bitmap of the NPU hardware abstraction layer to obtain the support status of each operator on the current NPU, and mark the primary execution device and fallback device for each operator.

[0229] (2) Graph partitioning stage: Merge consecutive operators labeled as the same device into subgraphs; use ONNXRuntime's GraphPartitioner in conjunction with the GetCapability() capabilities of each ExecutionProvider to divide the complete computation graph into multiple subgraph partitions;

[0230] (3) Step-by-step execution and result merging stage: each subgraph partition is executed in the topology order; intermediate tensors are transmitted across device edges via H2D / D2H copy or UMA zero copy method; the execution results of each subgraph are merged into a complete output and backfilled to the caller.

[0231] The hybrid execution occurs within the framework adaptation layer (EP / Plugin / Delegate layer), and for the application, it is still a standard inference API call. The application does not need to be aware of the internal process of model splitting and cross-device execution.

[0232] When the local computing power shortage condition is triggered, the computing power balancing scheduler marks the current task as pending overflow and sends an overflow request to the scheduling server. If the scheduling server has no available idle nodes or the network is unreachable, the task is downgraded to local CPU execution (fallback strategy).

[0233] Phase 3: Task Execution

[0234] Based on the decision, tasks are executed on the local CPU, local NPU, remote nodes, or simultaneously on multiple paths. Remotely executed tasks are isolated through a sandbox to ensure the security of the host terminal.

[0235] During task execution, the AI ​​computer terminal utilizes the Unified Memory Architecture (UMA) through the Unified Memory Management Module to achieve physical memory sharing between the CPU and NPU via page table mapping, eliminating cross-device DMA copying. The bandwidth awareness module monitors the memory bandwidth utilization of the NPU and CPU, and triggers rate limiting when the threshold is exceeded to avoid memory bandwidth contention affecting inference performance.

[0236] The scene-aware engine continuously identifies user scenes during the execution of the current task and dynamically adjusts hardware weights and overflow strategies based on the identified scenes, so that the system always adapts to the computing power requirements of the current scene.

[0237] During the model loading phase, the system constructs a heterogeneous execution graph offline and performs hardware affinity annotation (CPU / GPU / NPU) on each operator. At runtime, the system supports reactive tasks preempting the NPU computing power of proactive tasks and backfilling the gaps in the operator chains of the reactive tasks with the small operators of the proactive tasks, thereby improving SoC utilization.

[0238] Phase 4: Results Feedback and Auditing

[0239] The results of multi-path execution are aggregated and returned to the initiating AI application. The complete task lifecycle (CREATED→SCHEDULED→DISPATCHED→EXECUTING→COMPLETED / FAILED) is written into an immutable audit log chain. The audit log chain adopts a hash chain structure, and each log block contains the hash value of the previous block to ensure the tamper-proof nature of the logs.

[0240] like Figure 3 As shown, when the computing power of a single AI PC cannot meet the current task requirements, the system implements cross-terminal overflow scheduling through the server. The specific process is as follows.

[0241] Step 1: Initiate an overflow request

[0242] When terminal A (source terminal, overloaded) detects insufficient local computing power through the computing power balancing scheduler, it sends an overflow request (OVERFLOW_REQ) to the scheduling server, carrying task specification information, which includes:

[0243] (1) Estimating computational load: When the session is created, the ONNX model computation graph is traversed, and the floating-point operation load (FLOPs) of each operator is accumulated using shape inference and recorded in units of MFLOPs;

[0244] (2) Required NPU computing power: Based on the total computational load of each operator in the model and the expected latency, the minimum required computing power value (TOPS, INT8 caliber) is converted.

[0245] (3) Required operator type: When creating the ONNX graph, traverse each operator node in the model and encode the operator type into an operator bitmap for NPU operator compatibility matching during cross-terminal scheduling;

[0246] (4) Maximum tolerable latency: Explicitly specified by the application API, or assigned a default value according to the following rules: GuaranteedQo S / Real-time interactive tasks default 50ms, interactive inference tasks default 500ms, batch inference tasks default 5000ms, BestEffort / Background analytics tasks default 30000ms (or 0 = no constraint), Specification Example / Uncategorized tasks default 100ms;

[0247] (5) Data sensitivity marking: Public data (level 0) is allowed to be remotely accessed across domains; internal data (level 1) is only allowed within the same organization / cluster node; sensitive data (level 2) is prohibited from leaving the domain and is only allowed within the same region / local area network; high-density / high-sensitivity model (level 3) is prohibited from remote access and is only allowed to be executed locally or requires TEE and must remain within the same domain.

[0248] Preferably, the criteria for determining insufficient local computing power include at least one of the following:

[0249] (1) The core temperature of the CPU or the NPU exceeds a preset temperature threshold, preferably 90°C;

[0250] (2) QoS resource quota exhausted;

[0251] (3) The utilization rate of the CPU and / or the NPU exceeds a preset utilization rate threshold, preferably 70%.

[0252] The overflow request carries task specification information, including estimated computational load, required NPU computing power, required operator type, maximum tolerable latency, and data sensitivity flag.

[0253] The scheduling server supports two working modes:

[0254] (a) Local Area Network Decentralized Mode

[0255] When each AI computer terminal starts up, it starts the computing power service on a fixed port (preferably 16000) and periodically announces its own computing power information (including node ID, CPU / GPU / NPU computing power, current load status, IP address and port) through UDP broadcast or multicast.

[0256] Terminals discover the computing power services of other terminals by scanning the fixed ports within the local area network. The scanning method is as follows: the terminal sends a probe request to the fixed port of each IP address within the local area network. The terminal that receives the probe response is a usable computing power node, and its information is registered in the local list of usable nodes.

[0257] Terminals communicate directly in a point-to-point manner. Task distribution and result feedback are transmitted directly between terminals through the fixed port without needing to be forwarded through a central node.

[0258] Each terminal periodically (preferably every 5 seconds) sends a heartbeat message through the fixed port. If the heartbeat times out several times in a row, the corresponding node is marked as offline and removed from the list of available nodes.

[0259] When a terminal exits, it broadcasts an exit message to the local area network through the fixed port. Upon receiving the message, other terminals immediately remove the node from the list of available nodes.

[0260] (ii) Centralized mode across local area networks

[0261] When a central server exists, each AI computer terminal registers with the central server. The registration information includes node ID, IP address and port, CPU / GPU / NPU computing power information, network location, and security domain label. The central server maintains a global node list, allowing other terminals to query and discover computing power nodes across the local area network.

[0262] When scheduling across local area networks, the source terminal queries the central server for candidate nodes that meet the criteria. The central server returns a list of matching terminals, and the source terminal then establishes a direct communication connection with the target terminal to distribute tasks.

[0263] If the terminal does not register with the central server, the computing power service will only be discovered within the local area network by scanning the fixed port.

[0264] Preferably, the overall load is a weighted average of CPU utilization and NPU utilization, and the preset threshold is 85%.

[0265] Step 2: Two rounds of matching to filter target nodes

[0266] The global scheduling engine on the scheduling server performs two rounds of matching:

[0267] First round of hard constraint filtering: Select candidate nodes that satisfy all of the following conditions:

[0268] The load rate is lower than the preset value, which is preferably 70%.

[0269] Load < 5% 1.00

[0270] Load <35% 0.50

[0271] Load < 70% 0.20

[0272] Load > 70%

[0273] NPU operator compatibility (target terminal NPU supports the types of operators required for the task);

[0274] Incompatible with 0

[0275] It can just run 0.5.

[0276] Compatible with 1.0

[0277] The network round-trip time (RTT) is less than the task deadline;

[0278] RTT < 5ms 1.00

[0279] RTT < 15ms 0.70

[0280] RTT < 25ms 0.50

[0281] RTT < 50ms 0.20

[0282] RTT > 50ms 0.00

[0283] The security level meets the requirements;

[0284] Sensitive data will not leave the domain. Tasks marked as sensitive data will not be scheduled to external terminals, but will only be executed locally or downgraded. The task deadline is the value of the deadline_ns field in the task descriptor, which is specified by the application, configured by the user, or assigned by the system default. When the system performs subtask splitting, the source terminal calculates and assigns a corresponding deadline to each subtask based on the overall task response time requirements and parallel splitting strategy.

[0285] Second round of comprehensive scoring: For candidate nodes that pass the hard constraint filtering, the comprehensive score is calculated according to the following weights:

[0286] Load rate: 40%

[0287] Network latency: 25%

[0288] Energy consumption: 15%

[0289] Historical success rate: 10% Actual execution count / Total number of queries

[0290] NPU operator matching degree: 10%

[0291] The node with the highest overall score is selected as the optimal target node (terminal B).

[0292] Preferably, other idle AI computer terminals refer to AI computer terminals with a load rate of less than 35%. When there are multiple idle terminals with a load rate of less than 35%, the comprehensive score is prioritized for selection from the idle terminals; if there are no idle terminals, the optimal target is selected from the available terminals with a load rate of less than 70%.

[0293] Step 3: Task Distribution

[0294] The scheduling server distributes the task (TASK_DISPATCH) to terminal B (an idle node). The payload prioritizes carrying the model content hash (model_hash) input data: if the hash is already matched in terminal B's local model repository or cache, the model weights are not transmitted; only when the model size is below a preset threshold (preferably 32MB) and the cache is not matched, are the model parameters and input compressed together with Snappy or Zstd and transmitted along the path. Large parameter models (such as LLM 7B) are prohibited from transmitting weights in their entirety on the overflow hot path; if a match is missed, the candidate node is changed, the model is preheated, and then rescheduled, or it is downgraded to local execution at the source. Terminal B caches the weights and compilation session of the loaded model according to the hash to accelerate subsequent similar tasks.

[0295] Step 4: The target terminal performs inference.

[0296] Terminal B loads the model and executes NPU inference in the sandbox. The sandbox is a process-level sandbox, which ensures that remote tasks do not affect the normal use of the host terminal through process isolation and cgroup resource limits.

[0297] Step 5: Result Feedback

[0298] Terminal B sends the execution result (TASK_RESULT) back to Terminal A via the scheduling server. Terminal A then aggregates the multi-path execution results and returns them to the initiating AI application.

[0299] Terminal downtime fault tolerance mechanism: Preferably, the downtime of the target terminal is detected by the scheduling server through a heartbeat mechanism. Each AI computer terminal sends a heartbeat message to the scheduling server at a fixed period (preferably 1 second). The heartbeat message includes the terminal identifier, current load status, temperature, and network connectivity status. When the scheduling server fails to receive a heartbeat message from a terminal for N consecutive times (preferably 3 times), it determines that the terminal is down and triggers task rescheduling.

[0300] Preferably, the downtime includes one of the following situations:

[0301] (1) Terminal overheating restart: The terminal core temperature exceeds the hardware protection threshold (such as CPU / GPU / NPU temperature exceeding 95°C-100°C), triggering system restart or forced shutdown;

[0302] (2) Terminal network abnormality: The terminal network connection is interrupted (such as Wi-Fi disconnection or network cable being unplugged) or the network suffers severe packet loss, resulting in the inability to deliver heartbeat messages;

[0303] (3) Other abnormalities that cause the terminal to malfunction: including but not limited to operating system crashes, process crashes, hardware failures (memory errors, storage failures, power failures, etc.).

[0304] When the target terminal is determined to be down, the scheduling server checks whether the deadline of the current task has passed; if it has not passed, the task is rescheduled to other candidate nodes (such as terminal C); if it has passed, the source terminal is notified to be downgraded to local CPU execution (fallback strategy).

[0305] Terminal Disconnection Autonomy Mechanism: When terminal A loses network access, it automatically switches to pure local eDWB scheduling (terminal autonomy), and seamlessly reconnects to the scheduling server after network recovery. Seamless reconnection is achieved as follows: When the terminal detects network recovery, it automatically re-initiates a connection request to the scheduling server, carrying the terminal identifier (node_id) and identity token (node_token); after the scheduling server verifies the identity, it re-registers the terminal to the consistent hash ring, restoring its online status. After the terminal reconnects, it immediately synchronizes its current status via heartbeat messages, including current load (CPU / NPU utilization), temperature, cache model hash list, etc., enabling the scheduling server to update the terminal's available capabilities in real time. The time interval between detecting network recovery and rejoining the available node pool is less than 3 seconds, thus achieving seamless reconnection.

[0306] The terminal autonomy module enables the AI ​​computer terminal to accept global scheduling from the scheduling server when connected to the network, and automatically switch to a pure local scheduling mode when disconnected from the network, thereby realizing a dual-mode working mode of sharing computing power when connected to the network and local autonomy when disconnected from the network.

[0307] This invention also provides an AI computer, which is applied to the aforementioned centralized scheduling and management system for computing resources, serving as a terminal node in at least one AI computer terminal. The AI ​​computer includes a CPU, an NPU, a memory, and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0308] Step 1: Collect CPU and NPU utilization data in real time through the kernel-mode computing power monitoring driver;

[0309] Step 2: Intercept AI inference tasks through task interception and routing mechanisms;

[0310] Step 3: Execute a dynamic weight balancing algorithm based on the usage data to determine the scheduling strategy for AI inference tasks. The scheduling strategy includes allocating tasks to the CPU or NPU for execution.

[0311] Step 4: When the overall load of the NPU exceeds a preset threshold, non-critical AI inference tasks are migrated back from the NPU to the CPU for execution. When the overall load of the NPU exceeds a preset threshold, tasks are determined to be non-critical AI inference tasks based on task priority tags and / or QoS levels, and these non-critical AI inference tasks are migrated back from the NPU to the CPU for execution.

[0312] The AI ​​computer terminal is equipped with a unified memory management module, a bandwidth awareness module, a scene awareness engine, an NPU virtualization module, and a QoS resource isolation module. When the processor executes the computer program, it implements the following preferred steps by calling the above modules:

[0313] Preferably, when the processor executes the computer program, it further performs the following steps:

[0314] Step 5: Construct a heterogeneous execution graph. During the model loading phase, the execution graph is constructed offline, and hardware affinity annotation (CPU / GPU / NPU) is performed on each operator. At runtime, it supports reactive tasks preempting the NPU computing power of proactive tasks and backfilling the small operators of proactive tasks in the gaps between the operator chains of the reactive tasks.

[0315] Preferably, when the processor executes the computer program, it further performs the following steps:

[0316] Step Six: Utilize the Unified Memory Architecture (UMA) of the AI ​​computer to achieve physical memory sharing between the CPU and NPU through page table mapping, eliminating cross-device DMA copying; monitor the memory bandwidth utilization of the NPU and CPU, and trigger rate limiting when the threshold is exceeded.

[0317] Preferably, when the processor executes the computer program, it further performs the following steps:

[0318] Step 7: Identify the current user scenario, which includes at least one of video conferencing, gaming, office work, development, media playback, and idle time; preset differentiated hardware weights and overflow strategies for each scenario, and adjust scheduling parameters after confirmation by anti-jitter when the scenario changes.

[0319] Preferably, when the processor executes the computer program, it further performs the following steps:

[0320] Step 8: Configure three QoS levels for different AI inference tasks. The three QoS levels include: a guaranteed resource reservation level, a basic resource plus available resource level, and a level that only uses remaining resources.

[0321] Example 1: CPU-NPU collaborative scheduling of a standalone AI PC (pure local mode)

[0322] This example uses an AI PC equipped with an Intel Core Ultra 7 155H processor (16 CPU cores, Intel AI Boost NPU 11.5 TOPS) to run the following AI tasks simultaneously:

[0323] Windows Studio Effects background blurring, NPU intensive, requires real-time processing of 720p video streams;

[0324] Microsoft Copilot documentation summary: moderate computational load, batch processing nature;

[0325] Local photo AI classification, large number of small images, batch inference.

[0326] The existing approach involves all three tasks directly requesting NPU resources. Background blurring, due to its real-time requirements, consumes most of the NPU time, while document summarization and photo classification queue in the NPU, while the CPU operates at low load (<30%). The NPU becomes the bottleneck, with document summarization experiencing delays exceeding 2 seconds and photo classification progressing slowly.

[0327] The processing flow of this invention:

[0328] Step 1: The computing power monitoring driver detected that the NPU utilization rate was 95% and the CPU utilization rate was only 25%.

[0329] Step 2: The eDWB algorithm calculates the imbalance Δ = |25% - 95%| = 70%, which far exceeds the 15% threshold, indicating that the NPU is severely overloaded;

[0330] Step 3: The dispatch center makes the following decisions:

[0331] Background blurring (Guaranteed QoS, real-time task) → Maintain NPU execution, reserve 20% of NPU computing power;

[0332] Document summary (BurstShare QoS, non-real-time) → migrated back to CPU execution (reverse overflow);

[0333] Photo categorization (BestEffort QoS, background task) → migrated back to CPU execution, using an idle-time backfilling strategy;

[0334] Step 4: Final result: NPU utilization dropped to 65% (background blur only), CPU utilization rose to 55% (taking over document summarization and photo classification), background blur latency remained smooth with a delay of <10ms, document summarization latency dropped from 2 seconds to 300ms, and overall NPU utilization increased from 35% to over 65%.

[0335] Example 2: Sharing computing power among multiple AI PCs within an enterprise (LAN mode)

[0336] This example uses 10 AI PCs (5 Intel Core Ultra, 3 AMD Ryzen AI, and 2 older laptops without NPUs) owned by a design department, deployed on the same gigabit local area network. The design manager initiates a batch task: to perform AI image enhancement (super-resolution and noise reduction) on 1000 high-resolution design drafts, with an estimated time of 2 hours per machine.

[0337] The existing approach involves the task being executed only on the design supervisor's computer, taking approximately two hours. During this time, the computer's AI computing power is exclusively used, affecting other operations. The NPUs of the remaining nine computers remain idle, resulting in wasted computing power.

[0338] The processing flow of this invention:

[0339] Step 1: The design supervisor initiates a batch AI image enhancement task on their AI PC (source terminal);

[0340] Step 2: The source terminal's eDWB algorithm detects that the NPU is fully loaded (>90%), and the overall load exceeds 85%, automatically triggering cross-terminal overflow;

[0341] Step 3: The source terminal sends an overflow request to the department's scheduling server (running on a small server);

[0342] Step 4: After two rounds of matching, the server-side global scheduling engine:

[0343] The augmentation task for 600 of the images was distributed to 5 Intel Core Ultra AI PCs (each PC processed 120 images).

[0344] Distribute 300 of the images to 3 AMD Ryzen AI PCs (each PC processes 100 images).

[0345] Two old laptops without NPUs were not used for AI inference tasks;

[0346] The remaining 100 images will be processed by the source terminal itself;

[0347] Step 5: After approximately 12 minutes (parallel processing), all results are sent back to the source terminal and aggregated;

[0348] Step 6: Final result: The total task time was reduced from 120 minutes to 12 minutes (10 times faster). All scheduled terminals could still work normally during task execution. The remote task ran in the sandbox and did not affect the host.

[0349] Example 3: Intelligent Perception and Scheduling in Video Conferencing Scenarios (Scene Awareness Mode)

[0350] This example uses a user conducting a video conference using Teams (with background blurring and real-time noise reduction, and NPU load) and simultaneously receiving a document from a colleague that requires AI summarization. The Windows indexing service in the background is performing AI content analysis on the new file.

[0351] Current technology handles this by having three AI tasks compete for NPU resources. Background blurring in video conferencing may experience stuttering or image quality degradation due to NPU overload. Users need to manually disable the background indexing service to restore meeting quality.

[0352] The processing flow of this invention:

[0353] Step 1: The scene perception engine identifies the current scene as a "video conference" by using multi-dimensional signals (camera on, microphone activated, Teams foreground and continuous network uplink);

[0354] Step 2: The scenario strategy matrix automatically enables "Meeting Mode":

[0355] Background blurring / noise reduction (Guaranteed QoS) → 30% of NPU computing power is reserved to ensure real-time processing (60fps);

[0356] Document Summary (BurstShare QoS) → Allocate 10% of the NPU's base computing power; if the NPU is idle, it can be borrowed up to 40%.

[0357] Background index (BestEffort QoS) → Suspend NPU usage, route all CPUs to low-priority execution;

[0358] Step 3: If the NPU temperature exceeds the safety threshold, preferably 70°C, further non-critical tasks will be overflowed to other AC-powered AI PCs within the local area network.

[0359] Step 4: Meeting ends (camera off) → Scene automatically switches to "office" mode, restoring NPU access for document summarization and background indexing;

[0360] Step 5: Final result: The video conference was smooth throughout (background blurred at 60fps), and the document summary was completed normally at a slightly slower speed during the meeting without any manual intervention from the user.

[0361] Example 4: Cross-operating system hybrid terminal computing power sharing (heterogeneous compatibility mode)

[0362] This example uses a startup team's computers with multiple operating systems: 3 Windows AI PCs (Intel NPU), 2 macOS AI PCs (Apple Neural Engine), and 1 Linux workstation (NVIDIA GPU). The team needs to conduct AI model performance benchmark tests, requiring them to run the same inference task on multiple hardware platforms.

[0363] The existing approach involves team members manually running tests on their own computers and summarizing the results, which is time-consuming and prone to errors.

[0364] The processing flow of this invention:

[0365] Step 1: The 6 terminals register with the same scheduling server through their respective platform's scheduling client;

[0366] Step 2: Automatically report the NPU / GPU capabilities of each device during registration:

[0367] Windows terminal: Intel AI Boost, 11.5 TOPS INT8 / 5.75 TOPS FP16, supports Conv2D / MatMul / Softmax, etc.;

[0368] macOS terminal: Apple Neural Engine, 18 TOPS, supports Core ML operator subsets;

[0369] Linux terminal: NVIDIA RTX 4060, 15 TFLOPS FP16, supports TensorRT operator subsets;

[0370] Step 3: The test initiator submits a benchmark test task (ONNX model) in a unified format, and the server automatically routes the test based on the compatibility of the NPU operators of each terminal.

[0371] Step 4: The NPU hardware abstraction layer translates the ONNX standard operators into native operator calls of the corresponding vendor on each terminal;

[0372] Step 5: Unsupported standard operators, such as LayerNorm, are not supported on Intel NPUs and will automatically fall back to CPU execution;

[0373] Step 6: Final result: A benchmark test task is automatically executed in parallel on 6 heterogeneous terminals and the results are aggregated, taking about 1 / 6 of the time of the traditional method.

[0374] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A centralized scheduling and control system for computing resources, characterized in that, It includes a scheduling client, a scheduling server, and at least one AI computer terminal; The AI ​​computer terminal is equipped with a central processing unit (CPU) and a neural network processor (NPU). The scheduling client runs on the AI ​​computer terminal and includes a computing power monitoring driver, a task interception and routing mechanism, a computing power balancing scheduler, and an NPU hardware abstraction layer. The computing power monitoring driver is used to collect the utilization rate data of the CPU and the NPU according to a preset time granularity. The task interception and routing mechanism is used to intercept AI inference tasks initiated on the AI ​​computer terminal; the computing power balancing scheduler is used to execute a dynamic weight balancing algorithm based on real-time data collected by the computing power monitoring driver, and to decide the scheduling strategy of the AI ​​inference task between the CPU and the NPU; the scheduling strategy includes allocating the task to the CPU or the NPU for execution, and migrating non-critical AI inference tasks back from the NPU to the CPU for execution when the overall load of the NPU is detected to exceed a preset threshold; the non-critical AI inference tasks are determined according to task priority tags and / or QoS levels; wherein, responsive interactive tasks are determined to be critical tasks, and background batch processing tasks are determined to be non-critical tasks; the NPU hardware abstraction layer is used to shield the interface differences of NPUs from different manufacturers; The scheduling server runs on a remote server and includes a global scheduling engine, which is used to schedule tasks to other idle AI computer terminals for execution when the local computing power of the AI ​​computer terminal is insufficient.

2. The centralized scheduling and management system for computing resources as described in claim 1, characterized in that: The dynamic weight balancing algorithm considers the following factors when making decisions: The weighted utilization difference between the CPU and the NPU; The temperature status of the CPU and the NPU; Power consumption constraints; Task priority tags; Scene context; And oscillation suppression, automatically increasing the inertia threshold during continuous reverse migration.

3. The centralized scheduling and management system for computing resources as described in claim 1, characterized in that: The task interception and routing mechanism includes: The framework adaptation module, as the main interception path, runs in user space and is used to take over tasks at the device selection and inference submission interfaces of the AI ​​inference framework. The user-space library interception module, as an auxiliary interception path, runs in user space and is used to intercept processes that have not loaded the framework adaptation module. And a kernel-mode observation module, which runs in kernel mode, used to discover, audit and constrain the quality of service of bare hardware access that bypasses the standard AI inference API; The framework adaptation module and the user-state library interception module construct task descriptors in the user-state reasoning semantic layer and submit them to the scheduling center. The kernel-mode observation module does not construct task descriptors for cross-device or cross-terminal forwarding, nor does it redirect system calls or device I / O requests to scheduling execution paths.

4. The centralized scheduling and management system for computing resources as described in claim 1, characterized in that: The scheduling client also includes: The NPU virtualization module is used to uniformly abstract multiple computing power units, including at least the CPU and the NPU, and provide a unified API access interface to the upper-layer scheduling module, so that the computing power balance scheduler can access different computing power units through the same API. The system also includes a QoS resource isolation module, which is used to configure three levels of QoS for different AI inference tasks. The three levels of QoS include: a guaranteed resource reservation level, a basic resource plus available resource level, and a level that only uses the remaining resources level.

5. The centralized scheduling and management system for computing resources as described in claim 1, characterized in that: The NPU hardware abstraction layer supports at least one of the following vendors' NPUs: Intel NPU, Qualcomm NPU, AMD NPU, and Apple NeuralEngine. The NPU hardware abstraction layer is also used to automatically map the target operator to an equivalent operator combination or downgrade it to the CPU for execution when the target operator is not supported on the current NPU.

6. The centralized scheduling and management system for computing resources as described in claim 1, characterized in that: The global scheduling engine uses multi-factor weighted matching to select target nodes. The factors include: load rate, NPU operator compatibility, network latency, energy consumption, security level, task type, and historical success rate. The scheduling server also includes a computing resource registration and discovery module, which uses a consistent hash ring to manage the computing resources of each AI computer terminal.

7. The centralized scheduling and management system for computing resources as described in claim 1, characterized in that: The AI ​​computer terminal also includes a terminal autonomy module, which is used to automatically switch to a pure local scheduling mode when the AI ​​computer terminal is disconnected from the scheduling server, and to automatically reconnect to the scheduling server after the network is restored.

8. The centralized scheduling and management system for computing resources as described in claim 1, characterized in that: The AI ​​computer terminal also includes: The unified memory management module utilizes the unified memory architecture (UMA) of the AI ​​computer terminal to achieve physical memory sharing between the CPU and the NPU through page table mapping, eliminating cross-device DMA copying. And a bandwidth awareness module, used to monitor the memory bandwidth utilization of the NPU and the CPU, and trigger rate limiting when the threshold is exceeded.

9. The centralized scheduling and management system for computing resources as described in claim 1, characterized in that: The AI ​​computer terminal also includes a scene perception engine for identifying the current user scenario, which includes at least one of video conferencing, gaming, office work, development, media playback, and idle time. The scene perception engine is also used to preset differentiated hardware weights and overflow strategies for each scenario, and to adjust scheduling parameters after confirmation by anti-jitter when the scenario changes.

10. An AI computer, applied to a centralized scheduling and management system for computing resources as described in any one of claims 1-9, characterized in that, The AI ​​computer serves as a terminal node in the at least one AI computer terminal, including: CPU and NPU; A memory on which computer programs are stored; and a processor, which, when executing the computer program, performs the following steps: The kernel-mode computing power monitoring driver collects the utilization data of the CPU and the NPU in real time. Intercept AI inference tasks through task interception and routing mechanisms; A dynamic weight balancing algorithm is executed based on the usage data to determine the scheduling strategy for the AI ​​inference task. The scheduling strategy includes allocating the task to the CPU or the NPU for execution. And when the overall load of the NPU is detected to exceed a preset threshold, non-critical AI inference tasks are migrated back from the NPU to the CPU for execution.

Citation Information

Patent Citations

  • Task back-off between data processors

    CN121957794A