Kernel-level heterogeneous collaborative AI reasoning method, device and equipment and storage medium

Abstract

The invention discloses a kernel-level heterogeneous collaborative AI reasoning method and device, equipment and a storage medium, and relates to the field of AI reasoning acceleration. When the KDS receives a reasoning request submitted by a user space, generating a cross-device plan list according to a global system resource state and issuing the cross-device plan list to the HCEE; the HCEE drives the corresponding target execution equipment according to the task execution sequence, reads the data information from the UCMP, or restores intermediate / result data reasoned and output by the target execution equipment; after reasoning of the plan list is completed, the kernel space triggers an event notification mechanism, and a user process of the user space is awakened; and the user process directly reads the reasoning result from the UCMP. According to the scheme, deep collaboration of task scheduling, memory management and equipment execution is realized on a Linux kernel level, the problems of high overhead, resource islands, memory redundant copy, extensive scheduling granularity and the like of a user mode reasoning framework system are solved, and end-to-end low-delay, high-throughput and high-energy-efficiency AI reasoning acceleration is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of AI inference acceleration, in particular to a kernel-level heterogeneous collaborative AI inference method and device, equipment and storage medium. BACKGROUND

[0002] The rapid development of artificial intelligence technology promotes the wide deployment of AI inference tasks in cloud computing centers, edge computing nodes, terminal devices and embedded systems, etc. The underlying support platform is generally constructed by heterogeneous computing units such as central processing units (CPUs), graphics processing units (GPUs) and neural processing units (NPUs), showing high diversity and heterogeneity.

[0003] The current mainstream inference framework, such as TensorRT, ONNX Runtime and OpenVINO, runs in the user space of the operating system. These architectures have the following inherent defects: High system overhead: frequent system calls and kernel trap / in operations bring significant context switching overhead; Heterogeneous resource island: CPU and accelerator are managed by independent software stacks, lacking global resource view and collaborative scheduling capability at the system level, which easily leads to uneven resource utilization; Redundant memory copy: data needs to be copied multiple times between user memory, kernel memory and accelerator memory, occupying PCIe bandwidth and increasing transmission delay; Coarse scheduling granularity: existing frameworks usually schedule in units of entire models or large subgraphs, and cannot dynamically allocate tasks across devices at a finer-grained operator level according to real-time load conditions, lacking system flexibility.

[0004] The existence of these problems is mainly because the existing inference framework runs in the user space, which is isolated from the operating system kernel and underlying hardware resources, and cannot directly manage and schedule underlying resources. Therefore, an AI inference runtime architecture is needed that can be deeply integrated into the operating system kernel and uniformly manage and coordinate computing, memory and scheduling resources. SUMMARY

[0005] The embodiments of the present application provide a kernel-level heterogeneous collaborative AI inference method, device, equipment and storage medium, which has the advantages of reducing context switching overhead, improving heterogeneous resource utilization, eliminating data copying and realizing fine-grained cross-device scheduling.

[0006] In one aspect, the application provides a kernel-level heterogeneous collaborative AI inference method, which is used in the Linux system kernel space, and is provided with a kernel dynamic scheduler for multi-process scheduling, a shared memory pool for multi-process memory allocation, and a heterogeneous collaborative engine for scheduling heterogeneous computing resources; the method comprises: When the kernel dynamic scheduler receives an inference request submitted by the user space, a cross-device plan list is generated according to the global system resource state, and is issued to the heterogeneous collaborative engine; the plan list includes all planned tasks, and the execution devices and task execution orders allocated to the planned tasks; the execution devices include GPU / NPU and CPU; The heterogeneous collaborative engine obtains the plan list, drives the corresponding target execution device according to the task execution order, reads data information from the shared memory pool based on the planned task, and stores the intermediate / result data output by the target execution device; When the plan list inference is completed, the kernel space triggers an event notification mechanism to wake up the user process in the user space; the user process directly reads the inference result from the shared memory pool.

[0007] Specifically, before the kernel dynamic scheduler receives the inference request, the method further comprises: When the kernel space is started, the model weights are extracted according to the model configuration file, and the shared memory pool is constructed through the memory management mechanism and the IOMMU input / output memory management unit; The model weights are loaded into the target partition of the shared memory pool and are locked; The kernel thread of the kernel dynamic scheduler is initialized, and the Linux real-time scheduling strategy is set to receive the inference request submitted by the user space.

[0008] Specifically, the shared memory pool is constructed based on the system physical memory, and is respectively mapped to the user virtual address space through mmap, mapped to the kernel virtual address space through DMA, and mapped to the GPU / NPU device address space through IOMMU / UVA technology; The shared memory pool is provided with an input partition, a weight partition, an output partition, and a shared partition; the input partition stores the data to be inferred, the weight partition stores the model weights, the output partition stores the inference result, and the shared partition stores the intermediate data and temporary data.

[0009] Specifically, when the kernel space is started, the model weights are sent from the kernel virtual address space to the weight partition; when the thread in the user space initiates an inference request, the data to be inferred is mapped from the user virtual address space to the input partition; intermediate data and temporary data are sent from the shared partition to the target execution device or are sent back from the target execution device to the shared partition when the inference task is completed; At the end of the inference, the inference result is sent back from the result partition to the user virtual address space.

[0010] Specifically, the kernel dynamic scheduler generates a cross-device scheduling list according to the global system resource state, including: Based on the Linux real-time scheduling strategy management and receiving the inference request, the computing graph task unit (GTU) independent of the execution device is obtained; the GTU is loaded based on the user state model management library and is obtained by disassembling the original model file; Based on the monitoring mechanism of the performance monitoring unit, the system resource utilization rate is collected in real time, including the usage rate of the CPU matrix acceleration unit (AMX), the usage rate of the internal stream multi-processor (SM) of the GPU / NPU, and the PCIe bandwidth utilization rate; Based on the system resource utilization rate and the target optimization strategy, the GTU or independent operator is dynamically allocated to the target execution device, and the task execution order is determined according to the computing graph dependency relationship; the target optimization strategy is to minimize the end-to-end delay or maximize the system throughput.

[0011] Specifically, the heterogeneous collaborative engine includes a CPU AMX backend and a GPU / NPU backend, which are respectively used to submit CPU operator task to drive the CPU AMX and submit GPU operator task to drive the GPU / NPU device to execute.

[0012] Specifically, when the allocated scheduling task is a cross-device cascading task, an event synchronization point is inserted in the scheduling task at the event synchronization point; the heterogeneous collaborative engine issues the scheduling task according to the event order, after the completion of the previous scheduling task, the next scheduling task is triggered based on the intermediate data written back to the shared memory pool, and the intermediate data is sent to another execution device; After the inference task is completed, the heterogeneous collaborative engine sends a completion signal to the kernel dynamic scheduler, and the kernel dynamic scheduler feeds back an asynchronous event notification to the user process based on the completion signal.

[0013] On the other hand, the present application provides a kernel-level heterogeneous collaborative AI inference device, which is used in the Linux system kernel space, and is provided with a kernel dynamic scheduler for multi-process scheduling, a shared memory pool for multi-process memory allocation, and a heterogeneous collaborative engine for scheduling heterogeneous computing resources; the device comprises: A task generation module is configured to generate a cross-device plan list according to a global system resource state when the kernel dynamic scheduler receives a user space submitted inference request, and to send the plan list to the heterogeneous collaborative engine; the plan list includes all planned tasks, and the execution devices and task execution sequences allocated to the planned tasks; the execution devices include GPU / NPU and CPU; A task scheduling module is configured to drive the corresponding target execution device according to the task execution sequence, and to read data information from the shared memory pool and store the intermediate / result data output by the target execution device based on the planned task, when the heterogeneous collaborative engine obtains the plan list. A feedback module is configured to trigger an event notification mechanism when the plan list inference is completed, to wake up the user process of the user space; and the user process reads the inference result directly from the shared memory pool.

[0014] In another aspect, the present application provides a computer device, which includes a processor and a memory, and the memory stores at least one instruction, at least one program, a code set or an instruction set, which is loaded and executed by the processor to implement the kernel-level heterogeneous collaborative AI inference method of the above-mentioned aspect.

[0015] In another aspect, the present application provides a computer readable storage medium, which stores at least one instruction, at least one program, a code set or an instruction set, which is loaded and executed by a processor to implement the kernel-level heterogeneous collaborative AI inference method of the above-mentioned aspect.

[0016] The technical scheme provided by the embodiments of the present application has at least the following beneficial effects: the present application significantly reduces the system overhead in the AI inference process. By sinking the key components to the kernel space, the frequent switching between user mode and kernel mode is reduced, and the context switching overhead is reduced. The dynamic scheduling mechanism based on global resource state improves the utilization efficiency of heterogeneous computing resources and avoids the resource island problem. The unified shared memory pool design eliminates the redundant data copying between user mode, kernel mode and device memory, reduces the data transmission delay and bandwidth occupation. The fine-grained task scheduling capability makes the system more flexible to respond to real-time load changes, and improves the responsiveness and throughput of the overall system. These improvements work together to effectively solve the system performance bottleneck faced by the existing AI inference framework, and provide strong support for AI inference tasks in an efficient heterogeneous computing environment. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1FIG. 1 is a schematic diagram of a kernel-level heterogeneous collaborative AI inference system provided by an embodiment of the present application; Figure 2 FIG. 2 is a flowchart of a kernel-level heterogeneous collaborative AI inference method provided by an embodiment of the present application; Figure 3 FIG. 3 shows a mapping structure schematic diagram of a UCMP memory pool; Figure 4 FIG. 4 is an AI inference acceleration interaction process flowchart based on the system module described above; Figure 5 FIG. 5 shows a structure block diagram of a kernel-level heterogeneous collaborative AI inference apparatus provided by an embodiment of the present application; Figure 6 FIG. 6 shows a structure block diagram of a computer device provided by an example embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical scheme and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0019] As mentioned herein, "a plurality of" refers to two or more. The "and / or" describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent: A exists alone, A and B exist together, and B exists alone. The character " / " generally represents that the associated objects before and after are in an "or" relationship.

[0020] In the face of the above problems, the present application first considers sinking resource management and task scheduling to the operating system kernel layer to break through the user state isolation limit. The traditional scheme tries to optimize scheduling through thread pool and memory pool in the user state, but is limited by inter-process isolation and device driver access authority, and cannot realize the global resource view across processes. Another idea is to use a dedicated hardware middleware to coordinate heterogeneous computing units, but this will introduce additional hardware costs and compatibility problems. Further analysis shows that the kernel dynamic scheduler (KDS) can directly access hardware performance counters such as performance monitoring units (PMU), real-time sensing of CPU core AMX (Advanced Matrix Extensions), GPU streaming multiprocessor (SM), and PCIe bandwidth utilization, etc. to provide data support for dynamic scheduling. At the same time, the kernel-level shared memory pool can use IOMMU / UVA technology to unify the address space, fundamentally eliminating cross-device data copying. Based on this, the present application chooses to build a three-layer collaborative architecture in the kernel space, realizes real-time scheduling through kernel threads, uses the memory management subsystem to build a unified memory pool, and designs a heterogeneous engine to directly drive the computing unit.

[0021] Figure 1 is a schematic diagram of a kernel-level heterogeneous collaborative AI inference system provided by an embodiment of the present application, The acceleration system is divided into user space, kernel space and heterogeneous devices of hardware layer from the structure. The user space and the kernel space are two running states based on the operating system level. The user state is the normal running state of the application program, and the application program has limited permissions in this mode. Specifically in terms of permissions, only user space memory can be accessed, and hardware cannot be directly operated or privileged instructions cannot be executed. In terms of function implementation, system calls are used to request the operating system to complete file operations, network communication and other tasks. The kernel state is the running state of the operating system kernel, which has the highest permission. In terms of permission range, all instructions can be executed, all memory spaces can be accessed, and hardware resources can be directly operated. In terms of function execution, system calls, interrupt requests, memory management, process scheduling and other core tasks can be processed. Whenever a system call (such as file operation), exception or interrupt (such as hardware failure) operation occurs, the state switching mechanism will be triggered (the present application focuses on the AI program starting the AI acceleration inference request).

[0022] The hardware layer heterogeneous device is a device that specifically executes AI acceleration, such as an acceleration card or an acceleration chip composed of a CPU, a GPU, an NPU and the like. Cross-device running is to be able to split and distribute an AI acceleration task to multiple heterogeneous devices according to the actual running state of the system to maximize resource utilization.

[0023] The system is integrated in the Linux system kernel space, and the core components are set in the kernel space, including a kernel dynamic scheduler, a unified copyless memory pool (UCMP) and a heterogeneous co-execution engine (HCEE).

[0024] The KDS scheduler itself has the functions of dynamic priority adjustment, multi-scheduling strategy support, context switching optimization, and resource load balancing. It dynamically adjusts the priority of the process by monitoring the CPU resource usage, I / O waiting time and other indicators of each process in real time.

[0025] The UCMP memory pool is a technology for optimizing memory data sharing and transmission. Its core function is to eliminate CPU participation and memory bandwidth waste in traditional data copying through hardware and kernel collaboration mechanisms, reduce context switching, and bring low latency and high bandwidth utilization.

[0026] The HCEE engine is a system-level technology that realizes dynamic task allocation and collaborative processing by integrating computing units of different architectures (such as CPU, GPU, NPU, FPGA, etc.). It allocates CPU general-purpose computing, GPU parallel computing, DSP signal processing and other tasks to the most suitable computing unit through hardware layer division and cooperation, achieving double improvement of energy efficiency and computing density. The system uses the KDS scheduler to monitor system resources and task division, and then uses the HCEE engine to manage and allocate each hardware computing device, and realizes cross-domain data sharing and zero-copy through the combination of the UCMP memory pool, to solve the above-mentioned technical problems.

[0027] Figure 2 The flowchart of the kernel-level heterogeneous co-execution AI inference method provided by the embodiment of the application is used in the Linux system kernel space. A kernel dynamic scheduler KDS for multi-process scheduling, a shared memory pool UCMP for multi-process memory allocation, and a heterogeneous co-execution engine HCEE for scheduling heterogeneous computing resources are set in the kernel space. The method comprises: S1, when the KDS scheduler receives an inference request submitted by the user space, a cross-device plan list is generated according to the global system resource state, and is issued to the HCEE engine; The user space submitted inference request, i.e., a request initiated by an application layer AI program, because the core component is set in the kernel space in the present application, only needs to submit and deliver a task to the kernel space, and the KDS scheduler detects the system resource state, such as CPU, GPU / NPU, FPGA and the like resource utilization, and simultaneously combines the computing characteristics of the chip device to specifically allocate the task, and generates a cross-device plan list. The plan list includes all planned tasks and the execution device and task execution order allocated for the planned tasks. For example, a complex computing task in an entire AI inference task is executed by a CPU, and a parallel simple computing task is executed by a GPU, and each computing task is also marked if there is a sequence, and then is handed over to the HCEE engine.

[0028] S2, the HCEE engine acquires the plan list, drives the corresponding target execution device according to the task execution order, and reads data information from the UCMP memory pool based on the planned task, or stores the intermediate / result data output by the target execution device inference; The UCMP memory pool is a unified memory management area constructed in the kernel space, and can be implemented by using physical memory reservation combined with mmap user space mapping and IOMMU / UVA technology. By eliminating the data copy operation between the CPU and the accelerator and between the user state and the kernel state, the bandwidth occupation and delay problem caused by cross-device data transmission is solved.

[0029] For the request initiated by the AI application program, the related data is mapped to the UCMP memory pool by mmap, and in addition, the model parameter or weight data is also written into the UCMP memory pool in advance. In this way, when the HCEE engine allocates a specific planned task, the target execution device allocated by the task is determined, and then the corresponding data information is extracted from the UCMP memory pool according to the task number and sent into the target execution device for execution. Similarly, the scheduling of all heterogeneous execution devices is controlled by the HCEE engine, and the intermediate data generated by the execution device in the inference process needs to be stored back to the UCMP memory pool according to the task instruction, and the result data is the same.

[0030] S3, when the plan list inference is completed, the kernel space triggers an event notification mechanism to wake up the user process of the user space; the user process directly reads the inference result from the UCMP shared pool.

[0031] The event notification mechanism is an asynchronous communication mechanism between the kernel and the user state process, and can be implemented by using an eventfd file descriptor combined with an epoll event listening mechanism. When the inference task is completed, the kernel directly triggers a user process wake-up operation to solve the context switching overhead problem caused by frequent system calls.

[0032] In summary, KDS, as a multi-process scheduler, is responsible for receiving inference requests submitted by user space and generating a cross-device plan list based on the global system resource state. UCMP is used for multi-process memory allocation, providing a unified memory management mechanism. HCEE is responsible for scheduling heterogeneous computing resources to drive different execution devices to complete inference tasks. This design achieves unified management and collaborative scheduling of heterogeneous computing resources by sinking key components to the kernel space. KDS can make real-time decisions based on global resource status, UCMP eliminates data copying between different memory spaces, and HCEE provides fine-grained task scheduling capabilities. The collaborative work of the three components solves the problems of high system overhead, resource islands, memory redundant copying, and coarse-grained scheduling in user-space inference framework, and realizes end-to-end low-latency, high-throughput, and high-energy-efficient AI inference acceleration.

[0033] In one possible embodiment, KDS scheduler, UCMP memory pool, and HCEE engine can be integrated in the Linux kernel. Using the Linux real-time scheduling policy, the KDS scheduler priority is set to the highest. The UCMP memory pool reserves a continuous area in the physical memory through the memory management system and maps it to the kernel space, user space, and GPU / NPU device address space through the page table mapping. The HCEE engine, as a kernel module, contains CPU and GPU / NPU backend drivers.

[0034] When a user space application needs to perform an AI inference task, it submits an inference request to the KDS scheduler through a system call. After receiving the request, the KDS scheduler first obtains system resource utilization, such as CPU matrix acceleration unit (AMX), GPU / NPU internal stream multiprocessor (SM) usage, and PCIe bandwidth utilization. Based on this information, the KDS scheduler generates an execution list containing multiple planned tasks, each specifying the execution device (CPU or GPU / NPU) and the execution order.

[0035] After receiving the execution list from the KDS scheduler, the HCEE engine starts executing the tasks in order. For CPU tasks, the HCEE engine directly calls the corresponding kernel function; for GPU / NPU tasks, the HCEE engine submits the computing core through the device driver. During task execution, the HCEE engine reads input data and model parameters from the UCMP memory pool, and controls the target execution device to write intermediate results and final output back to the UCMP memory pool, which avoids frequent data copying operations in traditional methods.

[0036] When all tasks are executed, the HCEE engine triggers an event through the eventfd mechanism of the Linux kernel to wake up the user process waiting for the results. The user process can then read the inference results directly from the UCMP memory pool mapped to the user space memory area without additional data transfer operations.

[0037] Through the above scheme, the present application significantly reduces the system overhead in the AI inference process. By sinking the key components to the kernel space, the frequent switching between user mode and kernel mode is reduced, and the context switching overhead is reduced. The dynamic scheduling mechanism based on global resource state improves the utilization efficiency of heterogeneous computing resources and avoids the problem of resource island. The unified shared memory pool design eliminates the redundant data copying between user mode, kernel mode and device memory, reduces the data transmission delay and bandwidth occupation. The fine-grained task scheduling capability enables the system to respond more flexibly to real-time load changes, improving the responsiveness and throughput of the overall system. These improvements work together to effectively solve the system performance bottleneck faced by existing AI inference frameworks, providing strong support for AI inference tasks in an efficient heterogeneous computing environment.

[0038] In some embodiments, before the KDS scheduler receives the inference request, the system also needs to be initialized and configured, the process includes: When starting the kernel space, the model weights are extracted according to the model configuration file, and the UCMP memory pool is constructed through the memory management mechanism and the IOMMU / UVA technology; Load the model weights into the target partition of the UCMP memory pool and lock them; Initialize the kernel thread of the KDS scheduler and set the Linux real-time scheduling policy to receive the inference request submitted by the user space.

[0039] Specifically, in the system initialization phase, the kernel parses the model structure according to the predefined model configuration file and extracts the weight parameters. Through the memory management subsystem, a physically continuous memory region is allocated, and combined with the address translation capability of IOMMU / UVA, a unified physical address space view across CPU, GPU / NPU is established. After the model weight is loaded into the weight partition of UCMP, it is kept in resident memory through the memory page locking mechanism to avoid additional overhead due to page replacement during inference. In the initialization process of the kernel dynamic scheduler KDS, a dedicated kernel thread is created and a real-time scheduling strategy is configured to ensure that the scheduler can respond to high-priority inference requests in a timely manner. Under this architecture, all computing devices can directly access the data in the shared memory pool without going through data copying between the user state and the kernel state, and the model weight remains accessible throughout the inference life cycle. For example, in the target detection scenario, the weight data of the YOLOv5s model is fixed at the 0x0000_0000 offset address region of the UCMP, the input frame is directly written to the 0x4000_0000 input partition by the video capture thread, the intermediate activation value generated during inference is stored in the 0xC000_0000 shared partition, and the final detection result is output to the 0x8000_0000 output partition. This memory management mechanism enables the CPU AMX unit and the GPU computing unit to directly access the same physical memory region, eliminating the delay caused by copying the video memory through the PCIe bus in traditional solutions.

[0040] All inference data, including input / output tensors, intermediate activation values, and model weights, are stored in this memory pool, completely eliminating data mapping between CPU and accelerator, as well as between kernel state and user state.

[0041] The above-mentioned UCMP memory pool is constructed based on system physical memory and is mapped to user virtual address space through mmap, to kernel virtual address space through DMA, and to GPU / NPU device address space through IOMMU / UVA. Figure 3 The mapping structure of the UCMP memory pool is shown, and the physical memory construction can use continuous physical memory allocation or a set of non-continuous but persistent physical memory pages, such as reserving 8GB of continuous memory region through kernel startup parameters or dynamically allocating persistent physical pages. In the triple address mapping mechanism, the mmap system call maps the physical memory to the virtual address space of the user process; the kernel virtual address mapping is completed through the dma_alloc_coherent function, which realizes the direct access of the kernel module to the memory pool; based on the IOMMU / UVA technology, the conversion table items of device physical address and system physical address are established, the UCMP physical page is registered to the GPU page table, and the unified virtual address access of CPU and GPU is realized.

[0042] Specifically, in the system initialization phase, the physical memory is reserved by the kernel memory management subsystem, for example, 8GB of continuous memory region is reserved by the cma=8G startup parameter. In the address mapping process, the user process maps the UCMP memory pool to its virtual address space through the mmap() system call, and the mapping parameters include permission flags and partition flags; the GPU device driver establishes a DMA scatter-gather list through dma_map_sg() to map the physical memory page to the device address space; and the kernel module converts the physical address to the kernel virtual address through dma_alloc_coherent() to realize low-latency access.

[0043] Further, the UCMP memory pool is provided with an input input partition, a weight weight partition, an output output partition, and a shared shared partition; the input partition stores to-be-reasoned data, the weight partition stores model weights, the output partition stores reasoning results, and the shared partition stores intermediate data and temporary data.

[0044] When the kernel space is started, the model weights are sent from the kernel virtual address space to the weight partition; when the thread of the user space initiates a reasoning request, the to-be-reasoned data is sent from the user virtual address space to the input partition. When the reasoning plan task, the intermediate data and the temporary data are sent from the shared partition to the target execution device, or are reversely stored from the target execution device to the shared partition, and the reasoning result also needs to be stored in the output partition. Correspondingly, at the end of the reasoning, the reasoning result is output from the output partition to the user virtual address space.

[0045] In the above process, when the user thread submits a reasoning request, the direct mapping channel between the user space virtual address and the input partition physical address is used to realize one-time writing of the input data, which can be accessed by the kernel state and the accelerator device. When the intermediate data is transmitted between the heterogeneous devices, the bidirectional mapping feature of the shared partition is used to complete the data transmission through address space remapping instead of physical copying, for example, the calculation result of the AMX unit in the cross-device cascade task is directly mapped to the GPU device address space through the shared partition. At the end of the reasoning, the physical memory of the result partition is mapped back to the virtual address space of the user process through the mmap mechanism, and the user process can directly read the structured data without the need for data transfer from the kernel state to the user state. Through the pre-allocation of the physical memory and the multi-level address mapping mechanism, the whole process constructs a zero-copy transmission path throughout the whole life cycle of the data, reduces the data copying operation from at least four times in the traditional scheme to zero times, and reduces the address conversion times from twice per transmission to a single permanent mapping.

[0046] In some embodiments, the KDS scheduler generates a cross-device planning list according to the global system resource state, including: Based on the Linux real-time scheduling policy management and receiving inference requests, the computing graph task unit GTU independent of the execution device is obtained; Based on the monitoring mechanism of the performance monitoring unit PMU, the system resource utilization rate is collected in real time, including the usage rate of the CPU matrix acceleration unit AMX, the usage rate of the stream multi-processor SM inside the GPU / NPU, and the PCIe bandwidth utilization rate; Based on the system resource utilization rate and the target optimization strategy, the target execution device is dynamically allocated for the GTU or independent operator, and the task execution order is determined according to the computing graph dependency relationship; the target optimization strategy is to minimize the end-to-end delay or maximize the system throughput.

[0047] The GTU therein is loaded based on the user state model management library and is obtained by disassembling. The user state model management library is linked to the user application program as a lightweight runtime library, loads the original model file (such as.onnx,.pt, etc.), performs model analysis, graph optimization (such as constant folding, operator fusion), and other heavy and device-independent preprocessing work. A device-independent computing graph (Intermediate Representation, IR) is generated, so that the scheduling decision is decoupled from the hardware type, which only describes the type of operator, input / output tensor, and calculation dependency relationship, and is not bound to specific hardware. The management library also registers the computing graph IR to the KDS through a system call, and returns a model handle for subsequent inference, and submits a task to the kernel, which includes the model handle, input / output address, and other information.

[0048] The performance monitoring unit collects the CPU matrix acceleration unit usage rate through the kernel performance event interface, such as the instruction execution cycle proportion of Intel AMX or ARM SME; acquires the stream multi-processor utilization rate through the GPU driver interface, such as the active thread bundle proportion of SM in NVIDIA GPU; and monitors the bandwidth utilization rate through the PCIe performance counter, such as the number of data packets transmitted per millisecond. During dynamic allocation, when the target optimization strategy is to minimize the end-to-end delay, the computation-intensive operator is preferentially allocated to the device with a utilization rate lower than the preset threshold; when the target optimization strategy is to maximize the system throughput, a pipeline scheduling strategy is adopted to execute the independent operators in parallel among multiple devices. The analysis of the computing graph dependency relationship is based on the topological sorting algorithm, which ensures that the output data of the preceding operator is written in the UCMP memory pool before the subsequent operator is scheduled and executed.

[0049] Specifically, in the inference request processing process, the KDS scheduler first converts the model calculation graph submitted by the user into a set of device-independent task units, each of which contains operator types, input and output tensor descriptions, and dependency relationship identifiers. The performance monitoring unit collects real-time load indicators of each device at a fixed sampling period. For example, when the AMX usage rate reaches 60%, the GPU SM utilization rate is 75%, and the PCIe bandwidth occupancy rate is 50%, the scheduling algorithm re-evaluates the device allocation strategy according to the preset optimization target. For delay-sensitive tasks, the matrix multiplication operator is dynamically migrated to the CPU core with lower AMX usage rate; for throughput priority tasks, multiple convolution operators are executed in parallel on the GPU. The task execution order is determined by the topological sorting of the directed acyclic graph, ensuring that the data dependency relationship is strictly followed. When the device load fluctuation exceeds the dynamic threshold, the KDS scheduler triggers the online rescheduling mechanism to reassign the unexecuted task units to the load-balanced devices while maintaining the execution order of the submitted tasks. In this way, the system resource utilization and task execution efficiency are optimized simultaneously in the dynamic scheduling process, and the end-to-end delay is reduced by up to 30% and the system throughput is improved by up to 25%.

[0050] Through the above technical solution, the application realizes a fine-grained dynamic scheduling mechanism, and solves the problems of insufficient real-time performance and flexibility in heterogeneous resource collaborative scheduling. The inference request is abstracted into device-independent calculation graph task units GTU, which decouples the scheduling decision from the specific hardware and lays the foundation for cross-device dynamic allocation. The performance monitoring unit PMU collects multi-dimensional system resource states in real time, providing comprehensive resource utilization awareness for the scheduler and breaking through the limitations of traditional static allocation strategies. Based on real-time resource utilization and preset optimization targets, the execution device is dynamically selected to achieve accurate matching of resource allocation and business demand. At the same time, the task execution order is determined according to the calculation graph dependency relationship, avoiding device idle or execution conflict caused by data dependency between tasks. This dynamic and fine-grained scheduling mechanism can flexibly adjust the device allocation strategy according to real-time load changes, effectively balance the end-to-end delay and system throughput, and improve the overall resource utilization efficiency.

[0051] For the heterogeneous collaborative engine HCEE, the embodiment designs it as a set of kernel state execution components that work with the KDS scheduler to drive various computing devices to execute computing tasks and support operator-level hybrid execution, which distributes different operators in the same inference task to different computing devices for execution according to the scheduling decision of the KDS. The engine includes a CPU AMX backend and a GPU / NPU backend, which are used to submit CPU operator tasks to drive CPU AMX (such as Intel AMX, ARM SME) and submit GPU operator tasks to drive GPU / NPU devices to execute, respectively.

[0052] As a preferred embodiment, the scheme of the present application is implemented as follows: The CPU AMX backend is optimized for the advanced matrix extension unit of the CPU. When the KDS assigns CPU operator tasks, the CPU AMX backend submits the tasks through instruction-level optimization, fully utilizing the AMX instruction set to accelerate matrix operations. For example, for convolution operations, the CPU AMX backend decomposes it into matrix multiplication and uses the tdpbssd instruction to accelerate 8-bit integer matrix multiplication.

[0053] The GPU / NPU backend is responsible for submitting computing tasks to GPU or NPU devices. When the KDS assigns GPU / NPU operator tasks, the GPU / NPU backend uses a streaming task queue mechanism to submit tasks. Specifically, the GPU / NPU backend maintains a task descriptor queue, each descriptor containing operator type, input / output address, etc. The GPU / NPU driver periodically polls the queue, obtains and executes tasks.

[0054] The HCEE supports cross-device mixed execution of different operators in the same inference task. When the KDS generates a scheduling decision, the HCEE assigns operator tasks to the corresponding backend for execution according to the decision. For example, for a neural network inference task containing convolution, pooling, and fully connected layers, the HCEE may assign the convolution operation to the GPU backend, the pooling operation to the CPU AMX backend, and the fully connected layer calculation to the GPU backend again.

[0055] The HCEE implements cross-device data sharing through a unified memory management mechanism. All computing devices share the same physical memory pool, avoiding explicit data transfer between devices. The HCEE only passes memory addresses when submitting tasks, and the data access is automatically completed by the hardware.

[0056] In some embodiments, the HCEE engine is responsible for driving various computing devices to execute computing tasks and supports operator-level mixed execution. However, during the execution of cross-device cascading tasks, the lack of synchronization mechanism between different devices may cause the execution order of tasks to be disordered or intermediate data to be read incorrectly when not ready, thereby causing computing errors or resource competition problems.

[0057] Therefore, when the assigned scheduled task is a cross-device cascading task, an event synchronization point is inserted in the scheduled task at the event synchronization point; the HCEE engine issues the scheduled task according to the event order, and after the previous scheduled task is completed, triggers the next scheduled task based on the intermediate data written back to the UCMP memory pool, and sends the intermediate data to another execution device; After the inference task is completed, the HCEE engine sends a completion signal to the KDS scheduler, and the KDS scheduler feeds back an asynchronous event notification to the user process based on the completion signal.

[0058] The insertion position of the event synchronization point can be set at the cross-device operator boundary, for example, a synchronization point is inserted after the CPU AMX execution unit completes the convolution calculation and before the GPU starts to perform the matrix multiplication. The event trigger condition can be configured as the update of the write flag bit of a specific address in the shared memory pool, for example, the subsequent task is automatically triggered when the intermediate data is written to the UCMP shared partition offset 0xC000_0000. The execution device switching logic can include a hardware semaphore mechanism, for example, mutual exclusion access between devices is realized through PCIe atomic operation. The intermediate data transmission adopts a zero-copy mechanism, which is realized by mapping the device address space to the same physical page of the shared memory pool, for example, the GPU directly accesses the updated intermediate data in the UCMP through the HMM mechanism. The completion signal notification can adopt a kernel event descriptor mechanism, for example, the user process is triggered to wake up the epoll event through the eventfd counter increment.

[0059] Specifically, during the execution of the cross-device cascading task, first, the device switching node is identified in the computation graph IR parsing stage, and a synchronization mark is inserted at the corresponding position. When the HCEE drives the AMX execution unit to complete the previous operator, the intermediate result is written to the UCMP shared partition and the state register is updated, and after detecting the state change, the GPU execution context is activated through the inter-device interrupt signal. The GPU directly reads the intermediate data from the mapped UCMP address space without the need for DMA copying. After the GPU operator is executed, the HCEE sends a completion event to the KDS through the kernel semaphore mechanism, and the event carries the task identifier and the result address information. After the KDS receives the event, the user process event queue is updated through the registered callback function, and the condition variable notification is triggered. During the whole process, the device switching delay can be controlled within microseconds, for example, the task switching time from AMX to GPU is less than 50 μs, and the memory access delay is optimized to the nanosecond level through address prefetching. In this way, the ordered execution of the cross-device task is ensured, the redundant data transmission is eliminated through the memory sharing mechanism, and the user state polling overhead is reduced through the kernel-level event notification.

[0060] Taking a real-time target detection task deployed in an intelligent video monitoring edge server as an example, the implementation process of the system of the application is described in detail.

[0061] System configuration: CPU: Intel Sapphire Rapids (supporting AMX instruction set) or ARM Neoverse V2 (supporting SME); Accelerator: NVIDIA A100 GPU; Operating system: Linux Kernel 5.15+, IOMMU enabled; Application scenario: 1080p@30fps real-time target detection, model YOLOv5s.

[0062] Step one: build unified memory pool and address mapping This step is executed in the system initialization phase, and the core goal is to create a shared physical memory area that can be directly accessed by CPU and GPU.

[0063] Reserve 8GB of continuous physical memory through the kernel startup parameter cma=8G (or dynamically allocate non-continuous but persistent physical memory after system startup). The memory pool is divided into INPUT, WEIGHT, OUTPUT, SHARED four logical areas.

[0064] Create character device / dev / aicard, whose mmap() operation maps the UCMP area to user-mode virtual address; The kernel driver realizes kernel-mode access through dma_alloc_coherent; Enable IOMMU and GPU driver UVA support, register UCMP physical pages to GPU page table, realize CPU and GPU unified virtual address access; Step two: load and parse the model The user-mode model management process parses the YOLOv5s.onnx model, performs device-independent optimization such as constant folding and operator fusion, and generates a device-independent computation graph; Register IR to KDS through ioctl(), and write model weight data to the WEIGHT area of UCMP.

[0065] Step three: user task submission The video capture thread directly writes the RGB image frame to the INPUT area of UCMP; Call ioctl( / dev / aicard, AI_SUBMIT_TASK,&task_desc) to submit the task, and the task descriptor contains the model handle, input and output address, etc.

[0066] Step four: kernel dynamic scheduling decision KDS parses YOLOv5s computation graph IR, queries PMU real-time data (AMX utilization is 40%, GPU SM utilization is 70%); Based on real-time load and operator characteristics, light convolution operators are preferentially allocated to AMX, and large matrix multiplication is allocated to GPU, and a fine-grained execution plan containing synchronization points is generated.

[0067] Step five: heterogeneous device cooperative execution HCEE directly drives the AMX unit to execute the operators allocated to the CPU; For the operators assigned to the GPU, HCEE submits tasks through the NVIDIA kernel driver interface; At the AMX->GPU boundary, HCEE performs sfence and triggers GPU event wait, ensuring cross-device data consistency and execution order.

[0068] Step six: task completion notification and result acquisition After the task is completed, HCEE notifies the user process by writing eventfd. The user process returns from the blocked epoll_wait() and directly reads the structured target detection result from the OUTPUT area of UCMP. The entire process has no data copying, achieving extremely low end-to-end delay.

[0069] Figure 4 is an AI inference acceleration interaction process flowchart based on the above system modules, which can be summarized as follows: 1. The user process writes input data to the shared pool of UCMP; 2. The user process submits the ioctl task to the KDS scheduler; 3. The KDS scheduler queries the system state through the PMU interface; the PMU interface returns performance data; 4. The KDS scheduler makes dynamic scheduling decisions based on real-time system load, allocates execution devices, and generates a task execution plan list; 5. The KDS scheduler issues the generated plan list to the HCEE engine; 6. The HCEE engine executes CPU operators according to the plan tasks, and the CPU AMX performs inference calculation; 7. For GPU operator tasks, the HCEE engine needs to submit GPU operator tasks to the GPU end, execute and return the calculation results by calling the CUDA driver; 8. For cross-device tasks with inserted event synchronization points, the HCEE engine handles cross-device synchronization and maintains the sequential execution of subtasks; 9. After completing the inference task, the HCEE engine feeds back to the KDS scheduler to notify the completion of the task; 10. The KDS scheduler triggers an asynchronous event notification to wake up the user process; 11. The user process directly reads the inference result from the UCMP memory pool.

[0070] In summary, compared with the existing user state inference framework, the technical scheme of the present application has the following multiple significant advantages: Significant delay reduction: eliminate context switching and data copying overhead, combined with kernel-level real-time scheduling, significantly shorten the end-to-end inference delay; Overall throughput is improved: dynamic scheduling based on a global resource view balances the load between CPU and accelerator, significantly improving system task processing capacity; Significant bandwidth and power consumption are saved: the zero-copy architecture completely avoids redundant data transfer on the PCIe bus, greatly releases bandwidth resources, and reduces system-level transmission delay and power consumption; Efficient heterogeneous unit cooperation: the kernel execution engine realizes operator-level fine-grained cooperation, fully utilizing the advantages of various special hardware.

[0071] Figure 5 The structure block diagram of the kernel-level heterogeneous cooperative AI inference device provided by the embodiment of the application is shown, and the device is used in the Linux system kernel space. A kernel dynamic scheduler for multi-process scheduling, a shared memory pool for multi-process memory allocation, and a heterogeneous cooperative engine for scheduling heterogeneous computing resources are arranged in the kernel space. The device comprises: The task generation module 510 is configured to generate a cross-device plan list according to the global system resource state when the kernel dynamic scheduler receives an inference request submitted by the user space, and deliver the plan list to the heterogeneous cooperative engine. The plan list includes all planned tasks, the execution devices allocated to the planned tasks, and the task execution order; the execution devices include GPU / NPU and CPU; The task scheduling module 520 is configured to acquire the plan list by the heterogeneous cooperative engine, drive the corresponding target execution device according to the task execution order, read data information from the shared memory pool based on the planned task, and store the intermediate / result data output by the target execution device. The feedback module 530 is configured to trigger an event notification mechanism when the plan list inference is completed, and wake up the user process of the user space; the user process directly reads the inference result from the shared memory pool.

[0072] Figure 6A structural block diagram of a computer device provided by an example embodiment of the present application is shown. The computer device can be a desktop computer, a notebook computer, a palm computer, a cloud server, and the like. The computer device can include, but is not limited to, a processor and a memory, the memory storing at least one instruction, at least one program, a code set, or an instruction set, the at least one instruction, the at least one program, the code set, or the instruction set being loaded and executed by the processor to implement the kernel-level heterogeneous collaborative AI inference method described in the above aspects. The processor and the memory can be connected through a bus or other means. The processor can be a central processing unit (CPU), a GPU, an NPU, or other general-purpose processor, a DSP, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other dedicated deep learning co-processor, a discrete gate or transistor logic device, a discrete hardware component, or a combination thereof.

[0073] The embodiment of the present application also discloses a computer readable storage medium. Specifically, the computer readable storage medium is used to store a computer program, which is executed by a processor to implement the method in the above method embodiments. Those skilled in the art can understand that all or part of the processes in the above embodiments of the present application can be completed by a computer program instructing related hardware, and the program can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the above embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid state drive (SSD), etc. The storage medium can also include a combination of the above types of memories.

[0074] The specific embodiments are only an explanation of the present application, and are not a limitation of the present application. Those skilled in the art can make modifications to the embodiments without creative contributions after reading the present specification, as long as the modifications are within the scope of the claims of the present application.

Claims

1. A kernel-level heterogeneous collaborative AI inference method, characterized in that, The method is used in a Linux system kernel space, and a kernel dynamic scheduler for multi-process scheduling, a shared memory pool for multi-process memory allocation, and a heterogeneous collaborative engine for scheduling heterogeneous computing resources are arranged in the kernel space; the method comprises: When the kernel dynamic scheduler receives an inference request submitted by a user space, a plan list across devices is generated according to a global system resource state, and is delivered to the heterogeneous collaborative engine; the plan list comprises all planned tasks, and execution devices and task execution sequences allocated to the planned tasks; the execution devices comprise GPUs / NPUs and CPUs; The heterogeneous collaborative engine acquires the plan list, drives corresponding target execution devices according to the task execution sequences, reads data information from the shared memory pool based on the planned tasks, and stores intermediate / result data output by the target execution devices; When the plan list is completed, the kernel space triggers an event notification mechanism to wake up a user process in the user space; the user process reads inference results directly from the shared memory pool.

2. The kernel-level heterogeneous co-sim AI inference method according to claim 1, wherein, Before the kernel dynamic scheduler receives the inference request, the method further comprises: When the kernel space is started, model weights are extracted according to a model configuration file, and the shared memory pool is constructed by a memory management mechanism and IOMMU / UVA technology; The model weights are loaded into a target partition of the shared memory pool and are locked; Kernel threads of the kernel dynamic scheduler are initialized, a Linux real-time scheduling strategy is set, and an inference request submitted by the user space is received.

3. The kernel-level heterogeneous co-sim AI inference method according to claim 2, characterized in that, The shared memory pool is constructed based on system physical memory, and is mapped to a user virtual address space by mmap, to a kernel virtual address space by DMA, and to a GPU / NPU device address space by IOMMU / UVA; The shared memory pool is provided with an input partition, a weight partition, an output partition and a shared partition; the input partition stores to-be-inferred data, the weight partition stores the model weights, the output partition stores inference results, and the shared partition stores intermediate data and temporary data.

4. The kernel-level heterogeneous co-sim AI inference method according to claim 3, characterized in that, When the kernel space is started, the model weights are sent from a kernel virtual address space to the weight partition; when a thread in the user space initiates an inference request, to-be-inferred data is mapped from a user virtual address space to the input partition; When the planned tasks are inferred, intermediate data and temporary data are sent from the shared partition to the target execution devices, or are reversely stored from the target execution devices to the shared partition; When the inference is completed, inference results are stored from the result partition back to the user virtual address space.

5. The kernel-level heterogeneous co-sim AI inference method according to claim 1, wherein, The kernel dynamic scheduler generates a plan list across devices according to a global system resource state, comprising: Based on a Linux real-time scheduling strategy, an inference request is managed and received, and a graph task unit GTU independent of execution devices is acquired; the GTU is loaded based on a user state model management library, and is obtained by disassembling an original model file; The monitoring mechanism based on the performance monitoring unit collects system resource utilization in real time, including the usage rate of CPU matrix acceleration unit AMX, the usage rate of GPU / NPU internal stream multiprocessor SM, and the PCIe bandwidth utilization rate; The target optimization strategy is used to dynamically allocate the target execution device for the GTU or independent operator based on the system resource utilization and target optimization strategy, and determine the task execution order according to the computation graph dependency relationship; the target optimization strategy is used to minimize the end-to-end delay or maximize the system throughput.

6. The kernel-level heterogeneous co-sim AI inference method according to claim 1, wherein, The heterogeneous collaborative engine includes a CPU AMX backend and a GPU / NPU backend, which are respectively used to submit CPU operator tasks to drive the CPU AMX and submit GPU operator tasks to drive the GPU / NPU device to execute.

7. The kernel-level heterogeneous cooperative Al inference method according to claim 6, characterized in that, When the allocated scheduled task is a cross-device cascading task, an event synchronization point is inserted in the event synchronization point scheduled task; the heterogeneous collaborative engine issues the scheduled task according to the event order, triggers the next scheduled task based on the intermediate data written back to the shared memory pool after the completion of the previous scheduled task, and sends the intermediate data to another execution device; After the inference task is completed, the heterogeneous collaborative engine sends a completion signal to the kernel dynamic scheduler, and the kernel dynamic scheduler feeds back an asynchronous event notification to the user process based on the completion signal.

8. A kernel-level heterogeneous cooperative AI inference apparatus, characterized by, The device is used in the Linux system kernel space, and a kernel dynamic scheduler for multi-process scheduling, a shared memory pool for multi-process memory allocation, and a heterogeneous collaborative engine for scheduling heterogeneous computing resources are arranged in the kernel space. The device comprises: A task generation module is configured to generate a cross-device scheduled list according to the global system resource state when the kernel dynamic scheduler receives an inference request submitted by the user space, and issue the scheduled list to the heterogeneous collaborative engine; the scheduled list includes all scheduled tasks, the execution devices allocated for the scheduled tasks, and the task execution order; the execution devices include GPU / NPU and CPU; A task scheduling module is configured to drive the corresponding target execution device according to the task execution order, read data information from the shared memory pool based on the scheduled task, and store the intermediate / result data output by the target execution device in the shared memory pool. A feedback module is configured to trigger an event notification mechanism in the kernel space when the scheduled list inference is completed, and wake up the user process in the user space; the user process directly reads the inference result from the shared memory pool.

9. A computer device, comprising: The computer device comprises a processor and a memory, and the memory stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to implement the kernel-level heterogeneous collaborative AI inference method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to implement the kernel-level heterogeneous collaborative AI inference method according to any one of claims 1 to 7.

Images