An end-side heterogeneous device compilation runtime system
By compiling and running the system on heterogeneous devices at the edge, the problems of resource waste and insufficient scheduling in the existing technology are solved, and efficient parallel task scheduling and resource management on heterogeneous devices are realized, improving inference performance and user experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- 58TH RES INST OF CETC
- Filing Date
- 2026-04-29
- Publication Date
- 2026-07-10
AI Technical Summary
Existing technologies cannot achieve flexible and parallel task scheduling and resource management on edge chips, resulting in wasted computing resources and insufficient inference performance. They are unable to cope with complex scenarios and cannot support multi-task cascading and internal return scenarios.
A compilation and runtime system for heterogeneous devices on the edge is provided, including multi-layered abstraction layers on the host side and the device side. Through constant data processing, data layout format conversion, operator fusion and segmentation, memory allocation and reuse, quantization processing and other means, the system optimizes the execution of the model on heterogeneous devices, and realizes serial, parallel and cascaded scheduling of tasks through the user interface runtime layer and kernel driver layer.
It improves inference efficiency, reduces the waste of memory and GPU resources, supports multi-task parallel and cascaded execution, and enhances user experience and computing performance.
Smart Images
Figure CN122363706A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of system software technology, and in particular to a compilation and runtime system for heterogeneous devices on the edge. Background Technology
[0002] AI edge chips are crucial infrastructure for artificial intelligence computing power. To enable edge chips to collaborate with the host computer in executing inference programs, a compilation runtime system is required. Existing AI chips on the market typically require pre-prepared AI model files, which must be compiled and then deployed and executed on the chip. In other words, compilation and runtime management are separate, making it impossible to "compile and use on the spot" on the host side. At the same time, the device side lacks flexible and necessary scheduling and parallel capabilities.
[0003] Patent CN114398186A describes a host-side accelerator card runtime system and scheduling method. Its basic processing unit is also the Conv operator, and its software layer modules are highly similar to the CUDA architecture. Furthermore, it provides services in the form of a server board, with ample memory resources on the host side. Developers can allocate significant resources to designing host-side programs to schedule and execute AI inference tasks.
[0004] This patent lacks the concept of a "Model." A Model consists of a series of ordered, interdependent operators that cannot be reused on the model side. These operators must be placed into a Stream one by one for computation. A Stream manages deep learning operators like Conv using the first-in, first-out (FIFO) principle of a queue in computing; it's a container concept that directly manages operators. Each operator is executed sequentially from front to back, according to the order in which they enter the Stream.
[0005] Due to the lack of sufficient task granularity, Stream also cannot run operators without data or logical dependencies in parallel; it also cannot handle multi-task cascading or situations where multi-task cascading returns midway. When a Stream submits operators to the driver, it can only execute operators sequentially in the order they are submitted, and cannot schedule operators in the Stream with higher priority first, which has significant limitations in practical scenarios.
[0006] Because of the frequent round-trip notifications with the host, this method has significant room for performance improvement during inference; however, it also has the following problems: 1) Program developers can only prepare the model to be deployed on the device in advance, and do the compilation optimization and conversion into files that can be loaded and run on their own heterogeneous devices. They cannot compile the model on the device in real time. The "theoretical" and "actual" situations may be quite different, and they cannot respond to various complex situations on the site in a timely manner. 2) The same application can usually only process data serially and cannot process different data in parallel. That is, it cannot use a data-driven strategy to divide inference tasks, which greatly wastes computing resources and affects inference performance. 3) If users customize parallel programs to solve the problem in 1), they need to start multiple programs, processes, and threads for parallel inference. However, at this time, multiple programs need to apply for independent host memory and device video memory for their models, and existing resources cannot be reused, resulting in a huge waste of memory and video memory resources. This is extremely fatal on edge processors where memory resources are extremely precious. 4) Within the application, if there are subtasks that can be executed in parallel, they usually cannot be executed in parallel. 5) Multiple cascaded applications require manual code control by the user and cannot be automatically connected in sequence, nor can they enjoy the benefits of parallel programming; 6) Multiple cascaded applications with partial returns between them, i.e., the first application runs some tasks, then the second application executes, and then the remaining tasks in the first application are executed. This situation cannot be supported by existing task scheduling methods. Summary of the Invention
[0007] The purpose of this invention is to provide a compilation and runtime system for heterogeneous devices on the edge side to solve the problems in the background art.
[0008] To address the aforementioned technical problems, this invention provides a compilation and runtime system for edge-side heterogeneous devices, oriented towards host-connected heterogeneous chip devices, including a host end and a device end; The host side includes a compilation layer, a user interface runtime layer, a kernel driver layer, and a firmware layer; the device side includes an operator-level scheduling module and a resource management module. The compilation layer consists of constant data processing, data layout format conversion, operator fusion and segmentation, memory allocation and reuse, quantization processing, and model generation. The constant data processing preprocesses some data that the hardware cannot support; generates constant data that the original model does not have but the hardware requires; and processes the constant data of the model into the corresponding precision format according to the hardware requirements. The data arrangement format conversion requires rearranging the constant data arrangement format originally carried by the model if it does not meet the hardware requirements. The operator fusion and segmentation involves some hardware modules merging multiple operators using the same hardware module together to reduce data read and write consumption; due to the limitations of on-chip storage, and the excessive resources required by some operators, hardware or software segmentation is necessary. The memory allocation and reuse process pre-allocates memory for the input and output tensors of each operator and their constant data, adds address offset information, and uses different memory optimization allocation strategies to improve hardware execution efficiency. The heterogeneous computation subgraph segmentation is described above. Ideally, the entire graph would run on the heterogeneous device when the heterogeneous inference device performs the task. However, there may be operators that the heterogeneous device cannot support. Therefore, it is necessary to segment out such operators in the computation graph and place them on the CPU for execution, while ensuring that the topological dependencies between the subgraphs are correct. The user interface runtime layer consists of model parser abstraction, model abstraction, memory abstraction, stream abstraction, and task abstraction, which are used to support the Stream scheduling and management module in the user interface runtime layer and provide runtime interfaces to the outside world. The Stream scheduling and management module in the user interface runtime layer performs various controls on inference tasks, including serial and parallel execution of tasks, setting inference task priorities, and providing additional support for cascaded Streams and cascaded Streams with internal returns, which greatly improves inference efficiency and provides a good user experience. The kernel driver layer manages device parameters, performs IPC operations, manages video memory, manages global TaskIDs, and also provides driver interfaces to the outside world; the driver interfaces include video memory operation interfaces, device operation interfaces, global TaskID allocation and release operation interfaces, and IPC operation interfaces; The firmware layer provides chip-level scheduling, manages information in Task tasks, and performs preliminary parsing and management of Block subtask unit data. The operator-level scheduling module dispatches the latest Op task unit in the Block subtask unit to the corresponding computing device. The resource management module contains numerous functional modules and is responsible for managing and computing heterogeneous devices on the edge.
[0009] In one implementation, the constant data processing, the data layout format conversion, the operator fusion and segmentation, the video memory allocation and reuse, and the quantization processing include the unique hardware features of the heterogeneous device itself, and perform all the above functional processing to efficiently run AI programs on this heterogeneous device. The constant data processing transforms the constant information in the model from multiple sums into fewer data points, thereby significantly reducing the number of data points and memory loading time. The data layout format conversion transforms the model data layout format from NCHW format to NCHW32, making it conform to the hardware data reading rules, facilitating hardware reading, loading and calculation, and improving computing performance. The operator fusion and segmentation is to merge multiple operators into a single equivalent operator according to matching rules, so as to save one data loading, one data writing, and one Kernel Launch time, and achieve the effect of two calculations in one calculation; If a certain operator requires too many resources, neither the software nor the hardware can support it, and it is necessary to split it at the original model. The aforementioned memory allocation and reuse involves iterating through and calculating the on-chip memory required by all operators, finding its maximum value, and setting it as the corresponding register size. Memory is not actually allocated during the compilation phase, but only at the user interface runtime layer. Furthermore, all operators use this memory, eliminating frequent memory allocation and deallocation operations and saving a significant amount of system time.
[0010] In one implementation, the quantization process converts the weights and biases of the Conv operator in the original model from the default Float32 type to Int8 type, thereby reducing the memory occupied by each data point from 4B to 1B, a reduction to 25% of the original. Meanwhile, Int8 is fixed-point data, which loads into memory / video memory faster and is also faster to compute than Float32.
[0011] In one implementation, the model generation involves saving the model processed by the compilation method in step 2 into a Bin file. This Bin file stores all the Tasks of the entire model. Each Task contains several sub-task units (Blocks), and each sub-task unit (Block) contains several minimum task units (Op). These Ops map the Task, Block, and Op numbers, quantity, offset address in the file, register value, and required number of computation cores.
[0012] In one implementation, the user interface runtime layer is used by users to build upper-layer inference tasks, and includes a user-mode interface and a runtime interface. The user-mode interface is the interface that users need to call to develop upper-layer inference tasks, and the runtime interface is the interface that users use to develop more advanced features or support the development or operation of other features. The upper-layer inference task, unlike the Task, refers to an upper-layer AI application, including deep learning algorithms for face detection, face recognition, and face comparison.
[0013] In one implementation, in the scheduling management module of the user interface runtime layer, an upper-layer inference task uses a deep learning model to generate the Bin file mentioned above; an inference Model object is created in main memory, the Bin file is parsed, and all Task information of the model is obtained. To better control and manage the inference process, an inference stream object is created in main memory. This object is bound to a Model object and is used to manage the specific inference matters of the model. All Stream objects bound to the same Model can use the common information of Tasks in the Model, such as localTaskId, binary Kernel functions, dynamic link libraries, input and output Shape information, and input and output offset information.
[0014] In one implementation, since each Stream object performs inference on different data, it needs to apply for a global TaskID for its own and all Tasks to ensure that each Task received by the kernel driver layer is unique and easy to manage. At the same time, if a user uses the same model to start multiple different inference tasks, or uses different models to start different inference tasks, the global TaskID can ensure that the data and results of different inference tasks are not confused and that there is a one-to-one correspondence.
[0015] In one embodiment, the driver interface includes a video memory operation interface, a device operation interface, a global TaskID allocation and release operation interface, and an IPC operation interface.
[0016] A scheduling method for a compilation and runtime system based on the aforementioned heterogeneous end-side device includes the following steps: Step S0: The user takes the inference model to be run and uses a command-line tool to convert the *.onnx model into a *.bin file. This command-line tool will perform all compilation processing and optimization. Step S1: The user parses the obtained bin file and constructs the corresponding Model object; Step S11: For all Tasks under Model's management, send requests to the kernel driver layer one by one to request Data memory, which is used to store the weight information required by each Task. This memory can be shared. Step S2: The user creates and constructs a Stream object to manage the specific inference matters of the model; Step S21: The Stream object needs to call the runtime interface to explicitly bind to a specific Model object, and a Model can be bound to multiple Stream objects. Step S22: In order to reuse memory and video memory, each Stream object reuses the common information of Task in Model, including weight information, localTaskId, dynamic link library, binary executable program information of running function, input and output dimensions of Task, and shape information. It is not necessary to apply for duplicate information for Task in each Stream object, which greatly saves memory and video memory. Step S3: In order to distinguish between App inference tasks performed by the same model or different models, or to use the same model to process data from different time frames in parallel, each Stream object requests a global globalTaskId for each Task. This value is globally unique and cannot be changed. Step S31: Send requests to the kernel driver layer one by one for the Tasks managed by the Stream object to request Assembly memory, which is used to store the input and output Tensor data of the Task. Each Task in the Stream object has this memory, and each one is different and cannot be reused. Step S4: Inference begins. The Stream object dispatches the tasks to be completed by its dependencies to the kernel driver layer, specifically including: Step S41, Serial Distribution: If each Task has data dependencies or logical dependencies, and they are related, then each Task is distributed to the next layer in sequence. The next Task can only be distributed after the previous Task has been calculated. Step S42, Parallel Distribution: If there are multiple Tasks without dependencies, they are distributed to the lower layers in parallel to achieve parallel scheduling of multiple tasks and significantly improve task execution efficiency. Step S43: Set different priorities for each Stream task. If a certain Stream has a higher priority, then the Tasks under that Stream will be executed first. After all the Tasks under that Stream are executed, the Tasks of other Streams with the next higher priority will be executed. In step S44, when multiple streams are cascaded, the tasks in the preceding stream are executed first, then the tasks in the following stream are executed, and so on. In step S45, when multiple Streams are cascaded and return midway, the first Task in the preceding Stream is executed first, then the Task in the following Stream is executed, and then the subsequent Task in the preceding Stream is returned, and so on. Step S5: After receiving the task submitted by the user interface runtime layer, the kernel driver layer records the information of each task and sends it to the firmware. Step S6: The firmware receives the Task, parses its information, registers the information, and distributes it to the Kernel scheduler using the Block subtask unit. Step S7: The kernel scheduler dispatches the received Block tasks to heterogeneous devices for computation based on the processor usage. Step S8: After the calculation is completed, return the Block task execution status to the firmware; Step S9: The firmware returns the status to the kernel driver layer, and after the kernel driver layer records it, it returns the status to the user interface runtime layer. Step S10: After receiving the status, the user interface runtime layer modifies the Task status. If the Task is completed normally, it notifies its consumer task and repeats steps S4 to S11 until the end. Step 11: If the Task is in a failed state, the entire reasoning task ends.
[0017] In one implementation, step 4 provides additional support for serial, parallel, inference priority, cascading, and cascading with internal return conditions, including: Serial execution is the common case, with the Conv, ReLU, and AvgPool operators executed in sequence. In the parallel scenario, after task T0 is completed, tasks T1, T2, and T3 can be executed in parallel because their dependent parent tasks have been satisfied. Inference priority: For multiple Stream objects of the same model, different priorities are set for different Stream objects using different input data. When they are distributed to the kernel driver layer, they are sorted according to their priorities. Tasks in high-priority Stream objects are dispatched to the firmware for further processing first. After processing, tasks in low-priority Stream objects are dispatched. If two Stream objects, stream1 and stream2, are cascaded, the user can call the cascade interface with stream1 and stream2 as input parameters. The tasks in stream1 will be executed first, followed by the tasks in stream2, with synchronization enabled by default. If parallel execution is possible during the execution of the two streams, the user can enjoy the benefits of parallel operation. In cases of cascading with internal returns, refer to the case of model cascading and use the cascading return interface.
[0018] In one embodiment, the operator-level scheduling module receives information from the resource management module and tasks in the flow queue of the Task scheduling management module, and schedules different types of tasks, such as memory access, communication, floating-point calculation, and tensor calculation, to different execution components on the device side for execution based on the idle status of the relevant hardware components on the device side; the relevant hardware components include: at least one computing unit and at least one storage unit.
[0019] This invention provides a compilation and runtime system for edge-side heterogeneous devices, which provides an additional layer of abstraction on the edge-side heterogeneous devices, supports scheduling at both the host and edge-side heterogeneous device levels, can fully utilize the performance of the edge-side heterogeneous devices, and achieve efficient execution of artificial intelligence applications. This invention also provides a secondary scheduling method for operators within the edge-side heterogeneous devices, which can utilize various resources on the accelerator card more efficiently and flexibly. Attached Figure Description
[0020] Figure 1 This is a schematic diagram of the compilation and runtime system and scheduling of heterogeneous devices on the end side provided by the present invention.
[0021] Figure 2 This is a schematic diagram for processing constant data.
[0022] Figure 3 This is a schematic diagram of merging multiple operators into a single equivalent operator.
[0023] Figure 4 This is a schematic diagram of dividing a single operator into multiple operators.
[0024] Figure 5 This is a diagram illustrating Task information.
[0025] Figure 6 This is a diagram illustrating the relationship between Model, Stream, and Task.
[0026] Figure 7 This is a schematic diagram of parallel distribution.
[0027] Figure 8 This is a diagram illustrating the cascading of multiple streams.
[0028] Figure 9 This is a diagram illustrating the situation where multiple Streams are cascaded and return midway. Detailed Implementation
[0029] The following detailed description, in conjunction with the accompanying drawings and specific embodiments, provides a further detailed explanation of the end-side heterogeneous device compilation and runtime system proposed in this invention. The advantages and features of this invention will become clearer from the following description. It should be noted that the drawings are all in a very simplified form and use non-precise proportions, and are only used to facilitate and clarify the illustration of the embodiments of this invention.
[0030] This invention is based on the following basic concepts: CUDA (Compute Unified Device Architecture) is a unified programming language provided by NVIDIA for its own GPU chip products, and it is currently the mainstream language worldwide.
[0031] CUDA does not have the concept of Model; instead, it provides the concept of Stream to organize operator / task computations.
[0032] Stream, meaning queue, is a software concept and an important one in CUDA. It's a first-in, first-out (FIFO) structure, meaning that if two tasks are in the queue, the task at the front of the queue will be executed before the task at the back. A Stream is the unit of inference in CUDA computation. A Stream can typically be considered as a specific inference task, such as face detection, face recognition, or face analysis.
[0033] A Stream can contain a series of operators, such as Conv→Relu→AvgPool→Resize→... These operators must be placed into the Stream in sequence and executed sequentially.
[0034] In deep learning algorithms, the Conv operator is used to extract features from the data and plays a central role in the algorithm. Each Conv operator has weight / Bias information, which are the most important components of a deep learning model and constitute all the information stored in a deep learning model file.
[0035] When deploying deep learning model inference tasks using CUDA, in common scenarios such as parallel processing of camera data / video frame data, it is necessary to use the same inference method for single and double frame data. This requires repeatedly and mechanically constructing multiple Streams and putting all operators into these Streams in sequence. This will cause all Conv's Weight / Bias information to be copied multiple times, wasting a lot of host memory and on-chip memory during computation (if there are N identical inference programs, then (N-1) copies of Weight / Bias resources are wasted).
[0036] At the same time, users need to manually write programs to process different data in parallel streams to achieve the effect of parallel processing.
[0037] At the same time, operators in a Stream, if they can run independently, cannot be executed in parallel and must be executed sequentially, even if these operators have no data or logical dependencies.
[0038] If multiple inference tasks are assigned to different streams and are cascaded, the next stream can only begin computation after the previous stream has been completed and synchronized. There is no possibility of parallel processing in between.
[0039] CUDA Streams does not support multi-task cascading and internal return scenarios.
[0040] This invention provides a compilation and runtime system for heterogeneous devices on the edge, such as... Figure 1 As shown, the system consists of two parts: the host side and the device side. The host side includes four software abstraction layers: the compilation layer, the User Interface Runtime (UIR) layer, the Kernel Driver Layer (KDL) layer, and the firmware layer. The device side includes two functional modules: the operator-level scheduling module and the resource management module. The compilation layer processes ordinary AI models. After being called by the user, it converts ordinary AI models (such as ONNX models, a widely used AI model storage format) into the file that this invention runs. During the conversion process, various format conversions are performed to ensure that the model can run correctly and efficiently on heterogeneous devices. The UIR layer is the User Interface Runtime layer, which is the layer where users build upper-layer inference tasks. It includes a user-space interface and a runtime interface. The former is the interface used by users to develop upper-layer inference task calls; the latter is the interface for users to perform more advanced development (such as parallel and asynchronous operations). The KDL layer is the Kernel Driver Layer, which receives UIR tasks and forwards them to the firmware layer. The firmware layer parses large Tasks into medium-sized Block Tasks and forwards them to the Kernel scheduler, essentially acting as a tiny operating system.
[0041] The host-side compilation layer mainly performs operations such as constant data processing, data layout format conversion, operator fusion and segmentation, memory allocation and reuse, quantization processing, and Bin file generation. These operations are crucial for the correct operation of the model on heterogeneous devices.
[0042] The host-side UIR layer consists of model parser abstraction, device abstraction, model abstraction, main memory abstraction, video memory abstraction, stream abstraction, and task abstraction; The three-layer abstraction system of Model, Stream, and Task, combined with the chip-level scheduling and operator-level scheduling modules of the hardware, constructs the scheduling function of this invention.
[0043] In the Stream scheduling management module, a single upper-level inference task maintains a stream queue for task distribution; different upper-level inference tasks (regardless of whether they come from the same model) use different stream queues for task distribution.
[0044] The host-side KDL layer includes functions such as device parameter management, IPC operation, video memory management, and global TaskID management; it provides driver interfaces to the outside world, including interfaces for video memory operation, device operation, global TaskID allocation and release operations, and IPC.
[0045] like Figure 5As shown, a Task contains several sub-task units (Blocks), and multiple Blocks can be combined into a Task; a sub-task unit (Block) contains several smallest task units (Ops), such as deep neural operators like Conv and ReLU.
[0046] The device-side operator-level scheduling module is responsible for dispatching different Ops in a Block to different hardware computing units for computation. The resource management module manages multiple modules, including numerous functional modules, and is responsible for managing and computing heterogeneous devices on the device side. The resource management module can view and statistically analyze the running status of each computing core on the device and provide feedback to the firmware. The device-side operator-level scheduling module receives information from the resource management module and tasks in the stream queue of the Stream scheduling management module on the host side. Based on the idle status of relevant hardware components on the device side, it schedules different types of tasks, such as memory access, communication, floating-point computation, and tensor computation, to different execution components on the device side for execution.
[0047] The scheduling steps of this runtime system are as follows: Step S0: The user takes the inference model to be run, such as Resnet50.onnx or Yolov5S.onnx, and uses the command-line tool provided in this invention to convert the *.onnx model into a *.bin file. This command-line tool will perform all the compilation processing optimizations mentioned above. Step S1: The user parses the obtained bin file and constructs the corresponding Model object; Step S11: For all Tasks under Model's management, send requests to the kernel driver layer one by one to request Data memory, which is used to store the weight information required by each Task. This memory can be shared. Step S2: The user creates and constructs a Stream object to manage the specific inference matters of the model. The relationship between Model, Stream, and Task is as follows: Figure 6 As shown; Step S21: Stream needs to call the runtime interface to explicitly bind to a specific Model object, and a Model can be bound to multiple Streams. Step S22: In order to reuse memory and video memory, each Stream can reuse the common information of Task in Model, such as weight information (Data video memory), localTaskId, dynamic link library, binary executable program information of running function, input and output dimensions of Task, shape and other information, without having to apply for duplicate information for Task in each Stream, which greatly saves memory and video memory. Step S3: In order to distinguish between App inference tasks performed by the same model or different models, or to use the same model to process data from different time frames in parallel, each Stream requests a global globalTaskId for each Task. This value is globally unique and cannot be changed. Step S31: Send requests to the kernel driver layer one by one for the Task managed by the Stream to request Assembly memory to store the input and output Tensor data of the Task. Each Task in the Stream has this memory, and each one is different and cannot be reused. Step S4: Inference begins. Stream distributes the tasks completed by the dependencies to the kernel driver layer. Step S41, Serial Distribution: If each Task has data dependencies or logical dependencies, and is related to the previous and next Tasks, then each Task is distributed to the next layer in sequence. The next Task can only be distributed after the previous Task has been calculated. Step S42, Parallel Distribution: If multiple Tasks have no dependencies, they can be distributed to lower layers in parallel to achieve multi-task parallel scheduling, significantly improving task execution efficiency, such as... Figure 7 As shown, T1, T2, and T3 are independent of each other and can be distributed in parallel; Step S43: Each Stream task can be set with a different priority. If a certain Stream has a higher priority, the Tasks under that Stream will be executed first. After all the Tasks under that Stream are executed, the Tasks of other Streams with the next higher priority will be executed. Step S44: In the case of multiple cascaded Streams, the tasks in the preceding Stream are executed first, then the tasks in the following Stream are executed, and so on. Figure 8 As shown; Step S45: In the case of multiple cascaded Streams returning midway, the first task in the preceding Stream is executed first, then the tasks in the following Stream are executed, then the subsequent tasks in the preceding Stream are returned, and so on. Figure 9 As shown; Step S5: After receiving the task submitted by the user interface runtime layer, the kernel driver layer records the information of each task and sends it to the firmware. Step S6: The firmware receives the Task, parses its information, registers the information, and distributes it to the Kernel scheduler using the Block subtask unit. Step S7: The kernel scheduler dispatches the received Block tasks to heterogeneous devices for computation based on the processor usage. Step S8: After the calculation is completed, return the Block task execution status to the firmware; Step S9: The firmware returns the status to the kernel driver layer, and after the kernel driver layer records it, it returns the status to the user interface runtime layer. Step S10: After receiving the status, the user interface runtime layer modifies the Task status. If the Task is completed normally, it notifies its consumer task and repeats steps S4 to S11 until the end. Step 11: If the Task is in a failed state, the entire inference task ends.
[0048] The implementation plan is as follows: 1. Users utilize the ONNX model and the compiler / executable program of this invention to process it into a runtime executable loadable file. This compiler / executable program will traverse the information in the ONNX model, construct a DAG (Directed Acyclic Graph), and process the information within it: 1) Constant data processing, such as Figure 2 As shown: If you encounter multiple constant values, combine them into one value. For example, if the model contains 2.3 + 1.4, calculate the result 3.7 directly and write it into the new model. This can reduce main memory access. 2) Data layout format conversion: Convert the NHWC or NCHW data layout format in the default ONNX file to the NCHW32 format, which makes it easier for the heterogeneous devices of this invention to read data faster; 3) Operator fusion and segmentation, such as Figure 3 As shown: When traversing the ONNX model information of the model, when traversing each operator node, the information of its parent node and child nodes will be accessed. If an operator combination that meets the fusion or splitting conditions is encountered, these operators will be combined into a new operator and replaced in the original model; or an operator that meets the conditions will be split into multiple operators and replaced in the original model.
[0049] Fusion: For example, if the Conv+BiasAdd operator meets the fusion rules, it will be replaced with the ConvBiasAdd operator. The weights of the original Conv operator and the bias values of the original Bias operator will be retained and put into the weights and bias values of the new ConvBiasAdd operator. The ConvBiasAdd operator will replace the original Conv+BiasAdd operator.
[0050] Segmentation as Figure 4 As shown: If the Split operator is in the split list, it will be divided into multiple split operators along the channel direction, and this information will be added to the original model.
[0051] 4) Memory allocation and reuse: Traverse the memory values of all tasks in the model to obtain the maximum value. This value is the memory value that needs to be allocated. In the UIR layer, only the memory of this maximum value is requested. All subsequent tasks are calculated in this memory. This is the memory allocation and reuse strategy.
[0052] 5) Quantization: The weights of Conv are quantized, converting them from Float32 to Int8. The conversion formula is as follows: Val_Int8 = (Val_Fp32 - Val_Fp32_max) / Scale 6) Model Generation The ONNX model processed by the above steps is converted into a binary file to generate the model required by UIR.
[0053] 2. After compilation, proceed to the UIR layer. The UIR calls the interface to parse the model and uses Model objects to record model information. The Model object retains key deep learning information such as Conv Weights / Bias used by each Task, and maps this information to Tasks. This data can be shared by all Streams derived from the same Model. 3. Based on the Model object and the user's actual parallel requirements, create a Stream object and call the interface "aiStream stream; aiStreamCreate(stream, model)" to bind the stream and model together; 3. For each Task in the Stream, allocate Assembly memory. This information is recorded in the Stream object, and allocate a global GlobalTaskId for all Tasks. IV. Users can set the priority of a Stream. Once set, all Tasks under that Stream will have that priority and will have the right to be executed before lower-priority Tasks. If multiple streams are cascaded, the cascading relationship needs to be set by calling the interface. For example, if Stream1, Stream2, Stream3, and Stream4 are cascaded, the interface "aiCascade(Stream1, Stream2); aiCascade(Stream2, Stream3); aiCascade(Stream3, Stream4);" can be called to set their cascading relationship. The "aiCascade" interface adds an Event object after the last Task in the queue represented by the first parameter, pointing to the first Task in the queue represented by the second parameter, thus establishing a sequential connection. The Event object is a lightweight object structure used to represent sequential connections.
[0054] If there are multiple Streams and there are cases where the process returns midway, the interface needs to be called to set the cascading relationship. For example, if Stream1 and Stream2 are cascaded, the first 3 tasks in Stream1 will be executed first, then the tasks in Stream2 will be executed, and after that, the process will return to execute the 4th task in Stream1 until Stream1 finishes its calculation. This requirement can be achieved by calling the overloaded interface "aiCascade(Stream1, Stream2, 3)". The “aiCascade” interface is an overloaded interface with the same name as the “aiCascade” interface above, but with a different parameter list. This interface inserts an Event object after the third task in the first queue, pointing the Event object to the first task represented by the second parameter. Then, after the last task represented by the second parameter, another Event object is inserted, pointing to the fourth task represented by the first parameter, thus establishing a connection.
[0055] 5. Provide data to the Stream, call the Stream execution interface, and the Stream will submit its contained Tasks to the global task pool; 6. After receiving all Tasks submitted by Streams, the global task pool sorts the Tasks according to their priority. 7. The global thread pool analyzes the dependencies of all sorted Tasks. If no dependencies are found, all Tasks are dispatched to KDL and the system waits for their return status. If there are dependencies between Tasks, they are dispatched to KDL in sequence and the system waits for their return status. 8. KDL receives the task, records the information, and dispatches it to the firmware; 9. The firmware receives the Task, parses out the Block, and dispatches it to the Kernel scheduler. 10. The kernel scheduler records the load status of all edge processors, dispatches them to idle processors, and waits for the status to be returned. 11. Upon receiving the return status, return sequentially in the order described above. If all blocks of a Task complete successfully, the Task status returns UIR sequentially, and the Task is run again in the order described in steps 7-11 above. If a block in a task fails to compute, all tasks in the stream to which that task belongs will fail, the global task pool will remove them all, and an error will be reported to the terminal.
[0056] The above description is merely a description of preferred embodiments of the present invention and is not intended to limit the scope of the present invention in any way. Any changes or modifications made by those skilled in the art based on the above disclosure shall fall within the protection scope of the claims.
Claims
1. A compilation and runtime system for heterogeneous end-side devices, characterized in that, For host-connected heterogeneous chip devices, including both host and device sides; The host side includes a compilation layer, a user interface runtime layer, a kernel driver layer, and a firmware layer; the device side includes an operator-level scheduling module and a resource management module. The compilation layer consists of constant data processing, data layout format conversion, operator fusion and segmentation, memory allocation and reuse, quantization processing, and model generation. The constant data processing preprocesses some data that cannot be supported by the hardware. Generate constant data that is not present in the original model but is required by the hardware; process the constant data of the model into the corresponding precision format according to the hardware requirements; The data arrangement format conversion requires rearranging the constant data arrangement format originally carried by the model if it does not meet the hardware requirements. The operator fusion and segmentation involves some hardware modules merging multiple operators using the same hardware module together to reduce data read and write consumption; due to the limitations of on-chip storage, and the excessive resources required by some operators, hardware or software segmentation is necessary. The memory allocation and reuse process pre-allocates memory for the input and output tensors of each operator and their constant data, adds address offset information, and uses different memory optimization allocation strategies to improve hardware execution efficiency. The heterogeneous computation subgraph segmentation is described above. Ideally, the entire graph would run on the heterogeneous device when the heterogeneous inference device performs the task. However, there may be operators that the heterogeneous device cannot support. Therefore, it is necessary to segment out such operators in the computation graph and place them on the CPU for execution, while ensuring that the topological dependencies between the subgraphs are correct. The user interface runtime layer consists of model parser abstraction, model abstraction, memory abstraction, stream abstraction, and task abstraction, which are used to support the Stream scheduling and management module in the user interface runtime layer and provide runtime interfaces to the outside world. The Stream scheduling and management module in the user interface runtime layer performs various controls on inference tasks, including serial and parallel execution of tasks, setting inference task priorities, and providing additional support for cascaded Streams and cascaded Streams with internal returns, which greatly improves inference efficiency and provides a good user experience. The kernel driver layer manages device parameters, performs IPC operations, manages video memory, manages global TaskIDs, and also provides driver interfaces to the outside world; the driver interfaces include video memory operation interfaces, device operation interfaces, global TaskID allocation and release operation interfaces, and IPC operation interfaces; The firmware layer provides chip-level scheduling, manages information in Task tasks, and performs preliminary parsing and management of Block subtask unit data. The operator-level scheduling module dispatches the latest Op task unit in the Block subtask unit to the corresponding computing device. The resource management module contains numerous functional modules and is responsible for managing and computing heterogeneous devices on the edge.
2. The end-side heterogeneous device compilation and runtime system as described in claim 1, characterized in that, The constant data processing, the data arrangement format conversion, the operator fusion and segmentation, the video memory allocation and reuse, and the quantization processing all contain the unique hardware features of the heterogeneous device itself, and perform all the above functional processing to efficiently run AI programs on this heterogeneous device. The constant data processing transforms the constant information in the model from multiple sums into fewer data points, thereby significantly reducing the number of data points and memory loading time. The data layout format conversion transforms the model data layout format from NCHW format to NCHW32, making it conform to the hardware data reading rules, facilitating hardware reading, loading and calculation, and improving computing performance. The operator fusion and segmentation is to merge multiple operators into a single equivalent operator according to matching rules, so as to save one data loading, one data writing, and one Kernel Launch time, and achieve the effect of two calculations in one calculation; If a certain operator requires too many resources, neither the software nor the hardware can support it, and it is necessary to split it at the original model. The aforementioned memory allocation and reuse involves iterating through and calculating the on-chip memory required by all operators, finding its maximum value, and setting it as the corresponding register size. Memory is not actually allocated during the compilation phase, but only at the user interface runtime layer. Furthermore, all operators use this memory, eliminating frequent memory allocation and deallocation operations and saving a significant amount of system time.
3. The end-side heterogeneous device compilation and runtime system as described in claim 1, characterized in that, The quantization process converts the weights and biases of the Conv operator in the original model from the default Float32 type to Int8 type, thereby reducing the memory occupied by each data point from 4B to 1B, a reduction to 25% of the original. Meanwhile, Int8 is fixed-point data, which loads into memory / video memory faster and is also faster to compute than Float32.
4. The end-side heterogeneous device compilation and runtime system as described in claim 1, characterized in that, The model generation process involves saving the model processed by the method in step 2 above into a Bin file. This Bin file stores all the Tasks of the entire model. Each Task contains several sub-task units (Blocks), and each sub-task unit (Block) contains several smallest task units (Op). These Ops map the Task, Block, and Op numbers, quantity, offset address in the file, register value, and required number of computation cores.
5. The end-side heterogeneous device compilation and runtime system as described in claim 1, characterized in that, The user interface runtime layer is used by users to build upper-layer inference tasks. It includes a user-mode interface and a runtime interface. The user-mode interface is the interface that users need to call to develop upper-layer inference tasks. The runtime interface is the interface that users use to develop more advanced features or support the development or operation of other features. The upper-layer inference task, unlike the Task, refers to the upper-layer AI application, including deep learning algorithms for face detection, face recognition, and face comparison.
6. The end-side heterogeneous device compilation and runtime system as described in claim 1, characterized in that, In the scheduling and management module of the user interface runtime layer, an upper-layer inference task uses a deep learning model to generate the Bin file mentioned above; an inference Model object is created in main memory, the Bin file is parsed, and all Task information of the model is obtained. To better control and manage the inference process, an inference stream object is created in main memory. This object is bound to a Model object and is used to manage the specific inference matters of the model. All Stream objects bound to the same Model can use the common information of Tasks in the Model, such as localTaskId, binary Kernel functions, dynamic link libraries, input and output Shape information, and input and output offset information.
7. The end-side heterogeneous device compilation and runtime system as described in claim 6, characterized in that, Since each Stream object performs inference on different data, it needs to apply for a global TaskID for its own and all Tasks to ensure that each Task received by the kernel driver layer is unique and easy to manage. At the same time, if a user uses the same model to start multiple different inference tasks, or uses different models to start different inference tasks, the global TaskID can ensure that the data and results of different inference tasks are not confused and that there is a one-to-one correspondence.
8. The end-side heterogeneous device compilation and runtime system as described in claim 1, characterized in that, The scheduling method for the compilation and runtime system of the heterogeneous device on the terminal side includes the following steps: Step S0: The user takes the inference model to be run and uses a command-line tool to convert the *.onnx model into a *.bin file. This command-line tool will perform all compilation processing and optimization. Step S1: The user parses the obtained bin file and constructs the corresponding Model object; Step S11: For all Tasks under Model's management, send requests to the kernel driver layer one by one to request Data memory, which is used to store the weight information required by each Task. This memory can be shared. Step S2: The user creates and constructs a Stream object to manage the specific inference matters of the model; Step S21: The Stream object needs to call the runtime interface to explicitly bind to a specific Model object, and a Model can be bound to multiple Stream objects. Step S22: In order to reuse memory and video memory, each Stream object reuses the common information of Task in Model, including weight information, localTaskId, dynamic link library, binary executable program information of running function, input and output dimensions of Task, and shape information. It is not necessary to apply for duplicate information for Task in each Stream object, which greatly saves memory and video memory. Step S3: In order to distinguish between App inference tasks performed by the same model or different models, or to use the same model to process data from different time frames in parallel, each Stream object requests a global globalTaskId for each Task. This value is globally unique and cannot be changed. Step S31: Send requests to the kernel driver layer one by one for the Tasks managed by the Stream object to request Assembly memory, which is used to store the input and output Tensor data of the Task. Each Task in the Stream object has this memory, and each one is different and cannot be reused. Step S4: Inference begins. The Stream object dispatches the tasks to be completed by its dependencies to the kernel driver layer, specifically including: Step S41, Serial Distribution: If each Task has data dependencies or logical dependencies, and they are related, then each Task is distributed to the next layer in sequence. The next Task can only be distributed after the previous Task has been calculated. Step S42, Parallel Distribution: If there are multiple Tasks without dependencies, they are distributed to the lower layers in parallel to achieve parallel scheduling of multiple tasks and significantly improve task execution efficiency. Step S43: Set different priorities for each Stream task. If a certain Stream has a higher priority, then the Tasks under that Stream will be executed first. After all the Tasks under that Stream are executed, the Tasks of other Streams with the next higher priority will be executed. In step S44, when multiple streams are cascaded, the tasks in the preceding stream are executed first, then the tasks in the following stream are executed, and so on. In step S45, when multiple Streams are cascaded and return midway, the first Task in the preceding Stream is executed first, then the Task in the following Stream is executed, and then the subsequent Task in the preceding Stream is returned, and so on. Step S5: After receiving the task submitted by the user interface runtime layer, the kernel driver layer records the information of each task and sends it to the firmware. Step S6: The firmware receives the Task, parses its information, registers the information, and distributes it to the Kernel scheduler using the Block subtask unit. Step S7: The kernel scheduler dispatches the received Block tasks to heterogeneous devices for computation based on the processor usage. Step S8: After the calculation is completed, return the Block task execution status to the firmware; Step S9: The firmware returns the status to the kernel driver layer, and after the kernel driver layer records it, it returns the status to the user interface runtime layer. Step S10: After receiving the status, the user interface runtime layer modifies the Task status. If the Task is completed normally, it notifies its consumer task and repeats steps S4 to S11 until the end. Step 11: If the Task is in a failed state, the entire reasoning task ends.
9. The end-side heterogeneous device compilation and runtime system as described in claim 8, characterized in that, Step 4 provides additional support for serial, parallel, inference priority, cascading, and cascading with internal return conditions, including: Serial execution is the common case, with the Conv, ReLU, and AvgPool operators executed in sequence. In the parallel scenario, after task T0 is completed, tasks T1, T2, and T3 can be executed in parallel because their dependent parent tasks have been satisfied. Inference priority: For multiple Stream objects of the same model, different priorities are set for different Stream objects using different input data. When they are distributed to the kernel driver layer, they are sorted according to their priorities. Tasks in high-priority Stream objects are dispatched to the firmware for further processing first. After processing, tasks in low-priority Stream objects are dispatched. If two Stream objects, stream1 and stream2, are cascaded, the user can call the cascade interface with stream1 and stream2 as input parameters. The tasks in stream1 will be executed first, followed by the tasks in stream2, with synchronization enabled by default. If parallel execution is possible during the execution of the two streams, the user can enjoy the benefits of parallel operation. In cases of cascading with internal returns, refer to the case of model cascading and use the cascading return interface.
10. The end-side heterogeneous device compilation and runtime system as described in claim 8, characterized in that, The operator-level scheduling module receives information from the resource management module and tasks from the flow queue in the Task scheduling management module. Based on the idle status of relevant hardware components on the device side, it schedules different types of tasks, such as memory access, communication, floating-point calculation, and tensor calculation, to different execution components on the device side for execution. The relevant hardware components include: at least one computing unit and at least one storage unit.