Intelligent inference framework optimization method based on real-time operating system and multi-core CPU

Abstract

The present application relates to a kind of intelligent inference framework optimization method based on real-time operating system and multicore CPU, belong to real-time operating system field.The method of the present application includes: task splitting and core allocation: inference task is divided according to the way of tile, each core is responsible for processing the inference task of a tile;Each tile will be allocated to idle core and be processed in parallel;Real-time monitoring and dynamic adjustment: system will monitor the load condition of each core in real time, and dynamically adjust the allocation of task according to the idleness of core and the complexity of task;Through work stealing technique, if the load of a core is low, it will obtain task from other core to balance the load;Priority and task scheduling: when processing the task with higher real-time requirement, it is preferentially allocated to high-priority core.The present application effectively improves the inference efficiency and resource utilization of system.

Description

Technical Field

[0001] This invention belongs to the field of real-time operating systems, specifically relating to an optimization method for an intelligent inference framework based on a real-time operating system and a multi-core CPU. Background Technology

[0002] With the widespread application of artificial intelligence technology in various embedded systems, the efficiency of deep learning inference frameworks has become a core requirement, especially on resource-constrained devices (such as microcontrollers and edge devices), where optimization of inference frameworks is particularly important. Currently, TFLite Micro is designed for low-resource devices, but its native design is primarily based on single-core CPUs and bare-metal operating environments for inference computation. It lacks optimization for multi-core CPUs and real-time operating systems, failing to effectively utilize the parallel processing capabilities of multi-core processors and real-time operating systems, resulting in insufficient inference performance and wasted computing resources. Existing inference frameworks like TFLite Micro fail to fully utilize the computing resources of multi-core CPUs, leading to low inference efficiency. Furthermore, existing frameworks lack flexible task scheduling and load balancing mechanisms, failing to dynamically adjust task allocation based on real-time requirements and complexity, thus affecting the overall system performance and response speed.

[0003] To address this issue, this invention proposes an optimization method for an intelligent inference framework based on a real-time operating system and a multi-core CPU. It employs a load balancing algorithm and task scheduling strategy to ensure that inference tasks can be effectively allocated to multiple cores, thereby improving the overall inference speed and computing resource utilization. Summary of the Invention

[0004] (a) Technical problems to be solved The technical problem this invention aims to solve is how to provide an optimization method for an intelligent inference framework based on a real-time operating system and a multi-core CPU, in order to address the problem that existing frameworks lack flexible task scheduling and load balancing mechanisms, and cannot dynamically adjust task allocation according to the real-time requirements and complexity of tasks, thereby affecting the overall performance and response speed of the system.

[0005] (II) Technical Solution To address the aforementioned technical problems, this invention proposes an optimization method for an intelligent inference framework based on a real-time operating system and a multi-core CPU.

[0006] (III) Beneficial Effects This invention proposes an intelligent inference framework optimization method based on a real-time operating system and a multi-core CPU. Compared with existing technologies, this invention effectively improves the inference efficiency and resource utilization of the system by adopting an inference framework optimization method based on a real-time operating system and a multi-core CPU. Through dynamic load balancing and real-time task scheduling, this invention can intelligently allocate tasks, fully utilize the advantages of multi-core processors, and reduce the waste of computing resources. Simultaneously, the strategy of tile segmentation and overlapping region processing ensures the integrity and accuracy of the inference results, avoids the influence of boundary effects, and thus improves the overall performance and reliability of the system. Attached Figure Description

[0007] Figure 1 This is a structural diagram of the inference framework optimization system based on RTOS and multi-core CPU of the present invention; Figure 2 The system scheduling flowchart describes the corresponding task scheduling process in detail. Figure 3 A functional block diagram of the inference framework optimization system is provided, outlining the relationships between the modules and the data flow. Figure 4 This is a diagram illustrating tile segmentation. Taking a 640*640 image as an example, it is segmented using 160*160 tiles. The shaded areas represent overlapping content, which is used to reduce the boundary effect caused by tile segmentation and ensure the accuracy of the inference results. Figure 5 This diagram illustrates the reasoning process after tile segmentation. Reasoning efficiency is accelerated by assigning different tiles to different cores for reasoning. Detailed Implementation

[0008] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0009] This invention relates to an optimization method for a lightweight intelligent inference framework based on a real-time operating system (RTOS) and a multi-core CPU. Through innovative load balancing algorithms and task scheduling strategies, it enhances the parallel computing capability of inference tasks, reduces resource waste, increases inference speed, and ensures the execution of high real-time tasks.

[0010] The purpose of this invention is to propose an optimization method for inference frameworks based on real-time operating systems and multi-core CPUs, aiming to overcome the shortcomings of existing inference frameworks in fully utilizing computing resources and lacking flexible task scheduling in multi-core environments. By introducing innovative load balancing algorithms and task scheduling strategies, this invention can effectively improve the parallel processing capability of inference tasks, reduce inference time, and ensure the priority execution of high real-time tasks, thereby improving overall inference efficiency and meeting the real-time requirements of resource-constrained devices.

[0011] This invention proposes an optimization method for an intelligent inference framework based on a real-time operating system (RTOS) and a multi-core CPU. The aim is to improve the parallel computing capability of inference tasks, reduce resource waste, increase inference speed, and ensure the execution of high real-time tasks. This method is achieved through the following key technical measures: Task splitting and core allocation: The inference task is divided into tiles, with each core responsible for processing the inference task of one tile. Each tile is then assigned to an idle core for parallel processing.

[0012] Real-time monitoring and dynamic adjustment: The system monitors the load of each core in real time and dynamically adjusts task allocation based on core idle time and task complexity. Through work-stealing technology, if a core has a low load, tasks will be taken from other cores to balance the load.

[0013] Priority and Task Scheduling: When processing tasks with high real-time requirements, they are preferentially assigned to high-priority cores. The real-time operating system's scheduling algorithm ensures that tasks with high real-time requirements are executed first.

[0014] Example 1: Step 1: Decomposing the Reasoning Task S11. Obtain the input data (such as an image) for the inference task. Divide the inference task into multiple small subtasks according to the preset tile size (such as 160x160 or 80x80 pixels). Each subtask corresponds to one tile. Overlapping regions are used to ensure that the boundary information of each tile is fully processed. These tiles will be assigned to different cores for parallel computation.

[0015] S12. Each core is responsible for computing the inference task of one tile, ensuring the even distribution of tasks and the efficient utilization of multiple cores.

[0016] Step 2: Core Allocation and Load Balancing The system monitors the load of each core in real time. When a core's load is low, the load balancing module allocates more tasks to that core. When a core completes its tasks early and is idle or under low load, while other cores are heavily loaded and have a backlog of tasks, the low-load core will proactively retrieve tasks from the task queue of the high-load core. This combination of mechanisms ensures absolute load balance across all cores, thus avoiding resource waste.

[0017] Step 3: Hybrid Task Scheduling in the Inference Framework The system assigns priorities to each task based on the scheduling algorithm of the Real-Time Operating System (RTOS). High-real-time tasks are preferentially assigned to higher-priority cores to ensure that tasks with high real-time requirements can be executed in a timely manner. Low-priority tasks are scheduled to be executed on cores with lower load, so as not to affect the real-time performance of high-priority tasks.

[0018] Step 4: Perform the reasoning task Once the inference task is assigned to each core, the system initiates inference computation on each core. Each core performs parallel inference computation based on its assigned tiles. During computation, each core independently processes its assigned task but shares common data and model weights to ensure the accuracy of the inference.

[0019] Step 5: Merging and Outputting Reasoning Results S51. After each core completes its inference task, the system merges the inference results of all cores. For overlapping boundary areas, a weighted average or other fusion method is used to ensure the integrity and accuracy of the inference results.

[0020] S52. When performing tasks such as object detection, algorithms such as non-maximum suppression (NMS) are used to remove redundant detection boxes and effectively remove redundant information in tile overlapping areas, ensuring that the final inference output does not contain duplicate or erroneous results.

[0021] Step Six: Real-time Monitoring and Dynamic Adjustment The system continuously monitors the load of each core and dynamically adjusts the task allocation strategy based on task complexity, priority, and core idle status. When it detects that a core has a low load or that some tasks are executing too slowly, the system automatically adjusts the task allocation to ensure that the computing power of each core is fully utilized and that real-time tasks can be executed in a timely manner.

[0022] This invention innovates in task scheduling and load balancing by proposing a dynamic scheduling algorithm based on task complexity and core load. This algorithm dynamically adjusts task allocation by monitoring the load of each core in real time, ensuring that high real-time tasks are executed first and fully utilizing the computing resources of multi-core CPUs.

[0023] Key to this invention: The key point of this invention lies in effectively improving the parallel computing power and inference efficiency of an inference framework based on a real-time operating system (RTOS) and multi-core CPU by introducing innovative load balancing algorithms and real-time task scheduling strategies. By splitting inference tasks into multiple tiles and utilizing overlapping regions to ensure that the boundary information of each tile is fully processed, boundary effects are avoided, thus improving the accuracy of inference results. Simultaneously, this invention employs dynamic load balancing and work-stealing techniques during task allocation, which can adjust task allocation in real time according to the load of each core, ensuring full utilization of computing resources. By combining the scheduling mechanism of the real-time operating system, this invention provides fine-grained scheduling and management for tasks of different priorities, enabling high real-time tasks to be processed promptly and meeting the system's real-time requirements. Furthermore, the use of efficient inference result merging strategies, such as weighted averaging and non-maximum suppression (NMS), effectively removes redundant information from tile overlapping regions, ensuring that the final output inference results are accurate and free of duplication, thereby improving the performance and efficiency of the entire inference framework.

[0024] Beneficial effects: Compared with existing technologies, this invention effectively improves the inference efficiency and resource utilization of the system by adopting an inference framework optimization method based on a real-time operating system and multi-core CPU. Through dynamic load balancing and real-time task scheduling, this invention can intelligently allocate tasks, fully leverage the advantages of multi-core processors, and reduce the waste of computing resources. Simultaneously, the strategy of tile segmentation and overlapping region processing ensures the integrity and accuracy of inference results, avoids the influence of boundary effects, and thus improves the overall performance and reliability of the system.

[0025] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An optimization method for an intelligent inference framework based on a real-time operating system and a multi-core CPU, characterized in that, The method includes: Task splitting and core allocation: The reasoning task is divided into tiles, and each core is responsible for processing the reasoning task of one tile; each tile is allocated to an idle core for parallel processing; Real-time monitoring and dynamic adjustment: The system monitors the load of each core in real time and dynamically adjusts the task allocation based on the core's idle time and task complexity; through work-stealing technology, if the load of a certain core is low, tasks will be taken from other cores to balance the load. Priority and task scheduling: When processing tasks with high real-time requirements, prioritize assigning them to high-priority cores.

2. The optimization method for an intelligent inference framework based on a real-time operating system and a multi-core CPU as described in claim 1, characterized in that, The task splitting and core allocation specifically include: S11. Obtain the input data for the inference task, and divide the inference task into multiple small subtasks according to the preset tile size. Each subtask corresponds to one tile. Overlapping areas are used to ensure that the boundary information of each tile is fully processed. These tiles will be allocated to different cores for parallel computing. S12. Each core is responsible for computing the inference task of one tile, ensuring the even distribution of tasks and the efficient utilization of multiple cores.

3. The optimization method for an intelligent inference framework based on a real-time operating system and a multi-core CPU as described in claim 2, characterized in that, The input data is an image, with tile sizes of 160x160 or 80x80 pixels.

4. The optimization method for an intelligent inference framework based on a real-time operating system and a multi-core CPU as described in claim 1, characterized in that, In the real-time monitoring and dynamic adjustment, the system monitors the load of each core in real time. When the load of a core is low, the load balancing module will allocate more tasks to that core.

5. The optimization method for an intelligent inference framework based on a real-time operating system and a multi-core CPU as described in claim 4, characterized in that, In the real-time monitoring and dynamic adjustment, if the load on a certain core is low, the system will actively acquire tasks from other cores with higher loads through job stealing technology to ensure load balance across all cores and thus avoid resource waste.

6. The optimization method for an intelligent inference framework based on a real-time operating system and a multi-core CPU as described in claim 1, characterized in that, The priority and task scheduling includes: the system assigns a priority to each task according to the scheduling algorithm of the real-time operating system; high real-time tasks are given priority to be assigned to higher priority cores to ensure that tasks with high real-time requirements can be executed in a timely manner; low-priority tasks are arranged to be executed on cores with lower load, so as not to affect the real-time performance of high-priority tasks.

7. The optimization method for an intelligent inference framework based on a real-time operating system and a multi-core CPU as described in any one of claims 1-6, characterized in that, The method also includes: after the inference task is assigned to each core, the system will start the inference calculation of each core, and each core will perform parallel inference calculation according to the assigned tiles.

8. The optimization method for an intelligent inference framework based on a real-time operating system and a multi-core CPU as described in claim 7, characterized in that, During the computation process, each core independently processes its assigned task, but shares common data and model weights to ensure the accuracy of inference.

9. The optimization method for an intelligent inference framework based on a real-time operating system and a multi-core CPU as described in claim 8, characterized in that, This method also includes: merging and outputting inference results, specifically including: S51. After each core completes its inference task, the system will merge the inference results of all cores; for the handling of overlapping boundary areas, a weighted average method is used to ensure the integrity and accuracy of the inference results. S52. When performing object detection tasks, the Non-Maximum Suppression (NMS) algorithm is used to remove redundant detection boxes and effectively remove redundant information in tile overlapping areas, ensuring that the final inference output does not contain duplicate or erroneous results.

10. The optimization method for an intelligent inference framework based on a real-time operating system and a multi-core CPU as described in claim 9, characterized in that, The method also includes real-time monitoring and dynamic adjustment, specifically including: the system continuously monitors the load of each core and dynamically adjusts the task allocation strategy according to the complexity, priority and idle status of the tasks; when it is found that the load of a certain core is low or some tasks are executed too slowly, the system will automatically adjust the task allocation to ensure that the computing power of each core is fully utilized and that real-time tasks can be executed in a timely manner.

Images