Artificial intelligence model dynamic reasoning method and system based on consumption-level heterogeneous chip

Abstract

The invention discloses an artificial intelligence model dynamic reasoning method and system based on a consumer-level heterogeneous chip, and relates to the technical field of artificial intelligence and computer processing. The method is operated on a heterogeneous processor integrating a CPU (Central Processing Unit), a GPU (Graphics Processing Unit) and an NPU (Network Processing Unit), by monitoring hardware states such as load, temperature and available memory in real time and combining an AI model hierarchical structure and operator characteristics, reasoning calculation tasks are dynamically allocated to an optimal processing unit, namely, the CPU is responsible for task arrangement and serial logic, the GPU executes large-scale parallel calculation, and the NPU processes and optimizes a neural network layer. And the results are integrated cooperatively. The system adopts a dynamic inference engine, realizes efficient and low-energy-consumption operation of a large-scale AI model on resource-constrained equipment, improves response speed, protects local data privacy, and is suitable for various consumer electronics needing AI capability.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and computer hardware technology, specifically relating to a dynamic reasoning method and system for artificial intelligence models based on consumer-grade heterogeneous chips. Background Technology

[0002] In recent years, artificial intelligence technology, especially deep learning models represented by Large Language Models (LLMs), has made significant progress in fields such as natural language processing, computer vision, and multimodal understanding. For example, some publicly available large language models contain hundreds of billions to trillions of parameters and have demonstrated near-human-level reasoning and comprehension capabilities in various academic and professional benchmark tests. However, the reasoning process of such models places high demands on computing resources, memory bandwidth, and energy consumption. Traditional deployment methods often rely on high-performance cloud servers and specialized acceleration hardware (such as multi-GPU clusters), making it difficult to run them efficiently on consumer-grade devices.

[0003] Currently, the computing power and integration of consumer-grade chips have significantly improved. In terms of CPUs, the latest generation of desktop processors offers multi-core architectures with sixteen or more cores, coupled with large-capacity high-speed caches and advanced manufacturing processes, providing strong general-purpose computing and parallel processing capabilities. In terms of GPUs, consumer-grade graphics cards boast thousands of computing cores, tens of gigabytes of video memory, and hundreds of TFLOPS of half-precision floating-point performance, accelerating large-scale matrix operations and tensor calculations. In terms of mobile system-on-a-chip (SoC), representative products have integrated the CPU, GPU, and dedicated NPU onto a single chip, achieving high data transfer bandwidth and energy efficiency through high-speed unified memory.

[0004] Nevertheless, deploying large-scale LLMs directly to consumer hardware still presents several challenges:

[0005] (1) In terms of storage and memory, large language models usually require tens to hundreds of GB of memory to be fully loaded, while most consumer-grade GPUs have a memory capacity of 8GB to 24GB, and even high-end SoCs with unified memory usually have a range of 16GB to 128GB, making it difficult to accommodate a complete model at once.

[0006] (2) In terms of computing performance, LLM inference involves a large number of matrix multiplications and high-dimensional tensor operations. Consumer-grade CPUs are far inferior to professional accelerator cards in terms of floating-point computing power. Although GPUs have strong parallel capabilities, they may still be limited by memory bandwidth and core count when processing ultra-long contexts or larger-scale models.

[0007] (3) In terms of energy efficiency and thermal design, running large-scale models in power-constrained environments such as laptops or mobile devices can easily lead to increased heat generation and power consumption, thereby affecting system stability and battery life.

[0008] (4) Existing localization deployment methods mostly rely on model quantization, model pruning, or efficient parameter fine-tuning to adapt to limited hardware resources. However, these solutions usually require a trade-off between accuracy and inference speed, and most optimizations are targeted at specific hardware platforms, lacking the ability to adapt to a variety of consumer-grade hardware architectures.

[0009] A search of existing technologies revealed that, for example, Chinese Patent Publication CN120371566A proposes a tensor optimization method and system for GPU-accelerated deep learning inference engines. This method improves GPU inference efficiency through asymmetric quantization, tensor partitioning, and shared memory loading, and dynamically adjusts output based on GPU utilization and request queue length. While this approach can alleviate GPU memory bandwidth bottlenecks to some extent, it has the following shortcomings: First, this technology is only applicable to GPU environments and does not consider the joint scheduling of other heterogeneous computing units such as CPUs and NPUs. Second, the dynamic adjustment mechanism is based solely on GPU utilization and queue length, lacking real-time awareness and optimization of multi-dimensional hardware states such as power consumption and temperature. Third, the model optimization method is singular, limited to asymmetric quantization, and cannot flexibly select optimization strategies such as pruning and mixed precision based on different hardware conditions and inference tasks. Fourth, it does not introduce a fine-grained task partitioning method based on model network structure and operator type, making it difficult to achieve optimal task allocation across heterogeneous units.

[0010] In addition, while some open-source frameworks provide inference optimization interfaces for GPUs or specific hardware, they lack dynamic task decomposition and scheduling mechanisms across CPUs, GPUs, and NPUs, making it difficult to achieve optimal allocation among heterogeneous computing resources and limiting their universality and efficiency on various consumer-grade hardware.

[0011] In summary, existing technologies still have significant shortcomings in dynamic reasoning across heterogeneous hardware, real-time hardware status awareness, and multi-unit task collaborative scheduling. There is an urgent need for a general method and system that can realize dynamic reasoning of artificial intelligence models on consumer-grade heterogeneous chips, so as to simultaneously take into account inference speed, energy efficiency, and model accuracy. Summary of the Invention

[0012] To address the technical challenges mentioned in the background section regarding the deployment of artificial intelligence models on consumer-grade heterogeneous chips, such as storage and memory limitations, insufficient computing performance, energy efficiency issues, and imperfect task scheduling mechanisms, this invention provides a dynamic inference method and system for artificial intelligence models based on consumer-grade heterogeneous chips. The aim is to achieve efficient inference of artificial intelligence models on resource-constrained consumer devices through real-time hardware status monitoring, dynamic task decomposition and allocation, and optional model optimization.

[0013] The technical solution of the present invention is as follows:

[0014] This invention provides a dynamic inference method for an artificial intelligence model based on a consumer-grade heterogeneous chip. The heterogeneous chip includes a central processing unit (CPU), a graphics processing unit (GPU), and a neural network processing unit (NPU). The method includes the following steps: acquiring the current hardware state of each processing unit in the heterogeneous chip in real time; receiving an inference task for the artificial intelligence model; dynamically decomposing the inference task into multiple subtasks based on the network structure, operator type, and current hardware state of the artificial intelligence model; scheduling the multiple subtasks to one or more of the most suitable processing units among the CPU, GPU, and NPU for execution; and summarizing the execution results of each processing unit to generate the final inference output. The dynamic decomposition is based on a preset workload affinity model, which defines the execution efficiency relationship between different types of computational operators and different processing units. The CPU is mainly responsible for executing subtasks related to task flow control, data preprocessing, and serial computation. The GPU is mainly responsible for executing matrix operations and tensor computation subtasks with highly parallel characteristics. The NPU is mainly responsible for executing specific neural network layers that have undergone quantization or structural optimization, including some operations in convolutional layers or attention mechanisms. In addition, the method also includes a real-time model optimization step: before or during the execution of the inference task, the artificial intelligence model is dynamically pruned or quantized according to the power consumption limit or performance requirements of the device.

[0015] This invention also provides a dynamic inference system for artificial intelligence models based on consumer-grade heterogeneous chips. The system includes: a heterogeneous chip comprising a CPU, GPU, and NPU; a memory; and instructions stored in the memory and executable on the heterogeneous chip, wherein the instructions, when executed, enable the system to implement the aforementioned method. Specifically, the instructions include: a hardware status monitoring module for real-time acquisition of load, power consumption, and memory usage information of each processing unit; a workload partitioning module for decomposing and allocating inference tasks to different processing units according to preset rules and real-time hardware status; an execution coordinator module for managing the execution order, data synchronization, and result aggregation of each subtask; and a model optimization module for dynamically pruning, quantizing, or structurally optimizing the artificial intelligence model before or during task execution.

[0016] The present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described method. When running an artificial intelligence model on a consumer-grade heterogeneous chip, the technical solution of the present invention fully utilizes the general control capabilities of the CPU, the parallel computing advantages of the GPU, and the dedicated neural network acceleration characteristics of the NPU. Through dynamic task decomposition and real-time optimization mechanisms, it achieves balanced utilization of computing resources and optimization of the inference process.

[0017] Compared with the prior art, the present invention has the following beneficial effects:

[0018] (1) By dynamically decomposing tasks based on model structure and hardware status, subtasks are assigned to the most suitable processing units, avoiding excessive load or waste of resources on a single hardware. Compared with solutions that rely solely on GPUs (as described in CN120371566A), the overall inference speed can be improved, especially when dealing with large-scale parallel computing.

[0019] (2) Introducing a real-time optimization model that includes dynamic pruning and quantization, the model adapts to the memory limitations of consumer hardware and adjusts task allocation according to power consumption status, reducing unnecessary energy consumption and improving energy efficiency, which is especially suitable for mobile devices.

[0020] (3) This solution supports a variety of consumer-grade heterogeneous chip architectures, not limited to GPUs, but also including the collaboration of CPUs and NPUs. It solves the problem of insufficient scheduling of heterogeneous units in existing technologies, and achieves more refined dynamic adjustment through multi-dimensional hardware state awareness, thereby enhancing versatility and adaptability.

[0021] (4) The optimization process ensures the relative stability of model performance, while protecting local data privacy, avoiding delays and security risks in cloud deployment, and maintaining model accuracy.

[0022] Through the above technical solution, the present invention solves the shortcomings of the prior art mentioned in the background art and provides a feasible solution for the efficient deployment of artificial intelligence models on consumer devices. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1This is a schematic diagram of the architecture of an artificial intelligence model dynamic inference system based on a consumer-grade heterogeneous chip, according to an embodiment of the present invention.

[0025] Figure 2 This is a flowchart illustrating a dynamic reasoning method for an artificial intelligence model based on a consumer-grade heterogeneous chip, according to an embodiment of the present invention. Detailed Implementation

[0026] The specific embodiments of the present invention will be described in detail below with reference to the examples. It should be understood that the following examples are only used to explain the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention. Those skilled in the art can make appropriate modifications or equivalent substitutions based on the technical solutions of the present invention without departing from the spirit and scope of the present invention, and these modifications or substitutions should all fall within the scope of protection of the present invention.

[0027] The methods and systems of this invention can be implemented on various consumer computing devices, including but not limited to personal computers, laptops, smartphones, or embedded devices integrating CPUs, GPUs, and NPUs. Taking a consumer-grade SoC equipped with an Intel Core i7-13700K CPU, an NVIDIA RTX 3080 GPU (10GB VRAM), and an integrated NPU as an example, this device features a multi-core CPU (16 cores, 32 threads), a GPU (8704 CUDA cores), and an NPU (dedicated AI acceleration unit), with 32GB of DDR5 system memory. In this embodiment, the system runs in software, implementing instruction modules based on an open-source framework, and executing in conjunction with the operating system and hardware drivers.

[0028] like Figure 1 The diagram shows an architecture of a dynamic inference system for artificial intelligence models based on consumer-grade heterogeneous chips. The implementation of each module in this architecture is as follows:

[0029] Hardware status monitoring module implementation: This module collects real-time status information of each processing unit by calling the operating system interface, including CPU load utilization, GPU memory usage and utilization, and NPU temperature and power consumption. The sampling frequency is set to once per second to avoid excessive overhead. The collected data is stored in shared memory for access by other modules. When the GPU load exceeds 80%, the module marks it as a high-load state and notifies the workload partitioning module to adjust the allocation strategy.

[0030] The workload partitioning module is implemented as follows: This module decomposes tasks based on a preset workload affinity model. The affinity model is built through offline benchmarking, measuring the execution time and energy consumption of common operators on CPUs, GPUs, and NPUs, and establishing a mapping table: matrix operations are affinity-GPU (efficiency coefficient 0.9), serial logic is affinity-CPU (efficiency coefficient 0.8), and quantized neural network layers are affinity-NPU (efficiency coefficient 0.95). At runtime, the module parses the network structure of the artificial intelligence model with the Open Neural Network Exchange Format (ONNX), identifies the operator type, and dynamically decomposes the inference task into subtasks based on the current hardware status.

[0031] In the workload partitioning module, tasks are decomposed based on a preset workload affinity model; the affinity refers to the performance evaluation metric of an operator on a specific processing unit. The affinity is calculated using the following formula:

[0032]

[0033] Where op is the operator type, unit is the processing unit, and F op,unit T represents the efficiency coefficient of the operator op on the processing unit, i.e., the affinity score, ranging from [0,1]. A higher value indicates a better match; ref For reference execution time (in milliseconds), the minimum execution time of operators across all units is typically taken to standardize performance metrics and highlight the advantages of high-efficiency units; T op,unit The average execution time (in milliseconds) of the operator on the processing unit was measured through offline benchmarking. min The minimum energy consumption reference value for the operator is taken as the lowest energy consumption among all units to emphasize units with high energy efficiency; E op,unit The average energy consumption (in joules) of executing the operator on the processing unit was measured by benchmark testing; L unit The current load level of the processing unit, with a value ranging from [0,1], is collected in real time by the hardware status monitoring module. For example, if the CPU utilization is 80%, then L... unit It is 0.8; L max This is the maximum load threshold for the processing unit, usually set to 1, used to normalize load items.

[0034] Furthermore, to further achieve dynamic adjustment, the workload partitioning module introduces a dynamic affinity adjustment formula:

[0035]

[0036] Where k is the attenuation constant, with a value of 0.5; L thresholThe load threshold is set to 0.7. When the load exceeds the threshold, this formula reduces affinity exponentially to prevent tasks from being over-allocated to high-load units. This formula enhances the system's real-time adaptability and more effectively avoids overload compared to existing static scheduling schemes (such as the utilization-based adjustment in CN120371566A).

[0037] The implementation of the coordinator module manages subtask scheduling, using a multi-threaded framework to create task queues. Each subtask is encapsulated as an executable unit, containing input data and synchronization signals. Execution order is maintained through a dependency graph. Data synchronization employs a zero-copy mechanism to reduce transfer overhead between the CPU and GPU. Result aggregation is achieved through aggregation functions, returning the tensor output from the GPU to the CPU for final softmax activation function calculation and output generation. If a synchronization error occurs, the module rolls back and reassigns tasks.

[0038] The model optimization module is implemented as follows: Optimization is triggered before or during inference when hardware power consumption exceeds a preset threshold. Dynamic quantization uses a mean quantization algorithm with mini-batch datasets. Dynamic pruning is based on weight magnitude thresholds, employing a structured method to remove entire channels, with the pruning rate adaptively adjusted according to memory state. After optimization, the model is reloaded to the corresponding units to ensure inference continuity.

[0039] like Figure 2 The diagram shows a flowchart of a dynamic inference method for an artificial intelligence model based on consumer-grade heterogeneous chips. The steps of this method are as follows:

[0040] Step S1. The hardware status monitoring module collects data and obtains the current hardware status of each processing unit in the heterogeneous chip in real time through API calls, and stores it in shared memory; if the load of any unit exceeds the threshold, a notification mechanism is triggered to support subsequent dynamic adjustments.

[0041] Step S2. Receive the inference task for the artificial intelligence model, parse the user input, convert it into a tensor format that the model can process, and check the task complexity in preparation for subsequent decomposition. This step is implemented through the standard input interface and supports batch task queues.

[0042] Step S3. Check if model optimization is needed. If enabled, perform model optimization and proceed to step S4; otherwise, proceed directly to step S5.

[0043] Step S4. Model optimization module: This module performs dynamic quantization and pruning, adjusts model weights and structure, and reloads the model after optimization.

[0044] Step S5. The workload partitioning module decomposes the tasks, parses the model network structure, identifies the operator type, calculates affinity and dynamically adjusts affinity, decomposes the tasks into a list of subtasks, and assigns initial target units.

[0045] Step S6. Execute the coordinator module to schedule subtasks, such as CPU processing data preprocessing, GPU performing matrix operations, and NPU processing convolutional layers.

[0046] Step S7. Summarize the results and generate output.

[0047] The foregoing has provided a detailed description of a dynamic reasoning method and system for artificial intelligence models based on consumer-grade heterogeneous chips, as provided in the embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A dynamic inference method for artificial intelligence models based on consumer-grade heterogeneous chips, wherein the heterogeneous chips include a central processing unit (CPU), a graphics processing unit (GPU), and a neural network processing unit (NPU), characterized in that, The method includes the following steps: S1. Real-time acquisition of the current hardware status of each processing unit in the heterogeneous chip; S2. Receive the inference task for the artificial intelligence model; S3. Based on the network structure, operator type, and current hardware status of the artificial intelligence model, the inference task is dynamically decomposed into multiple sub-tasks; S4. Schedule the multiple subtasks to one or more of the most suitable processing units among the CPU, GPU and NPU for execution; S5. Summarize the execution results of each processing unit and generate the final inference output.

2. The method according to claim 1, characterized in that, In step S3, the dynamic decomposition is based on a preset workload affinity model, which defines the execution efficiency relationship between different types of computing operators and different processing units.

3. The method according to claim 1 or 2, characterized in that, The CPU is primarily responsible for executing subtasks related to task flow control, data preprocessing, and serial computation.

4. The method according to claim 1 or 2, characterized in that, The GPU is primarily responsible for performing matrix operations and tensor computation subtasks with highly parallel characteristics.

5. The method according to claim 1 or 2, characterized in that, The NPU is primarily responsible for executing specific neural network layers that have been quantized or structurally optimized, including some operations in convolutional layers or attention mechanisms.

6. The method according to claims 1 to 5, characterized in that, It also includes a real-time model optimization step: before or during the execution of the inference task, the artificial intelligence model is dynamically pruned or quantized according to the power consumption limit or performance requirements of the device.

7. A dynamic inference system for artificial intelligence models based on consumer-grade heterogeneous chips, characterized in that, The system includes: S1. A heterogeneous chip containing a CPU, GPU, and NPU; S2. Memory; S3. Instructions stored in the memory and executable on the heterogeneous chip, wherein, when executed, the instructions cause the system to implement the method of any one of claims 1 to 6.

8. The system according to claim 7, characterized in that, The instructions include: S1. Hardware status monitoring module, used to obtain the load, power consumption and memory usage information of each processing unit in real time; S2. Workload partitioning module, used to decompose and allocate inference tasks to different processing units according to preset rules and real-time hardware status; S3. The execution coordinator module is used to manage the execution order of each subtask, data synchronization, and result aggregation. S4. Model optimization module, used to dynamically prune, quantize or structurally optimize the artificial intelligence model before or during task execution.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method of any one of claims 1 to 6.

Images