An AI model inference heterogeneous computing acceleration system

Abstract

The application discloses a heterogeneous computing acceleration system for AI model reasoning, comprising a heterogeneous computing execution module, a performance monitoring and evaluation module, a strategy optimization and task division module, and a model retraining and updating module; the four modules are connected through data flow and control flow to constitute a complete reasoning evaluation optimization updating closed-loop feedback system, dynamic intelligent scheduling of heterogeneous resources is realized, and the deployment efficiency of medical AI application is improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and high-performance computing technology, and in particular to a heterogeneous computing acceleration system for AI model inference. Background Technology

[0002] With the deepening application of artificial intelligence in the medical field, such as medical image analysis (CT, MRI), pathological slide recognition, and multimodal vital sign prediction, extremely high demands are placed on the inference speed, accuracy, and system stability of AI models. Medical scenarios often require real-time or near-real-time diagnostic assistance, such as lung nodule detection in emergency systems, requiring single-sample inference latency of less than 50 milliseconds. At the same time, the sensitivity and privacy of medical data make cloud-based inference risky due to latency and privacy concerns, making edge or local deployment a growing trend.

[0003] Existing acceleration solutions mostly focus on optimizing a single hardware platform or homogeneous distributed computing. For example, using multiple FPGAs or NPUs for collaborative computing can improve performance, but the system lacks compatibility and flexibility, making it difficult to perform fine-grained scheduling based on different model layer characteristics and dynamic loads. Another approach is to allocate computationally intensive and regular pre- and post-processor computing areas to the Neural Processing Unit (NPU) and dynamically sparse areas to the Central Processing Unit (CPU) to achieve preliminary heterogeneous division of labor. However, this static partitioning strategy cannot adapt to model iteration, changes in data distribution, and fluctuations in hardware status. Furthermore, existing systems lack a closed-loop mechanism for feeding back inference results to model training and adjusting task scheduling strategies, making it impossible for the system to continuously optimize over long-term operation and difficult to cope with the complex and ever-evolving diagnostic needs in medical scenarios.

[0004] At the model training level, traditional reinforcement learning methods (such as swarm relative policy optimization algorithms) require computing a large number of samples to generate high-quality inference chains, resulting in huge training overhead and low efficiency. This limits the model's ability to rapidly iterate and update based on online feedback.

[0005] Therefore, there is an urgent need for an AI model inference acceleration system that can deeply integrate heterogeneous computing resources, possess intelligent dynamic task partitioning capabilities, and form a complete optimization closed loop based on inference performance feedback, in order to meet the medical field's demand for high-performance, high-reliability, and adaptive AI applications. Summary of the Invention

[0006] The purpose of this invention is to provide a heterogeneous computing acceleration system for AI model inference, so as to solve the problems existing in the prior art.

[0007] To achieve the above objectives, the present invention provides the following solution: The present invention provides a heterogeneous computing acceleration system for AI model inference, comprising:

[0008] The heterogeneous computing execution module, performance monitoring and evaluation module, strategy optimization and task partitioning module, and model retraining and update module are sequentially connected and form a closed loop.

[0009] The heterogeneous computing execution module is used to load the AI ​​inference model and execute medical data inference tasks on computing units containing multiple architectures.

[0010] The performance monitoring and evaluation module is used to collect performance data in real time during the inference process and calculate the system performance score.

[0011] The strategy optimization and task partitioning module is used to allocate strategies based on the performance data and system performance score, through the optimal computing units of each operator in the dynamic cost model and the reinforcement learning policy network dynamic decision AI inference model.

[0012] The model retraining and update module is used to retrain and update the AI ​​inference model based on data accumulated during the inference process when the performance data or system performance score indicates that the model performance has degraded.

[0013] Preferably, the system performance score calculated by the performance monitoring and evaluation module is as follows:

[0014] SES=α*(T_baseline / T_actual)+β*Accuracy+γ*(1-P_actual / P_max)-δ*(D_total / D_max),

[0015] Where T_actual is the actual inference time, T_baseline is the baseline time; Accuracy is the task-related accuracy; P_actual is the actual power consumption, P_max is the power consumption limit; D_total is the total data transfer volume between devices, D_max is the bandwidth limit; α, β, γ, δ are configurable weight coefficients.

[0016] Preferably, the dynamic cost model in the strategy optimization and task partitioning module is used to evaluate the cost of allocating operators to computing units (devices), and its calculation formula is as follows:

[0017] Cost(op,device)=ω_c*CompTime(op,device)^p_c+

[0018] ω_h*(1 / HardwareUtil(device))^p_h+ω_d*DataTransTime(op,device)^p_d,

[0019] Where CompTime, HardwareUtil, and DataTransTime are the estimation of computation time, equipment utilization, and data transmission time, respectively; ω_c, ω_h, and ω_d are the weighting coefficients, p_c, p_h, and p_d are the power coefficients, and op is the operator.

[0020] Preferably, the scheduling objective of the strategy optimization and task partitioning module is to minimize the longest execution time in the execution queue of each computing unit, so as to achieve heterogeneous computing load balancing.

[0021] Preferably, the state space of the reinforcement learning policy network includes model layer features and hardware load, the action space is the selection of computing units, and the reward function is associated with SES.

[0022] Preferably, the training and updating method of the model retraining and updating module includes: generating multiple inference results using the current model on the accumulated data; calculating the reward value of each result and its relative advantage value in the group of results; pruning based on the absolute value of the relative advantage value, retaining samples with high advantage values; and calculating the gradient and updating the model parameters only based on the retained samples.

[0023] Preferably, the model retraining and update module further includes a scheduling strategy: in a distributed training environment, when the number of effective training samples on a certain computing device is lower than a threshold, new, pruned and filtered samples are automatically sampled from the data pool and allocated to the device to maintain a balanced computing load.

[0024] Preferably, the computing units supported by the heterogeneous computing execution module include at least two of the following: central processing unit, graphics processing unit, neural network processor, field-programmable gate array, and digital signal processor.

[0025] The present invention discloses the following technical effects: The present invention realizes dynamic intelligent scheduling of heterogeneous resources; by integrating dynamic cost model and reinforcement learning policy network, the system can make optimal task partitioning decisions in real time according to model characteristics and hardware status, which significantly improves the overall utilization and energy efficiency of heterogeneous hardware such as GPU, NPU, and CPU.

[0026] This invention forms a complete performance optimization closed loop; through the "execution-monitoring-evaluation-optimization" closed loop, the system not only optimizes the task scheduling strategy, but also triggers iterative updates of model parameters, enabling the system to have continuous adaptive capabilities, maintain optimal performance over a long period of time, and cope with changes in the distribution of medical data.

[0027] This invention improves the deployment efficiency of medical AI applications. Through closed-loop optimization, the system can meet the stringent real-time (e.g., latency below 50ms) and accuracy requirements of medical scenarios while reducing reliance on a single top-tier hardware, thus improving system compatibility and cost-effectiveness. In actual testing, on similar medical image classification tasks, the single-sample inference latency can be reduced from 127ms to 38ms, while simultaneously improving task accuracy.

[0028] This invention reduces the cost of model iteration; the advantage-based pruning training method it employs, by focusing on high-value training samples, significantly reduces the computational overhead and time cost of incremental learning or fine-tuning of the model based on feedback, making rapid iteration of clinical models possible. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0030] Figure 1 This is a flowchart illustrating the overall architecture and closed-loop feedback of the system of this invention. Figure 2 This diagram illustrates the collaborative working principle of the dynamic cost model and the reinforcement learning policy network in the strategy optimization and task partitioning module. Figure 3 This is a schematic diagram of the training process for the model retraining and update module. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0033] This invention provides a heterogeneous computing acceleration system for AI model inference, comprising:

[0034] The heterogeneous computing execution module, performance monitoring and evaluation module, strategy optimization and task partitioning module, and model retraining and update module are sequentially connected and form a closed loop.

[0035] The heterogeneous computing execution module is used to load the AI ​​inference model and execute medical data inference tasks on computing units containing multiple architectures.

[0036] The performance monitoring and evaluation module is used to collect performance data in real time during the inference process and calculate the system performance score;

[0037] The strategy optimization and task partitioning module is used to allocate strategies to the optimal computing units of each operator in the dynamic decision-making AI inference model based on performance data and system efficiency scores through a dynamic cost model and a reinforcement learning policy network.

[0038] The model retraining and update module is used to retrain and update the AI ​​inference model based on data accumulated during the inference process when performance data or system performance scores indicate that the model performance has degraded.

[0039] The scheduling objective of the strategy optimization and task partitioning module is to minimize the longest execution time in the execution queue of each computing unit in order to achieve heterogeneous computing load balancing.

[0040] The state space of a reinforcement learning policy network includes model layer features and hardware load, the action space is the selection of computational units, and the reward function is associated with SES.

[0041] The training and updating methods of the model retraining and updating module include: generating multiple inference results using the current model on the accumulated data; calculating the reward value of each result and its relative advantage value in the group of results; pruning based on the absolute value of the relative advantage value, retaining samples with high advantage values; and calculating gradients and updating model parameters only based on the retained samples.

[0042] The model retraining and update module also includes a response scheduling strategy: in a distributed training environment, when the number of valid training samples on a computing device is lower than a threshold, new, pruned samples are automatically sampled from the data pool and allocated to that device to maintain a balanced computing load.

[0043] In one embodiment of the present invention, a heterogeneous computing execution module is used to receive medical data to be inferred (such as medical images and physiological signal sequences) and load an AI inference model. This module includes various heterogeneous computing units, including but not limited to a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), a field-programmable gate array (FPGA), and a digital signal processor (DSP). The module has a built-in unified device abstraction layer that encapsulates the calling interfaces of different hardware.

[0044] In one embodiment of the present invention, a performance monitoring and evaluation module is used to collect performance data of the heterogeneous computing execution module in real time during the inference process, including: execution time, power consumption, cache hit rate, data transmission bandwidth and latency of each model layer on different computing units, as well as clinical evaluation indicators such as the accuracy and confidence of the final inference result. This module calculates the core System Efficiency Score (SES) as a quantitative basis for closed-loop feedback. The SES calculation formula is as follows:

[0045] SES = α*(T_baseline / T_actual) + β*Accuracy + γ*(1-P_actual / P_max) - δ*(D_total / D_max); where T_actual and P_actual are the actual inference time and power consumption, respectively; T_baseline is the baseline time; P_max is the upper limit of power consumption; Accuracy is the task-related accuracy; D_total is the total data transfer volume between devices; D_max is the upper limit of bandwidth; α, β, γ, and δ are weighting coefficients, which are configured according to the real-time, accuracy, and energy efficiency requirements of the medical scenario; for example, the α value is higher in the emergency room scenario.

[0046] In one embodiment of the present invention, the strategy optimization and task partitioning module is the intelligent hub of the system. It receives the SES (Self-Evaluation System Execution) and detailed performance data calculated by the performance monitoring and evaluation module. This module maintains a dynamic cost model to evaluate the expected overhead of allocating any model operator to a specific heterogeneous computing unit. The cost model draws on ideas from search results, and its calculation method is: Cost(op,device)=ω_c*CompTime(op,device)^p_c+ω_h*(1 / HardwareUtil(device))^p_h+ω_d*DataTransTime(op,device)^p_d;

[0047] Where CompTime is the computation time estimate, HardwareUtil is the current device utilization, and DataTransTime is the required data transfer time; ω_c, ω_h, ω_d are the weights, and p_c, p_h, p_d are the power coefficients. The initial values ​​of these parameters are set based on the hardware specifications and will be dynamically adjusted by the strategy optimization and task partitioning module as the system runs.

[0048] The core of this module is a policy network based on reinforcement learning. Its state space (S) includes: features of the current model layer (type, input / output dimension, sparsity), real-time load status of each heterogeneous computing unit, and historical SES trends. The action space (A) is for selecting target computing units (such as CPU, GPU, NPU) for the current layer. The reward function (R) is directly linked to the improvement of SES. Through continuous learning, the policy network can output the optimal task partitioning scheme. Its scheduling objective can be formalized as minimizing max(T_CPU, T_GPU, T_NPU, ...), that is, minimizing the longest execution time in each device queue to achieve load balancing.

[0049] Furthermore, the model retraining and update module is used to optimize the AI ​​inference model itself based on closed-loop feedback. When the performance monitoring and evaluation module detects a continuous decline in model accuracy on specific categories of data, or the policy optimization module identifies certain model layers as performance bottlenecks that cannot be mitigated through scheduling, a retraining process will be triggered. This module employs an efficient training method based on dominance pruning; the training method specifically includes:

[0050] Use the currently deployed model (as the legacy strategy model) to generate multiple inference results (responses) from the accumulated samples of complex medical data.

[0051] Calculate the reward value for each outcome using a lightweight reward model (which can be based on doctor feedback or rules).

[0052] Calculate the relative advantage value of a set of results (reward value minus mean divided by variance), and calculate its absolute value as an indicator of pruning importance.

[0053] Based on the set pruning rate (e.g., 0.5), low-quality or low-information samples with low absolute dominance values ​​are preferentially pruned, and only important samples are retained for gradient calculation.

[0054] Based on the new policy model, the reference model, and the old policy model, the predicted probability is calculated only for the retained samples, and the gradient is calculated accordingly to update the new policy model. This method significantly reduces training overhead and enables rapid model iteration.

[0055] In one embodiment of the present invention, inference of a 3DCNN model for lung nodule detection in CT images is accelerated;

[0056] This embodiment uses the automatic detection and classification of lung nodules from lung CT images as an application scenario. The inference model is a complex 3D convolutional neural network (3DCNN).

[0057] System initialization and model loading:

[0058] The heterogeneous computing execution module connects to a server containing a CPU, GPU, and a dedicated medical imaging NPU. After system startup, the performance monitoring and evaluation module establishes a baseline, recording the time consumption (T_baseline), average power consumption, and nodule detection sensitivity and specificity of the 3DCNN model under pure GPU inference. The policy optimization and task partitioning module initializes the dynamic cost model parameters and loads the pre-trained reinforcement learning policy network.

[0059] First round of inference and performance monitoring:

[0060] A batch of CT image blocks was received. In the initial stage, the policy network adopted a conservative strategy, allocating most of the 3D convolutional computations to the GPU and NPU, and assigning some non-regular computations such as logical judgments and post-processing to the CPU. The performance monitoring and evaluation module recorded in detail: the execution time of the first 3D convolutional layer on the NPU was t1, and on the GPU it was t1'; a certain deconvolutional layer became a bottleneck on the GPU due to memory bandwidth limitations; the model's final detection rate of small nodules (<3mm) was lower than expected. The initial SES value was calculated to be SES0.

[0061] Strategy optimization and dynamic task partitioning:

[0062] The policy optimization and task partitioning module analyzes the data and finds that deconvolutional layers have better energy efficiency on the NPU. The dynamic cost model updates its internal parameters based on the measured CompTime and DataTransTime. The policy network outputs new actions based on the new state (including bottleneck layer information and NPU idle state): in the next batch of inference, the deconvolutional layer is migrated from the GPU to the NPU for execution, while some data preprocessing work is offloaded from the CPU to the GPU's CUDA cores to free up CPU resources. This scheduling objective aims to minimize max(T_GPU, T_NPU, T_CPU).

[0063] Closed-loop feedback and model update:

[0064] After several rounds of adjustments, the system's SES value steadily increased to SES1 (SES1 > SES0); however, the performance monitoring module found that the false negative rate of the model was trending upward for the specific type of "ground-glass opacity nodules". The model retraining and update module was triggered. It collects recent false negative samples, uses the current model to generate multiple possible bounding boxes and classification predictions, and scores them using the built-in reward model (which gives rewards based on the IoU of the bounding box and the doctor's annotation and the classification correctness).

[0065] Subsequently, a training method based on dominance pruning was applied, retaining only the top 30% of samples with the highest dominance values ​​for gradient calculation, which efficiently completed a round of model fine-tuning and improved the ability to identify this type of nodule.

[0066] Effect verification:

[0067] After running in a closed loop for one week, the system reached a stable state. Compared with the initial pure GPU solution, while maintaining an overall sensitivity of >95.7%, the average inference time per CT image (512x512x300 voxels) was reduced from 4.2 seconds to 2.1 seconds, and the overall power consumption of the system decreased by 35%, achieving a dual improvement in performance and energy efficiency.

[0068] In another embodiment of the present invention, the inference of the Transformer model for multimodal vital sign prediction in the ICU is accelerated; this embodiment takes the early prediction of disease deterioration based on multimodal time-series data (ECG, BP, SpO2) in the intensive care unit (ICU) as the application scenario.

[0069] System features and initialization:

[0070] This scenario utilizes an edge computing device, with hardware comprising a low-power CPU, a DSP, and an FPGA. The FPGA has been programmed with a customized attention mechanism and one-dimensional convolutional computation circuitry. The system is extremely sensitive to latency, requiring the prediction of 12 hours of data to be completed within 60 seconds. During initialization, the policy network tentatively allocates self-attention layers and fully connected layers in the Transformer.

[0071] Dynamic scheduling and cache optimization:

[0072] Performance monitoring revealed that self-attention computation is extremely fast on the FPGA, but the attention patterns corresponding to different patient data vary greatly, resulting in a low FPGA cache hit rate. The policy optimization module introduces a dynamic priority cache management mechanism. It maintains a dynamic priority score S for the weight parameters of each attention head, and updates it after each inference based on the activation intensity (routing score) of that head: S_new = α * Activation + (1-α) * S_old. When there is insufficient space to load the key attention weights of a new patient into the FPGA on-chip cache, the weight block with the lowest priority score S is replaced.

[0073] Pipeline parallelism and forward prefetching:

[0074] To further conceal the data transfer latency from CPU memory to the FPGA, the system implements a pipelined parallel design. Independent data flows are created: Flow 1 transfers multimodal data from the CPU to the FPGA for the current time step; Flow 2 performs computations on the FPGA on the ready data from the previous time step; Flow 3 performs scalar computations and subsequent logic on the CPU. These three flows execute concurrently. Simultaneously, the strategy module proactively predicts the attention heads most likely to be activated in the next few time steps based on current vital sign trends and prefetches their weights into the FPGA cache, further reducing computational latency.

[0075] Closed-loop model optimization:

[0076] Monitoring revealed that the predictive specificity for septic shock needed improvement. The model retraining module was activated. Utilizing data from cases collected over the past week that had low model prediction confidence but were ultimately diagnosed, it employed a highly efficient training method based on dominance pruning to quickly generate multiple prediction sequences. After pruning low-dominance samples, the model was fine-tuned, effectively improving the early identification specificity for this critical condition.

[0077] Effect verification:

[0078] Through closed-loop system optimization, the edge prediction device maintains an average prediction latency of less than 45 seconds during continuous operation, meeting real-time requirements. Compared to before optimization, the FPGA cache hit rate is improved by 40%, the overall system power consumption is reduced by 25%, and the prediction specificity for septic shock is increased from 88% to 93%, significantly improving the reliability of clinical early warning.

[0079] In the description of this invention, it should be understood that the terms "longitudinal", "lateral", "up", "down", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this invention, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.

[0080] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A heterogeneous computing acceleration system for AI model inference, characterized in that, include: The heterogeneous computing execution module, performance monitoring and evaluation module, strategy optimization and task partitioning module, and model retraining and update module are sequentially connected and form a closed loop. The heterogeneous computing execution module is used to load the AI ​​inference model and execute medical data inference tasks on computing units containing multiple architectures. The performance monitoring and evaluation module is used to collect performance data in real time during the inference process and calculate the system performance score. The strategy optimization and task partitioning module is used to allocate strategies based on the performance data and system performance score, through the optimal computing units of each operator in the dynamic cost model and the reinforcement learning policy network dynamic decision AI inference model. The model retraining and update module is used to retrain and update the AI ​​inference model based on data accumulated during the inference process when the performance data or system performance score indicates that the model performance has degraded.

2. The heterogeneous computing acceleration system for AI model inference according to claim 1, characterized in that, The system performance score calculated by the performance monitoring and evaluation module is as follows: SES=α*(T_baseline / T_actual)+β*Accuracy+γ*(1-P_actual / P_max)-δ*(D_total / D_max), Where T_actual is the actual inference time, T_baseline is the baseline time; Accuracy is the task-related accuracy; P_actual is the actual power consumption, P_max is the power consumption limit; D_total is the total data transfer volume between devices, D_max is the bandwidth limit; α, β, γ, δ are configurable weight coefficients.

3. The heterogeneous computing acceleration system for AI model inference according to claim 1, characterized in that, The dynamic cost model in the strategy optimization and task partitioning module is used to evaluate the cost of allocating operators to computing units (devices), and its calculation formula is as follows: Cost(op,device)=ω_c*CompTime(op,device)^p_c+ ω_h*(1 / HardwareUtil(device))^p_h+ω _d*DataTransTime(op,device)^p_d, Where CompTime, HardwareUtil, and DataTransTime are the estimation of computation time, equipment utilization, and data transmission time, respectively; ω_c, ω_h, and ω_d are the weighting coefficients, p_c, p_h, and p_d are the power coefficients, and op is the operator.

4. The heterogeneous computing acceleration system for AI model inference according to claim 3, characterized in that: The scheduling objective of the strategy optimization and task partitioning module is to minimize the longest execution time in the execution queue of each computing unit in order to achieve heterogeneous computing load balancing.

5. The heterogeneous computing acceleration system for AI model inference according to claim 3, characterized in that: The state space of the reinforcement learning policy network includes model layer features and hardware load, the action space is the selection of computing units, and the reward function is associated with SES.

6. The heterogeneous computing acceleration system for AI model inference according to claim 1, characterized in that, The training and updating methods of the model retraining and updating module include: generating multiple inference results using the current model on the accumulated data; calculating the reward value of each result and its relative advantage value in the group of results; pruning based on the absolute value of the relative advantage value, retaining samples with high advantage values; and calculating the gradient and updating the model parameters only based on the retained samples.

7. The heterogeneous computing acceleration system for AI model inference according to claim 6, characterized in that, The model retraining and update module also includes a scheduling strategy: in a distributed training environment, when the number of effective training samples on a computing device is lower than a threshold, new, pruned and filtered samples are automatically sampled from the data pool and allocated to that device to maintain a balanced computing load.

8. The heterogeneous computing acceleration system for AI model inference according to claim 1, characterized in that: The heterogeneous computing execution module supports computing units including at least two of the following: central processing unit, graphics processing unit, neural network processor, field-programmable gate array, and digital signal processor.

Images