ONNX-based deep learning reasoning adaptive acceleration method and device, and storage medium

Abstract

The invention discloses a deep learning reasoning adaptive acceleration method and device based on ONNX and a storage medium. The method comprises the steps of model loading and deployment, wherein a pathological diagnosis model file in an ONNX format is loaded; self-adaptive pre-acceleration optimization: according to the hardware characteristic information of the target computing device, calculating an optimal operation configuration parameter under a current hardware condition through a preset algorithm; executing optimization reasoning, namely driving a reasoning program of the model to run by applying the optimal running configuration parameters calculated in the accelerated optimization step; and S4, dynamic monitoring and adjustment: monitoring a hardware resource utilization rate index in a reasoning process in real time, and returning to the accelerated optimization step to recalculate the optimal operation configuration parameter when detecting that the fluctuation of the resource utilization rate exceeds a threshold value. According to the ONNX-based deep learning reasoning adaptive acceleration method provided by the invention, the optimal operation configuration parameters can be dynamically calculated according to the hardware characteristic information of the target calculation equipment in the microscope image pathology auxiliary diagnosis scene through adaptive pre-acceleration optimization.

Description

Technical Field

[0001] This invention belongs to the field of deep learning inference deployment technology, specifically relating to an adaptive acceleration method, device and storage medium for deep learning inference based on ONNX. Background Technology

[0002] In pathological auxiliary diagnosis, AI models need to perform high-precision analysis on massive amounts of pathological slide images, and their inference speed and stability directly affect diagnostic efficiency and accuracy. Against this backdrop, ONNX, as an open-source intermediate format for neural network models, has become a key link from training to clinical deployment of pathological auxiliary diagnostic models due to its compatibility with mainstream training frameworks such as PyTorch and TensorFlow, as well as its adaptability to multiple hardware platforms including CPUs and GPUs. This has promoted the large-scale application of deep learning in this field—for example, ONNX-based pathological image segmentation and lesion recognition models can be deployed on diverse equipment in different hospitals.

[0003] Currently, while mainstream inference engines such as ONNX Runtime support multiple systems and hardware backends, their adaptive optimization in pathological auxiliary diagnosis scenarios still has significant shortcomings: the inference engine's running parameters (such as the number of parallel threads within operators (intra_op_num_threads) and the number of parallel threads for model-level tasks (model_thread_num)) need to be configured manually or statically, and cannot be dynamically adjusted according to the hardware characteristics of diagnostic equipment (such as the number of CPU cores and load status), making it difficult to achieve optimal inference performance in diverse hardware environments. In pathological diagnosis, this manifests as follows: high-end equipment may waste computing power due to improper parameters, while low-end equipment in primary hospitals may prolong inference time due to parameter imbalances, directly affecting the timeliness of diagnosis. Simultaneously, existing technologies lack real-time awareness of system load. When pathology equipment processes multiple slides simultaneously, resource contention can easily lead to inference interruptions or result deviations, reducing diagnostic stability.

[0004] Existing improvements, such as preset parameter mapping tables, suffer from poor versatility and fail to consider the real-time load requirements of microscopic image pathological diagnosis, making it difficult to achieve the goal of "minimizing inference time while ensuring system stability." Therefore, addressing the shortcomings in the field of pathological auxiliary diagnosis, such as the lack of adaptive capabilities and the inability to balance inference speed and system stability, there is an urgent need for a dynamic optimization method based on ONNX pathological model inference acceleration that can dynamically optimize operating parameters such as the number of threads according to hardware characteristics and load conditions. Summary of the Invention

[0005] This invention provides an ONNX-based deep learning inference adaptive acceleration method, device, and storage medium to address the aforementioned technical problem of inaccurate traditional early warning methods. Specifically, the technical solution is as follows:

[0006] An adaptive acceleration method for deep learning inference based on ONNX includes the following steps:

[0007] S1. Model Loading and Deployment: Load the ONNX format pathological diagnostic model file and deploy it to the target computing device;

[0008] S2, Adaptive Pre-acceleration Optimization: Based on the hardware characteristics of the target computing device, the optimal operating configuration parameters under the current hardware conditions are adaptively calculated using a preset algorithm;

[0009] S3. Optimize inference execution: Apply the optimal running configuration parameters calculated in step S2 to drive the inference program of the model.

[0010] S4. Dynamic monitoring and adjustment: Monitor the hardware resource utilization rate during the inference process in real time. When the resource utilization rate fluctuation exceeds the threshold, return to step S2 to recalculate the optimal operating configuration parameters.

[0011] Further, in step S1, loading the ONNX format deep learning model file includes:

[0012] Model feature extraction: Parse the loaded ONNX model file and extract its computation graph structure feature information.

[0013] Furthermore, in step S2, the hardware characteristic information includes the number of physical cores N of the CPU.

[0014] Further, in step S2, the optimal operating configuration parameters include:

[0015] Number of parallel threads within the operator;

[0016] Number of parallel threads for model inference tasks;

[0017] Furthermore, the average CPU load rate and the average execution time of a single-threaded inference task are used as evaluation metrics.

[0018] Further, in step S2, the preset optimization algorithm includes:

[0019] S21. Traverse the possible values ​​of the number of parallel threads inside the operator in the first parameter search space, where the first parameter search space is an integer greater than or equal to 1 and less than or equal to N divided by 2.

[0020] S22. For each given number of parallel threads within an operator, traverse the possible values ​​of the number of parallel threads in the model inference task within the second parameter search space. The second parameter search space is an integer greater than or equal to 1 and less than or equal to N divided by the number of parallel threads within the current operator.

[0021] S23. For each parameter combination consisting of the number of parallel threads inside the operator and the number of parallel threads in the model inference task, perform multiple inference calculations and collect the average CPU load rate and the average execution time of the single-threaded inference task.

[0022] S24. Filter candidate parameter combinations that meet system stability constraints, that is, only retain combinations where the average CPU load rate is less than or equal to the preset safety threshold.

[0023] S25. Select the optimal configuration from the candidate combinations, that is, under the constraint of S24, select the parameter combination that minimizes the average execution time of the single-threaded inference task as the optimal runtime parallel configuration parameters.

[0024] Further, in step S3, the execution of the inference program driving the model includes:

[0025] Create a number of threads that match the number of parallel threads for the model inference task calculated in step S25.

[0026] Using the application programming interface provided by the ONNX runtime, set the value of the operator internal parallel thread count parameter in the session options object to the optimal value calculated in step S25.

[0027] Further, in step S4, the hardware resource utilization index includes:

[0028] Average load rate of CPU cores;

[0029] GPU memory usage;

[0030] Memory bandwidth utilization.

[0031] Furthermore, the ONNX-based deep learning inference adaptive acceleration method supports cross-operating system platform deployment, including Windows, Linux, and macOS, and the optimal running configuration parameters are hardware-independent on different operating systems.

[0032] A computer device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the aforementioned ONNX-based deep learning inference adaptive acceleration method.

[0033] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned ONNX-based adaptive acceleration method for deep learning inference.

[0034] The ONNX-based deep learning inference adaptive acceleration method provided in this invention dynamically calculates the optimal operating configuration parameters, namely the number of cores and threads, based on the hardware characteristics of the target computing device in a microscopic image-based pathology auxiliary diagnosis scenario through an adaptive pre-acceleration optimization step. This effectively avoids the limitations of manual configuration or static strategies in diverse diagnostic devices (such as ordinary terminals in primary hospitals to high-performance workstations in tertiary hospitals). This not only significantly improves the inference speed of AI models for pathological slide analysis and lesion identification, shortening the diagnostic result generation time, but also ensures stable system operation under a reasonable CPU load rate, avoiding diagnostic interruptions or delays caused by resource contention, thus supporting the continuity and reliability of pathological diagnosis.

[0035] The ONNX-based adaptive acceleration method for deep learning inference provided in this invention supports the deployment and running of ONNX format models exported from microscopic image pathology diagnostic model frameworks on multiple operating system platforms. By jointly optimizing the number of parallel threads within operators and the number of parallel threads executing model inference tasks, it achieves optimal cross-platform compatibility and inference performance. This provides developers of pathology-assisted diagnostic systems with an efficient and easy-to-use AI model inference solution, enabling rapid deployment in hardware environments across different levels of medical institutions. It broadens the applicability of this method in scenarios such as batch analysis of microscopic pathology slides and rapid emergency diagnosis, thereby contributing to improved overall pathology diagnostic efficiency. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a schematic diagram of an adaptive acceleration method for deep learning inference based on ONNX according to the present invention;

[0038] Figure 2 This is a flowchart of the adaptive pre-acceleration optimization algorithm of the present invention;

[0039] Figure 3 This is a schematic diagram of the dynamic monitoring and adjustment of the present invention. Detailed Implementation

[0040] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0041] The method described in this application is applied to the field of microscopic image-based pathological auxiliary diagnosis, specifically to the adaptive acceleration of deep learning model inference based on Open Neural Network Exchange (ONNX). This embodiment uses a Windows-based operating system scenario for intestinal polyp pathological diagnosis inference as an example to illustrate the ONNX-based deep learning inference adaptive acceleration method of this invention in detail.

[0042] like Figure 1 As shown, the ONNX-based adaptive acceleration method for deep learning inference in this embodiment includes:

[0043] Step S1: Model loading and deployment.

[0044] Specifically, the pathological diagnostic model file in the ONNX exchange format of the Open Neural Network is loaded and deployed to the target computing device. In step S1, loading the ONNX format deep learning model file includes:

[0045] Model feature extraction: This involves parsing the loaded ONNX model file and extracting its computational graph structure feature information for use in step S2.

[0046] Step S1 specifically includes:

[0047] Step S1.1: Model File Loading: In this embodiment, a PyTorch-trained intestinal polyp pathology classification model is used. First, the model is exported to ONNX format (filename resnet50.onnx) using PyTorch's torch.onnx.export() interface. The loading process calls the ONNX Runtime C++ API to complete the parsing and memory loading of the model file.

[0048] Step S1.2: Deploy to the target device: By calling session_options.SetIntraOpNumThreads(1), temporarily set it to 1 thread, and initially deploy the model to the computer CPU, waiting for the optimization parameters in step S2.

[0049] Step S2: Adaptive pre-acceleration optimization.

[0050] like Figure 2As shown, based on the hardware characteristics of the target computing device, the optimal operating configuration parameters under the current hardware conditions are adaptively calculated using a preset algorithm.

[0051] In step S2, the hardware characteristic information includes: the number of physical cores N of the CPU.

[0052] Optimal runtime configuration parameters include:

[0053] Number of parallel threads within the operator: intra_op_num_threads of ONNX Runtime;

[0054] Number of parallel threads for model inference task: model_thread_num.

[0055] Furthermore, the average CPU load rate (load_rate) and the average execution time (t) of a single-threaded inference task are used as evaluation metrics.

[0056] Step S2 is as follows:

[0057] Step S2.1: Obtain hardware information: Obtain the target device hardware characteristics through Windows API GetSystemInfo(): CPU physical core count N = 24.

[0058] Step S2.2: Determine the parameter search space.

[0059] Within the first parameter search space, iterate through the possible values ​​of the number of parallel threads inside the operator, `intra_op_num_threads`. The first parameter search space is an integer greater than or equal to 1 and less than or equal to N divided by 2, i.e., `intra_op_num_threads` ∈ [1, floor(N / 2)]. For each given number of parallel threads inside the operator, `intra_op_num_threads`, iterate through the possible values ​​of the number of parallel threads in the model inference task, `model_thread_num`, within the second parameter search space. The second parameter search space is an integer greater than or equal to 1 and less than or equal to N divided by the current number of parallel threads inside the operator, i.e., `model_thread_num` ∈ [1, floor(N / intra_op_num_threads)]. Specifically, the first parameter search space (`intra_op_num_threads`): [1, floor(24 / 2)] = [1, 12]. The second parameter search space (model_thread_num) has a range of [1, floor(24 / intra_op_num_threads)] for each intra_op_num_threads value. For example, when intra_op_num_threads = 2, the range of model_thread_num is [1, 12].

[0060] Step S2.3: Traversal and Data Acquisition: For each parameter combination (intra_op_num_threads, model_thread_num) consisting of the number of parallel threads inside the operator and the number of parallel threads in the model inference task, perform multiple inference calculations and collect the average CPU load rate (load_rate) and the average execution time (t) of the single-threaded inference task.

[0061] Specifically, perform the following operations for each parameter combination:

[0062] Set the ONNX Runtime parameter: session_options.SetIntraOpNumThreads(intra_op_num_threads), and use the C++ std::thread library to create model_thread_num parallel threads;

[0063] The input is image data captured by an external CCD camera connected to a microscope, and each thread performs inference calculations 20 times repeatedly.

[0064] The CPU average load rate is sampled every 100ms using the PerformanceCounter API, and the average value is taken during 20 inference cycles; the average execution time t of a single-threaded inference task is: total time / 20 / number of threads.

[0065] Step S2.4: Candidate parameter screening: Screen candidate parameter combinations that meet system stability constraints, that is, only retain combinations where the CPU average load rate is less than or equal to a preset safety threshold.

[0066] Specifically, a CPU load safety threshold of 75% is preset to ensure system responsiveness, L_threshold = 0.75, and combinations that meet the requirement of CPU average load rate load_rate ≤ 0.75 are selected.

[0067] Step S2.5: Optimal configuration selection: Select the optimal configuration from the candidate combinations, that is, under the constraints of step S2.4, select the parameter combination (intra_op_num_threads, model_thread_num) that minimizes the average execution time of the single-threaded inference task as the optimal runtime parallel configuration parameters.

[0068] The optimal configuration values ​​are obtained by selecting the parameter with the smallest t from the candidate combinations: intra_op_num_threads and model_thread_num.

[0069] Step S3: Optimize inference execution.

[0070] Specifically, the optimal operating configuration parameters calculated in step S2 are used to drive the model's inference program. Step S33, driving the model's inference program, includes:

[0071] Create a number of threads that match the number of parallel threads for the model inference task calculated in step S2.5, model_thread_num.

[0072] Using the application programming interface provided by the ONNX runtime, set the intra_op_num_threads parameter in the SessionOptions object to the optimal value calculated in step S2.5.

[0073] Specifically, step S3 includes:

[0074] Step S3.1: Create a thread pool: Create model_thread_num model-level inference threads using std::thread;

[0075] Step S3.2: Configure ONNX Runtime: Set the number of kernels enabled within a thread, i.e., call the function session_options.SetIntraOpNumThreads(intra_op_num_threads);

[0076] Step S3.3: Drive inference: Each thread calls session.Run() to execute inference and outputs the classification results.

[0077] Step S4: Dynamic monitoring and adjustment.

[0078] The hardware resource utilization index is monitored in real time during the inference process. When the fluctuation of resource utilization exceeds the threshold, the process returns to step S2 to recalculate the optimal operating configuration parameters.

[0079] In step S4, the hardware resource utilization metrics include: average CPU core load rate, GPU memory usage rate, and memory bandwidth utilization rate.

[0080] like Figure 3 As shown, step S4 specifically includes:

[0081] Step S4.1: Monitoring Metric Collection: While the program is performing model inference, the average CPU load rate is collected in parallel every 500ms using GetProcessorInfo(). Based on the threshold L_threshold = 0.75, a fluctuation threshold is set, and adjustments are triggered when the difference between the current average CPU load rate load_rate and L_threshold exceeds 10%.

[0082] Step S4.2: If a sudden increase in CPU load to 85% is detected, automatically return to step S2.2 and re-execute the parameter search. At this time, N is still 24, but the load baseline value is updated to the current system state. A new optimal configuration is recalculated: intra_op_num_threads, model_thread_num, which makes the load rate fall back below 75%, and the inference time is still the optimal solution.

[0083] The above steps can be repeated on Linux and macOS platforms to obtain the optimal parameter combination, which improves inference speed compared to the native ONNX Runtime without affecting system responsiveness.

[0084] It is understood that the ONNX-based adaptive acceleration method for deep learning inference in this application supports cross-operating system platform deployment, including Windows, Linux, and macOS, and the optimal running configuration parameters are hardware-independent on different operating systems.

[0085] This application also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the aforementioned ONNX-based deep learning inference adaptive acceleration method.

[0086] This application also discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned ONNX-based deep learning inference adaptive acceleration method.

[0087] As can be seen from the above specific implementation methods, the present invention can significantly improve inference performance in diverse hardware and operating system environments through adaptive parameter optimization and dynamic adjustment mechanisms, while ensuring system stability. It can be directly applied to industrial-grade AI inference deployment scenarios.

[0088] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any way, and all technical solutions obtained by equivalent substitution or equivalent transformation fall within the protection scope of the present invention.

Claims

1. An adaptive acceleration method for deep learning inference based on ONNX, characterized in that, Includes the following steps: S1. Model Loading and Deployment: Load the ONNX format pathological diagnostic model file and deploy it to the target computing device; S2, Adaptive Pre-acceleration Optimization: Based on the hardware characteristics of the target computing device, the optimal operating configuration parameters under the current hardware conditions are adaptively calculated using a preset algorithm; S3. Optimize inference execution: Apply the optimal running configuration parameters calculated in step S2 to drive the inference program of the model. S4. Dynamic monitoring and adjustment: Monitor the hardware resource utilization rate during the inference process in real time. When the resource utilization rate fluctuation exceeds the threshold, return to step S2 to recalculate the optimal operating configuration parameters.

2. The ONNX-based adaptive acceleration method for deep learning inference according to claim 1, characterized in that, In step S1, loading the ONNX format deep learning model file includes: Model feature extraction: Parse the loaded ONNX model file and extract its computation graph structure feature information.

3. The ONNX-based adaptive acceleration method for deep learning inference according to claim 1, characterized in that, In step S2, the hardware characteristic information includes the number of physical cores N of the CPU.

4. The ONNX-based adaptive acceleration method for deep learning inference according to claim 3, characterized in that, In step S2, the optimal operating configuration parameters include: Number of parallel threads within the operator; Number of parallel threads for model inference tasks; Furthermore, the average CPU load rate and the average execution time of a single-threaded inference task are used as evaluation metrics.

5. The ONNX-based adaptive acceleration method for deep learning inference according to claim 4, characterized in that, In step S2, the preset optimization algorithm includes: S21. Traverse the possible values ​​of the number of parallel threads inside the operator in the first parameter search space, where the first parameter search space is an integer greater than or equal to 1 and less than or equal to N divided by 2. S22. For each given number of parallel threads within an operator, traverse the possible values ​​of the number of parallel threads in the model inference task within the second parameter search space. The second parameter search space is an integer greater than or equal to 1 and less than or equal to N divided by the number of parallel threads within the current operator. S23. For each parameter combination consisting of the number of parallel threads inside the operator and the number of parallel threads in the model inference task, perform multiple inference calculations and collect the average CPU load rate and the average execution time of the single-threaded inference task. S24. Filter candidate parameter combinations that meet system stability constraints, that is, only retain combinations where the average CPU load rate is less than or equal to the preset safety threshold. S25. Select the optimal configuration from the candidate combinations, that is, under the constraint of S24, select the parameter combination that minimizes the average execution time of the single-threaded inference task as the optimal runtime parallel configuration parameters.

6. The ONNX-based adaptive acceleration method for deep learning inference according to claim 5, characterized in that, In step S3, the inference program of the driving model is executed as follows: Create a number of threads that match the number of parallel threads for the model inference task calculated in step S25. Using the application programming interface provided by the ONNX runtime, set the value of the operator internal parallel thread count parameter in the session options object to the optimal value calculated in step S25.

7. The ONNX-based adaptive acceleration method for deep learning inference according to claim 1, characterized in that, In step S4, the hardware resource utilization index includes: Average load rate of CPU cores; GPU memory usage; Memory bandwidth utilization.

8. The ONNX-based adaptive acceleration method for deep learning inference according to claim 1, characterized in that, The ONNX-based adaptive acceleration method for deep learning inference supports cross-operating system platform deployment, including Windows, Linux, and macOS, and the optimal running configuration parameters are hardware-independent on different operating systems.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the ONNX-based adaptive acceleration method for deep learning inference as described in any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the ONNX-based adaptive acceleration method for deep learning inference as described in any one of claims 1-8.

Images