Mobile terminal lightweight reasoning optimization method and system based on PyTorch Android Lite
The PyTorch Android Lite method, which utilizes quantization training and static graph optimization, solves the problems of mobile memory and hardware compatibility, achieves high inference latency, and meets the needs of real-time video processing.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-09
- Publication Date
- 2026-04-07
AI Technical Summary
PyTorch Android Lite faces issues such as excessive memory consumption, poor hardware compatibility, and high inference latency on mobile devices, making it difficult to run stably on resource-constrained devices and meet the needs of real-time interactive applications.
A lightweight neural network model is generated using a quantization-aware training method, converted to low-precision INT8 format, and optimized into a static graph representation using TorchScript. Combined with Vulkan backend hardware acceleration and asynchronous processing pipeline, training nodes and operators are separated and merged to achieve efficient utilization of hardware resources.
Significantly reduces memory usage to less than 50MB, improves hardware compatibility and inference speed, and reduces single-frame latency to 10-30 milliseconds, meeting the needs of real-time video processing.
Smart Images
Figure CN121809655A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of mobile artificial intelligence computing technology, specifically to a lightweight inference optimization method and system for mobile devices based on PyTorch AndroidLite. Background Technology
[0002] With the increasing computing power of mobile chips, real-time deep learning inference on terminal devices (On-Device AI) has become a development trend. This approach offers advantages such as low latency, strong privacy protection, and no reliance on the network. PyTorch, as a mainstream deep learning framework, offers its mobile version, PyTorch Android Lite (also known as PyTorch Mobile), designed to provide efficient inference capabilities for Android devices.
[0003] PyTorch Android Lite is a subset of PyTorch designed specifically for mobile devices, providing lightweight, fast, and efficient model inference capabilities. It features: 1) Cross-platform support: Supports various Android devices. 2) Optimized performance: Optimized for model size and execution time, suitable for deployment on resource-constrained devices. 3) Simplified deployment: Simplifies integration with the Android platform through C++ and Java APIs.
[0004] First, ensure that Android Studio and NDK are installed, and create a new Android project. Next, load and run the model by converting the trained PyTorch model to TorchScript format and loading it. Then, on the Android side, use code to load and run the model. Although PyTorch Android Lite offers cross-platform support and a simplified API, it still faces three core challenges in real-world industrial deployments, especially on resource-constrained mobile devices: 1) Excessive memory usage: Traditional PyTorch models rely on a complete runtime environment, including training components such as computation graph construction and automatic differentiation, which often results in memory usage exceeding 500MB on mobile devices, making it difficult to run stably on resource-constrained mobile devices.
[0005] 2) Poor hardware compatibility: Android devices suffer from severe hardware fragmentation, with diverse architectures for computing units such as CPU, GPU, and NPU. PyTorch Mobile's driver support for emerging dedicated accelerators (such as NPU and DSP) often lags behind industry standards (such as TensorFlow Lite), making it difficult to fully utilize the device's hardware acceleration potential and resulting in unstable performance.
[0006] 3) Excessive inference latency: Models that have not been deeply optimized have slow inference speeds on mobile CPUs, with single-frame processing latency often exceeding 200 milliseconds, which cannot meet the stringent requirements of real-time interactive applications (such as real-time video processing that requires latency ≤30ms / frame).
[0007] Therefore, a systematic optimization method is urgently needed to solve the aforementioned memory, heterogeneous hardware adaptation, and performance latency issues. Summary of the Invention
[0008] The purpose of this invention is to provide a lightweight inference optimization method and system for mobile devices based on PyTorch Android Lite, so as to solve the problems mentioned in the background art.
[0009] To achieve the above objectives, the present invention provides the following technical solution: a lightweight inference optimization method for mobile devices based on PyTorch Android Lite, comprising the following steps: 1) Model preparation stage: A pre-trained lightweight neural network model is obtained using the quantization-aware training method and converted into low-precision INT8 format; 2) Model conversion and optimization stage: The low-precision model is converted into a static graph representation using TorchScript, and the graph is optimized using the optimize_for_mobile tool, including stripping training nodes and performing operator fusion, and finally generating the optimized model file; 3) Runtime environment configuration stage: Integrate the PyTorch Lite Interpreter inference library into the Android application; 4) Model deployment and execution phase: Load the optimized model file into the PyTorch LiteInterpreter, perform inference calculations on the Android device and output the results.
[0010] Preferably, the model preparation stage includes: selecting a lightweight network architecture suitable for mobile devices as the base model; training the model using quantization-aware training technology to simulate the INT8 quantization effect during forward propagation, so that the model weights can adapt to low-precision calculations; and performing structured pruning on the trained model to remove redundant neuron connections and further compress the model size.
[0011] Preferably, the model conversion and optimization stage includes: converting the PyTorch model into a static computation graph in TorchScript format using the torch.jit.trace or torch.jit.script method; optimizing the TorchScript model by calling the torch.utils.mobile_optimizer.optimize_for_mobile function to remove training nodes that are irrelevant to inference, such as automatic differentiation and gradient calculation; and performing fusion optimization on continuous operators in the computation graph, fusing convolution, batch normalization, and activation function sequences into a single composite operator.
[0012] Preferably, the runtime environment configuration phase also includes: configuring PyTorch Lite Interpreter to use the Vulkan backend as a unified computing interface to utilize the GPU or NPU of the mobile device for hardware acceleration; and setting the number of inference threads according to the number of computing cores of the target device to achieve optimal allocation of computing resources.
[0013] Preferably, the model deployment and execution phase includes: establishing an asynchronous processing pipeline in the Android application to enable image acquisition, data preprocessing, model inference, and result post-processing operations to be executed in parallel; calling the Vulkan backend through PyTorch LiteInterpreter to perform asynchronous inference calculations on the input data; and outputting the inference results to the application for real-time video analysis, object detection, or image segmentation tasks.
[0014] A lightweight mobile inference optimization system based on PyTorch Android Lite, comprising: Model preparation module: Used to obtain a pre-trained lightweight neural network model using the quantization-aware training method and convert it into low-precision INT8 format; Model conversion and optimization module: This module converts low-precision models into static graph representations using TorchScript and optimizes the graph using the optimize_for_mobile tool, including stripping training nodes and performing operator fusion, ultimately generating the optimized model file. Runtime environment configuration module: Used to integrate the PyTorch Lite Interpreter inference library into Android applications; Model Deployment and Execution Module: This module loads the optimized model file into the PyTorch LiteInterpreter, performs inference calculations on the Android device, and outputs the results.
[0015] Preferably, the model preparation module includes: Base model selection unit: Used to select a lightweight network architecture suitable for mobile devices as the base model; Quantization-aware training unit: Used to train the model using quantization-aware training technology, simulating the INT8 quantization effect during forward propagation, so that the model weights can adapt to low-precision calculations. Structured pruning unit: Used to perform structured pruning on the trained model to remove redundant neuron connections and further compress the model size.
[0016] Preferably, the model conversion and optimization module includes: Static Graph Conversion Unit: Used to convert a PyTorch model into a static computation graph in TorchScript format using the torch.jit.trace or torch.jit.script methods; Training node stripping unit: Used to call the torch.utils.mobile_optimizer.optimize_for_mobile function to optimize the TorchScript model by stripping training nodes that are not related to inference, such as automatic differentiation and gradient calculation; Operator fusion optimization unit: used to fuse and optimize continuous operators in the computation graph, fusion convolution, batch normalization and activation function sequences into a single composite operator.
[0017] Preferably, the runtime environment configuration module also includes: Hardware Acceleration Configuration Unit: Used to configure PyTorch Lite Interpreter to use the Vulkan backend as a unified computing interface to leverage the GPU or NPU of mobile devices for hardware acceleration; Thread Count Setting Unit: Used to set the number of inference threads based on the number of computing cores of the target device to achieve optimal allocation of computing resources.
[0018] Preferably, the model deployment and execution module includes: Asynchronous processing pipeline establishment unit: used to establish an asynchronous processing pipeline in an Android application, enabling image acquisition, data preprocessing, model inference, and result postprocessing operations to be executed in parallel; Asynchronous inference computation unit: Used to call the Vulkan backend through PyTorch Lite Interpreter to perform asynchronous inference computation on the input data; Result output unit: Used to output inference results to applications for real-time video analysis, object detection, or image segmentation tasks.
[0019] Compared with the prior art, the beneficial effects of the present invention are: This invention proposes a lightweight inference optimization method and system for mobile devices based on PyTorch Android Lite. By separating training and inference and using the Lite Interpreter, the runtime memory usage is reduced from over 500MB to less than 50MB, enabling complex models to run stably on mid-to-low-end mobile phones. Utilizing Vulkan as an abstraction layer effectively unifies the calls to GPUs / NPUs from different manufacturers, overcoming the problem of insufficient support for new hardware in PyTorch Mobile and achieving better performance portability. Through end-to-end optimization of "model quantization + operator fusion + hardware acceleration," on mainstream Android devices, for typical visual tasks (such as object detection), the single inference latency can be stably maintained at 10-30 milliseconds, fully meeting the requirements of real-time video processing (≥30FPS). Attached Figure Description
[0020] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation
[0021] To make the objectives, technical solutions, and advantages of the present invention clear and complete, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only some, not all, embodiments of the present invention, and are merely illustrative of the embodiments of the present invention. They are not intended to limit the embodiments of the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0022] Example 1: This invention provides a technical solution: a lightweight inference optimization method for mobile devices based on PyTorch Android Lite, comprising the following steps: 1) Model preparation stage: A pre-trained lightweight neural network model is obtained using quantization-aware training method and converted into low-precision INT8 format; including: selecting a lightweight network architecture suitable for mobile devices as the base model; training the model using quantization-aware training technology, simulating the INT8 quantization effect during forward propagation to adapt the model weights to low-precision calculation; and performing structured pruning on the trained model to remove redundant neuron connections and further compress the model size.
[0023] 2) Model Conversion and Optimization Stage: The low-precision model is converted into a static graph representation using TorchScript, and graph optimization is performed using the optimize_for_mobile tool, including stripping training nodes and fusing operators, ultimately generating the optimized model file. This includes: using the torch.jit.trace or torch.jit.script method to convert the PyTorch model into a static computation graph in TorchScript format; calling the torch.utils.mobile_optimizer.optimize_for_mobile function to optimize the TorchScript model, stripping training nodes that are irrelevant to inference, such as automatic differentiation and gradient calculation; and fusing and optimizing continuous operators in the computation graph, fusing convolution, batch normalization, and activation function sequences into a single composite operator.
[0024] 3) Runtime environment configuration stage: Integrate the PyTorch Lite Interpreter inference library into the Android application; including: configuring PyTorch Lite Interpreter to use the Vulkan backend as a unified computing interface to utilize the GPU or NPU of the mobile device for hardware acceleration; setting the number of inference threads according to the number of computing cores of the target device to achieve optimal allocation of computing resources.
[0025] 4) Model Deployment and Execution Phase: The optimized model file is loaded into the PyTorch LiteInterpreter, inference calculations are performed on the Android device, and the results are output. This includes: establishing an asynchronous processing pipeline in the Android application to enable image acquisition, data preprocessing, model inference, and result post-processing operations to be executed in parallel; calling the Vulkan backend through PyTorch Lite Interpreter to perform asynchronous inference calculations on the input data; and outputting the inference results to the application for real-time video analysis, object detection, or image segmentation tasks.
[0026] Example 2, based on Example 1, proposes a lightweight mobile inference optimization system based on PyTorch Android Lite, which includes the following three core optimization levels: ① Model Layer Optimization: Lightweight model generation techniques with training-inference separation. This layer aims to reduce model size and computational cost from the source and optimize model structure for deployment.
[0027] Lightweight network architecture selection and training: Prioritize and train lightweight network architectures designed specifically for mobile devices, such as MobileNetV3 and ShuffleNetV2.
[0028] Model quantization: Quantization-aware training technology is employed to simulate INT8 quantization during the model training feedforward process. This allows the model to adapt to low-precision computation during the training phase, thereby maximizing the preservation of accuracy when deriving the INT8 model. This is a crucial step in reducing model size and computational latency.
[0029] Structured pruning: After training, the model is pruned through channels to remove redundant neuron connections and further compress the model.
[0030] ② Transformation Layer Optimization: Deployment-oriented static graph optimization and operator fusion techniques. This layer is responsible for efficiently converting the trained model into a format suitable for mobile execution.
[0031] TorchScript Conversion and Optimization: Use `torch.jit.trace` or `torch.jit.script` to convert a PyTorch model into a static graph structure in TorchScript. Then, call the `torch.utils.mobile_optimizer.optimize_for_mobile` function, which strips training nodes that are irrelevant to inference (such as automatic differentiation and gradient calculation).
[0032] Operator fusion: During the model export stage, PyTorch's graph optimization capabilities are used to fuse common sequential operations (such as Conv2D + BatchNorm + ReLU) into a single composite operator, significantly reducing the number of kernel calls and the memory read / write overhead of intermediate tensors.
[0033] ③ Runtime layer optimization: Heterogeneous hardware acceleration and efficient inference engine scheduling technology. This layer focuses on the final execution efficiency of the model on mobile devices.
[0034] Lightweight inference runtime: When deployed, it does not depend on the full LibTorch library, but integrates PyTorch Lite Interpreter which only contains core operators. This runtime removes Python dependencies and training components and is only a few MB in size.
[0035] Unified acceleration via Vulkan backend: Specify Vulkan as the computation backend when exporting models. Vulkan is a low-overhead, cross-platform graphics and computation API that effectively utilizes the parallel computing capabilities of the GPU and part of the NPU in most Android devices, solving hardware heterogeneity issues.
[0036] Asynchronous pipeline scheduling: At the application level, an asynchronous pipeline of "image acquisition → preprocessing → model inference → postprocessing" is constructed, so that the inference process overlaps with the preprocessing and postprocessing processes, thereby masking single-frame latency, improving overall throughput, and meeting real-time requirements.
[0037] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A lightweight inference optimization method for mobile devices based on PyTorch Android Lite, characterized in that: Includes the following steps: 1) Model preparation stage: A pre-trained lightweight neural network model is obtained using the quantization-aware training method and converted into low-precision INT8 format; 2) Model conversion and optimization stage: The low-precision model is converted into a static graph representation using TorchScript, and the graph is optimized using the optimize_for_mobile tool, including stripping training nodes and performing operator fusion, and finally generating the optimized model file; 3) Runtime environment configuration stage: Integrate the PyTorch Lite Interpreter inference library into the Android application; 4) Model deployment and execution phase: Load the optimized model file into the PyTorch Lite Interpreter, perform inference calculations on the Android device and output the results.
2. The lightweight inference optimization method for mobile devices based on PyTorch Android Lite according to claim 1, characterized in that: The model preparation phase includes: selecting a lightweight network architecture suitable for mobile devices as the base model; training the model using quantization-aware training techniques to simulate the INT8 quantization effect during forward propagation, making the model weights adapt to low-precision calculations; and performing structured pruning on the trained model to remove redundant neuron connections and further compress the model size.
3. The lightweight inference optimization method for mobile devices based on PyTorch Android Lite according to claim 2, characterized in that: The model conversion and optimization phase includes: converting the PyTorch model into a static computation graph in TorchScript format using the torch.jit.trace or torch.jit.script methods; optimizing the TorchScript model by calling the torch.utils.mobile_optimizer.optimize_for_mobile function to remove training nodes that are irrelevant to inference, such as automatic differentiation and gradient calculation; and fusing and optimizing continuous operators in the computation graph, merging convolution, batch normalization, and activation function sequences into a single composite operator.
4. The lightweight inference optimization method for mobile devices based on PyTorch Android Lite according to claim 3, characterized in that: The runtime environment configuration phase also includes: configuring PyTorch Lite Interpreter to use the Vulkan backend as a unified computing interface to leverage the GPU or NPU of mobile devices for hardware acceleration; and setting the number of inference threads based on the number of computing cores on the target device to achieve optimal allocation of computing resources.
5. The lightweight inference optimization method for mobile devices based on PyTorch Android Lite according to claim 4, characterized in that: The model deployment and execution phase includes: establishing an asynchronous processing pipeline in the Android application to enable image acquisition, data preprocessing, model inference, and result post-processing operations to be executed in parallel; calling the Vulkan backend through PyTorch LiteInterpreter to perform asynchronous inference calculations on the input data; and outputting the inference results to the application for real-time video analysis, object detection, or image segmentation tasks.
6. A lightweight mobile inference optimization system based on PyTorch Android Lite, applied to the method described in claim 5, characterized in that: include: Model preparation module: Used to obtain a pre-trained lightweight neural network model using the quantization-aware training method and convert it into low-precision INT8 format; Model conversion and optimization module: This module converts low-precision models into static graph representations using TorchScript and optimizes the graph using the optimize_for_mobile tool, including stripping training nodes and performing operator fusion, ultimately generating the optimized model file. Runtime environment configuration module: Used to integrate the PyTorch Lite Interpreter inference library into Android applications; Model Deployment and Execution Module: This module loads the optimized model file into the PyTorch LiteInterpreter, performs inference calculations on the Android device, and outputs the results.
7. A lightweight mobile inference optimization system based on PyTorch Android Lite according to claim 6, characterized in that: The model preparation module includes: Base model selection unit: Used to select a lightweight network architecture suitable for mobile devices as the base model; Quantization-aware training unit: Used to train the model using quantization-aware training technology, simulating the INT8 quantization effect during forward propagation, so that the model weights can adapt to low-precision calculations. Structured pruning unit: Used to perform structured pruning on the trained model to remove redundant neuron connections and further compress the model size.
8. The lightweight mobile inference optimization system based on PyTorch Android Lite according to claim 7, characterized in that: The model conversion and optimization module includes: Static Graph Conversion Unit: Used to convert a PyTorch model into a static computation graph in TorchScript format using the torch.jit.trace or torch.jit.script methods; Training node stripping unit: Used to call the torch.utils.mobile_optimizer.optimize_for_mobile function to optimize the TorchScript model by stripping training nodes that are not related to inference, such as automatic differentiation and gradient calculation; Operator fusion optimization unit: used to fuse and optimize continuous operators in the computation graph, fusion convolution, batch normalization and activation function sequences into a single composite operator.
9. A lightweight mobile inference optimization system based on PyTorch Android Lite according to claim 8, characterized in that: The runtime environment configuration module also includes: Hardware Acceleration Configuration Unit: Used to configure PyTorch Lite Interpreter to use the Vulkan backend as a unified computing interface to leverage the GPU or NPU of mobile devices for hardware acceleration; Thread Count Setting Unit: Used to set the number of inference threads based on the number of computing cores of the target device to achieve optimal allocation of computing resources.
10. A lightweight mobile inference optimization system based on PyTorch Android Lite according to claim 9, characterized in that: The model deployment and execution module includes: Asynchronous processing pipeline establishment unit: used to establish an asynchronous processing pipeline in an Android application, enabling image acquisition, data preprocessing, model inference, and result postprocessing operations to be executed in parallel; Asynchronous inference computation unit: Used to call the Vulkan backend through PyTorch Lite Interpreter to perform asynchronous inference computation on the input data; Result output unit: Used to output inference results to applications for real-time video analysis, object detection, or image segmentation tasks.