Visual benchmark model reasoning framework deployed by embedded terminal
By implementing model lightweighting, cross-format conversion, and hardware acceleration, the deployment compatibility and speed issues of the visual language model on domestic AI processors have been resolved, enabling efficient deployment and real-time processing on domestic AI processors.
Patent Information
- Application Number
- CN202511284652.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-10
- Publication Date
- 2025-12-23
AI Technical Summary
Traditional visual-language benchmark models are difficult to deploy on resource-constrained embedded devices, and existing model conversion tools have poor compatibility with domestic AI processors, resulting in slow inference speed, excessive loss of accuracy, and inability to meet real-time requirements.
The model employs a lightweight model module, a cross-format conversion module, a data preprocessing module, and an inference acceleration module. By pruning and quantization, the model parameter size and computational complexity are reduced. Format conversion and hardware acceleration are performed for domestic AI processors to ensure that the accuracy loss is within a preset threshold.
It has enabled the efficient deployment of the visual language model on domestic AI processors, improved inference speed, met the real-time processing requirements of edge devices, and ensured that the accuracy loss is within 2%.
Smart Images
Figure CN121189481A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application discloses an embedded end deployment visual reference model inference framework and belongs to the technical field of information. BACKGROUND
[0002] Traditional visual reference models (such as Transformer-based cross-modal positioning models) usually rely on high-performance GPUs for inference, have large model parameter sizes (billions of parameters) and high computational complexity, and are difficult to be directly deployed on resource-limited embedded devices. In addition, existing model conversion tools have poor compatibility with domestic AI processors (such as Huawei Ascend series), and there are problems such as format mismatch and excessive loss of inference accuracy, so a unique design is needed for this scenario to solve this problem.
[0003] Before the present application, when deploying a PyTorch model to an Atlas series development board, an ONNX intermediate format conversion is needed, but the traditional conversion process is not optimized for the multi-input characteristics (image + text) of visual models, and dimension mismatch errors are prone to occur; at the same time, models that have not been subjected to lightweight processing have slow inference speed on the embedded end (single frame processing time > 500 ms), which cannot meet the real-time requirements. SUMMARY
[0004] PURPOSE OF THE INVENTION
[0005] The purpose of the present application is to provide an embedded end deployment visual reference model inference framework to solve the above problems.
[0006] TECHNICAL SCHEME
[0007] To achieve the above purpose, the present application adopts the following technical scheme:
[0008] An embedded end deployment visual reference model inference framework comprises a model lightweight module, a cross-format conversion module, a data preprocessing module, an inference acceleration module and a result verification module. The model lightweight module is used to prune and quantize the original visual reference model, reducing the model parameter size and computational complexity; the cross-format conversion module is used to convert the lightweighted PyTorch model (.pth) into ONNX format, and then convert the ONNX format into OM format suitable for domestic AI processors; the data preprocessing module is used to standardize the input visual data (image) and text data, including size adjustment, channel conversion and normalization; the inference acceleration module is used to optimize the model inference path and support multi-thread parallel computing; the result verification module is used to compare the inference results of the converted model and the original model, ensuring that the accuracy loss is within the preset threshold.
[0009] Step one, the model lightweight module adopts INT8 quantization strategy to convert the model weight from 32-bit floating point to 8-bit integer, while preserving the floating point calculation precision of the key layers of the model.
[0010] Step two, the cross-format conversion module includes an ONNX conversion unit and an OM conversion unit. The ONNX conversion unit converts the PyTorch model to ONNX format through the pth2onnx script, where the input dimension is set to [batch_size, 3, 640, 640] (visual data) and [batch_size, 40] (text data); the OM conversion unit uses the ATC tool to execute the conversion command, and the parameters include specifying the ONNX file path, declaring the framework type, defining the input dimension, and adapting to domestic processors.
[0011] Step three, the conversion command of the ATC tool is:
[0012] atc --model=xxx.onnx --framework=5 --input_format=NCHW --input_shape="img:1,3,640,640;word_id:1,40" --output=xxx --soc_version=Ascend310B1
[0013] Where the quotes in the input parameters are in English format to avoid parsing errors.
[0014] Step four, the data preprocessing module performs size adjustment, channel conversion, and normalization on visual data. Size adjustment adjusts the image to 640x640 through bilinear interpolation; channel conversion is used to convert from HWC format to CHW format; normalization uses the mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225] for standardization.
[0015] Step five, the domestic development board is Atlas 200I A2, which integrates Ascend 310B processor and supports 20TOPS (INT8) computing power, and the inference acceleration module realizes hardware acceleration by calling the ACL interface of the CANN tool chain.
[0016] Step six, the result verification module calculates the IoU (target detection) or cosine similarity (feature matching) of the inference result to ensure that the accuracy difference between the converted model and the original model is ≤2%.
[0017] In "step one", the "model weight is converted from 32-bit floating point to 8-bit integer by using INT8 quantization strategy, while the floating point calculation precision of the key layer of the model is preserved", the method is as follows: using the pruning and quantization combined strategy, removing the redundant attention head in the model, reducing the calculation amount, and using the torch.quantization tool of PyTorch to convert the model weight from FP32 to INT8, reducing the memory occupation.
[0018] In "step two", the "cross-format conversion module", the method is as follows: PyTorch→ONNX: write a special conversion script (pth2onnx.py), define the input dimensions of the image (3×640×640) and text (1×40) for the multi-input characteristics of the visual language model, and ensure that the ONNX model retains the multi-input node; ONNX→OM: call the Huawei ATC tool, specify the Ascend 310B processor parameters, match the multi-input dimensions through the --input_shape parameter, and avoid format errors.
[0019] In "step four", the "data preprocessing module performs size adjustment, channel conversion and normalization on visual data", the method is as follows: size adjustment adjusts the image to 640×640 through bilinear interpolation; channel conversion is used to convert from HWC format to CHW format; normalization uses the mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225] for standardization.
[0020] In "step five", the "hardware acceleration is realized by calling the ACL interface of the CANN tool chain", the method is as follows: based on the Atlas 200I A2 development board, which integrates the Ascend 310B processor and supports 20TOPS computing power, meeting the real-time inference requirements of the visual language model; through the ACL interface of the CANN tool chain, the hardware acceleration unit is called, and the text feature extraction task is allocated to the NPU, and the image feature extraction task is allocated to the DSP, realizing parallel computing.
[0021] In "step six", the "IoU (target detection) or cosine similarity (feature matching) of the inference result is calculated to ensure that the precision difference between the converted model and the original model is ≤2%", the method is as follows: using the standard data set to compare the performance of the model before and after conversion, calculating the AP0.5 index on the target detection task, ensuring that the difference is ≤2%; on the inference speed, the single frame processing time on the board is <200ms.
[0022] The positive effects and advantages of the present application are:
[0023] The application provides an embedded end deployment visual language benchmark model inference framework, supports visual language model multi-input conversion, solves the ONNX to OM format mismatch problem, improves inference speed, and meets the real-time processing requirements of the edge end. At the same time, the application is optimized for Ascend processors, fills the technical gap of domestic development board visual language model deployment, realizes efficient deployment of visual language benchmark models on the Atlas 200I A2 development board, meets the accuracy and real-time requirements, and can be directly applied to edge end intelligent reconnaissance, target positioning and other scenes. BRIEF DESCRIPTION OF DRAWINGS
[0024] Figure 1 Overall diagram of the embedded end deployment visual language benchmark model inference framework
[0025] Figure 2 Model lightening module schematic diagram based on pruning and quantization strategy combination
[0026] Figure 3 Cross-format conversion module schematic diagram based on ONNX conversion unit and OM conversion unit
[0027] Figure 4 Data preprocessing module schematic diagram based on size adjustment, channel conversion and normalization
[0028] Figure 5 Inference acceleration module schematic diagram based on the Atlas 200I A2 development board
[0029] Figure 6 Result verification module schematic diagram DETAILED DESCRIPTION
[0030] In order to make the purpose, technical scheme and advantages of the application clearer and more understandable, the application will be further described in detail below in combination with examples. It should be understood that the specific examples described herein are only used to explain the application and not to limit the application.
[0031] Cooperation Figure 1As shown, the application discloses an embedded end deployment visual reference model inference framework, which comprises a model lightweight module, a cross-format conversion module, a data preprocessing module, an inference acceleration module and a result verification module. The model lightweight module is used for pruning and quantization processing of the original visual reference model, reducing the model parameter size and the calculation complexity. The cross-format conversion module is used for sequentially converting the lightweight PyTorch model (.pth) into the ONNX format, and then converting the ONNX format into the OM format suitable for the domestic AI processor. The data preprocessing module is used for standardizing the input visual data (image) and text data, including size adjustment, channel conversion and normalization. The inference acceleration module is used for optimizing the model inference path and supporting multi-thread parallel computing. The result verification module is used for comparing the inference results of the converted model and the original model, and ensuring that the accuracy loss is within the preset threshold.
[0032] As shown, Figure 2 The model lightweight module is used for pruning and quantization processing of the original visual reference model, reducing the model parameter size and the calculation complexity, and the steps are as follows: first, redundant structure pruning is performed. Based on the attention mechanism characteristics of the visual model, the contribution degree of each attention head is analyzed, redundant attention heads are identified and removed, heads with a contribution degree lower than a threshold are removed, invalid calculation branches are removed, and the calculation complexity is reduced without significantly affecting the model feature extraction capability. Then, mixed precision quantization is performed on the model. An INT8 quantization strategy is adopted, the torch.quantization tool of PyTorch is used to convert the 32-bit floating-point weights of the non-key layers of the model into 8-bit integers, the floating-point values are compressed to the integer range through linear mapping, the memory occupation is greatly reduced, and the floating-point calculation precision of the key layers (such as the cross-modal fusion layer) of the model is retained, avoiding damage to the core feature extraction capability and balancing the lightweight and precision requirements.
[0033] As shown, Figure 3As shown, the cross-format conversion module is used to convert the lightweight PyTorch model (.pth) into ONNX format in sequence, and then convert the ONNX format into OM format suitable for domestic AI processors: first, the conversion of the PyTorch model (.pth) into ONNX format is realized by writing the pth2onnx.py script, and for the multi-input characteristics of the visual language model "image + text", the image input dimension [batch_size, 3, 640, 640] and the text input dimension [batch_size, 40] are explicitly defined during conversion, and the torch.onnx.export function is used to simulate input data (image + text), ensuring that the ONNX model retains the img and word_id multi-input nodes, solving the problem of dimension mismatch in traditional conversion; then, the Huawei ATC tool is called to convert the ONNX format into OM format suitable for domestic Ascend 310B processor, through the atc--model=xxx.onnx--soc_version=Ascend310B1 command, the hardware parameters of the Ascend 310B processor are specified, and through the --input_shape parameter, the input dimensions of the image (img: 1, 3, 640, 640) and the text (word_id: 1, 40) are accurately matched, so that the model format is compatible with the instruction set and storage structure of the Ascend processor, realizing the deep adaptation of the model to the domestic AI processor and solving the problem of format mismatch.
[0034] As shown in Figure 4 The data preprocessing module is used to standardize the input visual data (image) and text data, including size adjustment, channel conversion and normalization, and the steps are as follows: first, the original image is adjusted to 640x640 by using the bilinear interpolation algorithm, the scaling is realized by weighted average of surrounding pixel values, and the image aspect ratio is maintained by edge 0, ensuring the accuracy of image spatial feature recognition by the subsequent model; then, the image is converted from HWC format (height x width x channel) to CHW format (channel x height x width) through transpose(2, 0, 1) operation, which adapts to the input data layout requirements of Ascend 310B processor, and facilitates more efficient parallel computing based on CHW format tensor of the processor; finally, the converted image tensor is applied to the formula (x-mean) / std (where mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]), which eliminates the scale difference of image pixel values, makes the input data distribution consistent with the data set distribution during model training, avoids the influence of data scale fluctuation on model inference accuracy, and ensures the stability of cross-modal feature fusion.
[0035] As shown in Figure 5As shown, the inference acceleration module is used to optimize the model inference path and supports multi-threaded parallel computing. The steps are as follows: Based on the Atlas 200I A2 development board, the hardware acceleration unit is called through the ACL (Ascend Computing Language) interface of the CANN toolchain. Utilizing the heterogeneous computing capabilities of the Ascend 310B, the text feature extraction task is assigned to the NPU, and the image feature extraction task is assigned to the DSP, thereby achieving specialized division of labor and improving computing efficiency.
[0036] like Figure 6 As shown, the result verification module is used to compare the inference results of the converted model with those of the original model to ensure that the accuracy loss is within a preset threshold. The steps are as follows: Using a standard dataset, inference is performed using both the original model and the converted model. For the object detection task, IoU is calculated, and for the feature matching task, cosine similarity is calculated. The AP0.5 index is also calculated to ensure that the accuracy difference between the converted model and the original model is ≤2%, verifying that lightweighting and format conversion have not significantly affected the core functions of the model. The single-frame processing time of the converted model is tested on the Atlas 200I A2 development board. By comparing the actual inference speed with the preset threshold, the effectiveness of the inference acceleration module is verified, ensuring that the model can still meet the real-time response requirements of the business in a resource-constrained environment.
Claims
1. An embedded-device-deployed vision-language benchmark model inference framework, characterized in that: The system includes a model lightweighting module, a cross-format conversion module, a data preprocessing module, an inference acceleration module, and a result verification module. The model lightweighting module prunes and quantizes the original visual-text baseline model to reduce the model parameter size and computational complexity. The cross-format conversion module sequentially converts the lightweighted PyTorch model (.pth) to ONNX format, and then converts the ONNX format to OM format suitable for domestic AI processors. The data preprocessing module standardizes the input visual (image) and text data, including resizing, channel conversion, and normalization. The inference acceleration module optimizes the model inference path and supports multi-threaded parallel computing. The result verification module compares the inference results of the converted model with the original model to ensure that accuracy loss is within a preset threshold.
2. The embedded-deployed vision-language baseline model inference framework as described in claim 1, characterized in that: The model lightweighting module adopts the INT8 quantization strategy to convert the model weights from 32-bit floating-point type to 8-bit integer type, while retaining the floating-point calculation precision of the key layers of the model.
3. The embedded-deployed vision-language baseline model inference framework as described in claim 1, characterized in that: The cross-format conversion module includes an ONNX conversion unit and an OM conversion unit. The ONNX conversion unit converts the PyTorch model to ONNX format using the pth2onnx script, where the input dimensions are set to [batch_size,3,640,640] (visual data) and [batch_size,40] (text data). The OM conversion unit uses the ATC tool to execute conversion commands, with parameters including specifying the ONNX file path, declaring the frame type, defining the input dimensions, and adapting to domestic processors.
4. The embedded-deployed visual-language benchmark model inference framework as described in claim 1, characterized in that: The conversion commands for the ATC tool are as follows: atc--model=xxx.onnx--framework=5--input_format=NCHW--input_shape="img:1,3,640,640;word_id:1,40"--output=xxx--soc_version=Ascend310B1 The quotation marks in the input parameters should be in English format to avoid parsing errors.
5. The embedded-deployed visual-language benchmark model inference framework as described in claim 1, characterized in that: The data preprocessing module performs resizing, channel conversion, and normalization on the visual data. Resizing adjusts the image to 640×640 using bilinear interpolation; channel conversion converts the image from HWC format to CHW format; normalization standardizes the data using the mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225].
6. The embedded-deployed vision-language baseline model inference framework as described in claim 1, characterized in that: The domestically developed board is the Atlas 200IA2, which integrates the Ascend 310B processor and supports 20 TOPS (INT8) computing power. The inference acceleration module achieves hardware acceleration by calling the ACL interface of the CANN toolchain.
7. The embedded-deployed vision-language baseline model inference framework as described in claim 1, characterized in that: The result verification module ensures that the accuracy difference between the converted model and the original model is ≤2% by calculating the IoU (object detection) or cosine similarity (feature matching) of the inference results.