A neural network edge deployment method based on heterogeneous computing partitioning, equivalent structure adaptation and dynamic precision correction

Abstract

The application discloses a neural network edge deployment method based on heterogeneous computing division, equivalent structure adaptation and dynamic precision correction, and belongs to the technical field of edge artificial intelligence, and comprises the following steps: jointly analyzing a model and hardware, determining a hardware execution type according to an operator type, and dividing a boundary; encapsulating NPU candidate nodes into a fixed input-output subgraph, and organizing CPU control nodes into a control module; exporting the subgraph and compiling; arranging effective features into a compact matrix, and padding to an inner product dimension with zero elements; collecting real intermediate tensors to form a phased calibration set; calculating a decision interval between the first two candidate scores, and performing CPU floating point rechecking when the interval is less than or equal to a threshold value; connecting the NPU and the CPU by using a bounded queue, and limiting thread affinity; implementing a multi-level precision access control, and falling back to the last version that passes all access controls when the implementation fails. The application can reduce the risks of operator rollback, cross-processor data replication and quantization decision flip, and improve the resource utilization efficiency of edge devices.

Description

Technical Field

[0001] This invention belongs to the field of edge artificial intelligence technology, specifically relating to a neural network edge deployment method based on heterogeneous computing partitioning, equivalent structure adaptation, and dynamic precision correction. Background Technology

[0002] As edge AI devices gradually integrate NPUs (Neural Processing Units), CPUs (Central Processing Units), DSPs (Digital Signal Processors), and image processing units, deep learning models have the ability to perform local inference on low-power platforms. However, models in training frameworks typically include dynamic shapes, dynamic indexes, reductions, sorting, conditional branching, and irregular matrix operations. Moreover, the target NPU only supports fixed shapes and a finite set of operators. If the complete model is directly exported, compilation failures, implicit CPU callbacks, large-scale intermediate tensors, and quantization errors that are difficult to locate can easily occur.

[0003] Existing deployment methods typically employ two paths: one is to force quantization of the entire network and have it compiled by the NPU, with the runtime library falling back to the CPU when unsupported operators are encountered; the other is for engineers to manually split the model, allocating the compilable parts to the NPU and keeping the rest on the CPU. In the first path, the callback location and overhead are uncontrollable, making it difficult to prove whether the neural network is truly running on the NPU; the second path often only focuses on whether it can run, without simultaneously considering the splitting boundaries, cross-processor data volume, critical quantization decisions, and board-level precision gating, which can easily lead to speed reduction or output changes.

[0004] Furthermore, certain local correlations or matrix calculations rely on repeated tensor expansions within the training framework. Generating NPU inputs directly according to the original data layout would result in tens or even hundreds of times more data writes and memory overhead. Non-standard channel dimensions may also exceed the support range of NPU matrix units. Using only a small number of original images for quantization calibration cannot cover the intermediate feature distribution generated by the preceding network, potentially leading to dynamic range truncation, candidate sorting reversal, and errors in the final business output.

[0005] Therefore, a deployment optimization method is needed that unifies model diagram partitioning, fixed shape export, NPU compilation, compact data organization, quantization calibration, selective precise verification, heterogeneous pipeline scheduling, and stepwise precision verification, so that the model can obtain verifiable, retestable, and rollback-capable board-side running results under the constraints of limited CPU cores, memory, and accelerator interface. Summary of the Invention

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0007] A neural network edge deployment method based on heterogeneous computation partitioning, equivalent structure adaptation, and dynamic precision correction includes:

[0008] S1: Jointly analyze the model to be deployed and the target hardware, traverse the model computation graph, record the operator type, input / output shape, data type, constant parameters, dynamic dimension, memory usage and dependencies of each node; read the NPU compiler support list and board resource information, determine the corresponding hardware execution type and partition boundary based on the operator type, input / output shape and dynamic characteristics of each node, and obtain the partitioning result of the computation graph.

[0009] S2, based on the division in step S1, encapsulate the continuous NPU candidate nodes into an NPU subgraph with fixed input and output, and freeze the batch, input size, candidate upper limit and number of channels; organize the CPU control nodes into CPU control modules according to the previous and next dependencies of the original computation graph.

[0010] S3 exports the NPU subgraph in the training framework as a fixed-shape intermediate model, which is then used by the NPU compiler to generate a quantized model and accelerator binary.

[0011] S4 directly arranges the effective features into a compact left matrix and a compact right matrix, and pads the compact left matrix with zero elements along the column direction to the inner product dimension, and pads the compact right matrix with zero elements along the row direction to the inner product dimension;

[0012] S5: Representative original samples are passed sequentially through the validated front-end NPU sub-graph to collect the real input, intermediate features and candidate regions of the back-end NPU sub-graph, forming a phased calibration set.

[0013] S6, calculate the decision interval between the scores of the top two candidates in the quantized output. When the decision interval is greater than the preset decision interval threshold, the NPU result is used directly. When the decision interval is less than or equal to the preset decision interval threshold, the candidate is subjected to CPU floating-point precise verification.

[0014] S7 uses a fixed-depth bounded queue to connect the NPU output and CPU post-processing, and restricts the entire program to a preset CPU core through thread affinity.

[0015] S8 implements multi-level precision access control checks. If any access control fails, it will revert to the most recent version that passed all access control checks.

[0016] The present invention has the following beneficial effects:

[0017] This invention, through heterogeneous computation partitioning, compact tensor organization and equivalent zero-filling, representative intermediate tensor quantization calibration and decision interval verification, as well as NPU-CPU bounded pipeline and multi-level precision gating, enables the NPU to handle rule-based, computationally intensive neural network subgraphs, while the CPU only handles dynamic control and a small amount of precision verification; it can eliminate redundant unrolling and reduce cross-processor memory traffic; it can reduce the risk of quantization flip through representative intermediate tensor calibration and decision interval guarding; and it can prove that the deployment results are actually running on the target accelerator through zero callback and multi-level gating, and retain a verifiable fallback path in case of version anomalies. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the neural network edge deployment method based on heterogeneous computation partitioning, equivalent structure adaptation, and dynamic precision correction of the present invention.

[0019] Figure 2 This is a schematic diagram of the fixed-shape computation graph partitioning and NPU and CPU task mapping of the present invention;

[0020] Figure 3 This is a schematic diagram of the compact tensor, equivalent zero-filling, quantization calibration, and critical result accurate verification of the present invention;

[0021] Figure 4 This is a schematic diagram of the bounded pipelined execution, multi-level precision access control, and rollback mechanism of the NPU-CPU of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0023] Deploying models on heterogeneous edge devices is not simply a matter of converting the trained model into a binary file. Dynamic control within the model, irregular memory access, quantization-sensitive nodes, and runtime callbacks all affect the final latency, memory usage, and business accuracy. This embodiment provides a complete implementation process from model analysis to interface delivery.

[0024] This embodiment uses a complex visual neural network that simultaneously incorporates convolutional backbones, feature interactions, global correlation matrices, local correlation calculations, and dynamic post-processing as an example. The target device includes an NPU (Neural Processing Unit) and a multi-core CPU (Central Processing Unit). The system can recognize four CPU cores, but the deployment program allows a maximum of two CPU cores to be used. The model input is fixed as a single-channel NCHW (batch, channel, height, width) format. Floating-point tensors. It should be noted that the model type and input dimensions are for illustrative purposes only and do not constitute a limitation on the scope of protection of this invention. For example... Figure 1 As shown, the neural network edge deployment method based on heterogeneous computation partitioning, equivalent structure adaptation, and dynamic precision correction of the present invention includes the following steps:

[0025] S1 performs joint analysis of the deployment model and the target hardware. Specifically, this includes:

[0026] S11, traverse the model computation graph, and record the operator type, input / output shape, data type, constant parameters, dynamic dimension, memory usage, and dependencies of each node.

[0027] S12 then reads the NPU compiler support list and board resource information, and determines the corresponding hardware execution type of each node based on its operator type, input / output shape, and dynamic characteristics. Convolution, fixed-shape matrix multiplication, normalization, activation, and regular attention are marked as NPU candidate nodes, while dynamic indexing, Top-Q (top Q maximum or minimum values), conditional branching, candidate number-related loops, reduction, cross-selection, and coordinate decoding are marked as CPU control nodes.

[0028] S13, calculate the number of intermediate tensor bytes and the estimated runtime for each candidate split boundary, and define the synthesis cost. :

[0029] ;

[0030] in, This indicates the total number of bytes that need to be transferred between the NPU and the CPU. Indicates the number of subgraph partition boundaries. Indicates the expected CPU hotspot time. This indicates the number of neural network callback risks. , , and They are set according to the target platform. , , , The weights are determined. The partition with the lower synthesis cost is selected while satisfying NPU memory and compilation constraints. Based on the selected partition, the partition boundary between NPU candidate nodes and CPU control nodes is determined, resulting in the partitioning of the computational graph.

[0031] S2 generates a fixed input / output NPU subgraph and a CPU control module. The CPU control module is formed by organizing the CPU control nodes obtained in S1 according to the dependencies of the original computation graph. It takes the output of the previous NPU subgraph as input, performs dynamic processing such as dynamic indexing, candidate filtering, reduction, mutual selection, loop control and result decoding, and passes the results to the subsequent NPU subgraph or post-processing stage.

[0032] Based on the S1 partitioning, consecutive NPU candidate nodes are encapsulated into NPU subgraphs with fixed input and output, and batch size, input size, candidate upper limit, and number of channels are frozen. For computations such as dynamic indexing, candidate filtering, reduction, mutual selection, loop control, and result decoding, which are divided into CPU control nodes, they are organized into CPU control modules according to the dependencies in the original computation graph. The CPU control modules take the output tensors or candidate information of the preceding NPU subgraphs as input, perform corresponding dynamic control and data organization processing, and output the processing results to the subsequent NPU subgraphs or the result post-processing stage. Each NPU subgraph generates an independent input / output list, recording the tensor name, NCHW or matrix layout, data type, number of elements, and number of bytes. Dynamic candidate selection remains with the CPU, avoiding the forced inclusion of unstable indexes and loops into the NPU graph.

[0033] In one embodiment, the complex visual network is divided into twelve unique NPU binary sub-models, comprising a single-image backbone, multiple feature interaction stages, a global scoring stage, a fine-grained feature stage, and a local principal correlation stage. All formal NPU binary sub-models have zero neural network callbacks; the CPU is used for dynamic reduction, compact data organization, critical value verification, and result decoding, and does not execute neural network operators that are rolled back by the compiler, such as... Figure 2 As shown.

[0034] S3 performs fixed-shape export, quantization, and compilation of each NPU subgraph.

[0035] The NPU subgraph in the training framework is exported as a fixed-shape intermediate model, which undergoes model structure checking, constant folding, operator fusion, tensor layout checking, and compiler optimization in sequence. The intermediate model is not included in the final on-board runtime package; it is only used as an exchange format for quantization compilation and numerical verification.

[0036] For each NPU subgraph, calibration and validation data are first generated using the training framework output. Then, the NPU compiler generates the quantized model and accelerator binary. After compilation, the model metadata is read to check the input / output dimensions, data types, and number of callbacks. Only when the compiler simulation output meets the tolerance with the reference output, the board can actually load and execute it, and the number of callbacks is zero, is the NPU subgraph included in the formal model set.

[0037] S4 employs compact tensor organization and equivalent zero-filling to reduce input flow.

[0038] The original model's local correlation calculations repeatedly expand the left and right local features into large matrices before feeding them into matrix multiplication. This method generates a large amount of writes on the CPU side and increases NPU input transmission. This embodiment no longer constructs expanded tensors, but directly arranges the effective features into a compact left matrix. and compact right matrix .

[0039] set up ;

[0040] in, Represents the real number field. This represents a compact left matrix formed by arranging the effective features on the left. This represents a compact right matrix formed by arranging the effective features on the right. This indicates the number of rows in the compact left matrix. This represents the effective feature dimension participating in matrix multiplication, i.e., the original inner product dimension of the matrix multiplication. This indicates the number of columns in a compact right matrix.

[0041] NPU matrix units require an inner product dimension of . and Then construct:

[0042] ;

[0043] ;

[0044] ;

[0045] in, This indicates that the left matrix will be compacted. Padding along the column direction with zero elements to the inner product dimension The equivalent left matrix obtained later This indicates that the right matrix will be compacted. Padged with zeros along the row direction to the inner product dimension. The resulting equivalent right matrix; and These represent zero matrices in their respective dimensions.

[0046] Therefore, zero-padding is only used to adapt to hardware dimensions and does not change the effective matrix multiplication. In one embodiment, the old path required writing two expanded inputs for each candidate, totaling approximately 6,553,600 bytes; with the compact matrix, each candidate input is 41,984 bytes, reducing the input throughput by approximately 156 times.

[0047] S5, construct a phased quantization calibration set based on real intermediate tensors.

[0048] If only a small amount of original input is used for calibration, the actual dynamic range of the subsequent local tensors may not be covered. In this embodiment, representative original samples are sequentially passed through the validated preceding NPU subgraph to collect the true input, intermediate features, and candidate regions of the subsequent NPU subgraph, forming a staged calibration set.

[0049] In one embodiment, the post-correlated NPU subgraph is recalibrated using all 1,730 local tensors generated from two representative scenarios, and the maximum and minimum values, distribution tails, and saturation ratios of the inputs and outputs are checked. For channels that have undergone truncation or discrete selection changes, instead of simply relaxing the thresholds, the issue is addressed by adjusting the calibration data, channel calculation location, or quantization method.

[0050] S6 utilizes decision intervals to trigger selective high-precision review.

[0051] Normally, quantization error does not change the final selection for candidates far from the decision boundary. However, when the scores of the first and second candidates are close, quantization error may cause the ranking to be reversed. Let the scores of the first two candidates in the quantized output be... and Define decision interval :

[0052] ;

[0053] when When the decision interval exceeds the preset threshold, the NPU result is used directly; when... When the decision interval is less than or equal to a preset threshold, CPU floating-point precision verification is performed only on the key main channel, auxiliary channel, dot product, or normalized result of the candidate (triggering selective high-precision verification). In this way, high-precision calculation only covers a small number of critical candidates, without having to restore the entire subsequent calculation to a CPU floating-point path.

[0054] In this embodiment, the decision interval threshold for the main branch is 0.001, and the board-level accuracy verification has been completed accordingly. Two sets of representative data triggered eighteen and eighty-five precise selective high-precision reviews, respectively, and corrected six discrete choices affected by quantization. The final output is consistent with the high-precision reference gate. Figure 3 As shown.

[0055] S7, building a bounded pipeline between the NPU and the dual-core CPU.

[0056] During the runtime phase, input, output, compact matrix, and result buffers are pre-allocated, and a fixed-depth bounded queue connects the NPU output and CPU post-processing. After the NPU completes a local computation, it copies only the necessary output to the queue and immediately releases the accelerator output slot; two CPU worker threads execute in parallel selective high-precision verification, result filtering, and continuous numerical decoding triggered by S6.

[0057] Thread affinity restricts the entire program to a preset CPU core, preventing business applications from consuming all cores without prior declaration. The queue depth is determined based on the number of bytes output by the NPU and the CPU processing speed; too small a depth will cause NPU waiting, while too large a depth will increase peak memory usage. This embodiment uses a bounded queue with a depth of four, achieving an effective parallelism of approximately two CPU worker threads, close to that of two cores.

[0058] S8 implements multi-level precision access control, version rollback, and interface-based delivery.

[0059] Each model or runtime upgrade is performed sequentially with the following verifications: fixed shape and tensor byte count checks; output-by-output comparison of the training framework and fixed intermediate model; comparison of the compiler simulation model and reference output; NPU binary callback count checks; board-side discrete decision, continuous numerical, output order, and result hash comparisons; dual-core thread limits, model loading and releasing, end-to-end time consumption, and peak physical memory checks.

[0060] If any access control fails, that version is marked as a candidate failure version, will not be included in the production directory, and will revert to the most recent version that has passed all access control. The production delivery package records the hashes of the binary, model, and reference results to ensure that colleagues or the production environment can retest.

[0061] Furthermore, the validated model implementation is encapsulated into a stable C language application binary interface. The model files are kept in an independent directory, while input format conversion, camera access, and result post-processing are presented as publicly available business source code. The business side only needs a general-purpose cross-compiler to recompile the interface program, eliminating the need for training frameworks, model conversion tools, or NPU compilers, thereby avoiding further coupling between the model implementation and business code.

[0062] On a certain type of edge device integrating an NPU and CPU, after adopting the method of this invention, the formally deployed model consists of twelve NPU sub-models with zero neural network callbacks, and the program uses a maximum of two CPU cores. Compared with the benchmark version of the same precision, the end-to-end time of two representative inputs was reduced from 3318.040 ms and 4694.260 ms to 2009.450 ms and 3141.640 ms, respectively, representing improvements of 39.44% and 33.07%; the peak physical memory of the process was approximately 82.5 MiB. The number, order, continuous coordinates, and confidence of the deployed outputs were all determined through item-by-item and result hash gating, such as... Figure 4 As shown.

[0063] The above description is merely an embodiment of the present invention and does not limit the scope of the invention. Any equivalent structural or procedural transformations made based on the description and drawings of this invention, or direct or indirect applications in other related system fields, are similarly included within the protection scope of this invention. Contents not described in detail in this specification are prior art known to those skilled in the art.

Claims

1. A neural network edge deployment method based on heterogeneous computation partitioning, equivalent structure adaptation, and dynamic precision correction, characterized in that, include: S1 performs joint analysis of the model to be deployed and the target hardware, traverses the model computation graph, and records the operator type, input and output shape, data type, constant parameters, dynamic dimension, memory usage and dependencies of each node. Read the NPU compiler support list and board resource information, determine the corresponding hardware execution type and partition boundary based on the operator type, input / output shape and dynamic characteristics of each node, and obtain the partition result of the computation graph; S2, based on the division in step S1, encapsulate the continuous NPU candidate nodes into an NPU subgraph with fixed input and output, and freeze the batch, input size, candidate upper limit and number of channels; The CPU control nodes are organized into CPU control modules according to the dependencies of the original computation graph. S3 exports the NPU subgraph in the training framework as a fixed-shape intermediate model, which is then used by the NPU compiler to generate a quantized model and accelerator binary. S4 directly arranges the effective features into a compact left matrix and a compact right matrix, and pads the compact left matrix with zero elements along the column direction to the inner product dimension, and pads the compact right matrix with zero elements along the row direction to the inner product dimension; S5: Representative original samples are passed sequentially through the validated front-end NPU sub-graph to collect the real input, intermediate features and candidate regions of the back-end NPU sub-graph, forming a phased calibration set. S6, calculate the decision interval between the scores of the top two candidates in the quantized output. When the decision interval is greater than the preset decision interval threshold, the NPU result is used directly. When the decision interval is less than or equal to the preset decision interval threshold, the candidate is subjected to CPU floating-point precise verification. S7 uses a fixed-depth bounded queue to connect the NPU output and CPU post-processing, and restricts the entire program to a preset CPU core through thread affinity. S8 implements multi-level precision access control checks. If any access control fails, it will revert to the most recent version that passed all access control checks.

2. The neural network edge deployment method based on heterogeneous computing partitioning, equivalent structure adaptation, and dynamic precision correction according to claim 1, characterized in that, In S1, convolution, fixed-shape matrix multiplication, normalization, activation, and regular attention are marked as NPU candidate nodes, while dynamic indexing, top Q maximum or minimum values, conditional branching, candidate number-related loops, reduction, mutual selection, and coordinate decoding are marked as CPU control nodes.

3. The neural network edge deployment method based on heterogeneous computing partitioning, equivalent structure adaptation, and dynamic precision correction according to claim 1, characterized in that, In S1, the number of intermediate tensor bytes and the expected running time are calculated for each candidate split boundary, and a comprehensive cost function is defined. The comprehensive cost function is equal to the first weight multiplied by the total number of bytes to be transferred between the NPU and the CPU, plus the second weight multiplied by the number of subgraph split boundaries, plus the third weight multiplied by the expected CPU hotspot time, plus the fourth weight multiplied by the number of neural network callback risks. The split with the smaller comprehensive cost is selected under the condition of satisfying the NPU memory and compilation constraints.

4. The neural network edge deployment method based on heterogeneous computing partitioning, equivalent structure adaptation, and dynamic precision correction according to claim 1, characterized in that, In S3, for each NPU subgraph, calibration and validation data are first generated using the training framework output, and then the NPU compiler generates the quantization model and accelerator binary. After compilation, the model metadata is read to check the input and output dimensions, data types, and number of callbacks. Only when the compiler simulation output meets the tolerance with the reference output, the board can actually load and execute the code, and the number of callbacks is zero, will the NPU subgraph enter the formal model set.

5. The neural network edge deployment method based on heterogeneous computing partitioning, equivalent structure adaptation, and dynamic precision correction according to claim 1, characterized in that, In S4, let the number of rows of the compact left matrix be m, the number of columns of the compact right matrix be n, the effective feature dimension for matrix multiplication be k, and the inner product dimension required by the NPU matrix unit be K, where K is greater than or equal to k. The equivalent left matrix is ​​obtained by padding the compact left matrix with zero elements along the column direction to the inner product dimension K, and the equivalent right matrix is ​​obtained by padding the compact right matrix with zero elements along the row direction to the inner product dimension K.

6. The neural network edge deployment method based on heterogeneous computing partitioning, equivalent structure adaptation, and dynamic precision correction according to claim 1, characterized in that, In S5, the post-correlated NPU subgraph is recalibrated using all local tensors generated from representative scenarios, and the maximum and minimum values, distribution tails, and saturation ratios of the input and output are checked. For channels that have undergone truncation or discrete selection changes, the issues are addressed by adjusting the calibration data, channel calculation location, or quantization method.

7. The neural network edge deployment method based on heterogeneous computing partitioning, equivalent structure adaptation, and dynamic precision correction according to claim 1, characterized in that, In S6, the scores of the first two candidates in the quantized output are set as the first score and the second score, and the decision interval is defined as the difference between the first score and the second score. When the decision interval is greater than the preset decision interval threshold, the NPU result is directly used. When the decision interval is less than or equal to the preset decision interval threshold, CPU floating-point precise verification is performed only on the key main channel, auxiliary channel, dot product or normalized result of the candidate.

8. The neural network edge deployment method based on heterogeneous computing partitioning, equivalent structure adaptation, and dynamic precision correction according to claim 1, characterized in that, In S7, after the NPU completes a local calculation, it copies only the necessary output to the queue and immediately releases the accelerator output slot; two CPU worker threads execute in parallel the selective high-precision verification, result filtering, and continuous numerical decoding triggered by step S6; the queue depth is determined according to the number of NPU output bytes and the CPU processing speed.

9. The neural network edge deployment method based on heterogeneous computing partitioning, equivalent structure adaptation, and dynamic precision correction according to claim 1, characterized in that, In S8, each model or runtime upgrade sequentially performs the following checks: fixed shape and tensor byte count checks, comparison of training framework and fixed intermediate model output by output, comparison of compiler simulation model and reference output, NPU binary callback count checks, board-side discrete decision and continuous numerical and output order and result hash comparison, dual-core thread limit and model loading and release, and end-to-end time consumption and peak physical memory checks.

10. The neural network edge deployment method based on heterogeneous computing partitioning, equivalent structure adaptation, and dynamic precision correction according to claim 1, characterized in that, In S8, the verified model execution implementation is encapsulated into a stable C language application binary interface, the model files are kept in an independent directory, and the input format conversion, camera access and result post-processing are made public business source code; The formal delivery package records the hashes of the binary, model, and reference results.

Images