End-side neural network inference methods, devices, computer equipment, and storage media
By compiling neural networks into static computation graphs and optimizing memory, a compact binary parameter file is generated, solving the problem that wearable devices are difficult to deploy AI models due to power consumption and cost limitations, and realizing efficient neural network inference on low-power devices.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHONGQING QI ALGORITHM TECHNOLOGY CO LTD
- Filing Date
- 2026-01-22
- Publication Date
- 2026-06-02
AI Technical Summary
Due to power consumption and cost constraints, current wearable devices cannot use chips equipped with neural network processing units (NPUs) as main controllers, making it difficult to support inference schemes based on dedicated AI accelerators.
By compiling the neural network computation process into a static computation graph, removing training framework dependencies, optimizing memory, generating a compact binary parameter file, and utilizing embedded operator library functions for neural network inference, the system architecture is simplified, and power consumption and cost are reduced.
It enables efficient neural network inference on low-power and low-cost wearable devices, reducing reliance on high-cost and high-power NPUs or SoCs and meeting the requirements of low power consumption, small size and high reliability for wearable devices.
Smart Images
Figure CN122133792A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer processing technology, and in particular to an edge-side neural network inference method, apparatus, computer device, and storage medium. Background Technology
[0002] Edge AI typically refers to the deployment and inference computation of deep learning models on mobile consumer electronics products. Current mainstream edge AI implementation solutions mostly use System-on-a-Chip (SoC) with parallel computing capabilities as the hardware platform, and utilize its integrated Neural Processing Unit (NPU) and other dedicated acceleration cores to efficiently execute neural network inference tasks.
[0003] However, due to power consumption and cost constraints, most wearable devices with integrated sensors do not currently use chips equipped with neural network processing units (NPUs) as the main controller, making it difficult to support inference schemes based on dedicated AI accelerators. Summary of the Invention
[0004] This application provides an edge-side neural network inference method, apparatus, computer device, and storage medium, aiming to solve the problem that most current wearable devices with integrated sensors do not use chips equipped with neural network processing units (NPUs) as main controllers due to power consumption and cost constraints, making it difficult to support inference schemes based on dedicated AI accelerators.
[0005] In a first aspect, embodiments of this application provide an edge-side neural network inference method, the method comprising the following steps: Obtain the neural network operation flow to be processed; The neural network operation process is compiled to obtain a computation graph; The computation graph is then subjected to memory optimization to obtain an optimized computation graph; Based on the optimized computation graph, a scheduling instruction sequence is generated, wherein the scheduling instruction sequence is used to call functions in the embedded operator library; Based on the optimized computational graph and target model file, a binary parameter file is obtained; Deploy the scheduling instruction sequence and the binary parameter file to the target end device; When the compiled computing program runs on the target device, the binary parameter file is loaded to obtain the target model parameters; Neural network inference is performed based on the scheduling instruction sequence, the embedded operator library, and the target model parameters.
[0006] A further technical solution involves performing memory optimization processing on the computation graph, including: In the computation graph, if there are two input matrices that are superimposed after performing matrix multiplication with the same weight matrix, and the intermediate results obtained by the matrix multiplication of the two input matrices are not independently referenced in subsequent calculations, then the two input matrices are pre-added to generate a merged input matrix, and a single matrix multiplication operation is performed between the merged input matrix and the weight matrix.
[0007] A further technical solution is that the binary parameter file is a binary parameter file, the header of the binary parameter file stores a preset encryption verification code and encrypted model hyperparameters, and the tail of the binary parameter file contains a verification code used to verify the integrity of the file.
[0008] A further technical solution is that the target model parameters are a parameter matrix, and the process of loading the binary parameter file to obtain the target model parameters includes: The decrypted model calculation process and model hyperparameters are used to generate a reconstructed calculation graph; The parameter matrices used by the nodes are read sequentially according to the reconstructed computation graph topology sequence.
[0009] A further technical solution is that compiling the neural network operation flow to obtain a computation graph includes: The neural network operation process is broken down to obtain multiple computing nodes, where each node corresponds to a PyTorch computing function; Based on the tensor data dependencies between nodes, and the input and output variable dimensions of each node, a computational graph is obtained.
[0010] A further technical solution is that the method further includes: Get the number of times the output result of each computing node is used; When the number of times a certain output result is used drops to zero, the memory space it occupies is immediately released.
[0011] A further technical solution is that the target end-side device is a microcontroller, which integrates a floating-point arithmetic unit. During the execution of neural network inference, the method further includes: When the matrix operation involved by the operator meets the preset scale condition, the FPU is called to perform floating-point calculation, and the input matrix and weight matrix are divided into multiple sub-matrices along the row or column dimension, so that the data volume of each sub-matrix does not exceed the capacity threshold of the microcontroller cache. The submatrix is subjected to block matrix multiplication and block accumulation operations to optimize parameter read and write efficiency by reducing the number of main memory accesses.
[0012] Secondly, embodiments of this application also provide an edge-side neural network inference device, which includes a unit for performing the above-described method.
[0013] Thirdly, embodiments of this application also provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.
[0014] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the above-described method.
[0015] This application provides an edge-side neural network inference method, apparatus, computer device, and storage medium. The method includes: acquiring a neural network computation flow to be processed; compiling the neural network computation flow to obtain a computation graph; performing memory optimization on the computation graph to obtain an optimized computation graph; generating a scheduling instruction sequence based on the optimized computation graph, wherein the scheduling instruction sequence is used to call functions in an embedded operator library; obtaining a binary parameter file based on the optimized computation graph and a target model file; deploying the scheduling instruction sequence and the binary parameter file to a target edge device; loading the binary parameter file during the execution of a compiled computation program on the target edge device to obtain target model parameters; and performing neural network inference on the target edge device based on the scheduling instruction sequence, the embedded operator library, and the target model parameters.
[0016] This application's embodiments compile the neural network computation flow into a static computation graph, removing the dependency on the training framework runtime and achieving platform-independent model representation. Memory optimization of the computation graph reduces peak inference memory usage, enabling the model to be deployed on a microcontroller with only tens of KB of RAM. The optimized computation graph is converted into a lightweight scheduling instruction sequence, avoiding the overhead of edge devices parsing complex structures or dynamic scheduling. A compact binary parameter file is generated based on the computation graph, with its parameter order strictly aligned with the scheduling instructions and supporting on-demand block loading to prevent memory overflow. The scheduling instruction sequence and binary parameter file can be directly burned into the Flash memory of the target edge device (e.g., a microcontroller), simplifying the system architecture and meeting the requirements of wearable devices for low power consumption, small size, and high reliability. Based on this, the target edge device can efficiently complete neural network inference by calling embedded operator library functions based on the scheduling instruction sequence and passing in the loaded model parameters, without relying on high-cost, high-power NPUs or SoCs, thus reducing power consumption and cost. Attached Figure Description
[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.
[0020] Figure 1 This is a flowchart illustrating a first embodiment of an edge-side neural network inference method provided in this application; Figure 2 A flowchart illustrating another embodiment of an edge-side neural network inference method provided in this application; Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation
[0021] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0022] The following disclosure provides numerous different embodiments or examples for implementing various structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.
[0023] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.
[0024] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.
[0025] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.
[0026] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."
[0027] To address the aforementioned issues, this application provides an edge-side neural network inference method that enables lightweight deployment by simplifying the neural network's computation process and optimizing operator implementation logic. This solves the problem that target edge devices, such as microcontrollers, struggle to efficiently deploy AI models due to resource constraints, and optimizes the neural network inference and computation process on microcontrollers used for parsing sensor time-series data.
[0028] See Figure 1 , Figure 1 This is a flowchart illustrating a first embodiment of an edge-side neural network inference method provided in this application. The method includes the following steps: Step 110: Obtain the neural network operation flow to be processed.
[0029] Step 120: Compile the neural network operation process to obtain a computation graph.
[0030] Step 130: Perform memory optimization on the computation graph to obtain an optimized computation graph.
[0031] Step 140: Based on the optimized computation graph, generate a scheduling instruction sequence, wherein the scheduling instruction sequence is used to call functions in the embedded operator library.
[0032] Step 150: Based on the optimized computational graph and target model file, obtain the binary parameter file.
[0033] Step 160: Deploy the scheduling instruction sequence and the binary parameter file to the target end device.
[0034] Step 170: When the compiled calculation program is running on the target device, the binary parameter file is loaded to obtain the target model parameters.
[0035] Step 180: The target end device performs neural network inference based on the scheduling instruction sequence, the embedded operator library, and the target model parameters.
[0036] This embodiment compiles the neural network computation process into a static computation graph, eliminating the dependency on the training framework runtime and achieving platform-independent model representation. Memory optimization of the computation graph reduces peak inference memory usage, enabling the model to be deployed on a microcontroller with only tens of KB of RAM. The optimized computation graph is converted into a lightweight scheduling instruction sequence, avoiding the overhead of edge devices parsing complex structures or dynamic scheduling. A compact binary parameter file is generated based on the computation graph, with its parameter order strictly aligned with the scheduling instructions and supporting on-demand block loading to prevent memory overflow. The scheduling instruction sequence and binary parameter file can be directly burned into the Flash memory of the target edge device (e.g., a microcontroller), simplifying the system architecture and meeting the requirements of wearable devices for low power consumption, small size, and high reliability. Based on this, the target edge device can efficiently complete neural network inference by calling embedded operator library functions based on the scheduling instruction sequence and passing in the loaded model parameters, without relying on high-cost, high-power NPUs or SoCs, thus reducing power consumption and cost.
[0037] Referring to a second embodiment of an edge-side neural network inference method provided in this application, it may specifically include the following steps: Step 210: Obtain the neural network operation flow to be processed.
[0038] The neural network operation process to be processed can be a PyTorch operation process.
[0039] Step 220: Compile the neural network operation process to obtain a computation graph.
[0040] In some embodiments, step 220, namely compiling the neural network computation flow to obtain a computation graph, includes: Step 221: Decompose the neural network operation process to obtain multiple computing nodes, where each node corresponds to a PyTorch computing function.
[0041] Step 222: Based on the tensor data dependencies between nodes, and the input and output variable dimensions of each node, a computation graph is obtained.
[0042] For example, for a PyTorch neural network computation process based on an ONNX model file, the computation process in the PyTorch file can be broken down. Each computation function (python__function) called within it is treated as a node in the computation graph. The dimension of the input variable (torch_tensor) vector received by the computation function when it is called in the PyTorch code is taken as the input dimension of that node, and the dimension of the output variable of the computation node is taken as the output dimension of that node.
[0043] Thus, by preprocessing the neural network operation process, the computing resources can be maximized.
[0044] In some embodiments, code libraries corresponding to different types of computing nodes can be implemented.
[0045] For example, the basic computational units of a neural network can be implemented using commonly used embedded C code. For instance, the header functions of the on-heap memory call functions (such as malloc, calloc, and free) used in the computation can be exposed as interfaces that can be modified and adapted to the corresponding on-chip memory management functions. These memory management functions can be used to implement the computational processes corresponding to different types of nodes. The computational processes can include, but are not limited to, linear algebra operations, element-wise operations, activation functions, statistical operations, shape transformations, and normalization operators.
[0046] Linear algebra operations can include matrix multiplication, vector dot product, and convolution.
[0047] Element-wise operations can include, for example, Adama multiplication (element-wise multiplication), element-wise addition, element-wise subtraction, and element-wise division.
[0048] Activation functions, such as Sigmoid, ReLU, and tanh functions.
[0049] Statistical operations, such as vector summation, average, maximum, and minimum values.
[0050] Shape transformation can be used to convert the shape of a matrix / vector, such as matrix flipping (Transpose) and the view function.
[0051] Normalization operators can be used to standardize a batch on a specific dimension.
[0052] Thus, during the execution of the compiled computation program, the memory required for neural network inference is uniformly managed internally by the SDK (which can be implemented based on malloc, calloc, and free). Users can call any computation process through the API interface provided by the SDK call layer without directly manipulating memory.
[0053] Step 230: Perform memory optimization on the computation graph to obtain an optimized computation graph.
[0054] In some embodiments, step 230, namely the memory optimization processing of the computation graph, includes: if there are two input matrices that are superimposed after performing matrix multiplication with the same weight matrix in the computation graph, and the intermediate results obtained by the matrix multiplication of the two input matrices are not independently referenced in subsequent calculations, then the two input matrices are pre-added to generate a merged input matrix, and a single matrix multiplication operation is performed between the merged input matrix and the weight matrix.
[0055] Since the computing resources of the target edge device, such as a microcontroller, are relatively limited, this embodiment can optimize the computation process based on the topology of the computation graph itself for the time-series data of lightweight neural networks, thereby ensuring low latency and high throughput for real-time data processing by the neural network.
[0056] In some embodiments, during the optimization of the computation graph, the method further includes: 1) Obtain the number of times the output result of each computing node is used; 2) When the number of times a certain output result is used drops to zero, the memory space it occupies is immediately released.
[0057] In this way, by tracking the number of times the output results of each computing node are used, matrix calculation results that are no longer used can be released directly to save computing memory consumption.
[0058] Step 240: Based on the optimized computation graph, generate a scheduling instruction sequence, wherein the scheduling instruction sequence is used to call functions in the embedded operator library.
[0059] Step 250: Based on the optimized computational graph and target model file, obtain the binary parameter file.
[0060] The target model file can be a PyTorch model file, and the binary parameter file can be a model parameter .bin file.
[0061] In some embodiments, the binary parameter file is a binary parameter file whose header stores a preset encryption check code and encrypted model hyperparameters, and whose tail contains a check code for verifying file integrity.
[0062] In other words, a corresponding encrypted checksum can be generated for each model exported from PyTorch based on its computational process. This encrypted checksum, along with the encrypted model hyperparameters, is stored in the header of the binary parameter file.
[0063] Furthermore, a checksum is added to the end of the binary parameter file to ensure that the size of the initialized model parameters is exactly the same as the size of the written model.
[0064] Step 260: Deploy the scheduling instruction sequence and the binary parameter file to the target end device.
[0065] The target edge device can be a lightweight, low-power microcontroller or a more complex SoC edge or edge chip, meaning that efficient and secure neural network inference can be achieved on different hardware platforms.
[0066] In addition, the computing tasks supported by this solution are not limited to time-series data processing from edge sensor inputs, but can also be general neural network application scenarios, including but not limited to data classification, feature extraction and pattern recognition.
[0067] Step 270: When the compiled calculation program is running on the target device, the binary parameter file is loaded to obtain the target model parameters.
[0068] In some embodiments, the target model parameters are a parameter matrix, and step 270, loading the binary parameter file to obtain the target model parameters, includes: Step 271: Generate the reconstructed computation graph based on the decrypted model calculation process and model hyperparameters.
[0069] Step 272: Read the parameter matrix used by each node in the reconstructed computation graph topology sequence.
[0070] For steps 271-272, that is, in the model parameter loading, the corresponding computation graph is first generated according to the decrypted model computation process and hyperparameters, and then the parameter matrix used by its nodes is read sequentially according to the topological sequence of the computation graph.
[0071] Step 280: The target end device performs neural network inference based on the scheduling instruction sequence, the embedded operator library, and the target model parameters.
[0072] In some embodiments, the target end-side device is a microcontroller, which integrates a floating-point arithmetic unit. Step 280, during the execution of neural network inference, further includes: Step 281: When the matrix operation involved by the operator meets the preset scale condition, the FPU is called to perform floating-point calculation, and the input matrix and weight matrix are divided into multiple sub-matrices along the row or column dimension, so that the data volume of each sub-matrix does not exceed the capacity threshold of the microcontroller cache.
[0073] Step 282: Perform block matrix multiplication and block accumulation operations on the submatrix to optimize parameter read and write efficiency by reducing the number of main memory accesses.
[0074] For steps 281-282, that is, the floating-point unit (FPU) on the microcontroller can be used to optimize the calculations involving floating-point calculations (fp32 and fp16) in the above calculations.
[0075] For example, for matrix operations exceeding a certain size (the parameter can be adjusted), the matrix can be divided into several smaller matrices (i.e., submatrices) that can be placed in the cache, and then added and multiplied in blocks to optimize the reading and writing speed of matrix parameters.
[0076] In some embodiments, in addition to utilizing the main control chip's own cache, this solution can also build a dedicated matrix calculation cache in the system memory.
[0077] Intermediate computation results that need to be repeatedly updated during inference (such as the hidden state output of a recurrent neural network (RNN) in time-series processing) can be pre-written into the matrix cache; When new time-series labeled data is received, only a single-step forward computation is required to generate the current output based on the historical state in the cache, without having to recompute the entire historical sequence, thereby significantly improving inference efficiency and reducing computational overhead.
[0078] Based on the above embodiments, combined with Figure 2 Before deployment, the edge neural network inference method provided in this application requires preprocessing the simple neural network into a complete computation graph. Each node in the computation graph represents a computational process that calls an operator. The input of this node is the vector output obtained from the previous step, and the output is the vector obtained after computation by that operator, which is used as the input for the next step.
[0079] In addition, commonly used neural network computations, such as 1D convolution operations, linear layer operations, term-by-term multiplication operations (hadamard-product), and various activation functions, can be integrated into a computation library. This library is based on the computation process, optimized, and the operators are called one by one during inference.
[0080] For example, such as Figure 2 As shown, the main processes include the following: 1) Compiling the neural network computation process Among these, the neural network operation process can be compiled using the compilation and computation optimization layer to obtain the computation graph.
[0081] 2) Implement code libraries for different types of computing nodes.
[0082] For example, the basic computational units of a neural network can be implemented using C code commonly used in embedded systems.
[0083] 3) Topology-based computational optimization For example, for time-series data of lightweight neural networks, the computation process can be optimized based on the topology of the computation graph itself.
[0084] In this way, the neural network computation graph process is compiled and optimized using PyTorch, making it suitable for deployment on lightweight platforms.
[0085] 4) Storage and loading of model parameters For example, for each model exported from PyTorch, a corresponding encrypted verification code is generated based on its computation process. This encrypted verification code and the encrypted model hyperparameters are stored at the beginning of the binary parameter file. During model parameter loading, the corresponding computation graph is first generated based on the decrypted model computation process and hyperparameters, and then the parameter matrix used by each node is read sequentially according to the topological sequence of the computation graph.
[0086] In this way, the computation of neural networks does not need to rely excessively on the computational optimization of the chip's own toolchain. Instead, it achieves lightweight deployment by simplifying the computational process of neural networks and optimizing the operator implementation logic. This solves the problem that target edge devices, such as microcontrollers, are unable to efficiently deploy AI models due to resource constraints. Furthermore, it optimizes the neural network inference and computation process on microcontrollers used for parsing sensor time-series data, improving the neural network inference efficiency and execution performance of microcontrollers in sensor time-series data parsing scenarios.
[0087] Corresponding to the above-described edge neural network inference method, this application also provides an edge neural network inference device. This edge neural network inference device includes a unit for executing the above-described edge neural network inference method, and can be configured in a desktop computer, tablet computer, laptop computer, or other terminal.
[0088] like Figure 3 As shown in the figure, this application provides a computer device including a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114. Memory 113 is used to store computer programs; In one embodiment of this application, when the processor 111 executes the program stored in the memory 113, it implements the edge neural network inference method provided in any of the foregoing method embodiments, including: Obtain the neural network operation flow to be processed; The neural network operation process is compiled to obtain a computation graph; The computation graph is then subjected to memory optimization to obtain an optimized computation graph; Based on the optimized computation graph, a scheduling instruction sequence is generated, wherein the scheduling instruction sequence is used to call functions in the embedded operator library; Based on the optimized computational graph and target model file, a binary parameter file is obtained; Deploy the scheduling instruction sequence and the binary parameter file to the target end device; When the compiled computing program runs on the target device, the binary parameter file is loaded to obtain the target model parameters; The target end-side device performs neural network inference based on the scheduling instruction sequence, the embedded operator library, and the target model parameters.
[0089] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program may be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.
[0090] Therefore, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, wherein when executed by a processor, the computer program implements the steps of the edge neural network inference method provided in any of the foregoing method embodiments, including: Obtain the neural network operation flow to be processed; The neural network operation process is compiled to obtain a computation graph; The computation graph is then subjected to memory optimization to obtain an optimized computation graph; Based on the optimized computation graph, a scheduling instruction sequence is generated, wherein the scheduling instruction sequence is used to call functions in the embedded operator library; Based on the optimized computational graph and target model file, a binary parameter file is obtained; Deploy the scheduling instruction sequence and the binary parameter file to the target end device; When the compiled computing program runs on the target device, the binary parameter file is loaded to obtain the target model parameters; The target end-side device performs neural network inference based on the scheduling instruction sequence, the embedded operator library, and the target model parameters.
[0091] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, external hard drive, read-only memory (ROM), magnetic disk, or optical disk, or any other physical storage medium capable of storing program code. The computer-readable storage medium can be non-volatile or volatile.
[0092] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.
[0093] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.
[0094] The steps in the methods of this application embodiment can be adjusted, merged, or deleted according to actual needs. The units in the apparatus of this application embodiment can be merged, divided, or deleted according to actual needs. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0095] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.
[0096] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.
[0097] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Since these modifications and variations fall within the scope of the claims and their equivalents, this application also intends to include these modifications and variations.
[0098] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A method for edge-side neural network inference, characterized in that, The method includes the following steps: Obtain the neural network operation flow to be processed; The neural network operation process is compiled to obtain a computation graph; The computation graph is then subjected to memory optimization to obtain an optimized computation graph; Based on the optimized computation graph, a scheduling instruction sequence is generated, wherein the scheduling instruction sequence is used to call functions in the embedded operator library; Based on the optimized computational graph and target model file, a binary parameter file is obtained; Deploy the scheduling instruction sequence and the binary parameter file to the target end device; When the compiled computing program runs on the target device, the binary parameter file is loaded to obtain the target model parameters; The target end-side device performs neural network inference based on the scheduling instruction sequence, the embedded operator library, and the target model parameters.
2. The edge-side neural network inference method according to claim 1, characterized in that, Memory optimization of the computation graph includes: In the computation graph, if there are two input matrices that are superimposed after performing matrix multiplication with the same weight matrix, and the intermediate results obtained by the matrix multiplication of the two input matrices are not independently referenced in subsequent calculations, then the two input matrices are pre-added to generate a merged input matrix, and a single matrix multiplication operation is performed between the merged input matrix and the weight matrix.
3. The edge-side neural network inference method according to claim 1, characterized in that, The binary parameter file is a binary parameter file. The header of the binary parameter file stores a preset encryption check code and encrypted model hyperparameters. The tail of the binary parameter file contains a check code used to verify the integrity of the file.
4. The end-side neural network inference method according to claim 3, characterized in that, The target model parameters are a parameter matrix. Loading the binary parameter file to obtain the target model parameters includes: The reconstruction calculation graph is generated based on the decrypted model calculation process and model hyperparameters; The parameter matrices used by the nodes are read sequentially according to the reconstructed computation graph topology sequence.
5. The end-side neural network inference method according to claim 1, characterized in that, The process of compiling the neural network operation flow to obtain a computation graph includes: The neural network operation process is broken down to obtain multiple computing nodes, where each node corresponds to a PyTorch computing function; Based on the tensor data dependencies between nodes, and the input and output variable dimensions of each node, a computational graph is obtained.
6. The end-side neural network inference method according to claim 5, characterized in that, The method further includes: Get the number of times the output result of each computing node is used; When the number of times a certain output result is used drops to zero, the memory space it occupies is immediately released.
7. The end-side neural network inference method according to claim 1, characterized in that, The target end-side device is a microcontroller, which integrates a floating-point arithmetic unit. During the execution of neural network inference, the method further includes: When the matrix operation involved by the operator meets the preset scale condition, the FPU is called to perform floating-point calculation, and the input matrix and weight matrix are divided into multiple sub-matrices along the row or column dimension, so that the data volume of each sub-matrix does not exceed the capacity threshold of the microcontroller cache. The submatrix is subjected to block matrix multiplication and block accumulation operations to optimize parameter read and write efficiency by reducing the number of main memory accesses.
8. An edge-side neural network inference device, characterized in that, Includes a unit for performing the method according to any one of claims 1-7.
9. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, can implement the method according to any one of claims 1-7.