Low-bit neural network reasoning method and system for microcontroller

Abstract

The invention discloses a low-bit neural network reasoning method and system for a microcontroller, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining the weight data and input data of a convolutional neural network, and carrying out the sub-byte quantization; calling an inference calculation library interface in the microcontroller firmware, and transmitting quantized weight data and input data into an inference calculation library for execution, specifically, respectively inputting the quantized weight data and input data into a neural network performance function, calling a neural network auxiliary function to pre-process the weight data and input data, and outputting the pre-processed weight data and input data into the microcontroller firmware; obtaining a weight vector and an input vector; the neural network performance function calls a calculation instruction based on the weight vector and the input vector to carry out parallel convolution calculation to obtain a convolution result; and calling a neural network auxiliary function to compress the convolution result to obtain a compressed convolution result, and outputting the compressed convolution result. On the premise of ensuring that the model precision is basically not reduced, the reasoning execution efficiency is effectively improved, and the system resource occupation is reduced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a low-bit neural network inference method and system for a microcontroller. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] With the rapid development of artificial intelligence (AI), convolutional neural networks (CNNs) have become the mainstream architecture in deep learning, widely used in fields such as image recognition, speech processing, object detection, and intelligent surveillance. In recent years, with the widespread adoption of smart terminals and IoT devices, an increasing number of deep learning tasks have begun migrating to the edge. However, these embedded devices often face challenges such as weak computing power, limited storage resources, and energy-intensive performance, severely hindering the deployment of complex neural network models.

[0004] Taking MobileNetV1 as an example, its full-precision (FP32) model requires approximately 17MB of storage and performs nearly 500 million floating-point operations on a 224×224 image input. Even with 8-bit quantization, the model still requires approximately 4.2MB of storage, making it difficult to run on typical microcontrollers (such as the STM32F7 series, which has only 1MB of FLASH and less than 512KB of RAM). Furthermore, most traditional microcontrollers lack floating-point hardware support, making inference latency and power consumption key factors limiting deployment.

[0005] To address these issues, model compression and quantization technologies have been extensively researched. By compressing network weights, activations, and intermediate computational results from 32-bit floating point to lower-width integers (e.g., 8-bit, 4-bit, or 2-bit), storage requirements and computational complexity can be significantly reduced. Currently, toolchains such as CMSIS-NN, X-CUBE-AI, and TensorFlow Lite support pure integer inference deployment for 8-bit quantized models. While the 8-bit solution maintains accuracy in most scenarios, its compression rate is limited.

[0006] Further research has shown that using 4-bit and 2-bit sub-byte quantization schemes can compress model storage to 1 / 4 or even 1 / 16. Combined with mixed-precision strategies, flexible allocation of quantization bit widths based on the network layer's sensitivity to precision is expected to achieve more efficient edge deployment while maintaining accuracy. However, while current mainstream low-bit inference solutions (such as CMix-NN) already support 4-bit and 2-bit mixed-precision computation, they still do not fully utilize microcontroller architectures, particularly with bottlenecks in multiplication instruction scheduling and data path optimization, failing to maximize the potential of the underlying hardware. Summary of the Invention

[0007] In order to overcome the above-mentioned deficiencies of the prior art, the present invention provides a low-bit neural network inference method and system for a microcontroller, which effectively improves the inference execution efficiency and reduces system resource usage while ensuring that the model accuracy is basically not reduced.

[0008] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: In a first aspect, the present invention provides a low-bit neural network inference method for a microcontroller, comprising: After the convolutional neural network training is completed, the weight data and input data thereof are obtained, and the weight data and input data are sub-byte quantized to obtain quantized weight data and input data; Call the inference calculation library interface in the microcontroller firmware, pass the quantized weight data and input data into the inference calculation library for execution, and output the final convolution result; The internal steps of the inference calculation library are executed by the central processing unit, specifically: The quantized weight data and input data are input into the neural network function respectively, and the neural network auxiliary function is called to pre-process them to obtain the weight vector and input vector; The neural network function calls the calculation instructions based on the weight vector and the input vector to perform parallel convolution calculations to obtain the convolution results; The neural network auxiliary function is called to compress the convolution result, obtain the compressed convolution result and output it; the neural network auxiliary function adopts the little endian byte order method during preprocessing and compression.

[0009] In a further technical solution, the preprocessing includes decompression and data alignment, which are implemented using an im2colQ operation with quantization processing.

[0010] A further technical solution is that the im2colQ operation with quantization processing is specifically as follows: extracting quantum bit precision data from the input data, accessing and unpacking the data in 32-bit blocks, calling the decompression and data alignment functions in the neural network auxiliary function, and restoring the input data to the corresponding vector format for convolution calculation.

[0011] According to a further technical solution, the decompression process is optimized at the instruction level according to the arrangement of data with different bit widths.

[0012] A further technical solution is that the parallel convolution calculation is specifically as follows: two sets of input vectors in standard format are generated according to the input vector, the data in the two sets of input vectors are rearranged and combined into a new set of input vectors using a staggered splicing strategy, and the new input vector is multiplied and accumulated with the weight vector.

[0013] A further technical solution is to split and reorganize the accumulated results through post-processing logic to extract two independent sets of accumulated output values, which correspond to the intermediate results of the two output channels respectively.

[0014] A further technical solution is to use integer multiplication and right shift to approximately replace floating-point division during the compression process, express the scaling factor in the form of fixed-point numbers, and convert it into multiplication and shift parameters in advance.

[0015] In a second aspect, the present invention provides a low-bit neural network inference system for a microcontroller, comprising: A quantization model acquisition module is configured to: after the convolutional neural network training is completed, obtain its weight data and input data, perform sub-byte quantization on the weight data and input data, and obtain quantized weight data and input data; The quantization model running module is configured to: call the inference calculation library interface in the microcontroller firmware, pass the quantized weight data and input data into the inference calculation library for execution, and output the final convolution result; The internal steps of the inference calculation library are executed by the central processing unit, specifically: The quantized weight data and input data are input into the neural network function respectively, and the neural network auxiliary function is called to pre-process them to obtain the weight vector and input vector; The neural network function calls the calculation instructions based on the weight vector and the input vector to perform parallel convolution calculations to obtain the convolution results; The neural network auxiliary function is called to compress the convolution result, obtain the compressed convolution result and output it; the neural network auxiliary function adopts the little endian byte order method during preprocessing and compression.

[0016] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a low-bit neural network inference method for a microcontroller as described in the first aspect.

[0017] In a fourth aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of a low-bit neural network inference method for a microcontroller as described in the first aspect are implemented.

[0018] One or more of the above technical solutions have the following beneficial effects: The present invention designs a sub-byte data stream compression and decompression method, adopts little-endian byte order storage and compression, effectively reduces memory usage, improves storage and bandwidth utilization, and is particularly suitable for resource-constrained microcontroller environments; designs a sub-byte data convolution calculation method, recombines input data based on a staggered splicing strategy, and utilizes the dual-channel multiplication characteristics of the SMLAD instruction in the multiplication and accumulation process to achieve parallel accumulation of two groups of product results within a single instruction, significantly improving the effective computational density of a single multiplication and accumulation instruction.

[0019] This method effectively improves inference execution efficiency and reduces system resource usage while maintaining minimal degradation in model accuracy. Compared to traditional 8-bit quantized convolution methods, measured results on a typical microcontroller platform (STM32F746-DISCO) show an average reduction of 13% to 20% in inference latency and approximately 22% in Flash program storage space, significantly improving the model's deployment capabilities on resource-constrained platforms.

[0020] This low-bit neural network inference library is designed specifically for neural network inference in embedded environments. It supports weight and activation operations with sub-byte quantization (i.e., 4-bit and 2-bit precision), enabling efficient pure integer quantization inference on resource-constrained microcontrollers. Compared to traditional 8-bit quantization libraries, this solution significantly reduces memory usage and improves operator execution speed while maintaining inference accuracy.

[0021] Compared with similar low-bit convolution algorithms proposed in CMix-NN, the present invention has better performance in execution efficiency: under a 4-bit quantization configuration, the running delay of the convolution layer is reduced by about 15%; under a 2-bit quantization configuration, the delay is reduced by 12%, and under the premise of maintaining the same memory structure and model scale, the overall memory usage remains basically unchanged. If the application scenario where the weight data memory usage is small is targeted, adding weight offline processing can reduce the delay by more than 20%, but the memory usage will be slightly increased. This shows that the present invention achieves better execution speed and energy efficiency without introducing additional hardware burden. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0023] Figure 1 This is a flow chart of a low-bit neural network inference method for a microcontroller according to an embodiment of the present invention; Figure 2 This is a structural diagram of a low-bit neural network inference calculation library according to an embodiment of the present invention; Figure 3 Schematic diagram of sub-byte data format and decompression according to an embodiment of the present invention; Figure 4 This is a schematic diagram of parallel computing of low-bit-width data according to an embodiment of the present invention; Figure 5 This is a schematic diagram of im2col and data alignment according to an embodiment of the present invention; Figure 6 Schematic diagram of the weight decompression process according to an embodiment of the present invention; Figure 7 This is a flow chart of data preprocessing and SMLAD in accordance with an embodiment of the present invention; Figure 8 Schematic diagram of the convolution output of a sub-cycle in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0025] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0026] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0027] Explanation of terms: Low-bit quantization: refers to quantizing neural network parameters and activation values ​​from 32-bit floating point numbers to integers lower than 8 bits.

[0028] Microcontroller: A resource-constrained embedded processor, typically with limited memory and computing power.

[0029] SIMD instruction: Single instruction multiple data instruction, which can process multiple data in parallel.

[0030] Mixed precision: Different layers of the neural network use quantization strategies with different bit widths.

[0031] The existing technology has the following problems: (1) Insufficient low-bit quantization basic operator library at the deployment end Despite extensive server-side research in recent years on low-bit quantization strategies (particularly 4-bit and 2-bit quantization), deployment of related neural network inference technologies on microcontrollers remains relatively limited. Current mainstream embedded inference libraries (such as CMSIS-NN, X-CUBE-AI, and TensorFlow Lite) primarily focus on 8-bit quantization optimization and lack native support for 4-bit / 2-bit data formats. For example, on the typical STM32F746 platform, in the absence of low-bit operator support, deploying a 4-bit quantized model requires decompressing the weights to 8 bits at runtime, then calling standard 8-bit convolution operators to perform the calculations, and finally requantizing the model. This process not only fails to reduce overall memory usage but also increases decoding and re-encoding time overhead, completely negating the performance advantages of low-bit quantization.

[0032] (2) Weak support for mixed precision Most current embedded inference frameworks use a homogeneous quantization strategy, where all layers or channels use a uniform precision setting, lacking native support for mixed precision between layers or channels. Deployment of mixed-precision networks often requires complex type conversions, memory reordering, and precision alignment, which increases deployment difficulty and incurs additional computational and memory overhead. Furthermore, the lack of a dynamic precision switching mechanism prevents the network from flexibly adjusting precision configuration based on computational requirements during runtime, limiting the widespread application of mixed-precision quantization on resource-constrained devices.

[0033] (3) Low operator computation efficiency While sub-byte quantization (such as 4-bit and 2-bit) can effectively reduce storage requirements, the actual improvement in computational efficiency is limited due to a lack of specialized optimization. Low-precision calculations often require specialized processing such as bit packing / unpacking, but existing libraries do not provide efficient bit manipulation primitives, resulting in additional computational overhead. For sub-byte data, existing libraries fail to fully utilize the Single Instruction Multiple Data (SIMD) instruction set to accelerate computations, failing to maximize the parallel computing capabilities of microcontrollers.

[0034] Therefore, there is an urgent need for a low-bit neural network inference computing library that can be used for microcontroller platforms and supports lower bit width and higher instruction efficiency. An innovative low-bit inference method for microcontrollers is proposed, which supports 4-bit and 2-bit mixed precision convolution kernel configuration, and through deep collaborative optimization of computing-storage paths, significantly improves computing efficiency and reduces storage requirements, thereby realizing the efficient deployment of complex neural networks on embedded devices.

[0035] Example 1 like Figure 1 As shown, this embodiment discloses a low-bit neural network inference method for a microcontroller, the method comprising the following steps: S1: After the convolutional neural network training is completed, its weight data and input data are obtained, and the weight data and input data are sub-byte quantized to obtain quantized weight data and input data; In this embodiment, in the neural network inference calculation, the input data is the original data received by the neural network, such as images, text, etc., which is obtained by the sensor or the upper module and stored in the SRAM or the internal cache of the microcontroller; the weight data is the parameters obtained after the neural network model is trained, which can be obtained by parsing the model at the training end, deployed to the microcontroller MCU, and stored in Flash / ROM.

[0036] The weight data and input data are sub-byte quantized using existing quantization methods, which will not be described in detail in this embodiment.

[0037] S2: Call the inference calculation library interface in the microcontroller firmware, pass the quantized weight data and input data into the inference calculation library for execution, and output the final convolution result.

[0038] In this embodiment, the present invention proposes a low-bit mixed-precision convolution library (i.e., a low-bit neural network inference library) for microcontroller platforms. Its implementation fully leverages the instruction set of the ARM Cortex-M series processors. The following steps use 4-bit convolution as an example. Designed specifically for neural network inference in embedded environments, this library supports weight and activation operations with sub-byte quantization (i.e., 4-bit and 2-bit precision), enabling efficient pure integer quantization inference on resource-constrained microcontrollers.

[0039] like Figure 2 As shown, the core of the neural network inference library consists of a complete set of convolution kernel functions. It supports any combination of 4-bit and 2-bit weights, inputs, and output activations, and is compatible with various quantization strategies (such as per-layer and per-channel). The library includes two modules: neural network function functions and neural network auxiliary functions.

[0040] Neural network functions include convolution functions for different quantization precisions. Each convolution function is optimized for high efficiency, capable of running low-bit-precision convolution calculations on microcontrollers. The simple and flexible API interface makes these convolution functions easily adaptable and embeddable in various machine learning frameworks. Through this module, users can conveniently deploy low-bit quantized neural network models directly to the target platform. Neural network auxiliary functions are used to implement key operations such as bit decoding, bit packing, format conversion, and storage compression for low-bit data. These functions are targeted and optimized for different data bit widths, supporting efficient data alignment and memory access, ensuring that data channel conversion in convolution calculations does not become a performance bottleneck.

[0041] Neural network functions are common operations in the neural network computing library, such as convolution, pooling, and fully connected functions. The neural network inference computing library in this embodiment primarily includes convolution operations of varying bit precision, which can be directly called to implement neural network inference. Neural network auxiliary functions provide underlying support for neural network functions, including data preprocessing, format conversion, calculation, and quantization operations. These functions are integrated into the internal implementation of the neural network functions and are called when the neural network functions are executed.

[0042] In this embodiment, a set of efficient compression and decompression mechanisms, namely, sub-byte data stream compression and decompression methods, that are compatible with the microcontroller architecture are designed in the neural network auxiliary function to meet the storage and computing requirements of sub-byte quantized data (such as 4 bits and 2 bits) on the embedded platform.

[0043] like Figure 3 As shown, the compression method of the present invention uses little-endian storage. That is, low-bit data is compressed and stored in uint8 data blocks using little-endian data compression. Accumulated value scaling, bit width writeback, and output packing in convolution calculations also use little-endian compression. 4-bit quantized data is compactly stored in uint8 data blocks using little-endian byte order. Whether used for convolution weights or activations, each uint8 data block is compressed with two 4-bit elements. 2-bit quantized data is stored with four 2-bit elements per uint8 data block, also following little-endian byte order. This compact storage method effectively reduces memory usage and improves storage and bandwidth utilization, making it particularly suitable for resource-constrained microcontroller environments.

[0044] Before the neural network function performs convolution calculations, the low-bit neural network inference calculation library uses specially designed decompression and data alignment functions to restore the 4-bit or 2-bit quantized data to a 16-bit signed integer format that can be used for calculations. This is to adapt to the SIMD multiplication and addition instructions (Signed Multiply with Addition Dual, SMLAD) in the ARM Cortex-M architecture. The SMLAD instruction requires the input data to be in a 2×16-bit vector format, so the original sub-byte data must be decompressed into this structure to participate in parallel multiplication and addition operations.

[0045] The decompression function extracts multiple 4-bit or 2-bit values ​​stored in a uint8 data block. The decompression process utilizes instruction-level optimizations tailored to the arrangement of data of varying bit widths. This leverages the low-level, efficient bit manipulation instructions in the ARM processor instruction set to efficiently and parallelize the unpacking and restoration of the low-bit values ​​stored in the uint8 data block, adapting it to the data format required for subsequent vectorization and parallel computation. This utilizes fundamental ARM instructions such as logical right shift (LSR), bitwise AND (bitwise AND), and byte extension (Sign / Unsigned Extend Byte to Halfword (SXTB / UXTB)). This combines these instructions to extract two 4-bit values ​​from a uint8 data block and restore them to a 2×16-bit vector. For 2-bit values, due to their higher data density, a uint8 data block is parsed into four 2-bit values, which are then restored to two 2×16-bit vectors, meeting the data format requirements for parallel convolution computation.

[0046] Further, such as Figure 5 As shown, the mixed instruction is used multiple times for 4-bit data and 2-bit data: Out = _UXTB16(_ROR(_SXTB16(in), x)), where x represents the number of right shifts. For 4-bit data, the mixed instruction is used four times to extract and convert from a 32-bit data block (containing 8 4-bit data) into four 2x16-bit vectors. For 2-bit data, the mixed instruction is used eight times to extract and convert from a 32-bit data block (containing 16 4-bit data) into eight 2x16-bit vectors.

[0047] The data alignment function rearranges the extracted low-bit data so that it conforms to the correspondence between the input feature map and the convolution kernel in the convolution calculation.

[0048] In this embodiment, during the multiplication-accumulation stage of the convolution core, given the inherently narrow bit width of subbyte data, directly loading it into a standard 2x16-bit multiplication-accumulation unit would result in significant memory redundancy and hinder the efficient use of SIMD computing resources. To address this, the present invention expands and structurally rearranges low-bitwidth data to improve data utilization and execution efficiency during the multiplication-accumulation stage. A subbyte data convolution computation method is designed within a neural network auxiliary function.

[0049] like Figure 4 As shown in the figure, after analyzing 32-bit multiplication and decompressing the activation data (i.e., input data / input feature map), the low-bit neural network inference library first generates two sets of standard 2×16-bit input vectors based on the activation data. Subsequently, using a staggered splicing strategy, the data in these two sets of vectors are rearranged and combined into a new 2×16-bit input vector containing four consecutive sub-byte data elements. Simultaneously, the weight data is decompressed to form a 2×16-bit vector containing two elements, which constitutes the weight input for the current convolution window. The activation data and weight data are multiplied and accumulated during the convolution calculation.

[0050] Furthermore, the data in the two sets of 2×16-bit input vectors in the standard format are rearranged and combined into a new set of 2×16-bit input vectors, such as Figure 7 In the data concatenation part, the first 2×16 input vector remains unchanged, and the second 2×16 input vector is shifted left by a certain number of bits and then added to the first 2×16 input vector to form a new 2×16 input vector.

[0051] In the multiplication and accumulation process, the dual-channel multiplication feature of the SMLAD instruction is used to implement four multiplication operations on the above four activation data and two weight data, and complete the parallel accumulation of two sets of product results within a single instruction. This operation significantly improves the effective computational density of a single multiplication and accumulation instruction. In order to obtain the final output feature map value, the low-bit neural network inference calculation library further splits and reorganizes the accumulation result through post-processing logic, and extracts two sets of independent accumulation output values, which correspond to the intermediate results of two output channels or two spatial positions. Specifically, Figure 7 As shown in the figure, according to the calculation rules of the SMLAD instruction and the previous data alignment, the 32-bit calculation result, bits 0-7 and bits 11-19, are the valid data in the output feature map. After calculation, these bits are extracted and stored in 32-bit wide data blocks. Then, through multiple iterations of the same multiplication and accumulation calculation, the final output feature map data is obtained.

[0052] The internal steps of the inference computing library are executed by the central processing unit, specifically: (1) The quantized weight data and input data are input to the neural network function respectively, and the neural network auxiliary function is called to preprocess them to obtain the weight vector and input vector, i.e., the normalized vector. The weight vector and input vector are temporarily stored in SRAM for subsequent convolution calculations.

[0053] In this embodiment, a neural network auxiliary function is called to preprocess the input feature map (input data), including decompression and data alignment. The neural network auxiliary function is used to convert the input feature map into a data layout suitable for matrix multiplication, thereby constructing an efficient convolutional expansion mechanism. Traditional image expansion is performed as a column-to-column matrix operation (im2col). This operation is inefficient on microcontroller platforms due to frequent memory accesses and high data redundancy. To address this, the present invention proposes an im2colQ operation path optimized for low-bit quantized inference. This not only completes the convolution window expansion but also integrates quantization format alignment and vectorized preprocessing.

[0054] like Figure 5As shown, the neural network helper function extracts qubit Q-precision data from the input tensor tensorIn, accessing and unpacking it in 32-bit chunks. With 4-bit quantization, each 32-bit word stores 8 compressed input elements; with 2-bit quantization, each 32-bit word contains 16 input elements. Precisely controlled data alignment and decoding logic efficiently convert the compressed input data into a vector format suitable for direct multiplication and addition operations. To further improve data handling efficiency, the im2colQ module employs a double buffering mechanism (ptrBuff0 and ptrBuff1) to enable parallel expansion of two adjacent output spatial locations, supporting batch data loading and pipeline processing. During data loading, quantization zero-point correction, sign extension, and packing into a 2x16-bit vector format are performed, avoiding repeated decoding and format conversion in subsequent stages.

[0055] (2) The neural network function calls the calculation instructions based on the weight vector and the input vector to perform parallel convolution calculations to obtain the convolution results; the intermediate results of the calculations are stored in SRAM.

[0056] In this embodiment, the computational instructions are SMLAD multiply-accumulate instructions. The module responsible for performing the matrix multiplication and accumulation operations in the backbone convolution is the computational core of this invention. While supporting low-precision quantization weights such as 4-bit and 2-bit, this module designs efficient unpacking (UnpackQ) and vector expansion mechanisms for weights of varying bit widths to balance execution efficiency and computational accuracy. These mechanisms are tightly integrated into the main multiply-accumulate loop (AccLoop), creating a convolution computation flow suitable for embedded processors.

[0057] like Figure 6 As shown in the figure, for 4-bit precision weight data (weight parameters), each 32-bit compressed word containing eight 4-bit quantized weights is loaded from the compressed weight tensor tensorW . The decompression process is efficiently implemented using three instructions: first, a logical right shift (LSR) operation aligns the target weights, then a logical AND (AND) mask is used to extract the low-order bits, and finally, an unsigned extension instruction (UXTB16) is used to extend the 4-bit integer to a 16-bit format. For 2-bit precision weights, the process is similar, with different bit masks and data density control logic used during decompression. Each 32-bit compressed word contains a maximum of 16 2-bit weight values.

[0058] Taking 4-bit convolution as an example, in the traditional path, each SMLAD instruction can only process the multiplication of two 16-bit signed integers at the same time and accumulate the results into a 32-bit register, although only the lower 4 bits of the input actually carry valid information. This leads to memory redundancy and underutilization of computing units in data representation. Figure 7As shown in the figure, a one-step vector packing optimization is introduced in the preprocessing stage. Through shift splicing, multiple 4-bit input values ​​are combined into a structured 2x16-bit vector. The sign bits of In2_ch5 and In2_ch1 at the head of the 16-bit vector can be automatically supplemented during the calculation process. This allows an SMLAD instruction to complete the original two multiplication and accumulation while maintaining architectural compatibility. At the same time, it can be merged into four effective multiplication and accumulation, corresponding to the intermediate results of two output channels or two spatial positions, respectively. Figure 8 shown.

[0059] Furthermore, for applications with smaller weight data, the neural network helper function also supports a weight preprocessing strategy: during the neural network model generation phase, the weights are pre-processed offline, vectorized to 2x16 bits, and sorted, and then deployed in Flash or on-chip SRAM as structured storage. This strategy effectively avoids the dynamic decoding and sorting overhead within the convolution loop, making the runtime decompression operation close to zero overhead, further improving the overall convolution operation speed. This is particularly suitable for the real-time response requirements of low latency and low power consumption in edge AI scenarios.

[0060] (3) Call the neural network auxiliary function to compress the convolution result, obtain the compressed convolution result and output it. The convolution result is compressed to facilitate the input or storage of the next layer, and the compressed convolution result is written back to SRAM or Flash.

[0061] In this embodiment, after the calculation is completed, the convolution output result is still temporarily stored in the buffer in the form of a 32-bit integer. In order to adapt to the storage and communication requirements, it needs to be quantized and compressed again. The present invention introduces the Compress module to perform scaling, bit width write-back and output packaging operations on the accumulated value. The module supports two quantization strategies: Per-Layer (uniform scaling) and Per-Channel (channel independent scaling). During the quantization process, the original 32-bit accumulated value will be linearly transformed according to the corresponding scaling factor and zero offset to achieve the mapping from the integer accumulated value to the low-bit integer output. The present invention uses integer multiplication and right shift to approximate instead of floating-point division, and converts the scaling factor into a fixed-point number ( ) representation and pre-converted into multiplication and shift parameters, so that the scaling process can be completed at runtime using only integer multiplication and logical right shift, significantly reducing the computational overhead.

[0062] To ensure that quantized values ​​do not overflow or experience precision abrupt changes, the Compress module also incorporates the ARM instruction set's Saturating Arithmetic Instructions (SSAT) and Signed Saturate (USAT). These instructions truncate the bit width and impose upper and lower bounds on the result, ensuring that the final output value always falls within the target Qbit range, improving system stability and numerical security. After compression, the output data is packed in order according to the specified bit width (e.g., 8-bit, 4-bit, or 2-bit) and written to the output tensor, tensorOut.

[0063] The pseudo code of the quantized convolutional layer is shown in Table 1: Table 1 Pseudocode of quantized convolutional layer

[0064] Example 2 This embodiment discloses a low-bit neural network inference system for a microcontroller, including: A quantization model acquisition module is configured to: after the convolutional neural network training is completed, obtain its weight data and input data, perform sub-byte quantization on the weight data and input data, and obtain quantized weight data and input data; The quantization model running module is configured to: call the inference calculation library interface in the microcontroller firmware, pass the quantized weight data and input data into the inference calculation library for execution, and output the final convolution result; The internal steps of the inference calculation library are executed by the central processing unit, specifically: The quantized weight data and input data are input into the neural network function respectively, and the neural network auxiliary function is called to pre-process them to obtain the weight vector and input vector; The neural network function calls the calculation instructions based on the weight vector and the input vector to perform parallel convolution calculations to obtain the convolution results; The neural network auxiliary function is called to compress the convolution result, obtain the compressed convolution result and output it; the neural network auxiliary function adopts the little endian byte order method during preprocessing and compression.

[0065] Example 3 The purpose of this embodiment is to provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method of embodiment 1 when executing the program.

[0066] Example 4 The purpose of this embodiment is to provide a computer-readable storage medium, a computer-readable storage medium having a computer program stored thereon, which performs the steps of the method of embodiment 1 when executed by a processor.

[0067] The steps involved in the apparatuses of Examples 3 and 4 above correspond to those of Method Example 1. For detailed implementation, please refer to the relevant description of Example 1. The term "computer-readable storage medium" should be understood to mean a single medium or multiple media containing one or more instruction sets; it should also be understood to include any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and causing the processor to perform any of the methods of the present invention.

[0068] Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computer device. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0069] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

[0070] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. A low-bit neural network inference method for a microcontroller, characterized in that: include: After the convolutional neural network training is completed, the weight data and input data thereof are obtained, and the weight data and input data are sub-byte quantized to obtain quantized weight data and input data; Call the inference calculation library interface in the microcontroller firmware, pass the quantized weight data and input data into the inference calculation library for execution, and output the final convolution result; The internal steps of the inference calculation library are executed by the central processing unit, specifically: The quantized weight data and input data are input into the neural network function respectively, and the neural network auxiliary function is called to pre-process them to obtain the weight vector and input vector; The neural network function calls the calculation instructions based on the weight vector and the input vector to perform parallel convolution calculations to obtain the convolution results; The neural network auxiliary function is called to compress the convolution result, obtain the compressed convolution result and output it; the neural network auxiliary function adopts the little endian byte order method during preprocessing and compression.

2. A low-bit neural network inference method for a microcontroller according to claim 1, characterized in that: The preprocessing includes decompression and data alignment, which is implemented by using the im2colQ operation with quantization processing.

3. A low-bit neural network inference method for a microcontroller according to claim 2, characterized in that: The im2colQ operation with quantization specifically extracts qubit-precision data from the input data, accesses and unpacks it in 32-bit chunks, calls the decompression and data alignment functions in the neural network auxiliary function, and restores the input data to the corresponding vector format for convolution calculation.

4. A low-bit neural network inference method for a microcontroller according to claim 2, characterized in that: The decompression process is optimized at the instruction level according to the arrangement of data with different bit widths.

5. The low-bit neural network inference method for a microcontroller according to claim 1, wherein: The parallel convolution calculation is specifically as follows: two sets of input vectors in standard formats are generated according to the input vector, the data in the two sets of input vectors are rearranged and combined into a new set of input vectors using a staggered splicing strategy, and the new input vector is multiplied and accumulated with the weight vector.

6. A low-bit neural network inference method for a microcontroller according to claim 5, characterized in that: The accumulated results are split and reorganized through post-processing logic to extract two independent sets of accumulated output values, corresponding to the intermediate results of the two output channels respectively.

7. The low-bit neural network inference method for a microcontroller according to claim 1, wherein: During the compression process, integer multiplication and right shift are used to approximately replace floating-point division, and the scaling factor is expressed in the form of fixed-point numbers and converted into multiplication and shift parameters in advance.

8. A low-bit neural network inference system for a microcontroller, characterized in that: include: A quantization model acquisition module is configured to: after the convolutional neural network training is completed, obtain its weight data and input data, perform sub-byte quantization on the weight data and input data, and obtain quantized weight data and input data; The quantization model running module is configured to: call the inference calculation library interface in the microcontroller firmware, pass the quantized weight data and input data into the inference calculation library for execution, and output the final convolution result; The internal steps of the inference calculation library are executed by the central processing unit, specifically: The quantized weight data and input data are input into the neural network function respectively, and the neural network auxiliary function is called to pre-process them to obtain the weight vector and input vector; The neural network function calls the calculation instructions based on the weight vector and the input vector to perform parallel convolution calculations to obtain the convolution results; The neural network auxiliary function is called to compress the convolution result, obtain the compressed convolution result and output it; the neural network auxiliary function adopts the little endian byte order method during preprocessing and compression.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of a low-bit neural network inference method for a microcontroller as described in any one of claims 1 to 7 are implemented.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of a low-bit neural network inference method for a microcontroller as described in any one of claims 1 to 7 are implemented.

Images