A low-bit neural network inference method and system for microcontrollers

Abstract

This invention discloses a low-bit neural network inference method and system for microcontrollers, relating to the field of artificial intelligence technology. The method involves acquiring convolutional neural network weight data and input data, and performing sub-byte quantization. Within the microcontroller firmware, an inference computation library interface is called to pass the quantized weight data and input data into the inference computation library for execution. Specifically, the quantized weight data and input data are input to the neural network function, and a neural network auxiliary function is called to preprocess them, obtaining a weight vector and an input vector. The neural network function then performs parallel convolution calculations based on the weight vector and input vector, obtaining the convolution result. Finally, the neural network auxiliary function is called to compress the convolution result, obtaining and outputting the compressed convolution result. This method effectively improves inference execution efficiency and reduces system resource consumption while maintaining essentially no reduction in model accuracy.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a low-bit neural network inference method and system for microcontrollers. Background Technology

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] With the rapid development of artificial intelligence technology, Convolutional Neural Networks (CNNs) have become the mainstream architecture in deep learning, widely used in fields such as image recognition, speech processing, object detection, and intelligent monitoring. In recent years, with the popularization of smart terminals and IoT devices, more and more deep learning tasks have begun to migrate to the edge. However, such embedded devices typically face problems such as weak computing power, limited storage resources, and energy sensitivity, which severely restrict the deployment of complex neural network models.

[0004] Taking MobileNetV1 as an example, its full-precision (FP32) model requires approximately 17MB of storage and performs nearly 500 million floating-point operations on a 224×224 image input. Even with 8-bit quantization, the model still requires approximately 4.2MB of storage space, making it difficult to run on typical microcontrollers (such as the STM32F7 series, which has only 1MB of FLASH and less than 512KB of RAM). Furthermore, traditional microcontrollers often lack floating-point hardware support, and inference latency and power consumption are also key factors limiting deployment.

[0005] To address the aforementioned issues, model compression and quantization techniques have been extensively studied. By compressing the weights, activations, and intermediate computation results in the network from 32-bit floating-point numbers to low-bit-width integers (such as 8-bit, 4-bit, and 2-bit), storage requirements and computational complexity can be significantly reduced. Currently, toolchains such as CMSIS-NN, X-CUBE-AI, and TensorFlow Lite support pure integer inference deployment of 8-bit quantized models. Although the 8-bit scheme can maintain accuracy in most scenarios, its compression ratio is limited.

[0006] Further research shows that using 4-bit and 2-bit sub-byte quantization schemes can compress model storage to 1 / 4 or even 1 / 16. Combined with mixed-precision strategies, the quantization bit width can be flexibly allocated according to the network layer's precision sensitivity, potentially achieving more efficient edge deployment while maintaining accuracy. However, while current mainstream low-bit inference schemes (such as CMix-NN) support 4 / 2-bit mixed-precision computation, their utilization of microcontroller architecture remains insufficient, particularly in multiplication instruction scheduling and data path optimization, failing to maximize the potential of the underlying hardware. Summary of the Invention

[0007] To overcome the shortcomings of the prior art, the present invention provides a low-bit neural network inference method and system for microcontrollers, which effectively improves inference execution efficiency and reduces system resource consumption while ensuring that the model accuracy is not reduced by much.

[0008] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0009] In a first aspect, the present invention provides a low-bit neural network inference method for a microcontroller, comprising:

[0010] After the convolutional neural network is trained, its weight data and input data are obtained, and the weight data and input data are subjected to sub-byte quantization to obtain quantized weight data and input data.

[0011] The microcontroller firmware calls the inference computing library interface, passes the quantized weight data and input data into the inference computing library for execution, and outputs the final convolution result.

[0012] The internal steps of the inference computation library are executed by the central processing unit, specifically:

[0013] The quantized weight data and input data are input into the neural network function, and the neural network auxiliary function is called to preprocess them to obtain the weight vector and input vector.

[0014] The neural network function calls computation instructions based on the weight vector and input vector to perform parallel convolution calculations and obtain the convolution result.

[0015] The neural network auxiliary function is called to compress the convolution result, and the compressed convolution result is output. The neural network auxiliary function uses little-endian byte order for preprocessing and compression.

[0016] A further technical solution is that the preprocessing includes decompression and data alignment, which is implemented using the im2colQ operation with quantization.

[0017] A further technical solution is that the im2colQ operation with quantization processing specifically involves: extracting qubit-precision data from the input data, accessing and unpacking the data in 32-bit blocks, calling the decompression and data alignment functions in the neural network auxiliary function, and restoring the input data to the corresponding vector format for convolution calculation.

[0018] A further technical solution involves optimizing the decompression process at the instruction level for different bit-width data arrangement methods.

[0019] A further technical solution is that the parallel convolution calculation specifically involves: generating two sets of standard format input vectors based on the input vectors; using a staggered concatenation strategy to rearrange and combine the data in the two sets of input vectors into a new set of input vectors; and multiplying and accumulating the new input vector with the weight vector.

[0020] A further technical solution involves splitting and recombining the accumulated results through post-processing logic to extract two independent sets of accumulated output values, which correspond to the intermediate results of the two output channels respectively.

[0021] A further technical solution involves using integer multiplication and right shift to approximate floating-point division during the compression process, representing the scaling factor in fixed-point form, and pre-converting it into multiplication and shift parameters.

[0022] In a second aspect, the present invention provides a low-bit neural network inference system for a microcontroller, comprising:

[0023] The quantization model acquisition module is configured to: after the training of the convolutional neural network is completed, acquire its weight data and input data, and perform sub-byte quantization on the weight data and input data to obtain quantized weight data and input data;

[0024] The quantization model execution module is configured to: call the inference computing library interface in the microcontroller firmware, pass the quantized weight data and input data into the inference computing library for execution, and output the final convolution result;

[0025] The internal steps of the inference computation library are executed by the central processing unit, specifically:

[0026] The quantized weight data and input data are input into the neural network function, and the neural network auxiliary function is called to preprocess them to obtain the weight vector and input vector.

[0027] The neural network function calls computation instructions based on the weight vector and input vector to perform parallel convolution calculations and obtain the convolution result.

[0028] The neural network auxiliary function is called to compress the convolution result, and the compressed convolution result is output. The neural network auxiliary function uses little-endian byte order for preprocessing and compression.

[0029] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a low-bit neural network inference method for a microcontroller as described in the first aspect.

[0030] Fourthly, the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a low-bit neural network inference method for a microcontroller as described in the first aspect.

[0031] The above one or more technical solutions have the following beneficial effects:

[0032] This invention designs a sub-byte data stream compression and decompression method, which uses little-endian byte order for storage and compression, effectively reducing memory usage and improving storage and bandwidth utilization, making it particularly suitable for resource-constrained microcontroller environments. It also designs a sub-byte data convolution calculation method, which recombines input data based on a staggered splicing strategy. During the multiplication and accumulation process, it utilizes the dual-channel multiplication characteristic of the SMLAD instruction to achieve parallel accumulation of two sets of product results within a single instruction, significantly improving the effective computational density of a single multiplication and accumulation instruction.

[0033] This invention effectively improves inference execution efficiency and reduces system resource consumption while maintaining essentially no reduction in model accuracy. Compared with the traditional 8-bit quantized convolution method, the experimental results of this invention on a typical microcontroller platform (STM32F746-DISCO) show that the inference latency is reduced by an average of 13% to 20%, and the Flash program storage space is reduced by about 22%, significantly improving the model's deployment capability on resource-constrained platforms.

[0034] This invention presents a low-bit neural network inference library specifically designed for neural network inference in embedded environments. It supports weight and activation operations at sub-byte quantization (i.e., 4-bit and 2-bit) precision, enabling efficient pure integer quantization inference on resource-constrained microcontrollers. Compared to traditional 8-bit quantization libraries, the proposed solution significantly reduces memory usage and improves operator execution speed while maintaining inference accuracy.

[0035] Compared to similar low-bit convolutional algorithms proposed in CMix-NN, this invention demonstrates superior execution efficiency: with 4-bit quantization, the latency of the convolutional layer is reduced by approximately 15%; with 2-bit quantization, the latency reduction reaches 12%, and the overall memory usage remains essentially unchanged while maintaining the same memory structure and model size. For applications with limited weight data memory usage, adding offline weight processing can reduce latency by more than 20%, but memory usage increases slightly. This indicates that this invention achieves superior execution speed and energy efficiency without introducing additional hardware burden. Attached Figure Description

[0036] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0037] Figure 1 This is a flowchart of a low-bit neural network inference method for a microcontroller according to an embodiment of the present invention;

[0038] Figure 2 This is a structural diagram of the low-bit neural network inference computing library according to an embodiment of the present invention;

[0039] Figure 3 This is a schematic diagram of the subbyte data format and decompression in an embodiment of the present invention;

[0040] Figure 4 This is a schematic diagram of low-bit-width data parallel computation according to an embodiment of the present invention;

[0041] Figure 5 This is a schematic diagram of im2col and data alignment in an embodiment of the present invention;

[0042] Figure 6 This is a schematic diagram of the weight decompression process according to an embodiment of the present invention;

[0043] Figure 7 This is a flowchart of data preprocessing and SMLAD in an embodiment of the present invention;

[0044] Figure 8 This is a schematic diagram of the convolution output of a first sub-loop in an embodiment of the present invention. Detailed Implementation

[0045] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0046] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0047] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0048] Terminology Explanation:

[0049] Low-bit quantization: refers to quantizing neural network parameters and activation values ​​from 32-bit floating-point quantization to integers with fewer than 8 bits.

[0050] Microcontroller: A resource-constrained embedded processor, typically with limited memory and computing power.

[0051] SIMD instruction: Single Instruction Multiple Data instruction, which can process multiple data in parallel.

[0052] Mixed precision: Different layers of the neural network employ different bit width quantization strategies.

[0053] The existing technology has the following problems:

[0054] (1) Insufficient low-bit quantization basic operator library on the deployment side

[0055] Although there has been extensive research on low-bit quantization strategies (especially 4-bit and 2-bit quantization) on the server side in recent years, the deployment of related neural network inference technologies on microcontrollers remains relatively lacking. Most mainstream embedded inference libraries (such as CMSIS-NN, X-CUBE-AI, and TensorFlow Lite) focus on 8-bit quantization optimization and lack native support for 4-bit / 2-bit data formats. Taking the typical STM32F746 platform as an example, without low-bit operator support, deploying a 4-bit quantized model requires decompressing the weights to 8 bits at runtime, then calling the standard 8-bit convolution operator for computation, and finally requantizing. This process not only fails to reduce overall memory usage but also increases the time overhead of decoding and re-encoding, completely negating the performance advantages that low-bit quantization should offer.

[0056] (2) Weak support for mixed precision

[0057] Most current embedded inference frameworks employ homogeneous quantization, meaning all layers or channels use a uniform precision setting, lacking native support for mixed precision across layers or channels. Mixed-precision networks often require complex type conversions, memory rearrangements, and precision alignment operations during deployment, increasing deployment complexity and incurring additional computational and memory overhead. Furthermore, the lack of a dynamic precision switching mechanism prevents the network from flexibly adjusting precision configurations based on computational needs at runtime, limiting the widespread application of mixed-precision quantization on resource-constrained devices.

[0058] (3) The operator has low computational efficiency.

[0059] While subbyte quantization (such as 4-bit and 2-bit) can effectively reduce storage requirements, the actual improvement in computational efficiency is limited due to a lack of dedicated optimizations. Low-precision calculations typically require special processing such as bit packing / unpacking, and existing libraries do not provide efficient bit manipulation primitives, resulting in additional overhead during computation. For subbyte data, existing libraries fail to fully utilize the Single Instruction Multiple Data (SIMD) instruction set to accelerate computation, thus failing to maximize the parallel computing capabilities of microcontrollers.

[0060] Therefore, there is an urgent need for a low-bit neural network inference computing library that can support lower bit width and higher instruction efficiency for microcontroller platforms. This paper proposes an innovative low-bit inference method for microcontrollers, which supports 4-bit and 2-bit mixed precision convolution kernel configurations. Through deep collaborative optimization of the computation-storage path, it significantly improves computational efficiency and reduces storage requirements, thereby enabling the efficient deployment of complex neural networks on embedded devices.

[0061] Example 1

[0062] like Figure 1 As shown, this embodiment discloses a low-bit neural network inference method for microcontrollers, which includes the following steps:

[0063] S1: After the convolutional neural network is trained, its weight data and input data are obtained, and the weight data and input data are sub-byte quantized to obtain quantized weight data and input data.

[0064] In this embodiment, during neural network inference calculation, the input data is the raw data received by the neural network, such as images and text, which is acquired by sensors or upper-level modules and stored in SRAM or the internal cache of the microcontroller; the weight data is the parameters obtained after training the neural network model, which can be obtained by parsing the model at the training end, deployed to the microcontroller MCU, and stored in Flash / ROM.

[0065] The existing quantization method is used to perform sub-byte quantization on the weight data and input data, which will not be elaborated further in this embodiment.

[0066] S2: Call the inference computing library interface in the microcontroller firmware, pass the quantized weight data and input data into the inference computing library for execution, and output the final convolution result.

[0067] In this embodiment, the present invention proposes a low-bit mixed-precision convolution computation library (i.e., a low-bit neural network inference computation library) for microcontroller platforms. Its implementation fully utilizes the instruction set of the ARM Cortex-M series processors. The following steps take 4-bit convolution computation as an example. This computation library is specifically designed for neural network inference in embedded environments, supporting weight and activation operations at sub-byte quantization (i.e., 4-bit and 2-bit) precision, and can achieve efficient pure integer quantization inference on resource-constrained microcontrollers.

[0068] like Figure 2 As shown, the core of the neural network inference computation library consists of a complete set of convolution kernel functions, supporting arbitrary combinations of 4-bit and 2-bit weights, inputs, and output activations, and is compatible with various quantization strategies (such as Per-Layer and Per-Channel). The computation library includes two modules: neural network function modules and neural network auxiliary functions.

[0069] The neural network function set includes convolution functions for different quantization precipitates. Each convolution function is an optimized and efficient implementation, capable of running low-bit-precision convolution calculations on microcontrollers. The simple and flexible API interface allows these convolution functions to be easily adapted and embedded into various machine learning frameworks. This module allows users to easily deploy low-bit-quantized neural network models directly to target platforms. Neural network auxiliary functions are used to implement key operations such as bit decoding, bit packing, format conversion, and storage compression of low-bit data. These functions are specifically optimized for different data bit widths, supporting efficient data alignment and memory access, ensuring that data channel conversion during convolution calculations does not become a performance bottleneck.

[0070] Neural network function calls are common operations in neural network computation libraries, such as convolution, pooling, and fully connected functions. In this embodiment, the neural network inference computation library mainly includes convolution operations with different bit precisions, which can be directly called to implement neural network inference. Neural network auxiliary functions provide underlying support for neural network function calls, including data preprocessing, format conversion, computation, and quantization operations. These are integrated into the internal implementation of the neural network function calls and are invoked when the neural network function is executed.

[0071] In this embodiment, the neural network auxiliary function designs a set of efficient compression and decompression mechanisms compatible with microcontroller architecture, namely, a subbyte data stream compression and decompression method, to meet the storage and computation requirements of subbyte quantized data (such as 4 bits and 2 bits) on the embedded platform.

[0072] like Figure 3 As shown, the compression method of this invention uses little-endian byte order storage. That is, low-bit data compressed and stored in uint8 type data blocks is arranged in little-endian byte order. Furthermore, in convolution calculations, scaling of accumulated values, bit-width write-back, and output packing operations also use little-endian byte order compression. 4-bit quantized data is compactly stored in uint8 type data blocks using little-endian byte order. Whether used for convolution weights or activation values, it is compressed according to the structure of each uint8 data block containing two 4-bit elements. 2-bit quantized data is stored with each uint8 data block containing four 2-bit elements, also following little-endian byte order. This compact storage method effectively reduces memory usage and improves storage and bandwidth utilization, making it particularly suitable for resource-constrained microcontroller environments.

[0073] Before the neural network function performs convolution calculations, the low-bit neural network inference library uses specially designed decompression and data alignment functions to restore 4-bit or 2-bit quantized data into a 16-bit signed integer format that can be used for computation. This is to adapt to the SIMD (Signed Multiply with Addition Dual) instruction in the ARM Cortex-M architecture. The SMLAD instruction requires the input data to be in a 2×16-bit vector format. Therefore, the original subbyte data must be decompressed into this structure before it can participate in parallel multiply-accumulate operations.

[0074] The decompression function extracts multiple 4-bit or 2-bit data stored in a uint8 data block. Its decompression process is optimized at the instruction level for different bit-width data arrangements. In other words, it fully utilizes some low-level, high-efficiency bit manipulation instructions in the ARM processor instruction set to achieve efficient parallel unpacking and restoration of low-bit data stored in the uint8 data block, adapting to the data format required for subsequent vectorization and parallel computation. This includes logical right shift, bitmasking, and byte extension instructions, as discussed later. Combining basic ARM instructions such as Logical Shift Right (LSR), bitwise AND, and byte extension instructions (Sign / Unsigned Extend Byte to Halfword, SXTB / UXTB), it extracts two 4-bit numbers from a uint8 data block at once and restores them to a 2×16 bit vector. For 2-bit data, due to the higher data density, a uint8 data block is parsed into four 2-bit values ​​and restored to two 2×16 bit vectors, meeting the data format requirements for parallel convolution computation.

[0075] Further, such as Figure 5 As shown, this mixed instruction is used multiple times for 4-bit and 2-bit data: Out=_UXTB16(_ROR(_SXTB16(in),x)), where x represents different right shift bits. For 4-bit data, this mixed instruction is used 4 times to extract and convert 4 2x16 bit vectors from a 32-bit data block (containing 8 4-bit data). For 2-bit data, this mixed instruction is used 8 times to extract and convert 8 2x16 bit vectors from a 32-bit data block (containing 16 4-bit data).

[0076] The data alignment function rearranges the extracted low-bit data to conform to the correspondence between the input feature map and the convolution kernel in the convolution calculation.

[0077] In this embodiment, during the multiplication-accumulation stage of the convolution kernel, considering the limited bit width of sub-byte data, directly loading it into a standard 2x16-bit multiplication-accumulation unit would result in significant memory redundancy, making it difficult to effectively utilize SIMD computing resources. Therefore, this invention expands and restructures the low-bit-width data to improve data utilization and execution efficiency during the multiplication-accumulation stage, and designs a sub-byte data convolution calculation method within the neural network auxiliary function.

[0078] like Figure 4As shown, through the analysis of 32-bit multiplication, after decompressing the activation data (i.e., input data / input feature map), the low-bit neural network inference library first generates two sets of standard-format 2×16-bit input vectors based on the activation data. Then, using a staggered concatenation strategy, the data in these two sets of vectors are rearranged and combined into a new 2×16-bit input vector, containing four consecutive sub-byte data elements. Simultaneously, the weight data, after decompression, forms a 2×16-bit vector containing two elements, constituting the weight input for the current convolution window. The convolution calculation involves multiplying and accumulating the activation data and weight data.

[0079] Furthermore, the data in the two sets of standard 2×16-bit input vectors are rearranged and combined into a new 2×16-bit input vector, such as... Figure 7 In the data concatenation part, the first 2×16 input vector remains unchanged, and the second 2×16 input vector is shifted to the left by a certain number of bits and added to the first 2×16 input vector to form a new 2×16-bit input vector.

[0080] During the multiplication-accumulation process, the dual-channel multiplication feature of the SMLAD instruction is utilized to perform four multiplication operations on the four activation data and two weight data, and to complete the parallel accumulation of the two sets of product results within a single instruction. This operation significantly improves the effective computational density of a single multiplication-accumulation instruction. To obtain the final output feature mapping value, the low-bit neural network inference library further splits and reassembles the accumulation result through post-processing logic, extracting two independent sets of accumulated output values, corresponding to intermediate results from two output channels or two spatial locations, respectively. Specifically, as shown... Figure 7 As shown, according to the calculation rules of the SMLAD instruction and the preceding data alignment, bits 0-7 and 11-19 of the 32-bit calculation result are partial results of the valid data in the output feature map. After calculation, these are extracted and stored in 32-bit wide data blocks. Then, after multiple iterations of the same multiplication and accumulation calculation, the final output feature map data is obtained.

[0081] The internal steps of the inference computation library are executed by the central processing unit, specifically:

[0082] (1) Input the quantized weight data and input data into the neural network function, and call the neural network auxiliary function to preprocess them to obtain the weight vector and input vector, i.e., the standardized vector. The weight vector and input vector are temporarily stored in SRAM for subsequent convolution calculation.

[0083] In this embodiment, a neural network auxiliary function is invoked to preprocess the input feature map (input data). Preprocessing includes decompression and data alignment. The neural network auxiliary function converts the input feature map into a data layout suitable for matrix multiplication to construct an efficient convolution unrolling mechanism. Traditional image unrolling is a column-to-column (im2col) operation, which suffers from low processing efficiency on microcontroller platforms due to frequent memory access and high data redundancy. Therefore, this invention proposes an im2colQ operation path optimized for low-bit quantization inference, achieving an integrated design of quantization format alignment and vectorization preprocessing while completing the convolution window unrolling.

[0084] like Figure 5 As shown, the neural network auxiliary function extracts qubit-level Q-precision data from the input tensor `tensorIn`, accessing and unpacking it in 32-bit blocks. In a 4-bit quantization configuration, each 32-bit word contains 8 compressed input elements; in a 2-bit configuration, each 32-bit word contains 16 input elements. Through precisely controlled data alignment and decoding logic, the compressed input data can be efficiently restored to a vector format that can directly participate in multiplication and addition operations. To further improve data transfer efficiency, the `im2colQ` module employs a double-buffering mechanism (`ptrBuff0` and `ptrBuff1`) to achieve parallel expansion of two adjacent output space locations, supporting batch data loading and pipelined processing. During data loading, quantization zero-point correction, sign extension, and 2x16-bit vector format packaging are completed, avoiding repeated decoding and format conversion in subsequent stages.

[0085] (2) The neural network function calls the calculation instructions based on the weight vector and the input vector to perform parallel convolution calculation and obtain the convolution result; the intermediate results of the operation are stored in SRAM.

[0086] In this embodiment, the calculation instruction is the SMLAD multiply-accumulate instruction. The module responsible for performing matrix multiplication and accumulation operations in the backbone convolution is the core of the invention. While supporting low-precision quantization weights such as 4-bit and 2-bit, to balance execution efficiency and computational accuracy, this module designs an efficient unpacking (UnpackQ) and vector expansion mechanism for weights of different bit widths, and tightly integrates them into the main multiply-accumulate loop (AccLoop), constructing a convolution calculation flow adapted to embedded processors.

[0087] like Figure 6As shown, for 4-bit precision weight data (weight parameters), a 32-bit compressed word containing eight 4-bit quantized weights is loaded from the compressed weight tensor `tensorW` each time. The decompression process is efficiently implemented using three instructions: first, the target weights are aligned using a logical right shift (LSR) operation; second, low-bit data is extracted using a logical AND mask; and finally, the 4-bit integers are expanded to 16-bit format using an unsigned extension instruction (UXTB16). For 2-bit precision weights, the processing flow is similar, except that a different bitmask and data density control logic are used during decompression. Each 32-bit compressed word contains a maximum of 16 2-bit weight values.

[0088] Taking 4-bit convolution as an example, in the traditional path, each SMLAD instruction can only process the multiplication of two 16-bit signed integers simultaneously and accumulate the result to a 32-bit register. Although only the lower 4 bits of the input actually carry valid information, this results in memory redundancy and underutilization of computational units in data representation. Therefore, as... Figure 7 As shown, a one-step vector packing optimization is introduced in the preprocessing stage. Through shifting and concatenation, multiple 4-bit input values ​​are combined into a structured 2x16-bit vector. The sign bits of In2_ch5 and In2_ch1 at the beginning of the 16-bit vector are automatically supplemented during the calculation. This allows a single SMLAD instruction to complete what was originally two multiplications and accumulations into four effective multiplications and accumulations while maintaining architectural compatibility. These accumulations correspond to intermediate results from two output channels or two spatial locations, as shown below. Figure 8 As shown.

[0089] Furthermore, for application scenarios with smaller weight data sizes, the neural network auxiliary function also supports a weight preprocessing strategy: during the neural network model generation stage, the weights are pre-processed offline by unpacking, 2x16-bit vectorization, and sorting, and then deployed to Flash or on-chip SRAM in a structured storage manner. This strategy effectively avoids the dynamic decoding and sorting overhead within the loop during convolution, making the runtime decompression operation close to zero overhead, thereby further improving the overall convolution operation speed. It is particularly suitable for the real-time response requirements of low latency and low power consumption in edge AI scenarios.

[0090] (3) Call the neural network auxiliary function to compress the convolution result, obtain the compressed convolution result and output it. Compress the convolution result to facilitate input or storage in the next layer. Write the compressed convolution result back to SRAM or Flash.

[0091] In this embodiment, after the calculation is completed, the convolution output is still temporarily stored in the buffer as a 32-bit integer. To adapt to storage and communication requirements, it needs to be quantized and compressed again. This invention introduces the Compress module to perform scaling, bit width write-back, and output packing operations on the accumulated value. This module supports two quantization strategies: Per-Layer (uniform scaling) and Per-Channel (channel-independent scaling). During the quantization process, the original 32-bit accumulated value is linearly transformed according to the corresponding scaling factor and zero-point offset, realizing the mapping from the integer accumulated value to the low-bit integer output. This invention uses integer multiplication and right shift to approximate floating-point division, and the scaling factor is expressed in fixed-point form (…). The expression is represented as ) and pre-converted into multiplication and shift parameters, so that the scaling process can be completed at runtime using only integer multiplication and logical right shift, which significantly reduces computational overhead.

[0092] To ensure that the quantized values ​​do not overflow or experience abrupt changes in precision, the Compress module also introduces saturating arithmetic instructions, SSAT (Signed Saturate), and USAT (Unsigned Saturate) from the ARM instruction set. These instructions are used to truncate the bit width and impose upper and lower bound constraints on the results, ensuring that the final output value always falls within the target Q-bit range, thus improving system stability and numerical safety. The compressed output data is then ordered and packaged according to the set bit width (e.g., 8-bit, 4-bit, or 2-bit) and written to the output tensor `tensorOut`.

[0093] The pseudocode for quantized convolutional layers is shown in Table 1:

[0094] Table 1. Pseudocode for quantized convolutional layers

[0095]

[0096] Example 2

[0097] This embodiment discloses a low-bit neural network inference system for a microcontroller, comprising:

[0098] The quantization model acquisition module is configured to: after the training of the convolutional neural network is completed, acquire its weight data and input data, and perform sub-byte quantization on the weight data and input data to obtain quantized weight data and input data;

[0099] The quantization model execution module is configured to: call the inference computing library interface in the microcontroller firmware, pass the quantized weight data and input data into the inference computing library for execution, and output the final convolution result;

[0100] The internal steps of the inference computation library are executed by the central processing unit, specifically:

[0101] The quantized weight data and input data are input into the neural network function, and the neural network auxiliary function is called to preprocess them to obtain the weight vector and input vector.

[0102] The neural network function calls computation instructions based on the weight vector and input vector to perform parallel convolution calculations and obtain the convolution result.

[0103] The neural network auxiliary function is called to compress the convolution result, and the compressed convolution result is output. The neural network auxiliary function uses little-endian byte order for preprocessing and compression.

[0104] Example 3

[0105] The purpose of this embodiment is to provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method of Embodiment 1.

[0106] Example 4

[0107] The purpose of this embodiment is to provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the method of Embodiment 1.

[0108] The steps and methods involved in the apparatuses of Embodiments 3 and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0109] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0110] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0111] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A low-bit neural network inference method for microcontrollers, characterized in that, include: After the convolutional neural network is trained, its weight data and input data are obtained. The input data is an image or text. The weight data and input data are then subjected to sub-byte quantization to obtain quantized weight data and input data. The microcontroller firmware calls the inference computing library interface, passes the quantized weight data and input data into the inference computing library for execution, and outputs the final convolution result. The internal steps of the inference computation library are executed by the central processing unit, specifically: The quantized weight data and input data are input to the neural network function, and the neural network auxiliary function is called to preprocess them to obtain the weight vector and input vector. The preprocessing includes decompression and data alignment, which is implemented using the im2colQ operation with quantization. The im2colQ operation with quantization is specifically as follows: extracting qubit-precision data from the input data, accessing and unpacking the data in 32-bit blocks, calling the decompression and data alignment functions in the neural network auxiliary function, and restoring the input data to the corresponding vector format for convolution calculation; the decompression process is optimized at the instruction level for different bit width data arrangement methods. The neural network function calls computation instructions based on the weight vector and input vector to perform parallel convolution calculations and obtain the convolution result. The neural network auxiliary function is called to compress the convolution result, and the compressed convolution result is output. The neural network auxiliary function uses little-endian byte order for preprocessing and compression.

2. The low-bit neural network inference method for microcontrollers as described in claim 1, characterized in that, The parallel convolution calculation specifically involves: generating two sets of standard format input vectors based on the input vector; using a staggered concatenation strategy to rearrange and combine the data in the two sets of input vectors into a new set of input vectors; and multiplying and accumulating the new input vector with the weight vector.

3. The low-bit neural network inference method for microcontrollers as described in claim 2, characterized in that, The post-processing logic splits and reassembles the accumulated result, extracting two independent sets of accumulated output values, which correspond to the intermediate results of the two output channels respectively.

4. The low-bit neural network inference method for microcontrollers as described in claim 1, characterized in that, During the compression process, integer multiplication and right shift are used to approximate floating-point division. The scaling factor is represented in fixed-point form and pre-converted into multiplication and shift parameters.

5. A low-bit neural network inference system for a microcontroller, characterized in that, include: The quantization model acquisition module is configured to: after the training of the convolutional neural network is completed, acquire its weight data and input data, wherein the input data is an image or text, and perform sub-byte quantization on the weight data and input data to obtain quantized weight data and input data; The quantization model execution module is configured to: call the inference computing library interface in the microcontroller firmware, pass the quantized weight data and input data into the inference computing library for execution, and output the final convolution result; The internal steps of the inference computation library are executed by the central processing unit, specifically: The quantized weight data and input data are input to the neural network function, and the neural network auxiliary function is called to preprocess them to obtain the weight vector and input vector. The preprocessing includes decompression and data alignment, which is implemented using the im2colQ operation with quantization. The im2colQ operation with quantization is specifically as follows: extracting qubit-precision data from the input data, accessing and unpacking the data in 32-bit blocks, calling the decompression and data alignment functions in the neural network auxiliary function, and restoring the input data to the corresponding vector format for convolution calculation; the decompression process is optimized at the instruction level for different bit width data arrangement methods. The neural network function calls computation instructions based on the weight vector and input vector to perform parallel convolution calculations and obtain the convolution result. The neural network auxiliary function is called to compress the convolution result, and the compressed convolution result is output. The neural network auxiliary function uses little-endian byte order for preprocessing and compression.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of a low-bit neural network inference method for a microcontroller as described in any one of claims 1-4.

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of a low-bit neural network inference method for a microcontroller as described in any one of claims 1-4.

Images