Microcontroller-oriented neural network model compression method and device

Abstract

The invention discloses a microcontroller-oriented neural network model compression method and device, and belongs to the technical field of neural networks. The method comprises the steps that image sample data are input into a neural network model, the neural network model is trained, in the training process, the neural network model conducts convolution sampling processing on the input image sample data to obtain a feature matrix X, and global feature information is extracted from the feature matrix X through MaSSA and CGFFN; obtaining the minimum pruning unit of the neural network model, calculating the sparse sensitivity of each minimum pruning unit under different sparseness, and generating a sparse sensitivity matrix; according to the sparse sensitivity matrix, taking a minimum pruning unit as a unit, dividing the whole pruning process into a plurality of stages, and performing adaptive pruning on the neural network model stage by stage; and performing full-integer quantization on the neural network model after adaptive pruning.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of neural network technology, and in particular to a neural network model compression method and device for a microcontroller. Background Art

[0002] Micro-machine learning refers to machine learning performed on microcontroller units (MCUs) with less than 1W of power consumption. It offers advantages such as low power consumption, low cost, pluggability, and strong privacy, and has broad real-world application scenarios. However, MCUs have limited computing power and memory resources, which restricts the deployment and execution of large neural network models. Furthermore, as micro-machine learning evolves, its tasks become more complex, placing higher demands on model performance.

[0003] Traditional micro-machine learning efforts primarily focus on deployment-level optimization, such as operator optimization, typically by scaling down conventional models for deployment. Micro-machine learning typically prioritizes Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) when selecting models. Existing technologies simply stacking CNNs and ViTs fails to effectively combine the strengths of the two structures. Therefore, for complex tasks, models deployed on MCUs may not meet the requirements.

[0004] Furthermore, in common neural network model compression, conventional CNNs are typically stacked convolutional layers, with one or more layers selected for pruning during the pruning process. However, to meet task requirements, as the model structure becomes more complex, with more internal computational branches and different unit functions, their sparsity sensitivity to pruning also varies. If a single layer is still pruned using the same sparsity parameters, model performance may be significantly degraded. Summary of the Invention

[0005] The embodiments of the present application provide a neural network model compression method and device for a microcontroller, which can solve the problem that traditional micro machine learning mainly focuses on optimization at the deployment level. For complex tasks, the model deployed on the MCU may not meet the task requirements. Moreover, in terms of model compression, using the same sparsity pruning on a single layer will lead to a decrease in model performance.

[0006] In order to solve the above technical problems, this application is implemented as follows:

[0007] In a first aspect, a neural network model compression method for a microcontroller is provided, comprising:

[0008] Input the image sample data into the neural network model and train the neural network model. During the training process, the neural network model performs convolution sampling processing on the input image sample data to obtain a feature matrix X, and uses a separable self-attention mechanism MaSSA based on multi-hidden word matching and a context-gated forward propagation network mechanism CGFFN to extract global feature information from the feature matrix X, wherein the global feature information includes: feature information in the word dimension and the channel dimension;

[0009] Obtaining the minimum prunable unit of the neural network model, calculating the sparse sensitivity of each of the minimum prunable units at different sparsities, and generating a sparse sensitivity matrix;

[0010] According to the sparse sensitivity matrix, the entire pruning process is divided into multiple stages with the minimum prunable unit as the unit, and the neural network model is adaptively pruned stage by stage;

[0011] The neural network model after adaptive pruning is fully integer quantized.

[0012] In a second aspect, a neural network model design device for a microcontroller is provided, comprising:

[0013] A neural network model construction module is used to input image sample data into the neural network model and train the neural network model. During the training process, the image sample data input into the neural network model is subjected to convolution sampling processing to obtain a feature matrix X, and global feature information is extracted from the feature matrix X using a separable self-attention mechanism MaSSA based on multi-hidden word matching and a context-gated forward propagation network mechanism CGFFN, wherein the global feature information includes feature information in the word dimension and the channel dimension;

[0014] A sparse sensitivity evaluation module is used to obtain the minimum prunable unit of the neural network model, calculate the sparse sensitivity of each of the minimum prunable units under different sparsities, and generate a sparse sensitivity matrix;

[0015] A model adaptive pruning module is used to divide the entire pruning process into multiple stages based on the sparse sensitivity matrix and take the minimum prunable unit as the unit, and adaptively prune the neural network model stage by stage;

[0016] The model quantization module is used to perform full integer quantization on the neural network model after adaptive pruning to generate a sparse model.

[0017] In a third aspect, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the above-mentioned neural network model compression method for microcontrollers.

[0018] In a fourth aspect, a readable storage medium is provided, in which at least one computer program is stored. The computer program is loaded and executed by a processor to implement the above-mentioned neural network model compression method for microcontrollers.

[0019] In a fifth aspect, a computer program product is provided, which includes at least one computer program, and the computer program is loaded and executed by a processor to implement the microcontroller-oriented neural network model compression method provided in the various optional implementations mentioned above.

[0020] The microcontroller-oriented neural network model compression method, device, electronic device, readable storage medium and computer program product provided by the embodiment of the present application effectively combine MaSSA and CGFFN through a gating mechanism, greatly improving the feature extraction capability of the neural network model and improving the model performance; accurately evaluate the sparsity sensitivity of convolution and fully connected operators of different layers and different functions to pruning, providing a basis for subsequent model adaptive pruning, so as to maximize the model performance while compressing the model storage occupancy. In the subsequent pruning process, the smallest prunable unit with lower sparsity sensitivity is preferentially selected for pruning, so that the compression ratio of the entire model after pruning can be maximized while maintaining the highest possible accuracy; in the adaptive pruning process, the sparsity is gradually increased in stages to reduce the impact of one-time pruning on the model performance, and the storage occupancy required under the hardware constraints of the microcontroller is used as the termination condition. While compressing the storage occupancy of the model, the model performance is maximized under the hardware constraints, so that the model maintains relatively stable performance during the pruning process. It overcomes the limitations of deploying neural network models on microcontrollers and makes it possible to deploy high-performance neural network models on resource-constrained devices.

[0021] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0023] Figure 1 A flowchart of a neural network model compression method for a microcontroller provided by an exemplary embodiment of the present application is shown;

[0024] Figure 2 A flowchart of another microcontroller-oriented neural network model compression method provided by an exemplary embodiment of the present application is shown;

[0025] Figure 3 A flowchart of another microcontroller-oriented neural network model compression method provided by an exemplary embodiment of the present application is shown;

[0026] Figure 4 A flowchart of another microcontroller-oriented neural network model compression method provided by an exemplary embodiment of the present application is shown;

[0027] Figure 5 A flowchart of adaptively pruning a neural network model stage by stage is shown according to an exemplary embodiment of the present application;

[0028] Figure 6 A flowchart of determining the expected sparsity value of each minimum prunable unit to be pruned at the current stage provided by an exemplary embodiment of the present application is shown;

[0029] Figure 7 A flowchart of performing full integer quantization on a neural network model after adaptive pruning is shown according to an exemplary embodiment of the present application;

[0030] Figure 8 A block diagram of a neural network model compression device for a microcontroller provided by an exemplary embodiment of the present application is shown;

[0031] Figure 9 A structural block diagram of an electronic device provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0032] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0033] In order to solve the problem that the existing technology based on traditional micro machine learning mainly focuses on optimization at the deployment level, for complex tasks, the model deployed on the MCU may not meet the task requirements, and in terms of model compression, using the same sparsity pruning on a single layer will lead to a decrease in model performance, the embodiment of the present application provides a neural network model compression method and device for a microcontroller.

[0034] First, the terms involved in this application are explained.

[0035] Separable Self-Attention (SSA): This mechanism automatically learns the correlations between different positions when processing sequential data, thereby capturing richer contextual information. Specifically, for each element in the sequence, the correlations with all other elements are calculated and a weighted vector is generated based on this correlation. This weighted vector can be regarded as a representation of the element in the current context.

[0036] Gate: In neural networks, gating is a common technique used to dynamically regulate the flow of information. A gate value is typically a scalar or vector, typically ranging from 0 to 1. When the gate value is close to 1, the corresponding feature is allowed to pass smoothly; when the gate value is close to 0, the corresponding feature is suppressed and has difficulty passing.

[0037] Pruning: It is a model optimization technique that achieves model sparsity by gradually reducing the minimum prunable unit of the model to zero during training.

[0038] Model quantization refers to converting a neural network's floating-point arithmetic to fixed-point arithmetic. Low-precision models represent model weight values ​​in the FP16 (half-precision floating point) or INT8 (8-bit fixed-point integer) format. Regular-precision models typically represent model weight values ​​in the FP32 (32-bit floating point, single precision) format. Mixed-precision models use both FP32 and FP16 weight value formats. FP16 reduces memory usage by half, but some parameters or operators must use the FP32 format to maintain accuracy.

[0039] Post-training Quantization (PTQ): This method converts a trained FP32 network directly into a fixed-point network. This process eliminates the need for any training of the original model and only requires adjustments to a few hyperparameters, significantly improving the model's inference speed.

[0040] The neural network model compression method and device for microcontrollers provided in the embodiments of the present application can be applied to a microcontroller MCU, which can be applied to electronic devices such as electronic doorbells, smart door locks, smart lights, smart air conditioners, smart meters or sensors, etc., without specific limitations. The application scenarios of the above methods and devices include scenarios where neural network models are deployed on microcontrollers. The neural network models involved mainly include: Convolutional Neural Networks (CNN), Transformers or ViT (Vision Transformer, visual transformer), etc.

[0041] Figure 1 FIG. 1 shows a flowchart of a microcontroller-oriented neural network model compression method according to an exemplary embodiment of the present application. Figure 1 As shown, the neural network model compression method for microcontrollers is applied to the microcontroller and specifically includes the following steps (S101-S104):

[0042] S101, inputting image sample data into a neural network model and training the neural network model. During the training process, the neural network model performs convolution sampling processing on the input image sample data to obtain a feature matrix X, and uses a separable self-attention mechanism MaSSA based on multi-hidden word matching and a context-gated forward propagation network mechanism CGFFN to extract global feature information from the feature matrix X, where the global feature information includes feature information in the word dimension and the channel dimension;

[0043] In the embodiments of the present application, the neural network model uses a CNN-ViT hybrid model structure, hereinafter referred to as MatchViT. This model structure includes convolutional layers and ViT units. Convolutional neural networks can extract local features from input image sample data through convolution operations, while ViT is better at extracting long-range feature dependencies.

[0044] In the ViT unit, the Matching Separable Self-Attention (MaSSA) mechanism based on word matching and the Context Gated Feedforward Network (CGFFN) mechanism are used to enable the MatchViT model to extract global feature information more efficiently and comprehensively, further improving the performance of the model deployed on the microcontroller. In an embodiment of the present application, the image sample data is input into the neural network model, and the neural network model is trained to obtain a dense model with high performance, small number of parameters, and low computational complexity, which is used for compression in subsequent processes.

[0045] As an optional implementation in the embodiment of the present application, the implementation of step S101 is described below.

[0046] like Figure 2 As shown, in another microcontroller-oriented neural network model compression method provided by an embodiment of the present invention, during the training process, the neural network model performs convolution sampling processing on the input image sample data to obtain a feature matrix X, and uses a separable self-attention mechanism MaSSA based on multi-hidden word matching and a context-gated forward propagation network mechanism CGFFN to extract global feature information from the feature matrix X. Specifically, the following steps (S201-S205) may be included:

[0047] S201, the neural network model downsamples the input image sample data by a factor of 2 through a 3×3 convolution and an inverse residual block with an expansion factor of 2 to obtain data to be processed with an increased number of channels;

[0048] In the embodiment of the present application, the neural network model uses the hybrid model structure of MatchViT, which consists of three parts: a convolution pre-processing module, a backbone structure, and a post-processing module.

[0049] In some embodiments, the convolution pre-processing module includes a 3×3 convolution and a reverse residual block with an expansion factor of 2. Both components downsample the input image sample data by a factor of 2 to reduce the resolution of the image data feature map and increase the number of channels of the image data feature map, which helps to extract richer feature information while reducing the amount of data.

[0050] S202, after downsampling the data to be processed by 2 times, encoding the position information through separable convolution to obtain a feature matrix X;

[0051] In some embodiments, the MatchViT backbone structure consists of four stages. Each stage starts with a 2x downsampled inverse residual block, followed by a MatchViT block that learns features from the global context, so that the resolution of the feature matrix X gradually decreases while the number of channels gradually increases, forming a pyramid structure. The MatchViT block is similar to the ViT block. It starts with a separable convolution, which is composed of a 3×3 depth-wise convolution stacked with a 1×1 point-wise convolution. After the separable convolution encodes the position information, the feature matrix X is obtained.

[0052] S203, using the separable self-attention mechanism MaSSA based on multi-hidden word matching to extract feature information on the word dimension from the feature matrix X;

[0053] S204, using the context gated feedforward network mechanism CGFFN, and based on the feature information on the word dimension, extracting global feature information from the feature matrix X;

[0054] In some embodiments, the feature matrix X is passed through multiple ViT encoders to extract global feature information of the feature matrix X. In this embodiment of the present application, each ViT encoder consists of a MaSSA module and a CGFFN module, connected using a reverse residual block. The expansion factor of the reverse residual block and the forward propagation network is set to 2.

[0055] S205: Perform global average pooling on the global feature information, and perform linear mapping classification through a classifier.

[0056] In some embodiments, the post-processing part of MatchViT includes global average pooling and a final classifier, which performs global average pooling on the global feature information and finally obtains the classification result through linear mapping.

[0057] The above specific implementation of step S101 is described below with a simple application example.

[0058] Convolution pre-processing module: The input image size is 224×224×3 (height×width×number of channels). After 3×3 convolution (with a stride of 2), the feature map resolution is reduced to 112×112, and the number of channels increases according to the number of convolution kernels, assuming it increases to 64. Then, after a reverse residual block with an expansion factor of 2 (also performing 2x downsampling), the feature map resolution is further reduced to 56×56, and the number of channels is increased again, for example, to 128.

[0059] The backbone architecture consists of four stages. Taking the first stage as an example, the input is a 56×56×128 feature map obtained by the convolutional pre-processing module. First, through a 2x downsampling inverse residual block, the feature map resolution is reduced to 28×28, and the number of channels is expanded to 256 (expansion factor 2). Next, the MatchViT block encodes the positional information through separable convolution, resulting in a feature matrix X, which then enters the ViT encoder. Assume that each stage's MatchViT block has two ViT encoders. In the first ViT encoder, MaSSA processes the feature matrix X to extract feature information at the word-unit dimension. CGFFN then uses this information extracted by MaSSA to further enhance the feature representation and extract global feature information from the feature matrix X. After two ViT encoders, the output feature map of this stage is 28×28×256. Subsequent stages repeat a similar process, with the feature map resolution further decreasing (e.g., to 14×14 after the second stage and 7×7 after the third stage) and the number of channels increasing accordingly (e.g., to 512 after the second stage and 1024 after the third stage).

[0060] Post-processing module: After the backbone architecture, the feature map is sized at 7×7×1024. After global average pooling, the feature map is compressed into a one-dimensional vector of length 1024. This vector is then fed into the final classifier, which maps it to the category space through a linear transformation (e.g., using a fully connected layer). For a 10-category classification task, the final output is a vector of length 10, with each element representing the score for the corresponding category. The softmax function is used to obtain the probability of each category, thereby determining the classification result.

[0061] As an optional implementation in the embodiment of the present application, the implementation of step S203MaSSA is described below.

[0062] In the implementation of this application, based on the separable self-attention mechanism, a separable self-attention mechanism based on word unit matching is proposed, and the separable self-attention mechanism MaSSA based on multi-hidden word unit matching is used to extract feature information on the word unit dimension of the feature matrix X.

[0063] The following is a brief introduction to the separable self-attention mechanism.

[0064] The separable self-attention (SSA) mechanism reduces the computational complexity of self-attention in ViT from quadratic to linear. Specifically, SSA splits the input matrix X into three branches through a linear transformation: I, K, and V. I is a vector corresponding to the separated hidden tokens, and K and V represent the key matrix and value matrix, respectively.

[0065] First, each word in I competes to obtain the context score CS, namely:

[0066] CS=Softmax(I) (1)

[0067] The context score can be understood as the importance of each word in representing the global feature.

[0068] Next, calculate the vector distance between the context score in the vector space and all the word units in the key matrix K, and sum them up to get the context vector CV, that is:

[0069]

[0070] Here, * represents element-by-element multiplication, and N is the number of tokens. The context vector can be understood as a weighted sum of the vector groups in the key matrix K, using the context scores as weights. The resulting vector represents the global features of the entire feature map.

[0071] Next, the context vector is shared with all the words in the value matrix after the ReLU function is activated (which can be understood as broadcasting the global features) to obtain the output result, namely:

[0072] Y i,j =CV j *ReLU(V) i,j (3)

[0073] SSA generally follows the process of global feature aggregation-broadcasting, separating the two processes and reducing the computational complexity to linear.

[0074] Based on the separable self-attention mechanism, such as Figure 3 As shown, in another microcontroller-oriented neural network model compression method provided by an embodiment of the present invention, a separable self-attention mechanism MaSSA based on multi-hidden word unit matching is used to extract feature information on the word unit dimension from the feature matrix X, which can specifically include the following steps (S301-S304):

[0075] S301. Generate four MaSSA matrices from the feature matrix X through linear transformation. The MaSSA matrices include: a hidden word unit matrix I, a key matrix K, a value matrix V, and a matching matrix R in the separable self-attention mechanism. The hidden word unit matrix I includes k hidden words, and the matching matrix R represents the word unit feature of each hidden word unit.

[0076] In the embodiments of this application, MaSSA mainly solves two problems in SSA:

[0077] Problem 1: In the separable self-attention mechanism, there is only one hidden word, which leads to limited feature information extracted in a single layer.

[0078] Question 2: After adding multiple hidden word units, it is still necessary to determine the matching relationship between the hidden word units and the output word unit matrix when interacting with them, otherwise redundancy will occur.

[0079] To address the first problem, in this embodiment, the shape of matrix I is expanded to increase the number of hidden tokens to multiple (k hidden tokens). Through matrix multiplication, MaSSA generates a context vector group containing k vectors (in contrast, the context vector in the separable self-attention mechanism only contains one vector), thereby enabling feature extraction at multiple levels.

[0080] To solve the second problem, the MaSSA matrix in the embodiment of the present application introduces a matching matrix R to represent the features required for each word unit to ensure that each hidden tag can selectively interact with the value matrix.

[0081] In addition, the meanings of I, K, and V in the embodiments of the present application are the same as those of the corresponding matrices in the separable self-attention mechanism and will not be repeated here.

[0082] S302. Calculate a context vector group CV based on the hidden word unit matrix I according to the separable self-attention mechanism, perform a dot product between the context vector group CV and the matching matrix R, perform a Softmax operation on the word unit number dimension, and obtain a matching score MS, where the matching score MS represents the amount of information that each word unit in the value matrix V needs to obtain from each hidden word unit in the hidden word unit matrix I;

[0083] Specifically, according to the formula (1) above, each word in I competes to obtain the context score CS, and according to the formula (2) above, each hidden word is weighted and aggregated with the global information of the key matrix K to generate a context vector group CV.

[0084] The context vector group CV is dot-producted with the matching matrix R using the following formula (4), and a Softmax operation is performed on the word unit dimension to obtain the matching score MS:

[0085] MS=Softmax(R·CS T ) (4)

[0086] Thus, the distance between each hidden word in the matrix I and each word in the matching matrix R can be obtained. The matching score represents the most relevant feature selected from the k hidden words for each input word (N).

[0087] S303, performing matrix multiplication on the matching score MS and the context vector group CV to obtain the matched context vector group MCV;

[0088] Specifically, the context vector group MCV is calculated by using the following formula (5) using the matching score MS as the weight and performing weighted summation on the context vector group CS:

[0089] MCV=MS·CV (5)

[0090] S304, the matched context vector group MCV is calculated by weight W S Map to the original dimension and perform element-wise multiplication with the value matrix V to obtain the MaSSA output result, where the original dimension is the dimension of the feature matrix X. The MaSSA output result includes: global feature information on the word dimension.

[0091] Specifically, the matched context features are multiplied element-by-element by the activated value matrix through the following formula (6) to obtain the MaSSA output result Y:

[0092] Y=(MCV·W S)*ReLU(V) (6)

[0093] Among them, Y represents the output result of MaSSA, and ReLU (V) represents the value matrix after activation.

[0094] The specific implementation of step S203 is described below with a simple application example.

[0095] A1. The input feature matrix X is 10×64 (N=10 tokens, C=64 dimensions), and the number of hidden tokens k=4 is set. The input feature matrix X is projected into the following four branches:

[0096] I: 4×64 (k=4), K: 10×64, V: 10×64, R: 10×64;

[0097] A2. Generate context vector group CV:

[0098] CS (Context Score): 4×6, represents the feature importance of 4 hidden words after Softmax.

[0099] CV (Context Vector Group): 4×64, global information of each hidden word aggregation key matrix.

[0100] A3. Matching mechanism:

[0101] MS (matching score): 10×4, each row corresponds to an input word, and the value represents the dependency weight of the word on the four hidden words. For example, the first word might be assigned weights [0.7, 0.1, 0.1, 0.1], indicating that it is primarily dependent on the first hidden word.

[0102] A4. Output generation:

[0103] MCV: 10×64, each word is fused with the most relevant hidden word features.

[0104] Final output: 10×64, through the gating effect of ReLU (V), important local features are retained.

[0105] For example, in the human posture recognition task of smart watches, MatchViT extracts multi-scale joint point features through MaSSA. Compared with the traditional ViT model, the inference speed is increased by 3 times, the memory usage is reduced by 60%, while maintaining a classification accuracy of more than 90%.

[0106] In the embodiment of the present application, MaSSA expands the shape of the matrix I, increasing the number of hidden tokens to multiple (k hidden tokens). Through matrix multiplication, MaSSA generates a context vector group containing k vectors, thereby including multiple levels of feature extraction and completing the feature information extraction of the token number dimension of the feature matrix X. In addition, the MaSSA matrix introduces a matching matrix R, which represents the features required for each token, to ensure that each hidden token can selectively interact with the value matrix, ensuring that no redundancy occurs after adding multiple hidden tokens.

[0107] As an optional implementation in the embodiment of the present application, the implementation of the context-gated forward propagation network CGFFN in step S204 is described below.

[0108] In the ViT network, CGFFN is responsible for extracting features from the feature maps of the feature matrix X at a higher dimension. This is orthogonal to the direction of multi-head self-attention and is a key component of the ViT network. In MaSSA, multi-scale features are aggregated into a context vector group and applied to each word through a matching mechanism. This operation is performed at the word level, while the aggregated feature information is not utilized at the channel level, which is underutilized by conventional FFNs.

[0109] In the embodiment of the present application, the core point of the CGFFN improvement is that the use of context vector groups in the forward propagation network can make fuller use of the multi-scale information extracted from MaSSA, so that the encoder has better performance. The computational flow of CGFFN is divided into two branches, which are combined through a gating mechanism to enhance the model feature extraction capability while slightly increasing the number of model parameters and the amount of computation. These two branches work together to form the ViT encoder together with MaSSA, and then form a complete neural network structure with the convolutional layer.

[0110] like Figure 4 As shown, in another microcontroller-oriented neural network model compression method provided by an embodiment of the present invention, a context-gated forward propagation network mechanism CGFFN is used, and global feature information is extracted from the feature matrix X based on feature information on the word dimension. Specifically, the following steps (S401-S404) may be included:

[0111] S401, the feature matrix X is upgraded to a high-dimensional space by weight W1, and activated by applying SiLU activation function to obtain a high-dimensional feature matrix X up ;

[0112] Step S401 is the first branch of the CGFFN calculation flow. Similar to the conventional forward propagation network, the feature matrix X is upgraded to a high-dimensional space (D) by the weight W1 through the following formula (7) to obtain a high-dimensional feature matrix X up :

[0113] X up =SiLU(X·W1) (7)

[0114] Among them, the input feature matrix is ​​X∈R N×C (N words, C-dimensional features). For example, if C = 4, D = 8, then Xup∈R N×8 .

[0115] S402: Upgrade the context vector group CV to a high-dimensional space using the weight W2 to obtain a high-dimensional context vector group CV. up ;

[0116] Step S402 is the second branch of the CGFFN computation flow. CGFFN reuses the feature information (context vector group CV) obtained from MaSSA as a gating function. This is one of the core improvements of CGFFN, which can dynamically adjust the features in FFN. CGFFN can use the context information of MaSSA and the gating mechanism to influence feature transfer.

[0117] Specifically, the context vector group CV in MaSSA is first mapped to a high-dimensional space through the weight W2 through the following formula (8), and the high-dimensional context vector group CV is obtained. up :

[0118] CV up =CV·W2 (8)

[0119] S403, the feature matrix X is passed through the weight W k Mapping to obtain feature information X k , the feature information X k CV with high-dimensional context vectors up Perform matrix multiplication and activate through SiLU activation function to obtain the gate value Gate;

[0120] Specifically, the gate value Gate is obtained by the following formula (9):

[0121] Gate=SiLU(X·W k CS up ) (9)

[0122] The above calculation process is actually measuring the input feature matrix X and the context vector group CV up A certain degree of correlation between them. Specifically, X·W kProject the feature matrix X into a new space so that it is aligned with the context vector CV up Matrix multiplication operations can be performed on the dimension. The result of the multiplication represents the strength of the association between each input feature word and the context vector. After the SiLU activation function, the obtained Gate value will be distributed between 0 and 1 according to the strength of this association. For example, for the features of a certain channel in the input feature map $X$, if it has a high correlation with the global features represented by the context vector group, then in the calculated Gate value, the element corresponding to this channel will be close to 1, which means that the features of this channel can pass smoothly and be retained for subsequent calculations; on the contrary, if the features of a certain channel have a low correlation with the context vector group, the corresponding Gate value element will be close to 0, the features of this channel will be suppressed, and the role it plays in subsequent calculations will be greatly reduced.

[0123] S404, apply the gate value Gate to the high-dimensional feature matrix X up , and then reduce the dimension to the original dimension through the weight W3 to obtain the CGFFN output result, where the CGFFN output result includes: global feature information.

[0124] After gating, the feature matrix X of the first branch is obtained by the following formula (10): up By reducing the dimension to the original dimension through the weight W3, the output result Y is obtained:

[0125] Y=(X up *Gate)·W3 (10)

[0126] In the embodiments of this application, the core improvements of CGFFN are: (1) Reusing global context: Using the context vector group CV generated by MaSSA as a gating signal to dynamically adjust the features in conventional FFN. (2) Channel dimension enhancement: Using a gating mechanism to fuse global context in the channel dimension, the model's sensitivity to multi-scale features is improved.

[0127] Compared to conventional feedforward networks, CGFFN reuses the context vector group of the MaSSA module, making better use of the multi-scale information extracted by MaSSA. It also uses gating on high-dimensional mapped inputs, further improving the model's feature extraction capabilities while only slightly increasing the number of model parameters and multiplication-addition computations. MaSSA and CGFFN work together to form the ViT encoder, which, together with the convolutional layers, completes the neural network architecture, significantly enhancing the neural network model's feature extraction capabilities and performance.

[0128] S102, obtaining the minimum prunable unit of the neural network model, calculating the sparse sensitivity of each minimum prunable unit under different sparsities, and generating a sparse sensitivity matrix;

[0129] After training the neural network model, the model is further compressed through pruning.

[0130] The traditional CNN model is usually a simple stack of convolutional layers. During the pruning process, it is only necessary to consider the different sparse sensitivities caused by the different depth positions of different convolutional layers in the model. However, the embodiment of the present application adopts the MatchViT model structure. Convolution and full connection operations are used multiple times during the calculation process, and different full connection layers play multiple roles such as mapping context vectors, mapping output feature maps, and mapping context gating. Traditional CNN pruning may only consider the position of the layer, but MatchViT combines convolution and full connection layers, and the pruning effects of different layers are different, so it is necessary to evaluate the sparse sensitivity of each module. Under such conditions, using the method of manually setting pruning parameters may cause a significant decline in model performance during the model pruning process. Therefore, in the embodiment of the present application, for different layers and different functions of convolution and full connection operators (defined as the minimum pruning unit in the embodiment of the present application), their sparse sensitivity to pruning is evaluated, a sparse sensitivity matrix is ​​generated, and it is passed as a result to the model adaptive pruning module for subsequent adaptive pruning.

[0131] Ideally, to prune a minimum pruneable unit in a neural network model, it is necessary to prune as many parameters of the minimum pruneable unit as possible, so that more storage space can be saved after sparse coding, and the degradation of model performance caused by pruning should be as small as possible. Based on the above ideas, in the embodiment of the present application, the sparse sensitivity of each minimum pruneable unit at different sparsities is calculated by the following formula (11):

[0132]

[0133] Among them, sp represents sparsity, m represents the minimum pruning unit, represents the sparse sensitivity of the minimum pruneable unit m under the sparsity sp, acc dense with acc sparse They represent the image classification TOP-1 accuracy of the unpruned model and the model after a single pruning of the minimum pruneable unit m, respectively, param dense with param sparse They represent the storage usage of the unpruned model and the model after a single pruning of the minimum prunable unit m.

[0134] In an embodiment of the present application, the minimum prunable unit includes one of the following: a convolutional layer, a fully connected layer, a weight, and a MaSSA matrix; in actual application, the minimum prunable unit can be a layer in a neural network model (such as a convolutional layer or a fully connected layer), or it can be a specific operator (such as a convolution kernel or a set of weights in a fully connected layer). The specific correspondence is determined according to the evaluation granularity. In actual operation, evaluation is usually performed in layers, but it can also be refined to a more specific operator level.

[0135] In the embodiments of the present application, the neural network model is split into minimum prunable units, such as a single convolutional layer, a fully connected layer, and the matching matrix R in MaSSA. For example, the minimum prunable unit can be a 3×3 depthwise convolution of the inverse residual block, the matching matrix R in MaSSA, or the weights W1, W2, and W3 in CGFFN. Furthermore, to more accurately understand the response of each part of the model to pruning, the minimum prunable unit can be a specific weight.

[0136] In the embodiment of the present application, in order to obtain sparse sensitivity more quickly, one-shot pruning is used when evaluating sparse sensitivity. Specifically, the weight values ​​with lower L1 norm are directly set to zero without fine-tuning.

[0137] From the above formula (11), it can be seen that under the same storage space compression ratio (param dense -param sparse Same), if the accuracy drops more after pruning (acc dense -acc spatse The larger the pruning accuracy, the higher the sparsity sensitivity. This means that the unit is more sensitive to pruning, and pruning the parameters of the unit will have a greater impact on the model performance. dense -acc sparse Same), the storage space compression ratio is higher (param dense -param sparse This indicates that this unit can compress the storage space to a greater extent while maintaining the model performance during pruning, making it a more suitable unit for pruning.

[0138] After traversing the minimum pruning units, a sparse sensitivity matrix Sens is obtained. The sparse sensitivity matrix Sens is a two-dimensional matrix, and the rows represent multiple minimum pruning units. Let M be the set of minimum pruning units, for example, M = {m1, m2, ... m k}, indicating that there are k minimum pruning units. The columns of the sparse sensitivity matrix represent different sparsities, and SP is a sparsity set. For example, in the embodiment of the present application, SP is set to an array with an initial value of 0.5, an end value of 0.7, and a step size of 0.05, that is, SP = {0.5, 0.55, 0.6, 0.65, 0.7}. The elements of the sparse sensitivity matrix Sens i,j Represents the minimum pruning unit m i At the corresponding sparsity sp j The lower the sparsity sensitivity value, the higher the cost-effectiveness of pruning.

[0139] For example, a simplified model contains three minimum prunable units M = {m1, m2, m3}, sparsity set SP = [0.5, 0.55, 0.6], and sparse sensitivity matrix These correspond to the sensitivities of m1, m2, and m3 at different sparsities. For example, in the sparse sensitivity matrix, the sparse sensitivity of the smallest prunable unit m1 at a sparsity of 0.5 is 0.5, meaning it is pruned by 50%.

[0140] In the embodiments of this application, the sparsity sensitivity of convolutional and fully connected operators at different layers and with different functions to pruning is accurately assessed, providing a basis for subsequent adaptive pruning of the model, thereby compressing model storage while maximizing model performance. Therefore, in the subsequent pruning process, the smallest prunable units with lower sparsity sensitivity are preferentially selected for pruning. This can maximize the compression ratio of the entire pruned model while maintaining the highest possible accuracy.

[0141] S103, based on the sparse sensitivity matrix, the entire pruning process is divided into multiple stages with the minimum pruning unit as the unit, and the neural network model is adaptively pruned stage by stage;

[0142] like Figure 5 As shown, as an optional implementation in the embodiment of the present application, according to the sparse sensitivity matrix, the entire pruning process is divided into multiple stages with the minimum prunable unit as the unit, and the neural network model is adaptively pruned stage by stage, including the following steps:

[0143] S501, determining the memory occupancy standard that needs to be achieved under the hardware constraints of the microcontroller, and determining the target memory occupancy of each stage based on the memory occupancy standard, wherein the target memory occupancy is the memory occupancy of the model after pruning in this stage, and decreases linearly with the number of stages until the memory occupancy standard is achieved;

[0144] In an application example, the initial model memory occupancy is S0, and the memory occupancy standard that needs to be achieved under the hardware constraints of the microcontroller is S target, the target storage occupancy of each stage i can be calculated by a linear function

[0145] In the above simplified model, the initial model memory occupation S0 is 100 units, and the memory occupation standard that needs to be achieved under the hardware constraints of the microcontroller is S target The unit is 50, the pruning stage I = 3, and the target storage occupancy S of each stage can be calculated according to the linear function i :

[0146] Stage 1: S1 = 100 - 1 / 3 (100 - 50) = 83.33 units;

[0147] Stage 2: S2 = 100 - 2 / 3 (100 - 50) = 66.67 units;

[0148] Stage 3: S3 = 100 - 3 / 3 (100 - 50) = 50 units.

[0149] In an embodiment of the present application, before performing the pruning operation, as an optional implementation method in the embodiment of the present application, the method provided in the embodiment of the present application also includes: initializing the priority queue pq, and enqueuing the elements corresponding to the initial sparsity values ​​in the sparse sensitivity matrix in order of sparse sensitivity from small to large as members of the priority queue pq.

[0150] Specifically, each row in the sparse sensitivity matrix is ​​regarded as an array, which represents the sparse sensitivity of a minimum prunable unit at different sparsities. For example, Sens 1j It can be the sparse sensitivity of m1 under the sparsity {0.5, 0.55, 0.6, 0.65, 0.7}. Each column in the sparse sensitivity matrix is ​​regarded as an array, representing the sparse sensitivity of the k smallest prunable units under the same sparsity.

[0151] When initializing the priority queue pq, the head element is first popped out of the array (for example, the sparse sensitivity of the k smallest prunable units with sp=0.5) and added to the priority queue.

[0152] For example, the sparse sensitivity matrix The head element of each row is popped out and added to the priority queue. At this time, the elements in the priority queue pq are: The elements in the priority queue pq represent the minimum prunable unit, sparsity, and sparsity sensitivity, respectively.

[0153] S502, dividing the entire pruning process into multiple stages, and adaptively pruning the neural network model stage by stage;

[0154] In each stage, the following pruning operations are performed:

[0155] S5021. Determine the expected sparsity value of each minimum pruneable unit to be pruned in the current stage, wherein the sparsity sensitivity of the minimum pruneable unit to be pruned is low and the model memory occupancy after pruning the minimum pruneable unit to be pruned meets the target memory occupancy of the current stage;

[0156] In an embodiment of the present application, before pruning, the pruning sparsity of each minimum prunable unit in each stage is configured. The logic of the pruning sparsity configuration is to select the minimum prunable unit with the lowest sensitivity and sparsity from the priority queue in sequence, and determine whether the storage occupancy of the model after pruning the selected minimum prunable unit meets the target storage occupancy of this stage. If so, it is used as the minimum prunable unit to be pruned and waits for subsequent pruning. If not, the search continues to meet the target storage occupancy of this stage until all elements in the sparse sensitivity matrix are traversed.

[0157] S5022: For each minimum pruneable unit to be pruned, the pruning of this stage is completed in a manner that the sparsity increases linearly with the number of cycles to an expected sparsity value.

[0158] After completing the sparsity configuration of each unit in each stage, the actual pruning of the neural network model begins.

[0159] During the model pruning process, if all units that need to be pruned are directly set to a higher sparsity at one time, for example, increasing the sparsity of the model from 0 to 0.7 (i.e., pruning 70% of the weights) at one time, it may cause the model performance to drop sharply. Therefore, a more gentle and gradual way to increase the sparsity is needed to give the model a chance to gradually adapt to the changes in weights and maintain relatively stable performance. In the embodiment of the present application, as an optional implementation method in the embodiment of the present application, the pruning of each minimum prunable unit to be pruned is completed in a manner that the sparsity increases linearly with the number of cycles to the expected value of the sparsity, including:

[0160] Each stage is divided into multiple cycles. In each cycle, each minimum prunable unit to be pruned is pruned according to the sparsity within the cycle until the pruning of this stage is completed. The sparsity within the cycle increases linearly with the number of cycles from the initial sparsity value to the expected sparsity value corresponding to the minimum prunable unit to be pruned.

[0161] Specifically, in the adaptive pruning process of each stage, each stage is divided into multiple cycles, and the sparsity is linearly increased from the initial value to the expected value, so that the model can maintain relatively stable performance. For example, if the starting sparsity of the current stage is 0.5 and the target sparsity is 0.7, which is completed in 4 cycles, then the sparsity is increased by 0.05 in each cycle until it reaches 0.7. Gradually improving the sparsity in this way helps the model maintain higher performance during fine-tuning. Compared to obtaining sparse parameters and pruning through neural architecture search, the pruning strategy in the embodiment of the present application takes less time to find sparse parameters, and the performance degradation of the model remains within a controllable range.

[0162] As an optional implementation in the embodiment of the present application, the implementation of step S5021 is described below.

[0163] like Figure 6 As shown, in another microcontroller-oriented neural network model compression method provided by an embodiment of the present invention, determining the expected sparsity value of each minimum prunable unit to be pruned at the current stage can specifically include the following steps (S601-S606):

[0164] S601: Obtain the team member with the lowest current sparsity sensitivity and the corresponding sparsity from the priority queue as the minimum pruning unit to be pruned and the expected sparsity value corresponding to the minimum pruning unit to be pruned;

[0165] S602: Obtain the current model storage occupancy after pruning the minimum prunable unit to be pruned;

[0166] S603: Determine whether the current model storage occupancy meets the target storage occupancy of this stage. If the current model storage occupancy does not meet the target storage occupancy of this stage, execute step S601; if the current model storage occupancy does not meet the target storage occupancy of this stage, execute step S604.

[0167] S604: If the current model storage occupancy meets the target storage occupancy of this stage, record the minimum pruning unit to be pruned and the corresponding expected sparsity value;

[0168] S605. Check whether the minimum pruneable unit to be pruned currently has a corresponding higher-level sparsity in the sparse sensitivity matrix, where the higher-level sparsity is higher than the sparsity corresponding to the minimum pruneable unit to be pruned currently. If so, execute step S606. If not, the pruning sparsity configuration process of the current stage ends.

[0169] S606: The elements corresponding to the high-level sparsity in the sparse sensitivity matrix are queued into a priority queue in ascending order of sparse sensitivity, and the process returns to step S601.

[0170] In steps S604-S605, after a minimum prunable unit is selected as the unit with the lowest sensitivity, the queue corresponding to the minimum prunable unit is checked to see if there is a higher sparsity option. If a higher sparsity option exists, the minimum prunable unit with the higher sparsity is added to the priority queue (and may be pruned at the higher sparsity later). Otherwise, the unit has already reached its maximum sparsity and no further pruning is performed on it.

[0171] In an embodiment of the present application, through the pruning configuration strategy of "sparse sensitivity sorting" + "staged pruning", the global optimal pruning unit (the one with the lowest sparsity sensitivity and meeting the target storage occupancy) is continuously tracked through the priority queue to avoid the local optimal trap, and the pruning unit and sparsity with the least impact on accuracy are automatically selected; the sparsity is gradually increased in stages to reduce the impact of one-time pruning on model performance, and the storage occupancy that needs to be achieved under the hardware constraints of the microcontroller is used as the termination condition. While compressing the storage occupancy of the model, the model performance is maximized under the hardware constraints to adapt to the resource limitations of different microcontrollers.

[0172] Taking the simplified model above as an example, pruning is divided into three stages. The following example illustrates the pruning configuration in an application example.

[0173] The pruning configuration for phase 1 is as follows:

[0174] Extract the smallest prunable unit (m2, 0.5, 0.4) with the lowest sensitivity from the priority queue pq, set the sparsity of m2 to (0.5), and check the current model storage usage. Assume that the model storage usage is 90 units at this time, which does not meet the requirement of S1 = 83.33 units. Continue to extract the smallest prunable unit (m1, 0.5, 0.5) from the priority queue, set the sparsity of m1 to 0.5, and check the current model storage usage again. Assume that it has reached 83 units, meeting the requirements of stage 1.

[0175] Check whether there are higher sparsity options in the queues corresponding to m2 and m1. If both are available, add (m2, 0.55, 0.5) and (m1, 0.55, 0.6) to the priority queue.

[0176] The pruning configuration for phase 2 is as follows:

[0177] Take the smallest prunable unit with the lowest sensitivity from the priority queue pq, assuming it is (m2, 0.55, 0.5), set the sparsity of m2 to 0.55, check the current model storage usage, assuming it is 75 units at this time, which has not yet met the requirement of S2 = 66.67 units. Continue to take the smallest prunable unit from the priority queue and operate until the requirement is met.

[0178] Phase 3 pruning configuration:

[0179] The above process is repeated until the current model storage occupancy reaches the target of S3=50 units.

[0180] S104, performing full integer quantization on the adaptively pruned neural network model.

[0181] In this embodiment, post-training quantization (PTQ) is implemented for a neural network that has undergone weight pruning and dynamic sparsification. The purpose is to convert the network parameters into INT8 format for full integer quantization calculation. During the full integer quantization calculation process, not only are the weights and activation values ​​stored using fixed-point numbers, but the corresponding operators are also used in the fixed-point domain during the calculation process. This calculation method is more in line with the hardware characteristics of the microcontroller, further reducing the storage usage of the model and improving inference efficiency.

[0182] like Figure 7 As shown, as an optional implementation method of this embodiment, the neural network model after adaptive pruning is fully integer quantized, including the following steps:

[0183] S701, perform layer-by-layer normalization on the input feature matrix X, and amplify the normalized data after layer normalization by 2 7 times to obtain the first standardized data; 2 7

[0184] According to the structural characteristics of ViT, the layer normalization operation (LayerNorm) is performed using the following formula (12):

[0185]

[0186] Where x is the input eigenvector, E(x) is the mean of x, Var(x) is the variance of x, γ and β are the scaling and offset parameters of layer normalization, and ∈ takes a very small value to prevent division by zero. After the layer normalization operation, most of the data is distributed in the range [-2, 2]. If the rounding operation is performed directly in the floating-point domain, most of the data will fall into the integer value in the range [-2, 2], resulting in a large loss of numerical precision. In addition, the storage layer normalization output data uses a 16-bit integer, which is in the range [-2 16 ,2 16 -1], in order to make full use of the range of 16-bit integer, in this embodiment, the normalized data after the feature matrix X layer is normalized is amplified by 2 7 times to obtain the first standardized data, thereby reducing the loss of numerical precision caused by rounding during quantization.

[0187] Specifically, the first normalized data is calculated by the following formulas (13) and (14):

[0188] x=x q -Z i (13)

[0189]

[0190] Among them, Z i The zero point of the input determined during the quantization process is shifted by zero point (INT16) using formula (13), and the value is amplified by formula (14) by 2 7 Multiply and normalize (INT16).

[0191] For example, a feature vector x=[0.5,-1.0,1.5] output by the layer normalization operation is magnified by 2 7 =128 times to become [0.5×128,-1.0×128,1.5×128]=[64,-128,192], and the data distribution range is between [-256, 256]. This allows for more accurate representation of data during subsequent quantization, reducing the loss of precision caused by rounding during quantization.

[0192] S702, when performing linear transformation on the first standardized data, amplify the offset parameter by 2 7 times, and obtain the first quantitative data;

[0193] Specifically, the first quantized data is obtained by performing linear transformation using the following formula (15):

[0194] y=x norm *γ+β*2 8-1 (15)

[0195] In formula (15), the offset parameter β is also amplified by the same multiple. This is to ensure that the overall value after linear transformation is amplified by a factor of 27, thereby maintaining consistency in calculation with the amplified normalized result. Because both the weights and input data are amplified by the same multiple, although the data representation changes during the calculation process, the overall calculation result and logical relationship remain unchanged.

[0196] S703, rounding off the first quantized data and reducing it by 2 7 times to obtain the quantized output result.

[0197] Specifically, the quantized output result is obtained by the following formula (16):

[0198]

[0199] Where S is the scaling parameter in the quantization process, Z0 is the zero point of the input determined in the quantization process, and after completing the linear transformation and rounding, the first quantized data after the layer normalization linear transformation is reduced by 2 7 times, ensuring that the final calculation result is mathematically equivalent to the original floating-point calculation.

[0200] This quantization method in the embodiments of this application achieves pure integer layer normalization reasoning with almost no loss of model performance, accelerating the computational efficiency of layer normalization. Through these optimization measures for the ViT structure, the accuracy issues caused by the special data distribution during the quantization process of the ViT structure are overcome, allowing the quantized model to run more efficiently on hardware while maintaining high accuracy.

[0201] The neural network model compression method for microcontrollers provided by the embodiment of the present application effectively combines MaSSA and CGFFN through a gating mechanism, greatly improving the feature extraction capability of the neural network model and improving the model performance; for convolution and fully connected operators of different layers and different functions, accurately evaluate their sparsity sensitivity to pruning, and provide a basis for subsequent model adaptive pruning, so as to maintain the model performance to the greatest extent while compressing the model storage occupancy. In the subsequent pruning process, the smallest prunable unit with lower sparsity sensitivity is preferentially screened out for pruning, so that the compression ratio of the entire model after pruning can be maximized while maintaining the highest possible accuracy; in the adaptive pruning process, the sparsity is gradually increased in stages to reduce the impact of one-time pruning on the model performance, and the storage occupancy required under the hardware constraints of the microcontroller is used as the termination condition. While compressing the storage occupancy of the model, the model performance is maximized under the hardware constraints, so that the model maintains relatively stable performance during the pruning process. It overcomes the limitations of deploying neural network models on microcontrollers and makes it a reality to deploy high-performance neural network models on resource-constrained devices.

[0202] An exemplary embodiment of the present application provides a neural network model compression device for a microcontroller. Figure 8 FIG1 shows a block diagram of a microcontroller-oriented neural network model compression device 10 provided by an exemplary embodiment of the present application. The above-mentioned microcontroller-oriented neural network model compression device is applied to a microcontroller, and can achieve the following: Figures 1 to 7All or part of the contents of any of the illustrated embodiments. The following is a brief description of the structure and function of the neural network model compression device 10 for microcontrollers. For other matters not covered, please refer to the relevant description in the above-mentioned neural network model compression method for microcontrollers. The embodiment of the neural network model compression device 10 for microcontrollers corresponds to the embodiment of the neural network model compression method for microcontrollers. The various implementation processes and implementation methods of the above-mentioned method embodiments can be applied to the embodiment of the neural network model compression device for microcontrollers, and can achieve the same technical effects.

[0203] like Figure 8 As shown, the neural network model compression device 10 for microcontrollers includes: a neural network model construction module 100, a sparse sensitivity evaluation module 200, a model adaptive pruning module 300 and a model quantization module 400, wherein:

[0204] The neural network model construction module 100 is used to input image sample data into the neural network model and train the neural network model. During the training process, the image sample data input to the neural network model is subjected to convolution sampling processing to obtain a feature matrix X, and global feature information is extracted from the feature matrix X using a separable self-attention mechanism MaSSA based on multi-hidden word matching and a context-gated forward propagation network mechanism CGFFN, wherein the global feature information includes feature information in the word dimension and the channel dimension.

[0205] A sparse sensitivity evaluation module 200 is configured to obtain the minimum pruneable unit of the neural network model, calculate the sparse sensitivity of each minimum pruneable unit at different sparsities, and generate a sparse sensitivity matrix;

[0206] The model adaptive pruning module 300 is configured to divide the entire pruning process into multiple stages based on the sparse sensitivity matrix and take the minimum prunable unit as the unit, and adaptively prune the neural network model stage by stage;

[0207] The model quantization module 400 is used to perform full integer quantization on the adaptively pruned neural network model to generate a sparse model.

[0208] As an optional implementation in the embodiment of the present application, the implementation of the neural network model construction module 100 is described below.

[0209] The neural network model building module 100 uses the hybrid model structure of MatchViT, which consists of three parts: a convolutional pre-processing module, a backbone structure, and a post-processing module.

[0210] In some embodiments, the convolution pre-processing module includes a 3×3 convolution and a reverse residual block with an expansion factor of 2. Both components downsample the input image data by a factor of 2 to reduce the resolution of the image data feature map, while increasing the number of channels in the image data feature map, which helps to extract richer feature information while reducing the amount of data. The neural network model construction module 100 downsamples the input image sample data by a factor of 2 using a 3×3 convolution and a reverse residual block with an expansion factor of 2 to obtain the data to be processed with an increased number of channels.

[0211] In some embodiments, the backbone structure of MatchViT includes four stages, each of which starts with a 2x downsampled reverse residual block, followed by a MatchViT block to learn features from the global context, so that the resolution of the feature matrix X gradually decreases, while the number of channels gradually increases, forming a pyramid structure. The MatchViT block is similar to the ViT block. It starts with a separable convolution, which is composed of a 3×3 depth-wise convolution and a 1×1 point-wise convolution stacked together. After the separable convolution encodes the position information, the feature matrix X is obtained. After the neural network model construction module 100 downsamples the processed data by 2x, it encodes the position information through separable convolution to obtain the feature matrix X.

[0212] In some embodiments, the feature matrix X will pass through multiple ViT encoders to extract the global feature information of the feature matrix X. In the embodiment of the present application, each ViT encoder consists of a MaSSA module and a CGFFN module, and is connected using a reverse residual block, and the expansion factor of the reverse residual block and the forward propagation network is set to 2. This enables the MatchViT model to extract global feature information more efficiently and comprehensively, further improving the performance of the model deployed on the microcontroller. The neural network model construction module 100 utilizes the context-gated forward propagation network mechanism CGFFN, and based on the feature information on the word dimension, extracts global feature information for the feature matrix X.

[0213] In some embodiments, the post-processing portion of MatchViT includes global average pooling and a final classifier, which performs global average pooling on the global feature information and ultimately obtains the classification result through linear mapping. The neural network model construction module 100 performs global average pooling on the global feature information and performs linear mapping classification through the classifier.

[0214] The application examples used in the embodiments of the present application correspond to the application examples of the above-mentioned neural network model compression method for microcontrollers, and will not be repeated here.

[0215] As an optional implementation in the embodiment of the present application, the implementation of the MaSSA mechanism of the neural network model construction module 100 is described below.

[0216] The neural network model building module 100 uses the separable self-attention mechanism MaSSA based on multi-hidden word matching to extract feature information on the word dimension from the feature matrix X in the following way:

[0217] The feature matrix X is linearly transformed to generate four MaSSA matrices. The MaSSA matrix includes: the hidden word matrix I, the key matrix K, the value matrix V, and the matching matrix R in the separable self-attention mechanism. The hidden word matrix I includes k hidden words, and the matching matrix R represents the word feature of each hidden word.

[0218] The context vector group CV is calculated based on the hidden word matrix I according to the separable self-attention mechanism. The context vector group CV is dot-producted with the matching matrix R. A Softmax operation is performed on the word number dimension to obtain the matching score MS, where the matching score MS represents the amount of information that each word in the value matrix V needs to obtain from each hidden word in the hidden word matrix I.

[0219] Perform matrix multiplication on the matching score MS and the context vector group CV to obtain the matched context vector group MCV;

[0220] The matched context vector group MCV is passed through the weight W S Map to the original dimension and perform element-wise multiplication with the value matrix V to obtain the MaSSA output result, where the original dimension is the dimension of the feature matrix X. The MaSSA output result includes: global feature information on the word dimension.

[0221] For specific formulas and application examples, see Figure 3 The embodiment of the neural network model compression method for microcontrollers shown will not be described in detail here.

[0222] In the embodiment of the present application, MaSSA expands the shape of the matrix I, increasing the number of hidden tokens to multiple (k hidden tokens). Through matrix multiplication, MaSSA generates a context vector group containing k vectors, thereby including multiple levels of feature extraction and completing the feature information extraction of the token number dimension of the feature matrix X. In addition, the MaSSA matrix introduces a matching matrix R, which represents the features required for each token, to ensure that each hidden token can selectively interact with the value matrix, ensuring that no redundancy occurs after adding multiple hidden tokens.

[0223] As an optional implementation in the embodiment of the present application, the implementation of the CGFFN mechanism of the neural network model construction module 100 is described below.

[0224] The neural network model building module 100 utilizes the context gated feedforward network mechanism CGFFN in the following manner and extracts global feature information from the feature matrix X based on the feature information on the word dimension:

[0225] The feature matrix X is upgraded to a high-dimensional space by weight W1, and activated by the SiLU activation function to obtain a high-dimensional feature matrix X up ;

[0226] The context vector group CV is upgraded to a high-dimensional space by weight W2 to obtain a high-dimensional context vector group CS up ;

[0227] The feature matrix X is passed through the weight W k Mapping to obtain feature information X k , the feature information X k CV with high-dimensional context vectors up Perform matrix multiplication and activate through SiLU activation function to obtain the gate value Gate;

[0228] Apply the gate value Gate to the high-dimensional feature matrix X up , and then reduce the dimension to the original dimension through the weight W3 to obtain the CGFFN output result, where the CGFFN output result includes: global feature information.

[0229] For specific formulas and application examples, see Figure 4 The embodiment of the neural network model compression method for microcontrollers shown will not be described in detail here.

[0230] In the embodiments of this application, the core improvements of CGFFN are: (1) Reusing global context: Using the context vector group CV generated by MaSSA as a gating signal to dynamically adjust the features in conventional FFN. (2) Channel dimension enhancement: Using a gating mechanism to fuse global context in the channel dimension, the model's sensitivity to multi-scale features is improved.

[0231] Compared to conventional feedforward networks, CGFFN reuses the context vector group of the MaSSA module, making better use of the multi-scale information extracted by MaSSA. It also uses gating on high-dimensional mapped inputs, further improving the model's feature extraction capabilities while only slightly increasing the number of model parameters and multiplication-addition computations. MaSSA and CGFFN work together to form the ViT encoder, which, together with the convolutional layers, completes the neural network architecture, significantly enhancing the neural network model's feature extraction capabilities and performance.

[0232] In the embodiment of the present application, the sparse sensitivity evaluation module 200 calculates the sparse sensitivity of each minimum prunable unit at different sparsities using the above formula (11).

[0233] In an embodiment of the present application, the minimum prunable unit includes one of the following: a convolutional layer, a fully connected layer, a weight, and a MaSSA matrix; in actual application, the minimum prunable unit can be a layer in a neural network model (such as a convolutional layer or a fully connected layer), or it can be a specific operator (such as a convolution kernel or a set of weights in a fully connected layer). The specific correspondence is determined according to the evaluation granularity. In actual operation, evaluation is usually performed in layers, but it can also be refined to a more specific operator level.

[0234] In the embodiment of the present application, the sparse sensitivity assessment module 200 decomposes the neural network model into minimum prunable units, such as a single convolutional layer, a fully connected layer, and the matching matrix R in MaSSA. For example, the minimum prunable unit can be a 3×3 depthwise convolution of the inverse residual block, the matching matrix R in MaSSA, or the weights W1, W2, and W3 in CGFFN. Moreover, in order to more accurately understand the response of each part of the model to pruning, the minimum prunable unit can be a certain weight.

[0235] In the embodiment of the present application, in order to obtain sparse sensitivity more quickly, one-shot pruning is used when evaluating sparse sensitivity. Specifically, the weight values ​​with lower L1 norm are directly set to zero without fine-tuning.

[0236] From the above formula (11), it can be seen that under the same storage space compression ratio (param dense -param sparse Same), if the accuracy drops more after pruning (acc dense -acc sparse The larger the pruning accuracy, the higher the sparsity sensitivity. This means that the unit is more sensitive to pruning, and pruning the parameters of the unit will have a greater impact on the model performance. dense -acc sparse Same), the storage space compression ratio is higher (param dense -param sparse This indicates that this unit can compress the storage space to a greater extent while maintaining the model performance during pruning, making it a more suitable unit for pruning.

[0237] After traversing the minimum pruning units, a sparse sensitivity matrix Sens is obtained. The sparse sensitivity matrix Sens is a two-dimensional matrix, and the rows represent multiple minimum pruning units. Let M be the set of minimum pruning units, for example, M = {m1, m2, ... mk}, indicating that there are k minimum pruning units. The columns of the sparse sensitivity matrix represent different sparsities, and SP is a sparsity set. For example, in the embodiment of the present application, SP is set to an array with an initial value of 0.5, an end value of 0.7, and a step size of 0.05, that is, SP = {0.5, 0.55, 0.6, 0.65, 0.7}. The elements of the sparse sensitivity matrix Sens i,j Represents the minimum pruning unit m i At the corresponding sparsity sp j The lower the sparsity sensitivity value, the higher the cost-effectiveness of pruning.

[0238] For example, a simplified model contains three minimum prunable units M = {m1, m2, m3}, sparsity set SP = [0.5, 0.55, 0.6], and sparse sensitivity matrix These correspond to the sensitivities of m1, m2, and m3 at different sparsities. For example, in the sparse sensitivity matrix, the sparse sensitivity of the smallest prunable unit m1 at a sparsity of 0.5 is 0.5, meaning it is pruned by 50%.

[0239] For specific formulas and application examples, please refer to the description of step S102 of the embodiment of the neural network model compression method for microcontrollers above, which will not be repeated here.

[0240] In the embodiments of this application, the sparsity sensitivity of convolutional and fully connected operators at different layers and with different functions to pruning is accurately assessed, providing a basis for subsequent adaptive pruning of the model, thereby compressing model storage while maximizing model performance. Therefore, in the subsequent pruning process, the smallest prunable units with lower sparsity sensitivity are preferentially selected for pruning. This can maximize the compression ratio of the entire pruned model while maintaining the highest possible accuracy.

[0241] As an optional implementation in the embodiment of the present application, the implementation of the model adaptive pruning module 300 adaptively pruning the neural network model stage by stage is described below.

[0242] The model adaptive pruning module 300 divides the entire pruning process into multiple stages based on the sparse sensitivity matrix and takes the smallest prunable unit as the unit, and adaptively prunes the neural network model stage by stage in the following manner:

[0243] (1) Determine the memory occupancy standard that needs to be achieved under the hardware constraints of the microcontroller, and determine the target memory occupancy of each stage based on the memory occupancy standard. The target memory occupancy is the memory occupancy of the model after pruning in this stage, and decreases linearly with the number of stages until the memory occupancy standard is reached.

[0244] In an application example, the initial model memory occupancy is S0, and the memory occupancy standard that needs to be achieved under the hardware constraints of the microcontroller is S target , the target storage occupancy of each stage i can be calculated by a linear function

[0245] In the above simplified model, the initial model memory occupation S0 is 100 units, and the memory occupation standard that needs to be achieved under the hardware constraints of the microcontroller is S target The unit is 50, the pruning stage I = 3, and the target storage occupancy S of each stage can be calculated according to the linear function i :

[0246] Stage 1: S1 = 100 - 1 / 3 (100 - 50) = 83.33 units;

[0247] Stage 2: S2 = 100 - 2 / 3 (100 - 50) = 66.67 units;

[0248] Stage 3: S3 = 100 - 3 / 3 (100 - 50) = 50 units.

[0249] In an embodiment of the present application, before performing the pruning operation, as an optional implementation method in the embodiment of the present application, the method provided in the embodiment of the present application also includes: initializing the priority queue pq, and enqueuing the elements corresponding to the initial sparsity values ​​in the sparse sensitivity matrix in order of sparse sensitivity from small to large as members of the priority queue pq.

[0250] Specifically, each row in the sparse sensitivity matrix is ​​regarded as an array, which represents the sparse sensitivity of a minimum prunable unit at different sparsities. For example, Sens 1j It can be the sparse sensitivity of m1 under the sparsity {0.5, 0.55, 0.6, 0.65, 0.7}. Each column in the sparse sensitivity matrix is ​​regarded as an array, representing the sparse sensitivity of the k smallest prunable units under the same sparsity.

[0251] When initializing the priority queue pq, the head element is first popped out of the array (for example, the sparse sensitivity of the k smallest prunable units with sp=0.5) and added to the priority queue.

[0252] For example, the sparse sensitivity matrix The head element of each row is popped out and added to the priority queue. At this time, the elements in the priority queue pq are: The elements in the priority queue pq represent the minimum prunable unit, sparsity, and sparsity sensitivity, respectively.

[0253] (2) Divide the entire pruning process into multiple stages and perform adaptive pruning on the neural network model stage by stage;

[0254] In each stage, the following pruning operations are performed:

[0255] B1. Determine the expected sparsity value of each minimum pruneable unit to be pruned in the current stage, where the sparsity sensitivity of the minimum pruneable unit to be pruned is low and the model memory usage after pruning the minimum pruneable unit to be pruned meets the target memory usage of this stage;

[0256] In an embodiment of the present application, before pruning, the pruning sparsity of each minimum prunable unit in each stage is configured. The logic of the pruning sparsity configuration is to select the minimum prunable unit with the lowest sensitivity and sparsity from the priority queue in sequence, and determine whether the storage occupancy of the model after pruning the selected minimum prunable unit meets the target storage occupancy of this stage. If so, it is used as the minimum prunable unit to be pruned and waits for subsequent pruning. If not, the search continues to meet the target storage occupancy of this stage until all elements in the sparse sensitivity matrix are traversed.

[0257] B2. For each minimum pruneable unit to be pruned, the pruning of this stage is completed in such a way that the sparsity increases linearly with the number of cycles until the sparsity reaches the expected value.

[0258] After completing the sparsity configuration of each unit in each stage, the actual pruning of the neural network model begins.

[0259] During model pruning, if all units that need to be pruned are directly set to a higher sparsity at once, for example, increasing the sparsity of the model from 0 to 0.7 (i.e., pruning 70% of the weights) at once, this may cause a sharp drop in model performance. Therefore, a more gradual and gradual approach to increasing sparsity is needed to allow the model to gradually adapt to the weight changes and maintain relatively stable performance.

[0260] In the embodiment of the present application, as an optional implementation in the embodiment of the present application, the model adaptive pruning module 300 completes the pruning of this stage in the following manner for each minimum pruneable unit to be pruned, in a manner in which the sparsity increases linearly with the number of cycles to an expected sparsity value:

[0261] Each stage is divided into multiple cycles. In each cycle, each minimum prunable unit to be pruned is pruned according to the sparsity within the cycle until the pruning of this stage is completed. The sparsity within the cycle increases linearly with the number of cycles from the initial sparsity value to the expected sparsity value corresponding to the minimum prunable unit to be pruned.

[0262] Specifically, in the adaptive pruning process of each stage, each stage is divided into multiple cycles, and the sparsity is linearly increased from the initial value to the expected value, so that the model can maintain relatively stable performance. For example, if the starting sparsity of the current stage is 0.5 and the target sparsity is 0.7, which is completed in 4 cycles, then the sparsity is increased by 0.05 in each cycle until it reaches 0.7. Gradually improving the sparsity in this way helps the model maintain higher performance during fine-tuning. Compared to obtaining sparse parameters and pruning through neural architecture search, the pruning strategy in the embodiment of the present application takes less time to find sparse parameters, and the performance degradation of the model remains within a controllable range.

[0263] As an optional implementation in the embodiment of the present application, the model adaptive pruning module 300 is as follows: Figure 6 The steps (S601-S606) shown in FIG. 6 determine the expected sparsity value of each minimum pruning unit to be pruned at the current stage. For specific steps and application examples, please refer to Figure 6 The embodiment of the neural network model compression method for microcontrollers shown will not be described in detail here.

[0264] In an embodiment of the present application, through the pruning configuration strategy of "sparse sensitivity sorting" + "staged pruning", the global optimal pruning unit (the one with the lowest sparsity sensitivity and meeting the target storage occupancy) is continuously tracked through the priority queue to avoid the local optimal trap, and the pruning unit and sparsity with the least impact on accuracy are automatically selected; the sparsity is gradually increased in stages to reduce the impact of one-time pruning on model performance, and the storage occupancy that needs to be achieved under the hardware constraints of the microcontroller is used as the termination condition. While compressing the storage occupancy of the model, the model performance is maximized under the hardware constraints to adapt to the resource limitations of different microcontrollers.

[0265] As an optional implementation in the embodiment of the present application, the following describes an implementation in which the model quantization module 400 performs full integer quantization on the adaptively pruned neural network model.

[0266] The model quantization module 400 performs full integer quantization on the adaptively pruned neural network model in the following manner:

[0267] Normalize the input feature matrix X layer by layer and amplify the normalized data after layer normalization by 2 7 times to obtain the first standardized data;

[0268] When the first normalized data is linearly transformed, the offset parameter is amplified by 2 7 times, and obtain the first quantitative data;

[0269] After rounding off the first quantized data, reduce it by 2 7 times to obtain the quantized output result.

[0270] For specific formulas and application examples, see Figure 7 The embodiment of the neural network model compression method for microcontrollers shown will not be described in detail here.

[0271] This quantization method in the embodiments of this application achieves pure integer layer normalization reasoning with almost no loss of model performance, accelerating the computational efficiency of layer normalization. Through these optimization measures for the ViT structure, the accuracy issues caused by the special data distribution during the quantization process of the ViT structure are overcome, allowing the quantized model to run more efficiently on hardware while maintaining high accuracy.

[0272] The neural network model compression device for microcontrollers provided by the embodiment of the present application effectively combines MaSSA and CGFFN through a gating mechanism, greatly improving the feature extraction capability of the neural network model and improving the model performance; for convolution and fully connected operators of different layers and different functions, accurately evaluate their sparsity sensitivity to pruning, and provide a basis for subsequent model adaptive pruning, so as to maintain the model performance to the greatest extent while compressing the model storage occupancy. In the subsequent pruning process, the smallest prunable unit with lower sparsity sensitivity is preferentially screened out for pruning, so that the compression ratio of the entire model after pruning can be maximized while maintaining the highest possible accuracy; in the adaptive pruning process, the sparsity is gradually increased in stages to reduce the impact of one-time pruning on the model performance, and the storage occupancy required under the hardware constraints of the microcontroller is used as the termination condition. While compressing the storage occupancy of the model, the model performance is maximized under the hardware constraints, so that the model maintains relatively stable performance during the pruning process. It overcomes the limitations of deploying neural network models on microcontrollers and makes it a reality to deploy high-performance neural network models on resource-constrained devices.

[0273] Figure 9 FIG2 shows a block diagram of an electronic device 1000 according to an exemplary embodiment of the present application. The electronic device 1000 can be implemented as the aforementioned microcontroller-oriented neural network model compression device, which can be configured in electronic devices such as smartphones, tablet computers, laptops, desktop computers, smart watches, televisions, or servers.

[0274] Typically, the electronic device 1000 includes a processor 1001 and a memory 1002 .

[0275] The processor 1001 may include one or more processing cores, such as a 4-core processor, a 10-core processor, etc. The processor 1001 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 1001 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1001 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1001 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0276] The memory 1002 may include one or more computer-readable storage media, which may be non-transitory. The memory 1002 may also include a high-speed random access memory and a non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1002 is used to store at least one instruction, which is used to be executed by the processor 1001 to implement all or part of the steps in the microcontroller-oriented neural network model compression method shown in the method embodiment of the present application.

[0277] Those skilled in the art will understand that Figure 9 The structure shown in the figure does not constitute a limitation on the electronic device 1000, and the electronic device 1000 may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0278] In an exemplary embodiment, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores a program or instruction that, when executed by a processor, implements all or part of the steps in the above-described microcontroller-oriented neural network model compression method. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, or an optical data storage device.

[0279] In an exemplary embodiment, a computer program product is further provided, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, which, when executed by a computer, causes the computer to perform the above-mentioned Figures 1 to 7 All or part of the steps of the neural network model compression method for microcontrollers shown in any embodiment.

[0280] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered merely as exemplary, and the true scope and spirit of the present application are indicated by the claims.

[0281] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A neural network model compression method for microcontrollers, characterized in that: include: Input the image sample data into the neural network model and train the neural network model. During the training process, the neural network model performs convolution sampling processing on the input image sample data to obtain a feature matrix X, and uses a separable self-attention mechanism MaSSA based on multi-hidden word matching and a context-gated forward propagation network mechanism CGFFN to extract global feature information from the feature matrix X, wherein the global feature information includes: feature information in the word dimension and the channel dimension; Obtaining the minimum prunable unit of the neural network model, calculating the sparse sensitivity of each of the minimum prunable units at different sparsities, and generating a sparse sensitivity matrix; According to the sparse sensitivity matrix, the entire pruning process is divided into multiple stages with the minimum prunable unit as the unit, and the neural network model is adaptively pruned stage by stage; The neural network model after adaptive pruning is fully integer quantized.

2. The method according to claim 1, characterized in that The neural network model performs convolution sampling on the input image sample data to obtain a feature matrix X, and uses the separable self-attention mechanism MaSSA based on multi-hidden word matching and the context-gated forward propagation network mechanism CGFFN to extract global feature information from the feature matrix X, including: The neural network model downsamples the input image sample data by 2 times through 3×3 convolution and an inverse residual block with an expansion factor of 2 to obtain data to be processed with an increased number of channels; After downsampling the data to be processed by a factor of 2, encoding the position information through separable convolution to obtain the feature matrix X; Utilize the separable self-attention mechanism MaSSA based on multi-hidden word matching to extract feature information on the word dimension from the feature matrix X; Utilizing the context gated feedforward network mechanism CGFFN and based on the feature information on the word-unit dimension, extracting the global feature information from the feature matrix X; Global average pooling is performed on the global feature information, and linear mapping classification is performed through a classifier.

3. The method according to claim 1, characterized in that The method of extracting feature information on the word-unit dimension from the feature matrix X using the separable self-attention mechanism MaSSA based on multi-hidden word-unit matching includes: Generate four MaSSA matrices from the feature matrix X through linear transformation. The MaSSA matrices include: a hidden word unit matrix I, a key matrix K, a value matrix V, and a matching matrix R that can be separated from the attention mechanism. The hidden word unit matrix I includes k hidden words, and the matching matrix R represents the word unit feature of each hidden word unit. A context vector group CV is calculated based on the hidden word unit matrix I according to the separable self-attention mechanism, a dot product is performed on the context vector group CV and the matching matrix R, and a Softmax operation is performed on the word unit number dimension to obtain a matching score MS, wherein the matching score MS represents the amount of information that each word unit in the value matrix V needs to obtain from each hidden word unit in the hidden word unit matrix I; Perform matrix multiplication on the matching score MS and the context vector group CV to obtain a matched context vector group MCV; The matched context vector group MCV is divided into two groups by weight W. S Map to the original dimension and perform element-wise multiplication with the value matrix V to obtain a MaSSA output result, wherein the original dimension is the dimension of the feature matrix X, and the MaSSA output result includes: global feature information on the word dimension.

4. The method according to claim 3, characterized in that The method of utilizing the context gated feedforward network mechanism CGFFN and extracting global feature information from the feature matrix X based on the feature information on the word dimension includes: The feature matrix X is upgraded to a high-dimensional space by weight W1 and activated by SiLU activation function to obtain a high-dimensional feature matrix X up ; The context vector group CV is upgraded to a high-dimensional space by weight W2 to obtain a high-dimensional context vector group CV up ; The feature matrix X is passed through the weight W k Mapping to obtain feature information X k , the feature information amount X k With the high-dimensional context vector group CV up Perform matrix multiplication and activate through SiLU activation function to obtain the gate value Gate; Apply the gate value Gate to the high-dimensional feature matrix X up , and then reduce the dimension to the original dimension through the weight W3 to obtain the CGFFN output result, wherein the CGFFN output result includes: the global feature information.

5. The method according to claim 1, wherein The minimum pruneable unit includes one of the following: a convolutional layer, a fully connected layer, a weight, and the MaSSA matrix; The sparse sensitivity of each minimum pruneable unit at different sparsities is calculated using the following formula: Wherein, sp represents sparsity, m represents the minimum pruning unit, represents the sparse sensitivity of the minimum pruning unit m under the sparsity sp, acc dense with acc sparse They represent the image classification TOP-1 accuracy of the unpruned model and the model after a single pruning of the minimum prunable unit m, respectively, param dense with param sparse They represent the storage occupancy of the unpruned model and the model after a single pruning of the minimum prunable unit m.

6. The method according to claim 5, characterized in that The rows of the sparse sensitivity matrix represent a plurality of the minimum prunable units, the columns of the sparse sensitivity matrix represent different sparsities, and the elements of the sparse sensitivity matrix represent the sparse sensitivity of the minimum prunable unit under the corresponding sparsity; The method further comprises dividing the entire pruning process into multiple stages based on the sparse sensitivity matrix and taking the minimum pruning unit as a unit, and adaptively pruning the neural network model stage by stage, including: Determine the memory usage standard that needs to be achieved under the hardware constraints of the microcontroller, and determine the target memory usage of each stage based on the memory usage standard, where the target memory usage is the memory usage of the pruned model in this stage and decreases linearly with the number of stages until the memory usage standard is reached; In each stage, the following pruning operations are performed: Determining an expected sparsity value of each minimum pruneable unit to be pruned in the current stage, wherein the sparsity sensitivity of the minimum pruneable unit to be pruned is low and the model memory occupancy after pruning the minimum pruneable unit to be pruned meets the target memory occupancy of the current stage; The pruning of this stage is completed for each of the minimum prunable units to be pruned in a manner that the sparsity increases linearly with the number of cycles to the expected sparsity value.

7. The method according to claim 6, characterized in that The pruning of each of the minimum pruneable units to be pruned is completed in this stage in a manner that the sparsity increases linearly with the number of cycles to the expected sparsity value, including: Each stage is divided into multiple cycles. In each cycle, each of the minimum prunable units to be pruned is pruned according to the sparsity within the cycle until the pruning of this stage is completed. The sparsity within the cycle increases linearly with the number of cycles from the initial sparsity value to the expected sparsity value corresponding to the minimum prunable unit to be pruned.

8. The method according to claim 6, characterized in that Before performing the following pruning operations in each stage, the method further includes: Initialize the priority queue pq, and queue the elements corresponding to the initial sparsity values ​​in the sparse sensitivity matrix in ascending order of sparsity sensitivity as members of the priority queue pq; The determining of the expected sparsity value of each minimum prunable unit to be pruned in the current stage, wherein the sparsity sensitivity of the minimum prunable unit to be pruned is low and the model memory occupancy after pruning the minimum prunable unit to be pruned meets the target memory occupancy of the current stage, includes: S1. Obtaining the player with the lowest current sparsity sensitivity and the corresponding sparsity from the priority queue as the minimum pruning unit to be pruned and the expected sparsity value corresponding to the minimum pruning unit to be pruned; S2. Obtain the current model storage occupancy after pruning the minimum prunable unit to be pruned; S3. If the current model storage occupancy does not meet the target storage occupancy of this stage, execute step S1; S4. When the current model storage occupancy meets the target storage occupancy of this stage, record the current minimum prunable unit to be pruned and the corresponding expected sparsity value; and query whether the current minimum prunable unit to be pruned has a corresponding high-level sparsity in the sparse sensitivity matrix. If so, add the elements corresponding to the high-level sparsity in the sparse sensitivity matrix to the priority queue in ascending order of sparsity sensitivity, and return to step S1, wherein the high-level sparsity is higher than the sparsity corresponding to the current minimum prunable unit to be pruned; if not, the pruning sparsity configuration process of the current stage ends.

9. The method according to claim 1, characterized in that The neural network model after adaptive pruning is subjected to full integer quantization, including: The input feature matrix X is layer-normalized and the normalized data after layer normalization is amplified by 2 7 times to obtain the first standardized data; When the first standardized data is linearly transformed, the offset parameter is amplified by 2 7 times, and obtain the first quantitative data; The first quantized data is rounded off and then reduced by 2 7 times to obtain the quantized output result.

10. A neural network model design device for a microcontroller, characterized in that: include: A neural network model construction module is used to input image sample data into the neural network model and train the neural network model. During the training process, the image sample data input into the neural network model is subjected to convolution sampling processing to obtain a feature matrix X, and global feature information is extracted from the feature matrix X using a separable self-attention mechanism MaSSA based on multi-hidden word matching and a context-gated forward propagation network mechanism CGFFN, wherein the global feature information includes feature information in the word dimension and the channel dimension; A sparse sensitivity evaluation module is used to obtain the minimum prunable unit of the neural network model, calculate the sparse sensitivity of each of the minimum prunable units under different sparsities, and generate a sparse sensitivity matrix; A model adaptive pruning module is used to divide the entire pruning process into multiple stages based on the sparse sensitivity matrix and take the minimum prunable unit as the unit, and adaptively prune the neural network model stage by stage; The model quantization module is used to perform full integer quantization on the neural network model after adaptive pruning to generate a sparse model.

Images