An edge device image recognition method and system based on pulse neural network compression optimization

Abstract

The application discloses an edge device image recognition method based on pulse neural network compression optimization, comprising the following steps: obtaining an image to be recognized in a target edge device and an ANN model, constructing an SNN model according to the ANN model, optimizing the SNN model to obtain an optimized SNN model, processing the optimized SNN model by using a structured pruning method to obtain a pruned SNN model, performing 4bit quantization processing on the pruned SNN model to obtain a quantized SNN model, deploying the quantized SNN model on the target edge device to obtain an executable SNN model, and using the executable SNN model to recognize the image to be recognized in the target edge device to obtain an image recognition result. The application can solve the technical problem that the image recognition precision of the SNN model is low due to the difficulty in effectively reducing the uneven error generated in the process of converting the ANN into the SNN.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of artificial intelligence and brain-like computing, and more specifically, relates to an edge device image recognition method and system based on compression optimization of spiking neural networks. Background Technology

[0002] In recent years, with the rapid development of edge computing, the Internet of Things (IoT), and smart terminal technologies, image recognition for edge devices has been increasingly widely applied in scenarios such as smart security, industrial inspection, smart homes, unmanned terminals, and vehicle vision. However, edge devices are typically limited by computing power, storage capacity, and power supply, placing higher demands on the accuracy, parameter scale, computational complexity, and energy consumption of image recognition models. Spiking Neural Networks (SNNs), as the third generation of artificial neural networks, are based on event-driven binary pulse signal representation and processing of information. They possess characteristics of low power consumption and high biocompatibility, making them highly compatible with the resource-constrained characteristics of edge devices and thus becoming an important research direction for image recognition on edge devices.

[0003] Currently, existing image recognition methods for edge devices mainly fall into the following categories: The first category involves converting Artificial Neural Networks (ANNs) into SNN models, mapping the weights in the ANN model to the SNN model to achieve rapid transfer learning from ANN to SNN; the second category involves directly deploying ANN models to edge devices and using models such as convolutional neural networks and residual networks to complete image feature extraction and classification. This type of method typically relies on high-precision floating-point weights and dense connection structures to achieve high recognition accuracy; the third category involves directly training SNN models for image recognition on edge devices, utilizing SNNs based on event-driven pulse signals for information representation and computation, aiming to achieve low-power inference performance; and the fourth category involves lightweighting ANN models before deploying them to edge devices to reduce the number of ANN model parameters and computational load, thereby reducing the storage and inference overhead of edge devices.

[0004] However, all of the above methods have some drawbacks that cannot be ignored: First, existing methods for converting ANN models to SNN models typically employ the Integrate-and-Fire (IF) neuron model, which has relatively simple membrane potential dynamics and is difficult to effectively reduce the non-uniform errors generated during the ANN conversion process, resulting in low image recognition accuracy of the SNN model. Second, existing methods for directly deploying ANN models typically employ high-precision floating-point weights and dense connection structures, resulting in a large number of model parameters, high computational load, and high energy consumption, which makes it difficult to meet the application requirements of edge devices for low power consumption and lightweight design. Third, existing methods for directly training SNN models suffer from problems such as training difficulties, slow convergence speed, unstable gradient propagation, and high training costs, which are not conducive to efficiently obtaining high-precision models suitable for image recognition of edge devices. Fourth, while existing methods for deploying lightweight ANN models can reduce model size and computational overhead to some extent, they mainly optimize traditional ANN computation paradigms and still cannot fully leverage the advantages of event-driven computing in low-power edge device scenarios. Summary of the Invention

[0005] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides an edge device image recognition method and system based on spiking neural network compression optimization. Its purpose is to: solve the technical problems of existing edge device image recognition methods where the membrane potential dynamic characteristics of the neuron models used after converting artificial neural networks (ANNs) to SNNs are relatively simple, making it difficult to effectively reduce the non-uniform errors generated during the ANN-to-SNN conversion process, resulting in low image recognition accuracy of the SNN model; the technical problems of existing methods that directly deploy ANN models using a large number of model parameters, high computational cost, and high energy consumption, making it difficult to meet the low-power and lightweight application requirements of edge devices; the technical problems of existing methods that directly train SNN models experiencing training difficulties, slow convergence speed, unstable gradient propagation, and high training costs; and the technical problems of existing methods that deploy lightweight ANN models not being optimized for the SNN computational paradigm, making it difficult to fully leverage the advantages of event-driven computing in low-power edge device scenarios.

[0006] To achieve the above objectives, according to one aspect of the present invention, an edge device image recognition method based on spiking neural network compression optimization is provided, comprising the following steps: (1) Obtain the image to be identified and the ANN model in the target edge device, construct the SNN model based on the ANN model, and optimize the SNN model to obtain the optimized SNN model; (2) The optimized SNN model obtained in step (1) is processed using the structured pruning method to obtain the pruned SNN model; (3) Perform 4-bit quantization on the pruned SNN model obtained in step (2) to obtain the quantized SNN model; (4) Deploy the quantized SNN model obtained in step (3) to the target edge device to obtain an executable SNN model; (5) Use the executable SNN model obtained in step (4) to identify the image to be identified in the target edge device obtained in step (1) to obtain the image recognition result.

[0007] Preferably, step (1) includes the following steps: (1-1) Obtain the image to be identified from the image acquisition module of the target edge device, and construct an ANN model using the ANN model parameter file that has been trained and saved on the training device in advance. Replace the activation function of the ANN model with ST-BIF neurons to construct an SNN model. In the process of constructing the SNN model in this step, the weights of the SNN model are set to be equal to the weights of the ANN model. The membrane potential update rule of the ST-BIF neurons is set for the SNN model. The threshold of the ST-BIF neurons in the SNN model is set to be equal to the trainable parameter in the activation function of the ANN model. The membrane potential of the ST-BIF neurons in the SNN model is initialized to half of the threshold of the ST-BIF neurons. The membrane potential reset strategy of the ST-BIF neurons in the SNN model is set to soft reset to obtain the SNN model. (1-2) The convolutional layer and the batch normalization layer in the SNN model constructed in step (1-1) are fused to obtain the optimized SNN model.

[0008] Preferably, the activation function expression used in the ANN model is: ; in This represents the output of the activation function. This represents the input received by the activation function. It is a trainable parameter. () indicates rounding down to the nearest integer. This represents a truncation function that truncates the first parameter to a value between 0 and 1. The membrane potential update rule for ST-BIF neurons in the SNN model is as follows: ; ; ; in This represents the membrane potential of the ST-BIF neuron at time t. The input received by the ST-BIF neuron at time t. This represents the threshold of the ST-BIF neuron. This is a pulse tracker at time t, used to record the number of pulses that the ST-BIF neuron has fired to the next layer of ST-BIF neurons up to time t. To allow the maximum number of pulses that an ST-BIF neuron can fire to the next layer of ST-BIF neurons, To determine the minimum number of positive pulses that must be fired to the next layer of ST-BIF neurons before the ST-BIF neuron can fire a negative pulse to the next layer of ST-BIF neurons, This represents a function that determines whether an ST-BIF neuron fires a positive pulse, a negative pulse, or no pulse to the next layer of ST-BIF neurons, based on the membrane potential and threshold of the ST-BIF neuron and a pulse tracker. This indicates the current membrane potential of the ST-BIF neuron. This indicates the value currently recorded by the pulse tracker.

[0009] Preferably, step (2) includes the following steps: (2-1) Iterate through the optimized SNN model obtained in step (1) to obtain the pruning score of each output channel of the convolutional layer and fully connected layer in the SNN model; (2-2) Sort the pruning scores of all output channels of the convolutional and fully connected layers in the optimized SNN model obtained in step (2-1) in descending order, compress the optimized SNN model according to the set SNN model compression ratio to obtain the compressed SNN model, and copy the weights of the optimized SNN model to the compressed SNN model to obtain the SNN model with completed weight copying. (2-3) Fine-tune the SNN model with completed weight replication obtained in step (2-2) to obtain the pruned SNN model.

[0010] Preferably, step (2-1) includes the following steps: (2-1-1) Iterate through the optimized SNN model obtained in step (1) to obtain the scaling factor (scale) for each output channel of the convolutional and fully connected layers in the SNN model: ; Where i∈[1, the total number of output channels in the convolutional and fully connected layers of the SNN model], It is the maximum value among all the absolute values ​​of weights in the i-th output channel; (2-1-2) Based on the scaling factor (scale) of each output channel of the convolutional layer and fully connected layer obtained in step (2-1-1), obtain the quantized weight of the output channel and the threshold of the corresponding ST-BIF neuron. : ; ; in This represents the weight of the i-th output channel before quantization. This represents the weight of the quantized i-th output channel. This represents the threshold value of the ST-BIF neuron corresponding to the i-th output channel before quantization. This represents the threshold value of the ST-BIF neuron corresponding to the i-th output channel after quantization. (2-1-3) Based on the quantized weights of each output channel of the convolutional and fully connected layers obtained in step (2-1-2) and the threshold of the ST-BIF neuron corresponding to that output channel, obtain the pruning score of each output channel of the convolutional and fully connected layers in the optimized SNN model. : + ; in This indicates taking the absolute value. This represents the average weight of all weights in the i-th output channel.

[0011] Preferably, step (2-2) includes the following steps: (2-2-1) Obtain the number of output channels of the i-th layer in the compressed SNN model based on the number of output channels of the i-th layer in the optimized SNN model obtained in step (2-1). And based on the number of output channels of the i-th layer Construct a compressed SNN model; where the number of output channels in the i-th layer of the compressed SNN model is... The calculation formula is: ; in This represents the number of output channels in the i-th layer of the optimized SNN model obtained in step (2-1). This represents the compression ratio set for the i-th layer in the optimized SNN model; (2-2-2) Sort the pruning scores of all output channels of the convolutional and fully connected layers in the optimized SNN model obtained in step (2-1) in descending order. Based on the sorting result, determine the index of each output channel that needs to be retained in each layer of the convolutional and fully connected layers. Based on the index, copy the weights of the corresponding output channels of the optimized SNN model obtained in step (1) to the compressed SNN model constructed in step (2-2-1) to obtain the SNN model with completed weight copying.

[0012] Preferably, step (2-2-2) specifically involves the following steps: If a layer in the compressed SNN model has num output channels, then from all the output channels of that layer, select the num output channels corresponding to the top num results obtained in step (2-1), and obtain multiple indices of these num output channels in all the output channels of that layer. The weights of these num output channels in the optimized SNN model are then copied to the output channels corresponding to these indices in the compressed SNN model. Here, num is a natural number greater than or equal to 1, and the copying process is divided into the following four cases: First, the layer above is compressed, and the input channels of this layer must correspond to the output channels retained by the previous layer. Then, the weights are copied by combining the index of the output channels retained by this layer. Second, if the layer is compressed and the previous layer is not compressed, the weights are copied according to the index of the output channel retained by the layer. Third, if the layer is not compressed but the layer above it is compressed, match the indices of the input channels of this layer with the indices of the output channels that are preserved in the layer above, and then copy the weights. Fourth, since neither the previous nor the current layer is compressed, the weights are directly copied; finally, an SNN model with completed weight copying is obtained.

[0013] Preferably, step (3) includes the following steps: (3-1) Iterate through the pruned SNN model obtained in step (2) to obtain the scaling factor of each output channel of the convolutional layer and fully connected layer of the model; (3-2) Using the scaling factor of each output channel of the convolutional layer and fully connected layer obtained in step (3-1), obtain the quantized weight of the output channel and the threshold of the ST-BIF neuron corresponding to the output channel; (3-3) Copy the quantized weights of all output channels of the convolutional and fully connected layers obtained in step (3-2) and the thresholds of the ST-BIF neurons corresponding to all output channels to the pruned SNN model obtained in step (2) to obtain the quantized SNN model.

[0014] Preferably, step (4) includes the following steps: (4-1) Obtain the hardware parameter information of the target edge device, and use the hardware parameter information to adapt the quantized SNN model obtained in step (3) to obtain the adapted SNN model. (4-2) Use the ONNX export tool to convert the adapted SNN model obtained in step (4-1) into an executable SNN model. Step (5) includes the following steps: (5-1) Using the executable SNN model obtained in step (4), the image to be identified in the target edge device obtained in step (1) is processed sequentially by image size normalization, pixel value standardization, channel number matching, and tensor format conversion to obtain tensor T; (5-2) Input the tensor T obtained in step (5-1) into the executable SNN model obtained in step (4) for inference calculation to obtain the image recognition result.

[0015] According to another aspect of the present invention, an edge device image recognition method based on spiking neural network compression optimization is provided, comprising the following modules: The first module is used to acquire the image to be identified and the ANN model in the target edge device, construct an SNN model based on the ANN model, and optimize the SNN model to obtain the optimized SNN model. The second module is used to process the optimized SNN model obtained from the first module using a structured pruning method to obtain a pruned SNN model. The third module is used to perform 4-bit quantization on the pruned SNN model obtained from the second module to obtain the quantized SNN model. The fourth module is used to deploy the quantized SNN model obtained in the third module to the target edge device to obtain an executable SNN model. The fifth module is used to identify the image to be identified in the target edge device obtained by the first module using the executable SNN model obtained by the fourth module, so as to obtain the image recognition result.

[0016] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects: (1) Since the present invention adopts step (1), the SNN model can utilize the richer membrane potential dynamic characteristics and pulse firing mechanism of the bipolar integral-and-fire with spike tracing (ST-BIF) neuron model, thus solving the technical problem that the existing method of converting ANN to SNN has low image recognition accuracy due to the single dynamic characteristics of neuronal membrane potential; (2) Since the present invention adopts steps (2) to (3), it eliminates redundant parameters of the SNN model and greatly reduces storage overhead and computational complexity by using structured pruning and 4-bit quantization collaborative compression method. Therefore, it can solve the technical problems of existing methods for directly deploying ANN models, which have large number of parameters, high computational load, and high energy consumption, and are difficult to meet the low power consumption and lightweight requirements of edge devices. (3) Since the present invention adopts steps (1) to (3), it constructs an SNN model based on the ANN to SNN method and performs pruning and quantization, without directly training the SNN model. Therefore, it can solve the technical problems of difficult training, slow convergence speed, unstable gradient propagation and high training cost of the existing direct training SNN model method. (4) Since the present invention adopts steps (1) to (5), the entire process revolves around the SNN event-driven computing paradigm to perform structural and quantization adaptation, maximizing the advantages of low-power computing. Therefore, it can solve the technical problem that the existing lightweight ANN model optimization method is not optimized for the SNN paradigm and is difficult to fully utilize the low-power advantages of edge devices. (5) The present invention can flexibly adjust the compression ratio and quantization bit width of the SNN model according to the storage capacity and computation bit width of the edge device, adapt to different edge devices, and has wide applicability; (6) The operation sequence of pruning and quantization in this invention can avoid a sharp drop in model accuracy after quantization, and achieve a balance between lightweight and high accuracy. Attached Figure Description

[0017] Figure 1 This is a flowchart of the edge device image recognition method based on pulse neural network compression optimization according to the present invention; Figure 2 This is a comparison diagram of the SNN model structure before and after using the spiking neural network compression optimization method of this invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0019] The basic idea of ​​this invention is to provide an edge device image recognition method based on spiking neural networks (SNN) compression optimization. This method is based on the conversion from artificial neural networks (ANNs) to SNNs. It constructs the SNN model by introducing bipolar integral-and-fire with spike tracking (ST-BIF) neurons, improving the image recognition accuracy of the SNN. Then, through structured pruning and 4-bit quantization co-compression, a lightweight SNN model is obtained. A comparison of the before and after model structures is shown in [link to previous invention]. Figure 2Ultimately, lightweight SNN models can be deployed on edge devices to perform image recognition, adapting to the storage and power-constrained characteristics of edge devices.

[0020] like Figure 1 As shown, this invention provides an edge device image recognition method based on spiking neural network compression optimization, comprising the following steps: (1) Obtain the image to be identified and the ANN model (using an activation function with trainable parameters) in the target edge device, construct the SNN model based on the ANN model, and optimize the SNN model to obtain the optimized SNN model; This step includes the following steps: (1-1) Obtain the image to be identified in the target edge device from the image acquisition module (camera, industrial camera or other vision sensor), and construct an ANN model using the ANN model parameter file that has been trained and saved on the training device in advance. Replace the activation function of the ANN model with ST-BIF neurons to construct an SNN model. Specifically, in the process of constructing the SNN model in this step, the weights of the SNN model are set to be equal to the weights of the ANN model, the membrane potential update rule of the ST-BIF neuron is set for the SNN model, the threshold of the ST-BIF neuron in the SNN model is set to be equal to the trainable parameter in the activation function of the ANN model, the membrane potential of the ST-BIF neuron in the SNN model is initialized to half of the threshold of the ST-BIF neuron, and the membrane potential reset strategy of the ST-BIF neuron in the SNN model is set to soft reset to obtain the SNN model.

[0021] Specifically, the activation function expression used in this ANN model is: ; in This represents the output of the activation function. This represents the input received by the activation function (from the previous layer or the input image). It is a trainable parameter. () indicates rounding down to the nearest integer. This represents a truncation function that truncates the first parameter to between 0 and 1.

[0022] Specifically, the membrane potential update rule for ST-BIF neurons in this SNN model is as follows: ; ; ; in This represents the membrane potential of the ST-BIF neuron at time t. The input received by the ST-BIF neuron at time t (from the previous layer ST-BIF neuron or the input image). This represents the threshold of the ST-BIF neuron. This is a pulse tracker at time t, used to record the number of pulses that the ST-BIF neuron has fired to the next layer of ST-BIF neurons up to time t. To allow the maximum number of pulses that an ST-BIF neuron can fire to the next layer of ST-BIF neurons, To determine the minimum number of positive pulses that must be fired to the next layer of ST-BIF neurons before the ST-BIF neuron can fire a negative pulse to the next layer of ST-BIF neurons, This represents a function that determines whether an ST-BIF neuron fires a positive pulse, a negative pulse (crucial for mitigating non-uniform errors during ANN to SNN conversion), or no pulse to the next layer of ST-BIF neurons, based on the membrane potential and threshold of the ST-BIF neuron and a pulse tracker. This indicates the current membrane potential of the ST-BIF neuron. This indicates the value currently recorded by the pulse tracker.

[0023] (1-2) The convolutional layer and batch normalization layer in the SNN model constructed in step (1-1) are fused to obtain the optimized SNN model (which has low computational cost).

[0024] (2) Use the structured pruning method to process the optimized SNN model obtained in step (1) (i.e., use the new pruning score function to remove redundant parameters in the SNN model and fine-tune the SNN model) to obtain the pruned SNN model. This step includes the following steps: (2-1) Iterate through the optimized SNN model obtained in step (1) to obtain the pruning score of each output channel of the convolutional layer and fully connected layer in the SNN model; This step includes the following steps: (2-1-1) Iterate through the optimized SNN model obtained in step (1) to obtain the scaling factor (scale) of each output channel of the convolutional and fully connected layers in the SNN model. Specifically, the scaling factor of the i-th output channel The calculation formula is: ; Where i∈[1, the total number of output channels in the convolutional and fully connected layers of the SNN model], It is the maximum value among all the absolute values ​​of all weights in the i-th output channel (the numerator is 7 because the largest integer that a 4-bit signed number can represent is 7). (2-1-2) Based on the scaling factor (scale) of each output channel of the convolutional layer and the fully connected layer obtained in step (2-1-1), obtain the quantized weight of the output channel and the threshold of the ST-BIF neuron corresponding to the output channel; Specifically, the weights after quantization of the i-th output channel and the ST-BIF neuron threshold corresponding to this output channel The calculation formula is: ; ; in This represents the weight of the i-th output channel before quantization. This represents the weight of the quantized i-th output channel. This represents the threshold of the ST-BIF neuron corresponding to the i-th output channel before quantization (before quantization, all output channels in each convolutional layer and fully connected layer share the same ST-BIF neuron threshold). This represents the threshold value of the ST-BIF neuron corresponding to the i-th output channel after quantization. (2-1-3) Based on the quantized weights of each output channel of the convolutional and fully connected layers obtained in step (2-1-2) and the threshold of the ST-BIF neuron corresponding to that output channel, obtain the pruning score of each output channel of the convolutional and fully connected layers in the optimized SNN model. Specifically, the pruning score of the i-th output channel The calculation formula is: + ; in This indicates taking the absolute value. This represents the average weight of all weights in the i-th output channel; (2-2) Sort the pruning scores of all output channels of the convolutional and fully connected layers in the optimized SNN model obtained in step (2-1) in descending order, compress the optimized SNN model according to the set SNN model compression ratio to obtain the compressed SNN model, and copy the weights of the optimized SNN model to the compressed SNN model to obtain the SNN model with completed weight copying. This step includes the following steps: (2-2-1) Obtain the number of output channels of the i-th layer (including convolutional and fully connected layers) in the compressed SNN model based on the number of output channels of the i-th layer in the optimized SNN model obtained in step (2-1). And based on the number of output channels of the i-th layer Construct the compressed SNN model; Specifically, the number of output channels of the i-th layer in the compressed SNN model The calculation formula is: ; in This represents the number of output channels in the i-th layer of the optimized SNN model obtained in step (2-1). This represents the compression ratio set for the i-th layer in the optimized SNN model.

[0025] (2-2-2) Sort the pruning scores of all output channels of the convolutional and fully connected layers in the optimized SNN model obtained in step (2-1) in descending order. Based on the sorting result, determine the index of each output channel that needs to be retained in each layer of the convolutional and fully connected layers. Based on the index, copy the weights of the corresponding output channels of the optimized SNN model obtained in step (1) to the compressed SNN model constructed in step (2-2-1) to obtain the SNN model with completed weight copying. Specifically, if a certain layer (convolutional layer or fully connected layer) in the compressed SNN model has num (where num is a natural number greater than or equal to 1) output channels, then from all the output channels of that layer, select the num output channels corresponding to the first num results in the sorting results obtained in step (2-1), and obtain multiple indices of these num output channels in all the output channels of that layer (assuming num=3, and the indices are 1, 3, 8, that is, the pruning scores of these three channels are the highest in that layer), and copy the weights of these num output channels in the optimized SNN model to the output channels corresponding to these indices in the compressed SNN model respectively. The weight copying process is divided into four cases: First, if both the current and previous layers are compressed, the input channels of this layer must correspond to the retained output channels of the previous layer, and the weights are copied using the indices of the retained output channels. Second, if the current layer is compressed but the previous layer is not, the weights are copied using the indices of the retained output channels. Third, if the current layer is not compressed but the previous layer is compressed, the input channels of this layer are matched with the indices of the retained output channels of the previous layer, and the weights are copied. Fourth, if both the current and previous layers are not compressed, the weights are directly copied. This process ultimately yields the SNN model with completed weight copying.

[0026] (2-3) Fine-tune the SNN model with completed weight replication obtained in step (2-2) to obtain the pruned SNN model; Specifically, when fine-tuning the SNN model with completed weight replication obtained in step (2-2), a cosine annealing learning rate scheduler and a stochastic gradient descent optimizer are used to train for 30-100 rounds to obtain the pruned SNN model.

[0027] (3) Perform 4-bit quantization on the pruned SNN model obtained in step (2) to obtain the quantized SNN model; This step includes the following steps: (3-1) Iterate through the pruned SNN model obtained in step (2) to obtain the scaling factor of each output channel of the convolutional layer and fully connected layer of the model; Specifically, the scaling factor calculation formula for all output channels of the pruned SNN model is the same as the scaling factor calculation formula in step (2-1-1).

[0028] (3-2) Using the scaling factor of each output channel of the convolutional layer and fully connected layer obtained in step (3-1), obtain the quantized weight of the output channel and the threshold of the ST-BIF neuron corresponding to the output channel; Specifically, the formula for calculating the weight of each output channel after quantization and the corresponding ST-BIF neuron threshold is the same as the formula for calculating the weight of each output channel after quantization and the corresponding ST-BIF neuron threshold in step (2-1-2).

[0029] (3-3) Copy the quantized weights of all output channels of the convolutional and fully connected layers obtained in step (3-2) and the thresholds of the ST-BIF neurons corresponding to all output channels to the pruned SNN model obtained in step (2) to obtain the quantized SNN model.

[0030] The advantage of steps (1) to (3) above is that by optimizing the ST-BIF neuron model, pruning the structured branches, and quantizing the 4-bit quantization workflow, a balance between accuracy and lightweighting of the SNN model is achieved, thereby improving the technical problems of low accuracy, high storage overhead, and high energy consumption of the existing SNN model when running on edge devices.

[0031] (4) Deploy the quantized SNN model obtained in step (3) to the target edge device to obtain an executable SNN model; This step includes the following steps: (4-1) Obtain the hardware parameter information of the target edge device, and use the hardware parameter information to adapt the quantized SNN model obtained in step (3) to obtain the adapted SNN model. Specifically, firstly, the hardware parameter information of the target edge device is obtained, including storage capacity and supported computational bit width; then, based on the storage capacity, the allowed model storage occupancy threshold of the target edge device is determined (preferably, the threshold is set to 20% to 40% of the storage capacity) to adjust the size of the pruned SNN model obtained in step (2) (if the size of the pruned SNN model is greater than the threshold, the set SNN model compression ratio is increased, and step (2) is executed again); and based on the supported computational bit width, the quantization bit width in step (3) is adjusted (if the target edge device supports 8-bit computational bit width, the numerator in the scaling factor calculation formula for each output channel in step (2-1-1) is changed from 7 to 7). =127, repeat step (3) to obtain the adapted SNN model; The advantage of step (4-1) is that the compression rate and quantization bit width of the SNN model can be flexibly adjusted according to the storage capacity and computation bit width of the edge device to adapt to different edge devices.

[0032] (4-2) Perform model format conversion on the adapted SNN model obtained in step (4-1) to obtain an executable SNN model; Specifically, the adapted SNN model obtained in step (4-1) is converted into a model representation format supported by the target edge device using the ONNX export tool (or the model export interface provided by the deep learning framework), and the converted SNN model is loaded into the target edge device to obtain an executable SNN model. (5) Use the executable SNN model obtained in step (4) to identify the image to be identified in the target edge device obtained in step (1) to obtain the image recognition result; This step includes the following steps: (5-1) Using the executable SNN model obtained in step (4), the image to be identified in the target edge device obtained in step (1) is processed sequentially by image size normalization, pixel value standardization, channel number matching, and tensor format conversion to obtain tensor T (which meets the input requirements of the SNN model). (5-2) Input the tensor T obtained in step (5-1) into the executable SNN model obtained in step (4) for inference calculation to obtain the image recognition result; Specifically, this step involves inputting the tensor T obtained in step (5-1) into the executable SNN model obtained in step (4). The SNN model outputs the probability distribution of multiple image recognition categories and selects the image recognition category with the highest probability as the image recognition result.

[0033] Simulation results To verify the effectiveness of the method of this invention and to illustrate its application potential on resource-constrained edge devices, this experiment selected the CIFAR-10 and CIFAR-100 datasets as test datasets. The test results of the SNN model obtained using the method of this invention (ST-BIF neurons + structured pruning + 4-bit quantization) and the original ANN model on the above two datasets were compared.

[0034] This embodiment mainly uses the accuracy, size, and energy consumption of the compressed and optimized SNN model as the evaluation criteria for the feasibility of deployment to edge devices. When calculating the model's energy consumption, it is taken as 12.5 pJ for a single floating-point operation (FLOP) and 77 fJ for a single synaptic operation (SOP).

[0035] The experimental parameters are set as follows: ANN model: VGG16; ST-BIF neuron parameter settings: =32, =0; Structured pruning parameters: ① On the CIFAR-10 dataset, the compression rate is set to 57%, the learning rate used for fine-tuning is 0.002, and the training is conducted for 30 epochs; ② On the CIFAR-100 dataset, the compression rate is set to 71%, the learning rate used for fine-tuning is 0.003, and the training is conducted for 70 epochs.

[0036] The experimental results are shown in Table 1 (CIFAR-10 dataset) and Table 2 (CIFAR-100 dataset): Table 1 Weighted bit width Model accuracy Model size Model energy consumption ANN 32bit 95.02% 128.32MB 8.30mJ SNN 4bit 90.73% 7.04MB 0.22mJ Table 2 Weighted bit width Model accuracy Model size Model energy consumption ANN 32bit 76.28% 129.73MB 8.31mJ SNN 4bit 66.59% 4.96MB 0.04mJ As can be seen from Tables 1 and 2 above, on the CIFAR-10 dataset, the method of this invention significantly compresses the model size from 128.32MB to 7.04MB, a compression ratio of 94.5%; the model energy consumption is reduced from 8.30mJ to 0.22mJ, a reduction of 97.3%, while the image recognition accuracy only decreases by 4.29%. On the more complex CIFAR-100 dataset, the method of this invention compresses the model size from 129.73MB to 4.96MB, a compression ratio of 96.2%; the energy consumption is only 0.04mJ, a reduction of 99.5% compared to the original ANN model. It achieves extreme lightweight and low power consumption advantages with minimal accuracy loss, fully demonstrating the superiority of the ST-BIF neuron optimization, structured pruning, and 4-bit quantization collaborative compression strategy, which can meet the high-precision, low-storage, and low-energy consumption requirements of edge device image recognition in complex scenarios.

[0037] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An edge device image recognition method based on pulse neural network compression optimization, characterized in that, Includes the following steps: (1) Obtain the image to be identified and the ANN model in the target edge device, construct the SNN model based on the ANN model, and optimize the SNN model to obtain the optimized SNN model; (2) The optimized SNN model obtained in step (1) is processed using the structured pruning method to obtain the pruned SNN model; (3) Perform 4-bit quantization on the pruned SNN model obtained in step (2) to obtain the quantized SNN model; (4) Deploy the quantized SNN model obtained in step (3) to the target edge device to obtain an executable SNN model; (5) Use the executable SNN model obtained in step (4) to identify the image to be identified in the target edge device obtained in step (1) to obtain the image recognition result.

2. The edge device image recognition method based on spiking neural network compression optimization according to claim 1, characterized in that, Step (1) includes the following steps: (1-1) Obtain the image to be identified from the image acquisition module of the target edge device, and construct an ANN model using the ANN model parameter file that has been trained and saved on the training device in advance. Replace the activation function of the ANN model with ST-BIF neurons to construct an SNN model. In the process of constructing the SNN model in this step, the weights of the SNN model are set to be equal to the weights of the ANN model. The membrane potential update rule of the ST-BIF neurons is set for the SNN model. The threshold of the ST-BIF neurons in the SNN model is set to be equal to the trainable parameter in the activation function of the ANN model. The membrane potential of the ST-BIF neurons in the SNN model is initialized to half of the threshold of the ST-BIF neurons. The membrane potential reset strategy of the ST-BIF neurons in the SNN model is set to soft reset to obtain the SNN model. (1-2) The convolutional layer and the batch normalization layer in the SNN model constructed in step (1-1) are fused to obtain the optimized SNN model.

3. The edge device image recognition method based on spiking neural network compression optimization according to claim 1 or 2, characterized in that, The activation function expression used by the ANN model is: ； in This represents the output of the activation function. This represents the input received by the activation function. It is a trainable parameter. () indicates rounding down to the nearest integer. This represents a truncation function that truncates the first parameter to a value between 0 and 1. The membrane potential update rule for ST-BIF neurons in the SNN model is as follows: ； ； ； in This represents the membrane potential of the ST-BIF neuron at time t. The input received by the ST-BIF neuron at time t. This represents the threshold of the ST-BIF neuron. This is a pulse tracker at time t, used to record the number of pulses that the ST-BIF neuron has fired to the next layer of ST-BIF neurons up to time t. To allow the maximum number of pulses that an ST-BIF neuron can fire to the next layer of ST-BIF neurons, To determine the minimum number of positive pulses that must be fired to the next layer of ST-BIF neurons before the ST-BIF neuron can fire a negative pulse to the next layer of ST-BIF neurons, This represents a function that determines whether an ST-BIF neuron fires a positive pulse, a negative pulse, or no pulse to the next layer of ST-BIF neurons, based on the membrane potential and threshold of the ST-BIF neuron and a pulse tracker. This indicates the current membrane potential of the ST-BIF neuron. This indicates the value currently recorded by the pulse tracker.

4. The edge device image recognition method based on spiking neural network compression optimization according to any one of claims 1 to 3, characterized in that, Step (2) includes the following steps: (2-1) Iterate through the optimized SNN model obtained in step (1) to obtain the pruning score of each output channel of the convolutional layer and fully connected layer in the SNN model; (2-2) Sort the pruning scores of all output channels of the convolutional and fully connected layers in the optimized SNN model obtained in step (2-1) in descending order, compress the optimized SNN model according to the set SNN model compression ratio to obtain the compressed SNN model, and copy the weights of the optimized SNN model to the compressed SNN model to obtain the SNN model with completed weight copying. (2-3) Fine-tune the SNN model with completed weight replication obtained in step (2-2) to obtain the pruned SNN model.

5. The edge device image recognition method based on spiking neural network compression optimization according to claim 4, characterized in that, Step (2-1) includes the following steps: (2-1-1) Iterate through the optimized SNN model obtained in step (1) to obtain the scaling factor (scale) for each output channel of the convolutional and fully connected layers in the SNN model: ； Where i∈[1, the total number of output channels in the convolutional and fully connected layers of the SNN model], It is the maximum value among all the absolute values ​​of weights in the i-th output channel; (2-1-2) Based on the scaling factor (scale) of each output channel of the convolutional layer and fully connected layer obtained in step (2-1-1), obtain the quantized weight of the output channel and the threshold of the corresponding ST-BIF neuron. : ； ； in This represents the weight of the i-th output channel before quantization. This represents the weight of the quantized i-th output channel. This represents the threshold value of the ST-BIF neuron corresponding to the i-th output channel before quantization. This represents the threshold value of the ST-BIF neuron corresponding to the i-th output channel after quantization. (2-1-3) Based on the quantized weights of each output channel of the convolutional and fully connected layers obtained in step (2-1-2) and the threshold of the ST-BIF neuron corresponding to that output channel, obtain the pruning score of each output channel of the convolutional and fully connected layers in the optimized SNN model. : + ； in This indicates taking the absolute value. This represents the average weight of all weights in the i-th output channel.

6. The edge device image recognition method based on spiking neural network compression optimization according to claim 5, characterized in that, Step (2-2) includes the following steps: (2-2-1) Obtain the number of output channels of the i-th layer in the compressed SNN model based on the number of output channels of the i-th layer in the optimized SNN model obtained in step (2-1). And based on the number of output channels of the i-th layer Construct a compressed SNN model; where the number of output channels in the i-th layer of the compressed SNN model is... The calculation formula is: ； in This represents the number of output channels in the i-th layer of the optimized SNN model obtained in step (2-1). This represents the compression ratio set for the i-th layer in the optimized SNN model; (2-2-2) Sort the pruning scores of all output channels of the convolutional and fully connected layers in the optimized SNN model obtained in step (2-1) in descending order. Based on the sorting result, determine the index of each output channel that needs to be retained in each layer of the convolutional and fully connected layers. Based on the index, copy the weights of the corresponding output channels of the optimized SNN model obtained in step (1) to the compressed SNN model constructed in step (2-2-1) to obtain the SNN model with completed weight copying.

7. The edge device image recognition method based on spiking neural network compression optimization according to claim 6, characterized in that, Step (2-2-2) specifically involves the following steps: If a layer in the compressed SNN model has num output channels, select the num output channels corresponding to the top num results obtained in step (2-1) from all output channels of that layer. Obtain the multiple indices of these num output channels in all output channels of that layer, and copy the weights of these num output channels in the optimized SNN model to the output channels corresponding to these indices in the compressed SNN model. Here, num is a natural number greater than or equal to 1, and the copying process is divided into the following four cases: First, the layer above is compressed, and the input channels of this layer must correspond to the output channels retained by the previous layer. Then, the weights are copied by combining the index of the output channels retained by this layer. Second, if the layer is compressed and the previous layer is not compressed, the weights are copied according to the index of the output channel retained by the layer. Third, if the layer is not compressed but the layer above it is compressed, match the indices of the input channels of this layer with the indices of the output channels that are preserved in the layer above, and then copy the weights. Fourth, if the layer above is not compressed, the weights are directly copied. The final result is an SNN model with weight replication completed.

8. The edge device image recognition method based on spiking neural network compression optimization according to claim 7, characterized in that, Step (3) includes the following steps: (3-1) Iterate through the pruned SNN model obtained in step (2) to obtain the scaling factor of each output channel of the convolutional layer and fully connected layer of the model; (3-2) Using the scaling factor of each output channel of the convolutional layer and fully connected layer obtained in step (3-1), obtain the quantized weight of the output channel and the threshold of the ST-BIF neuron corresponding to the output channel; (3-3) Copy the quantized weights of all output channels of the convolutional and fully connected layers obtained in step (3-2) and the thresholds of the ST-BIF neurons corresponding to all output channels to the pruned SNN model obtained in step (2) to obtain the quantized SNN model.

9. The edge device image recognition method based on spiking neural network compression optimization according to claim 8, characterized in that, Step (4) includes the following steps: (4-1) Obtain the hardware parameter information of the target edge device, and use the hardware parameter information to adapt the quantized SNN model obtained in step (3) to obtain the adapted SNN model. (4-2) Use the ONNX export tool to convert the adapted SNN model obtained in step (4-1) into an executable SNN model. Step (5) includes the following steps: (5-1) Using the executable SNN model obtained in step (4), the image to be identified in the target edge device obtained in step (1) is processed sequentially by image size normalization, pixel value standardization, channel number matching, and tensor format conversion to obtain tensor T; (5-2) Input the tensor T obtained in step (5-1) into the executable SNN model obtained in step (4) for inference calculation to obtain the image recognition result.

10. An edge device image recognition method based on compression optimization of a spiking neural network, characterized in that, Includes the following modules: The first module is used to acquire the image to be identified and the ANN model in the target edge device, construct an SNN model based on the ANN model, and optimize the SNN model to obtain the optimized SNN model. The second module is used to process the optimized SNN model obtained from the first module using a structured pruning method to obtain a pruned SNN model. The third module is used to perform 4-bit quantization on the pruned SNN model obtained from the second module to obtain the quantized SNN model. The fourth module is used to deploy the quantized SNN model obtained in the third module to the target edge device to obtain an executable SNN model. The fifth module is used to identify the image to be identified in the target edge device obtained by the first module using the executable SNN model obtained by the fourth module, so as to obtain the image recognition result.

Images