Edge end spiking neural network compression and deployment method and system

Abstract

The invention relates to an edge end spiking neural network compression and deployment method and system, and belongs to the technical field of machine learning, the edge end spiking neural network compression and deployment method performs forward propagation on an initial spiking neural network model based on acquired multi-modal data, obtaining a memory use state in a forward propagation process based on hardware sensing initialization, and performing dynamic sparsification on the initial pulse neural network model based on the memory use state to construct a sparsified pulse neural network; based on the acquired activation statistical information of the sparse spiking neural network, performing dynamic pruning on the sparse spiking neural network by adopting a pruning strategy based on neuron activeness so as to realize compression of a spiking neural network model; the compressed spiking neural network model is optimized according to the edge end configuration, and the optimized spiking neural network model is deployed to the edge end to execute the reasoning task, so that the storage requirement and the calculation complexity are reduced.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and in particular to a method and system for compressing and deploying edge spiking neural networks. Background Technology

[0002] Spiking Neural Networks (SNNs) are third-generation neural networks that use pulse signals as the basic unit of information transmission. They simulate the pulse firing and information processing of biological neurons and are currently the neural network models that are closest to the operating mechanism of the biological brain. SNNs can process spatial and temporal information simultaneously and encode and process dynamically changing data through the time series of pulses. This gives them a significant advantage in processing time series data (such as speech, action recognition, and optical flow estimation).

[0003] However, due to the sparseness of spikes and the non-differentiability of spike events, traditional SNNs cannot be directly trained using mature backpropagation algorithms. Large-scale SNNs require a large number of neurons and synapses to buffer the weight matrix, resulting in high memory requirements. Furthermore, as the complexity of the task increases, the size of the SNN often increases, leading to high computational complexity. Moreover, SNNs consume a lot of energy when processing complex tasks, making it difficult to deploy spiking neural networks on low-power devices. Summary of the Invention

[0004] In view of this, it is necessary to provide a method and system for compressing and deploying edge spiking neural networks to solve the technical problems of large memory requirements and high computational complexity when deploying spiking neural networks on low-power devices.

[0005] To address the aforementioned problems, in a first aspect, the present invention provides a method for compressing and deploying edge-end spiking neural networks, comprising: The initial spiking neural network model is forward propagated based on the acquired multimodal data. The memory usage state during the forward propagation is obtained based on hardware-aware initialization. The initial spiking neural network model is then dynamically sparsified based on the memory usage state to construct a sparse spiking neural network. Based on the activation statistics of the obtained sparse spiking neural network, a pruning strategy based on neuron activity is used to dynamically prune the sparse spiking neural network in order to compress the spiking neural network model. The compressed spiking neural network model is optimized based on the edge configuration, and the optimized spiking neural network model is deployed to the edge to perform inference tasks.

[0006] In one possible implementation, the initial spiking neural network model includes a feature separation module, an encoding module, a hidden module, and a decoding module. The encoding module includes convolutional layers, normalization layers, and spiking neuron layers. The hidden module includes a cross-modal fusion submodule and a spiking enhancement submodule. The multimodal data includes ultrasound data, electromyography data, and temperature data. The forward propagation of the spiking neural network model based on the acquired multimodal data includes: The multimodal data is input into an initial spiking neural network model, and the feature separation module performs feature separation on the multimodal data to obtain acoustic features, electromyographic features, and temperature features. The acoustic features, electromyographic features, and temperature features are fused through the cross-modal fusion submodule of the hidden module, wherein the acoustic features, electromyographic features, and temperature features are fused by feature weighting through a cross-modal attention mechanism; The fused features are input into the encoding module. The fused features are processed sequentially by the convolutional layer and normalization layer of the encoding module. Then, the normalized features are processed by the spiking neuron layer to obtain a spiking signal. After the pulse signal is optimized and fused by the pulse enhancement submodule of the hidden module, the fused pulse signal is decoded by the decoding module to output the classification result.

[0007] In one possible implementation, the forward propagation of the initial spiking neural network model based on the acquired multimodal data further includes: Obtain the input tensor, calculate the scaling factor based on the input tensor and the quantization bit depth at the edges, obtain the quantized tensor based on the input tensor and the scaling factor, and dynamically quantize the weights of the initial spiking neural network model based on the quantized tensor.

[0008] In one possible implementation, obtaining the memory usage state during the forward propagation based on hardware-aware initialization, and dynamically sparsifying the initial spiking neural network model based on the memory usage state to construct a sparse spiking neural network, includes: The memory usage state during forward propagation is obtained based on hardware-aware initialization. When the memory usage state exceeds the memory threshold, a random tensor corresponding to the weight matrix is ​​obtained, and a Boolean mask is generated based on the preset sparsity and the random tensor. Obtain the weight matrix of the fully connected layer in the decoding module, and dynamically sparsify the initial spiking neural network model based on the weight matrix and Boolean mask to construct a sparse spiking neural network.

[0009] In one possible implementation, the hardware-aware initialization includes: The weights of the encoding modules are quantized according to the configuration at the edge. Register a forward propagation hook for the decoding module to monitor memory usage.

[0010] In one possible implementation, the activation statistics based on the acquired sparsed spiking neural network include: Initialize the statistical dictionary, traverse the samples of multimodal data input to the sparse spiking neural network, obtain the activation data of each layer in the sparse spiking neural network during the forward propagation, accumulate the activation data into the statistical dictionary, obtain the number of pulse firings of the sparse spiking neural network during the forward propagation, and determine the activation statistics of each layer in the sparse spiking neural network based on the accumulated value of the activation data in the statistical dictionary and the number of pulse firings.

[0011] In one possible implementation, the pruning strategy includes spatiotemporal joint pruning and progressive gradient-aware pruning; the dynamic pruning of the sparse spiking neural network based on the acquired activation statistics of the sparse spiking neural network, using a pruning strategy based on neuron activity, to achieve compression of the spiking neural network model, includes: Based on the activation statistics of the sparsed spiking neural network, a spatiotemporal joint pruning score is calculated, wherein the spatiotemporal joint pruning score includes the spatial domain importance score of the weight matrix and the temporal domain importance score of the temporal activity. Based on the set pruning score threshold and the spatiotemporal joint pruning score, the sparse spiking neural network is spatiotemporally joint pruned to obtain the pruned sparse spiking neural network. Progressive gradient-aware pruning is performed on the pruned sparse spiking neural network to compress the spiking neural network model.

[0012] In one possible implementation, optimizing the compressed spiking neural network model based on the edge configuration includes: Based on the obtained cache size at the edge, a pruner corresponding to the cache size is constructed, and the weights of each layer of the compressed spiking neural network model are structurally pruned based on the pruner.

[0013] In one possible implementation, deploying the optimized spiking neural network model to the edge includes: The optimized spiking neural network model is converted into a format supported by the edge, and the converted spiking neural network model is deployed to the edge. After initialization and deployment monitoring at the edge, the running effect of the converted spiking neural network model on the edge is verified, and the ADC configuration is optimized according to the characteristics of the edge.

[0014] Secondly, the present invention also provides an edge-end spiking neural network compression and deployment system, comprising: The network construction module is used to perform forward propagation on an initial spiking neural network model based on acquired multimodal data. Specifically, the memory usage state during the forward propagation process is obtained based on hardware-aware initialization, and the initial spiking neural network model is dynamically sparsified based on the memory usage state to construct a sparse spiking neural network. The network compression module is used to dynamically prune the sparse spiking neural network based on the obtained activation statistics of the sparse spiking neural network, using a pruning strategy based on neuron activity, so as to achieve compression of the spiking neural network model. The network deployment module is used to optimize the compressed spiking neural network model according to the edge configuration, and then deploy the optimized spiking neural network model to the edge to perform inference tasks.

[0015] The beneficial effects of this invention are as follows: A forward propagation of an initial spiking neural network model is performed based on acquired multimodal data. Specifically, memory usage during the forward propagation is obtained based on hardware-aware initialization. The initial spiking neural network model is then dynamically sparsified based on this memory usage to construct a sparse spiking neural network. This ensures efficient memory usage during operation and reduces memory requirements. Based on the activation statistics of the acquired sparse spiking neural network, a pruning strategy based on neuron activity is used to dynamically prune the sparse spiking neural network, thereby compressing the model. This dynamic pruning improves computational efficiency and further reduces memory requirements. Finally, the compressed spiking neural network model is optimized according to edge configuration and deployed to the edge to perform inference tasks, further reducing storage requirements and computational complexity. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of an embodiment of the edge-end spiking neural network compression and deployment method provided by the present invention; Figure 2 This is a schematic diagram of an embodiment of the edge-end spiking neural network compression and deployment system provided by the present invention. Detailed Implementation

[0018] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0019] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0020] A specific embodiment of the present invention discloses a method for compression and deployment of edge-end spiking neural networks, such as... Figure 1 As shown, the edge-end spiking neural network compression and deployment method includes: S101. The initial spiking neural network model is propagated forward based on the acquired multimodal data. The memory usage state during the forward propagation process is obtained based on hardware-aware initialization. The initial spiking neural network model is dynamically sparsified based on the memory usage state to construct a sparse spiking neural network. It should be noted that hardware-aware initialization reduces storage requirements and lowers memory pressure by quantizing the weights of the encoding layer and registering memory monitoring hooks.

[0021] S102. Based on the activation statistics of the obtained sparse spiking neural network, a pruning strategy based on neuron activity is used to dynamically prune the sparse spiking neural network in order to compress the spiking neural network model. It should be noted that by pruning the spiking neural network using a pruning strategy based on neuron activity, the characteristics and activation statistics of the sparse spiking neural network are fully utilized. The refined pruning strategy achieves effective model compression, which reduces computational complexity while maintaining high performance.

[0022] S103. Optimize the compressed spiking neural network model according to the edge configuration, and deploy the optimized spiking neural network model to the edge to perform inference tasks; It should be noted that the compressed spiking neural network model is optimized based on the edge configuration, which further reduces storage requirements and computational complexity.

[0023] In some embodiments, in step S101, the initial spiking neural network model is forward-propagated based on the acquired multimodal data. Specifically, the memory usage state during forward propagation is obtained based on hardware-aware initialization, and the initial spiking neural network model is dynamically sparsified based on the memory usage state to construct a sparse spiking neural network. The multimodal data includes ultrasound data, electromyography data, and temperature data. The initial spiking neural network model (SNN) includes a feature separation module, an encoding module, a hidden module, and a decoding module. The encoding module includes convolutional layers, normalization layers, and spiking neuron layers. The hidden module includes a cross-modal fusion submodule and a spiking enhancement submodule. The weights of the encoding module are quantized, and the decoding... The module registers a forward propagation hook to achieve hardware-aware initialization. The weights of the quantization encoding module are quantized by quantizing the weights of the first convolutional layer of the encoding module. The quantization bit depth is specified according to the edge configuration via a JSON / YAML configuration file or command-line parameters. During initialization, the edge configuration is read to obtain the quantization bit depth, which is then passed to the encoding layer. Specifically, the first-layer convolutional weights of the encoding module are compressed according to the configured quantization bit depth (e.g., 8 bits), and a scaling factor is dynamically calculated based on the maximum absolute value of the input tensor. Floating-point weights are converted to low-precision integers to reduce storage requirements. A forward propagation hook is registered for the decoding module to monitor the system's peak memory usage in real time. If it exceeds a preset threshold (e.g., 100MB), a trigger is activated. Dynamic sparsity is implemented to reduce memory pressure. The spiking neural network model is forward-propagated based on the acquired multimodal data. Specifically, multimodal data is input into the initial spiking neural network model, and a feature separation module separates the multimodal data to obtain acoustic, electromyographic (EMG), and temperature features. In the feature separation stage, the input multimodal data (such as acoustic and EMG signals) is separated into independent acoustic and EMG features. The acoustic, EMG, and temperature features are then fused through a cross-modal fusion submodule of the hidden module. Specifically, a cross-modal attention mechanism is used to perform weighted fusion of acoustic, EMG, and temperature features. This cross-modal attention mechanism is a fully connected layer that receives multiple signals. After inputting modal features, acoustic, electromyographic (EMG), and temperature features are concatenated through an attention mechanism. Acoustic features undergo cross-modal attention processing (e.g., interaction with EMG features) to generate enhanced acoustic features. EMG features are processed through independent or symmetrical attention modules (e.g., interaction with acoustic features) to generate enhanced EMG features. The enhanced acoustic and EMG features are then fused along the feature dimension (e.g., channel dimension or time step feature weighting) to form a unified fused feature. The fused feature is then input into the encoding module. The fused feature is processed sequentially through the convolutional and normalization layers of the encoding module. Finally, the normalized feature is processed through a spiking neuron layer to obtain a spiking signal.The spiking neuron layer uses an Integrate-and-Fire (IF) model to convert normalized data into pulse signals. After optimization and fusion of these pulse signals through a pulse enhancement submodule within the hidden module, which consists of a dynamic weight generation unit and a pulse gating unit, the submodule achieves feature optimization and fusion. It generates dynamic weights and optimizes the pulse signals. The submodule performs pulse coding preprocessing on the input multimodal features, uses an SNN to generate dynamic weights, extracts the weight values, performs basic fusion, and returns the fusion result. Finally, a decoding module decodes the fused pulse signals and outputs the classification result.

[0024] During the forward propagation of the initial spiking neural network, the input tensor is acquired, and the scaling factor is calculated based on the input tensor and the quantization bit depth at the edges. The quantized tensor is obtained based on the input tensor and the scaling factor. The weights of the initial spiking neural network model are dynamically quantized based on the quantized tensor. Based on the maximum absolute value of the input tensor and the quantization bit depth, the tensor is divided by the scaling factor and rounded to the nearest integer, converted to a byte type, and finally multiplied by the scaling factor to return the quantized tensor.

[0025] During the forward propagation of the initial spiking neural network, a cross-modal fusion submodule performs basic fusion of multimodal data. This includes anomaly filtering for temperature data, updating adaptive noise weights, checking the quality of electromyography (EMG) data, adjusting another weight based on the quality, and executing a backup fusion strategy if the EMG quality is substandard. Otherwise, the fusion score is calculated based on the weights and the result is returned. Real-time optimization of the initial spiking neural network involves checking if it's in a GPU environment using a memory monitoring hook; if not, the process returns directly. The memory monitoring hook also retrieves the current peak GPU memory usage and converts it to K. B. If memory usage exceeds a preset threshold, sparsity acceleration is activated. When memory exceeds the limit, each layer of the model is traversed. If it is a fully connected layer, a random mask is generated, and the weights are multiplied by this mask, making some weights zero, thereby achieving sparsity. Specifically: the memory usage state during forward propagation is obtained based on hardware-aware initialization. When the memory usage state exceeds the memory threshold, a random tensor corresponding to the weight matrix is ​​obtained. A Boolean mask (0 / 1 matrix) is generated according to the preset sparsity and the random tensor. A random tensor with the same shape as the weight matrix is ​​generated, and the value of the random tensor is... Within the specified range, a Boolean mask is generated based on the configured sparsity. Values ​​less than the sparsity are set to True, and the rest are set to False. Finally, the Boolean mask is converted to a floating-point number. Its sparsity is the proportion of the attention weight matrix that is sparsified (e.g., 60% of the weights are set to zero). Redundant calculations are forcibly reduced through masking operations. The weight matrix of the fully connected layer in the decoding module is obtained. Based on the weight matrix and the Boolean mask, the initial spiking neural network model is dynamically sparsified. The generated mask is multiplied element-wise with the weight matrix of the linear layer (fully connected layer), thereby achieving partial zeroing of the weight matrix and achieving the purpose of sparsification, thus constructing a sparse spiking neural network.

[0026] The forward propagation of the initial spiking neural network is achieved through feature separation, basic fusion of multimodal data, cross-modal attention mechanism, spatiotemporal pulse processing, pulse-enhanced fusion, and dynamic weight quantization. Feature separation provides initial feature representation for forward propagation, basic fusion of multimodal data initially integrates features from different modalities, cross-modal attention mechanism enhances the interaction between different modalities and improves the expressive power of features, spatiotemporal pulse processing captures the temporal and spatial information of the data to support dynamic data processing, pulse-enhanced fusion further optimizes the fusion effect of multimodal data, and dynamic weight quantization optimizes computational efficiency in the inference stage of forward propagation.

[0027] In some embodiments, in step S102, based on the obtained activation statistics of the sparse spiking neural network, a pruning strategy based on neuron activity is used to dynamically prune the sparse spiking neural network to achieve compression of the spiking neural network model. The pruning strategy based on neuron activity includes spatiotemporal joint pruning. Obtaining the activation statistics of the sparse spiking neural network involves initializing a statistical dictionary, traversing each sample, obtaining activation data for each layer through forward propagation, accumulating the data for each layer, recording the number of pulse firings, and calculating the average firing frequency of each layer. Specifically, this involves: initializing a statistical dictionary, traversing the samples of multimodal data input to the sparse spiking neural network, obtaining activation data for each layer in the sparse spiking neural network during forward propagation, accumulating the activation data into the statistical dictionary, and obtaining the forward propagation... The process involves determining the activation statistics of each layer in the sparse spiking neural network based on the cumulative activation data in the statistical dictionary and the number of pulses fired. Based on these activation statistics, the sparse spiking neural network is pruned using spatiotemporal joint pruning and progressive gradient-aware pruning. Specifically, a spatiotemporal joint pruning score is calculated based on the activation statistics of the sparse spiking neural network. This score includes the spatial importance score of the weight matrix and the temporal importance score of the temporal activity. The sparse spiking neural network is pruned based on a set pruning score threshold and the spatiotemporal joint pruning score to obtain a pruned sparse spiking neural network. Progressive gradient-aware pruning is then applied to the pruned sparse spiking neural network to achieve compression of the spiking neural network model.Based on the activation statistics of the sparse spiking neural network, the spatial domain importance score and the temporal domain importance score of the weight matrix are calculated. Combining these spatial and temporal importance scores, a comprehensive score is generated. The spatiotemporal joint pruning scores are ranked. Based on a set pruning score threshold and the ranking of the spatiotemporal joint pruning scores, neurons to be retained are selected, resulting in the pruned sparse spiking neural network. Progressive gradient-aware pruning is then performed on the pruned sparse spiking neural network. This involves traversing the model modules, finding all linear layers (fully connected layers), generating masks based on gradients, updating existing masks, selecting the positions of the k largest values, generating new masks, and pruning the sparse spiking neural network based on these new masks. Specifically, this involves recursively checking for pruning... In the pruned sparse spiking neural network, the fully connected layers are selected, and their weight gradient matrices are obtained. A binary mask matrix of the same size as the weight gradient matrix is ​​created, and gradient scores are calculated. The gradient scores of each layer's weights are sorted, and the top k% of the highest-scoring positions (e.g., k=20%) are retained, while the remaining positions are set to 0. A new mask is generated, thus completing the progressive gradient-aware pruning of the sparse spiking neural network. After pruning, the sparse spiking neural network is updated. This involves obtaining the original layer objects of the sparse spiking neural network, pruning the weight matrix and biases based on the indices of the neurons to be retained, and returning the pruned weights and biases to reconstruct the sparse spiking neural network.

[0028] In some embodiments, in step S103, the compressed spiking neural network model is optimized according to the edge configuration, and the optimized spiking neural network model is deployed to the edge to perform inference tasks. The compressed spiking neural network model is optimized through hardware-aware pruning optimization, joint optimization, dynamic pruning, quantization compression, and sparse coding. The hardware-aware pruning optimization involves: constructing a pruner corresponding to the obtained edge cache size; and performing structured pruning on the weights of each layer of the compressed spiking neural network model based on the pruner, thus completing the hardware-aware pruning optimization. The joint optimization involves: obtaining the cache configuration (L1 and L2 cache sizes) of the target chip, and... The pruning, quantization, and sparse coding strategies are adjusted based on the cache configuration to achieve joint optimization. When the target chip at the edge is not ESP32S3 or K210, the default configuration is returned. Dynamic pruning is achieved by calculating the dynamic pruning rate based on the edge cache size, creating an SNNPruner instance and passing in the pruning rate, using the SNNPruner instance's methods to prune the model, and updating the model. Dynamic quantization is achieved by applying dynamic quantization to the model, calling methods to perform dynamic quantization, and updating the model. Sparse coding is achieved by traversing the model layers; when a layer contains impulse data, run-length encoding is performed on the impulse data, and the result is saved to achieve sparse coding.After optimizing the compressed spiking neural network (SNN) model, the optimized SNN model is deployed to the edge. The optimized SNN model is then converted to a format supported by the edge device, and the format-converted SNN model is deployed to the edge. After initialization and deployment monitoring at the edge, the performance of the format-converted SNN model on the edge is verified. ADC configuration optimization is performed based on the characteristics of the edge device. First, the optimized SNN model undergoes format conversion: specifically, the instruction format is converted according to the target chip type at the edge. When the target device at the edge is ESP32S3, each layer of the optimized SNN model is traversed, the weights of the current layer are obtained, the weights are converted from dense format to CSR format, and the sparse weights are obtained. The process involves refactoring, indexing, and pointers, updating the layer's format to CSR, and saving the converted data to the layer; thus completing the format conversion of the optimized spiking neural network model. Next, after format conversion, the converted spiking neural network model is deployed to the edge, where initialization and deployment monitoring are performed. Initialization and deployment monitoring are achieved through initializing PPOAgent, searching for automatic pruning strategies, selecting pruning branches, and monitoring storage. The initialization of PPOAgent involves initializing a reinforcement learning-based agent PPOAgent, with the PPOAgent action space defined as the pruning rate adjustment step size (-0.1, -0.05, 0, +0.05, +0.1), and the reward function as: R = 0.7 (rate of change in precision) − 0.3 (rate of change in computational latency) + 0.2(1−compression ratio), input environment configuration, setting the action space to 5 discrete actions for adjusting the pruning rate; its automatic pruning strategy search is as follows: obtain the initial state of the optimized spiking neural network model, select an action based on the current state, prune the optimized spiking neural network model based on the selected action, evaluate the performance of the pruned model and calculate the reward, update the agent's state, action and reward, and realize automatic pruning strategy search; its selective pruning is as follows: define a pruning rate list, select the corresponding pruning rate based on the action index, return the pruned model, and realize selective pruning; its storage monitoring is as follows: check whether the memory usage exceeds 80%, if it does, call emergency pruning. The branch function processes the model and finally returns the processed model. Finally, based on the edge characteristics, ADC configuration optimization is performed. This involves configuring the ADC module, optimizing SIMD instructions for sparse spiking neural network (SNN) inference, initializing the SNN library, handling pulse events, performing mixed-precision inference, decoding pulse data, quantizing inference, initializing the neural network engine, processing sensor data, and executing inference. This completes data processing and ADC configuration. After ADC configuration, the acquired sensor data is used as multimodal input and transmitted to the spiking neural network for inference. The ADC module configuration includes: configuring the ADC to 12-bit precision and an adjustable sampling rate of 1-2MHz, setting channel attenuation, configuring quantized sampling parameters, enabling a precision clock and setting the oversampling factor and clock interval, switching to double buffering to achieve zero-wait switching, stopping and restarting ADC sampling, and setting the ADC configuration to 0.The buffer data processed within 5ms is used to initiate RMT pulse emission, setting a 150kHz drive pulse, and then starting continuous ADC acquisition to complete DMA interrupt configuration. After DMA interrupt configuration, the ADC configuration is completed by registering the interrupt service function. The purpose of the configuration is to acquire input data. Its sparse spiking neural network (SNN) inference SIMD instruction optimization is as follows: traverse each neuron of the sparse spiking neural network, use SIMD vectorization to process the sum of the product of the weights and inputs, process the non-vectorized parts of the remaining elements, and determine whether to issue a pulse based on the summation result. This achieves sparse spiking neural network inference SIMD instruction optimization. If the compilation environment is ESP32-S3, it outputs information indicating the use of SIMD optimization. Its SNN library initialization is as follows: if the compilation environment is ESP32-S3, it outputs information indicating the use of SIMD optimization; otherwise, it outputs information indicating the use of general-purpose ARM implementation. Its pulse event processing is as follows: use inline assembly to load the weights and pulse data into registers, execute SIMD instruction-optimized multiply-accumulate operations and update the membrane potential, and repeat this process until all data is processed. The input data is used for model inference. Its mixed-precision inference is as follows: Initialize the accumulator acc to 0, traverse each row of the weight matrix, and for each row, traverse each column of the weight matrix, calculate the product of the 8-bit weight and the 16-bit pulse data, accumulate it to acc, and then right-shift the accumulation result by 8 bits and store it in a 32-bit output array; Its decoded pulse data is as follows: The input consists of a compressed data pointer and an output buffer pointer, as well as the maximum length. Each two bytes represent a value and the number of repetitions. The data is decoded sequentially and filled into the output buffer until the maximum length is reached or the data ends; Its quantization inference is as follows: The input consists of quantized weights, pulse data, The algorithm outputs a pointer, scaling factor, and zero point. It iterates through the weight matrix and pulse data using a double loop, calculates a weighted sum, and considers the quantization zero point. Finally, it directly applies the quantization scaling factor in the pulse domain, skipping the dequantization step, and outputs a pulse sequence to achieve quantization inference. Its neural network engine initialization involves setting up a model parser, error reporter, and memory pool; loading model data and allocating tensors. Processing sensor data and performing inference involves copying input data to an input tensor, performing inference operations, logging errors and returning if inference fails, and acquiring the output tensor and sending the result if inference succeeds.

[0029] After configuring the ADC module, the neural network engine is initialized to prepare for inference, and the SNN library is initialized to support sparse SNN inference. After the SNN library is initialized, sparse SNN inference is optimized by using the SIMD (Single Instruction Multiple Data) instruction set to accelerate the inference process. The sensor data collected by the ADC module is sent to the neural network engine to start the inference task. During SNN inference, impulse events are processed and decoded into a data format that can be understood by subsequent processing units. To further optimize inference performance, mixed precision inference technology is adopted. Based on mixed precision inference, the model is further quantized to further reduce storage requirements and computational complexity.

[0030] In summary, the edge-end spiking neural network compression and deployment method provided by this invention performs forward propagation of an initial spiking neural network model based on acquired multimodal data. Specifically, it obtains the memory usage state during forward propagation based on hardware-aware initialization, and dynamically sparsifies the initial spiking neural network model based on the memory usage state to construct a sparse spiking neural network. Based on the acquired activation statistics of the sparse spiking neural network, it employs a pruning strategy based on neuron activity to dynamically prune the sparse spiking neural network, thereby compressing the spiking neural network model. Finally, it optimizes the compressed spiking neural network model according to the edge configuration and deploys the optimized spiking neural network model to the edge to perform inference tasks, reducing storage requirements and computational complexity.

[0031] To better implement the edge-end spiking neural network compression and deployment method in the embodiments of the present invention, based on the edge-end spiking neural network compression and deployment method, correspondingly, as follows: Figure 2 As shown, this embodiment of the invention also provides an edge-end spiking neural network compression and deployment system. The edge-end spiking neural network compression and deployment system 200 includes: The network construction module 201 is used to perform forward propagation on an initial spiking neural network model based on acquired multimodal data. Specifically, the memory usage state during the forward propagation process is obtained based on hardware-aware initialization, and the initial spiking neural network model is dynamically sparsified based on the memory usage state to construct a sparse spiking neural network. The network compression module 202 is used to dynamically prune the sparse spiking neural network based on the obtained activation statistics of the sparse spiking neural network and adopt a pruning strategy based on neuron activity to achieve compression of the spiking neural network model. The network deployment module 203 is used to optimize the compressed spiking neural network model according to the edge configuration, and then deploy the optimized spiking neural network model to the edge to perform inference tasks.

[0032] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for compressing and deploying an edge-end spiking neural network, characterized in that, include: The initial spiking neural network model is forward propagated based on the acquired multimodal data. The memory usage state during the forward propagation is obtained based on hardware-aware initialization. The initial spiking neural network model is then dynamically sparsified based on the memory usage state to construct a sparse spiking neural network. Based on the activation statistics of the obtained sparse spiking neural network, a pruning strategy based on neuron activity is used to dynamically prune the sparse spiking neural network in order to compress the spiking neural network model. The compressed spiking neural network model is optimized based on the edge configuration, and the optimized spiking neural network model is deployed to the edge to perform inference tasks.

2. The edge-end spiking neural network compression and deployment method according to claim 1, characterized in that, The initial spiking neural network model includes a feature separation module, an encoding module, a hidden module, and a decoding module. The encoding module includes convolutional layers, normalization layers, and spiking neuron layers. The hidden module includes a cross-modal fusion submodule and a spiking enhancement submodule. The multimodal data includes ultrasound data, electromyography data, and temperature data. The forward propagation of the spiking neural network model based on the acquired multimodal data includes: The multimodal data is input into an initial spiking neural network model, and the feature separation module performs feature separation on the multimodal data to obtain acoustic features, electromyographic features, and temperature features. The acoustic features, electromyographic features, and temperature features are fused through the cross-modal fusion submodule of the hidden module, wherein the acoustic features, electromyographic features, and temperature features are fused by feature weighting through a cross-modal attention mechanism; The fused features are input into the encoding module. The fused features are processed sequentially by the convolutional layer and normalization layer of the encoding module. Then, the normalized features are processed by the spiking neuron layer to obtain a spiking signal. After the pulse signal is optimized and fused by the pulse enhancement submodule of the hidden module, the fused pulse signal is decoded by the decoding module to output the classification result.

3. The edge-end spiking neural network compression and deployment method according to claim 2, characterized in that, The forward propagation of the initial spiking neural network model based on the acquired multimodal data also includes: Obtain the input tensor, calculate the scaling factor based on the input tensor and the quantization bit depth at the edges, obtain the quantized tensor based on the input tensor and the scaling factor, and dynamically quantize the weights of the initial spiking neural network model based on the quantized tensor.

4. The edge-end spiking neural network compression and deployment method according to claim 3, characterized in that, The process of obtaining memory usage status during forward propagation based on hardware-aware initialization, and dynamically sparsifying the initial spiking neural network model based on the memory usage status and the obtained activation statistics of the sparsed spiking neural network to construct a sparse spiking neural network includes: The memory usage state during forward propagation is obtained based on hardware-aware initialization. When the memory usage state exceeds the memory threshold, a random tensor corresponding to the weight matrix is ​​obtained, and a Boolean mask is generated based on the preset sparsity and the random tensor. Obtain the weight matrix of the fully connected layer in the decoding module, and dynamically sparsify the initial spiking neural network model based on the weight matrix and Boolean mask to construct a sparse spiking neural network.

5. The edge-end spiking neural network compression and deployment method according to claim 4, characterized in that, The hardware awareness initialization includes: The weights of the encoding modules are quantized according to the configuration at the edge. Register a forward propagation hook for the decoding module to monitor memory usage.

6. The edge-end spiking neural network compression and deployment method according to claim 4, characterized in that, The activation statistics based on the acquired sparsed spiking neural network include: Initialize the statistical dictionary, traverse the samples of multimodal data input to the sparse spiking neural network, obtain the activation data of each layer in the sparse spiking neural network during the forward propagation, accumulate the activation data into the statistical dictionary, obtain the number of pulse firings of the sparse spiking neural network during the forward propagation, and determine the activation statistics of each layer in the sparse spiking neural network based on the accumulated value of the activation data in the statistical dictionary and the number of pulse firings.

7. The edge-end spiking neural network compression and deployment method according to claim 6, characterized in that, The pruning strategy includes spatiotemporal joint pruning and progressive gradient-aware pruning; the dynamic pruning of the sparse spiking neural network based on the acquired activation statistics of the sparse spiking neural network, using a pruning strategy based on neuron activity, to achieve compression of the spiking neural network model, includes: Based on the activation statistics of the sparsed spiking neural network, a spatiotemporal joint pruning score is calculated, wherein the spatiotemporal joint pruning score includes the spatial domain importance score of the weight matrix and the temporal domain importance score of the temporal activity. Based on the set pruning score threshold and the spatiotemporal joint pruning score, the sparse spiking neural network is spatiotemporally joint pruned to obtain the pruned sparse spiking neural network. Progressive gradient-aware pruning is performed on the pruned sparse spiking neural network to compress the spiking neural network model.

8. The edge-end spiking neural network compression and deployment method according to claim 7, characterized in that, The optimization of the compressed spiking neural network model based on the edge configuration includes: Based on the obtained cache size at the edge, a pruner corresponding to the cache size is constructed, and the weights of each layer of the compressed spiking neural network model are structurally pruned based on the pruner.

9. The edge-end spiking neural network compression and deployment method according to claim 8, characterized in that, Deploying the optimized spiking neural network model to the edge includes: The optimized spiking neural network model is converted into a format supported by the edge, and the converted spiking neural network model is deployed to the edge. After initialization, deployment and monitoring are performed at the edge, the ADC configuration is optimized according to the characteristics of the edge.

10. An edge-end spiking neural network compression and deployment system, characterized in that, include: The network construction module is used to perform forward propagation on an initial spiking neural network model based on acquired multimodal data. Specifically, the memory usage state during the forward propagation process is obtained based on hardware-aware initialization, and the initial spiking neural network model is dynamically sparsified based on the memory usage state to construct a sparse spiking neural network. The network compression module is used to dynamically prune the sparse spiking neural network based on the obtained activation statistics of the sparse spiking neural network, using a pruning strategy based on neuron activity, so as to achieve compression of the spiking neural network model. The network deployment module is used to optimize the compressed spiking neural network model according to the edge configuration, and then deploy the optimized spiking neural network model to the edge to perform inference tasks.

Images