Pulse neural network-based visual sensor data classification system, training method, and classification method

Abstract

The application discloses a visual sensor data classification system based on a pulse neural network, a training method and a classification method, and belongs to the field of visual sensor data classification. The classification system performs structural compression on input pulse features in the following manner: after Haar wavelet decomposition of the pulse features along the width dimension, low-frequency subband features L and high-frequency subband features H are separated and stacked into features S_1 along the channel dimension; after sparsification of S_1 according to the subband features, the features are converted into pulse features S_hat_1; after Haar wavelet decomposition of S_hat_1 along the height dimension, LL, HL, LH and HH subband features are separated; the energy of each channel is mapped into a gating weight by using a Sigmoid function, multiplied by the corresponding channel, and then the four subband features are stacked into features S_2; and after conversion into pulse features, inverse wavelet transformation is performed. The application can improve the SNN compression rate to reduce resource consumption, and improve the classification accuracy.

Description

Technical Field

[0001] This invention belongs to the field of visual sensor data classification, and more specifically, relates to a visual sensor data classification system, training method, and classification method. Background Technology

[0002] Spiking Neural Networks (SNNs) encode and exchange information using discrete pulse sequences, mimicking the pulse transmission mechanism of biological neurons. Unlike mainstream Artificial Neural Networks (ANNs), SNNs perform sparse computation only when the membrane potential accumulates to a threshold and triggers a pulse, fundamentally reducing the number of multiply-accumulate (MAC) operations. This event-driven characteristic enables SNNs to exhibit extremely low power consumption and extremely high computational efficiency when processing visual sensor data classification tasks such as image classification and event stream classification. In particular, with the maturity of neuromorphic chips, SNNs can maintain high classification accuracy while improving energy efficiency by several orders of magnitude compared to traditional Convolutional Neural Networks (CNNs), thus showing significant application prospects in real-time vision tasks such as edge computing, autonomous driving, and robotic disaster relief.

[0003] To further improve the accuracy of SNNs in complex image / event stream classification tasks, researchers have begun to introduce Transformer architectures. For example, SpikFormer achieves long-range feature capture through a pulse-driven self-attention mechanism. However, these high-performance SNN models typically involve a large number of parameters and computational complexity. When deployed to resource-constrained edge devices, model compression becomes essential. Existing compression methods for SNNs, such as Token-weight joint pruning in SparseSpikFormer and evolutionary search in AutoSpikFormer, mostly follow the compression approach of Artificial Neural Networks (ANNs), directly performing pruning operations on the feature maps in the temporal domain. These methods fail to explicitly utilize the frequency-domain structured redundancy inherent in the SNN pulse sequences, resulting in limited compression rates achievable in image / event stream classification tasks.

[0004] More seriously, existing compression methods suffer from structural flaws in their information processing flow. In the SNN-Transformer architecture, exemplified by Spikformer, the self-attention mechanism essentially acts as a low-pass filter, filtering out high-frequency details in pulse sequences—high-frequency components crucial for dynamic event stream classification (such as fast-moving edge detection) and fine-grained image classification. Existing methods perform pruning on feature maps that have already lost key high-frequency information, essentially optimizing the storage of distorted features without achieving frequency-selective preservation from the source. For example, while SWformer introduces wavelet transform, it uses a fixed Haar transform and lacks an adaptive pruning mechanism, resulting in significant information redundancy. Furthermore, its adherence to traditional MLP design introduces a large number of parameters that fundamentally contradict the inherent sparsity objective of SNNs.

[0005] In summary, in the existing technology, when SNN is applied to image classification or event stream classification, the compression rate that can be achieved is limited, and the classification accuracy needs to be further improved. Summary of the Invention

[0006] To address the shortcomings and improvement needs of existing technologies, this invention provides a visual sensor data classification system, training method, and classification method. Its purpose is to improve the compression rate to reduce resource consumption and improve classification accuracy while applying SNN to image classification or event stream classification.

[0007] To achieve the above objectives, according to one aspect of the present invention, a sparse frequency-aware visual sensor data classification system is provided, comprising: a pulse segmentation module for converting visual sensor data into pulse features; an encoding module for structurally compressing the pulse features to obtain encoded features; and a classification head for predicting the category corresponding to the visual sensor data based on the encoded features; wherein the visual sensor data is an image or an event stream. The encoding module includes L cascaded SFAT encoding blocks; the SFAT encoding blocks are used to perform structural compression on the input pulse features through two-stage frequency-aware structured sparse processing; L is a positive integer; Two-level frequency-aware structured sparse processing includes: After performing Haar wavelet decomposition on the input pulse features along the width dimension, low-frequency sub-band features L and high-frequency sub-band features H are separated, and the two are stacked along the channel dimension to form feature S_1; according to S_tilde = sign(S_1) ReLU(|S_1| - tau_1) sparsifies feature S_1 to obtain feature S_tilde; spiking neurons are used to convert feature S_tilde into spiking feature S_hat_1; sign represents the sign function, ReLU represents the linear rectified function; tau_1 is a learnable soft threshold used to represent the weight coefficient of each pixel in feature S_1, and the weight coefficient of pixels in the low-frequency sub-band feature L is less than the weight coefficient of pixels in the high-frequency sub-band feature H; || represents the absolute value; After performing Haar wavelet decomposition on the pulse feature S_hat_1 along the height dimension, low-frequency approximate sub-band features LL, vertical detail sub-band features HL, horizontal detail sub-band features LH, and diagonal detail sub-band features HH are separated. The energy of each channel within each sub-band feature is mapped to the corresponding channel gating weight using the Sigmoid function and multiplied with the corresponding channel to achieve channel-level sparsity. The four channel-level sparsified sub-band features are stacked to form feature S_2. The pulse neuron is used to convert feature S_2 into pulse feature S_hat_2. Perform inverse wavelet transform on the pulse feature S_hat_2.

[0008] Furthermore, the SFAT encoding block includes: a sparse frequency-aware token mixing layer and a dual-path MLP routing layer; The Token mixing layer includes a hierarchical sparse wavelet layer, a spatial learner, a channel mixer, and a first residual connection layer. The hierarchical sparse wavelet layer is used to perform two-level frequency-aware structured sparse processing on the input pulse features. The spatial learner is used to model the local spatial context of the input pulse features. The channel mixer is used to perform channel information interaction on the input pulse features. The first residual connection layer is used to add the features output by the hierarchical sparse wavelet layer, the spatial learner, and the channel mixer to the input pulse features. The dual-path MLP routing layer is used to further extract features from the token mixing layer and add the extracted features to the input impulse features.

[0009] In some optional embodiments, the dual-path MLP routing layer includes: a second residual connection layer, and a frequency compensation layer, a fully connected layer, and a batch normalization layer connected in sequence; The frequency compensation layer consists of: a feature segmentation layer, four compensation branches, a global gating network, and a first fusion layer; The feature partitioning layer is used to partition the input features It is divided into four groups of features corresponding to the low-frequency approximate subband feature LL, the vertical detail subband feature HL, the horizontal detail subband feature LH, and the diagonal detail subband feature HH; The four compensation branches take the four groups of features obtained from the division as input, and each compensation branch is used to perform a 1×1 convolution operation on the corresponding group of features. Global gating networks are used to determine features Generate compensation weights for four sets of features; The first fusion layer is used to multiply each group of features by the corresponding compensation weights and then concatenate them; The second residual connection layer is used to combine the features output by the batch normalization layer with the features... Add them together.

[0010] In some optional embodiments, the dual-path MLP routing layer includes: a frequency compensation layer, a multilayer perceptron, a second fusion layer, and a third residual connection layer; The frequency compensation layer consists of: a feature segmentation layer, four compensation branches, a global gating network, and a first fusion layer; The feature partitioning layer is used to partition the input features It is divided into four groups of features corresponding to the low-frequency approximate subband feature LL, the vertical detail subband feature HL, the horizontal detail subband feature LH, and the diagonal detail subband feature HH; The four compensation branches take the four groups of features obtained from the division as input, and each compensation branch is used to perform a 1×1 convolution operation on the corresponding group of features. Global gating networks are used to determine features Generate compensation weights for four sets of features; The first fusion layer is used to multiply each group of features by the corresponding compensation weights and then concatenate them; Multilayer perceptrons use features For input; The second fusion layer is set between the hidden layer and the output layer of the multilayer perceptron. It is used to add the features output by the hidden layer of the multilayer perceptron to the features output by the frequency compensation layer and then input them into the output layer of the multilayer perceptron. The third residual connection layer is used to combine the features output by the multilayer perceptron with the features of the output layer of the multilayer perceptron. Add them together.

[0011] In some alternative embodiments, the spatial learner is a depthwise separable convolutional unit or a local window self-attention unit.

[0012] In some alternative embodiments, the channel mixer is a 1×1 convolutional unit or a channel attention unit.

[0013] According to another aspect of the present invention, a training method for the above-described sparse frequency-aware visual sensor data classification system is provided, comprising: Obtain labeled visual sensor data and construct a training dataset; the visual sensor data can be images or event streams. The sparse frequency-aware visual sensor data classification system provided by this invention is trained end-to-end using the training dataset. After training, the trained visual sensor data classification system is obtained.

[0014] According to another aspect of the present invention, a method for classifying visual sensor data is provided, comprising: The visual sensor data to be classified is input into a trained visual sensor data classification system to obtain the classification result; the visual sensor data to be classified is an image or an event stream. The trained visual sensor classification system is obtained by the training method provided by this invention.

[0015] According to another aspect of the present invention, a computer-readable storage medium is provided, including a stored computer program; when the computer program is executed by a processor, it implements the training method provided by the present invention, or the visual sensor data classification method provided by the present invention.

[0016] According to another aspect of the present invention, an electronic device is provided, comprising: A computer-readable storage medium for storing computer programs; And a processor, used to read a computer program stored in a computer-readable storage medium to implement the training method provided by the present invention, or the visual sensor data classification method provided by the present invention.

[0017] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects: (1) The sparse frequency-aware visual sensor data classification system provided by the present invention is a spiking neural network structure. The encoding module therein realizes the structural compression of sparse frequency-awareness through two-level frequency-aware structured sparse processing. Specifically, the pulse features are sparsified in two levels. The first level is coefficient-level sparsification, which distinguishes low-frequency sub-band features and high-frequency sub-band features through a learnable soft threshold tau_1. The second level is channel-level sparsification, which distinguishes four sub-band features of LL / HL / LH / HH ​​through the gate weights obtained by energy mapping of each channel. Thus, the visual sensor data can be compressed into frequency-related weight parameters, and the coarse-to-fine processing strategy of biological vision can be explicitly simulated. While utilizing the structured redundancy characteristics of the frequency domain to improve the compression rate, the classification accuracy is effectively improved.

[0018] (2) In the preferred embodiment of the sparse frequency-aware visual sensor data classification system provided by the present invention, a spatial learner and a channel mixer are introduced in each SFAT coding block in the coding module, which are respectively used to perform local spatial context modeling and channel information interaction on the input pulse features. They complement the features after two-level frequency-aware structured sparse processing, improve the spatial discriminativeness and channel expressive ability of the compressed features, and thus further improve the classification accuracy.

[0019] (3) In a preferred embodiment of the sparse frequency-aware visual sensor data classification system provided by the present invention, the dual-path MLP routing layer in the SFAT coding block includes a frequency compensation layer. This frequency compensation layer divides the input features into K=4 groups along the channel dimension. Each group corresponds to the four sub-band features of wavelet decomposition (LL / HL / LH / HH). After each group undergoes a lightweight 1x1 convolution transformation, a global gating network generates compensation weights for each sub-band based on global spatial information. After weighted concatenation, the weights are pulsed and output, thereby selectively recovering the high-frequency detail information lost due to sparsity. In a further preferred embodiment, the dual-path MLP routing layer includes only a frequency compensation layer, a fully connected layer, and a batch normalization layer, without including the large-scale fully connected layer in the MLP, significantly reducing the number of parameters and computational energy consumption.

[0020] (4) Traditional spiking neural network training methods often train artificial neural networks (ANNs) first and then convert them into spiking neural networks (SNNs), which is an ANN-to-SNN conversion. This conversion has a long time delay. The present invention uses an end-to-end approach to train the proposed sparse frequency-aware visual sensor data classification system, which is consistent with the current SpikingTransformer training paradigm. It does not rely on pre-trained models or ANN-to-SNN conversion, thus getting rid of the long time delay constraint of ANN-to-SNN conversion. The training process is stable and converges. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the structure of a sparse frequency-aware visual sensor data classification system provided in an embodiment of the present invention.

[0022] Figure 2 This is a schematic diagram of the SFAT encoding block structure provided in an embodiment of the present invention.

[0023] Figure 3 This is a schematic diagram of two-level frequency-sensing structured sparse processing provided in an embodiment of the present invention.

[0024] Figure 4 A schematic diagram of the dual-path MLP routing layer structure provided in this embodiment of the invention.

[0025] Figure 5A schematic diagram of the frequency compensation layer structure provided in this embodiment of the invention.

[0026] Figure 6 This is a schematic diagram of another dual-path MLP routing layer structure provided in an embodiment of the present invention.

[0027] Figure 7 This diagram illustrates the accuracy comparison of different methods on the CIFAR-100 dataset.

[0028] Figure 8 This diagram illustrates the comparison of accuracy and energy consumption of different methods on the Tiny-ImageNet dataset.

[0029] Figure 9 This diagram illustrates the accuracy comparison of different methods on a neuromorphic dataset.

[0030] Figure 10 This is a schematic diagram of the ablation experiment results of the parallel mode of this invention on the CIFAR-100 dataset.

[0031] Figure 11 This is a schematic diagram of the ablation experiment results of the lightweight mode of this invention on the CIFAR-100 dataset.

[0032] Figure 12 This diagram illustrates the comparison of parameter count and computational cost for different methods on the CIFAR-100 dataset.

[0033] Figure 13 This is a schematic diagram comparing the Pareto frontier of energy efficiency using different methods on the CIFAR-100 dataset. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0035] In this invention, the terms "first," "second," etc. (if present) in the invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0036] To improve compression ratio and reduce storage and computing resource consumption while simultaneously increasing classification accuracy when applying SNNs to image classification or event stream classification, this invention provides a visual sensor data classification system, training method, and classification method. The overall concept involves improving the structure of the spiking neural network to achieve sparse frequency domain perception-based structural compression. This explicitly utilizes the structured redundancy features in the frequency domain, effectively improving the compression ratio. This allows the system to effectively perform image / event stream classification tasks even in scenarios with limited storage and computing resources, such as edge computing, autonomous driving, and robotic disaster relief. Furthermore, it avoids blind pruning on the temporal feature map, thus preventing the loss of high-frequency information and improving classification accuracy.

[0037] Based on the above technical concepts, this invention first proposes a novel spiking neural network architecture, namely a sparse frequency-aware visual sensor data classification system, and further proposes an end-to-end training method for the visual sensor data classification system, as well as a visual sensor data classification method for performing visual sensor classification tasks based on the trained visual sensor data classification system.

[0038] The following is an example.

[0039] Example 1: A sparse frequency-aware visual sensor data classification system. For example... Figure 1 As shown, this embodiment includes: a pulse segmentation module for converting visual sensor data into pulse features; an encoding module for structurally compressing the pulse features to obtain encoded features; and a classification head for predicting the category corresponding to the visual sensor data based on the encoded features. The visual sensor data is either an image or an event stream.

[0040] The following provides further explanation of the specific implementation methods of each module.

[0041] In this embodiment, the pulse segmentation module (i.e. Figure 1 The SPS in the model further includes a pulse mapping module (PSM) and a relative position encoding module (RPE). The pulse mapping module is used to segment the input visual sensor data into local image patches and convert them into pulse feature sequences through linear projection and spiking neurons. The relative position encoding module is used to inject spatial position information into the pulse feature sequences to compensate for the absolute position information lost during the pulse mapping process and enhance the spatial discriminative ability of the features.

[0042] like Figure 1In this embodiment, the encoding module includes L cascaded SFAT encoding blocks, where L is a positive integer. In the L cascaded SFAT encoding blocks, the first SFAT encoding block takes the pulse features output by the pulse segmentation module as input, the second to Lth SFAT encoding blocks take the pulse features output by the previous SFAT encoding block as input, and the pulse features output by the Lth SFAT encoding block serve as the input to the classification head. The SFAT encoding block is a sparse frequency-aware encoding block, and each SFAT encoding block is used to structurally compress the input pulse features through two levels of frequency-aware structured sparse processing.

[0043] like Figure 3 As shown in this embodiment, in the two-level frequency-aware structured sparse processing implemented by each SFAT coding block, the first level is coefficient-level sparseness, and the second level is channel-level sparseness, as follows: First-order coefficient-level sparsity: The horizontal Haar wavelet decomposition is achieved by performing Haar wavelet decomposition on the pulse features of the input along the width dimension. Low-frequency subband features L and high-frequency subband features H are separated from the pulse features after horizontal Haar wavelet decomposition, and the two are stacked along the channel dimension to form feature S_1; According to S_tilde = sign(S_1) ReLU(|S_1| - tau_1) sparsifies feature S_1 to obtain feature S_tilde; where sign represents the sign function, ReLU represents the linear rectified function; tau_1 is a learnable soft threshold used to represent the weight coefficient of each pixel in feature S_1, and the weight coefficient tau_L of pixels in low-frequency sub-band feature L is less than the weight coefficient tau_H of pixels in high-frequency sub-band feature H. Therefore, this embodiment can distinguish between low-frequency sub-band and high-frequency sub-band through this coefficient-level soft threshold. In practical applications, high-frequency information has a certain degree of redundancy and is easily activated. This embodiment sets a higher threshold for high-frequency sub-band features, which can suppress high-frequency activation to a certain extent, reduce the pulse firing rate of high-frequency information, thereby reducing energy consumption, and suppressing the imbalance between low-frequency sub-band and high-frequency sub-band components, thus improving accuracy; | represents absolute value; The feature S_tilde is converted into a pulse feature S_hat_1 using a spiking neuron, i.e., S_hat_1 = Spk_sym(S_tilde; V_th 1 ), where Spk_sym represents a spiking neuron, The threshold value for the spiking neuron in the first-order coefficient-level sparse region is determined through training, and C represents the channel dimension of the input spiking feature.

[0044] Second-level channel-level sparsity: Haar wavelet decomposition is performed on the pulse feature S_hat_1 along the height dimension, thereby realizing vertical Haar wavelet decomposition. From the pulse features after vertical Haar wavelet decomposition, the low-frequency approximate subband feature LL, the vertical detail subband feature HL, the horizontal detail subband feature LH, and the diagonal detail subband feature HH are separated; The Sigmoid function is used to map the energy of each channel within each sub-band feature to the corresponding channel's gate weight and multiply it by the corresponding channel, achieving channel-level sparsity; for a channel c within a sub-band feature, its energy... The absolute average of the channel across time, batch, and spatial dimensions; energy The specific method of mapping energy to gating weights is as follows: Convert to coefficients ,in, This represents the Sigmoid function. The average energy of all channels. The preset temperature coefficient is used; the final mapped gate weight is... , , and All are thresholds, and Based on the Haar wavelet frequency separation mechanism, wavelet domain energy and information content are positively correlated. High-energy subbands (such as LL) correspond to greater information loss, while low-energy subbands (such as HH) have a weak impact, providing a quantifiable decision basis for differentiated sparsity. In this embodiment, the gating weights obtained by mapping the energy of each channel can distinguish the features of the four subbands LL / HL / LH / HH. The four sub-band features after channel-level sparsification are stacked into feature S_2; feature S_2 is then converted into a spiking feature S_hat_2 using a spiking neuron, i.e., S_hat_2 = Spk_sym(S_2; V_th 2 ), where Spk_sym represents a spiking neuron, The threshold value represents the spiking neuron in the second-level channel-level sparse region, and its specific value is determined through training.

[0045] After the second-level channel-level sparsity, the pulse feature S_hat_2 is subjected to wavelet inverse transform to complete the two-level frequency-sensing structured sparsity processing.

[0046] This invention discovers that the frequency domain separation characteristics inherent in SNN pulse sequences exhibit similarities to the organizational principles of biological visual systems. Specifically, the primate visual system achieves parallel frequency processing through the M / P pathway; the M pathway is sensitive to low-frequency spatial frequencies, rapidly transmitting the overall contour; the P pathway is sensitive to high-frequency spatial frequencies, slowly encoding fine details, and the coarse-to-fine processing strategy further reduces neural coding redundancy. In this embodiment, Haar wavelet transform is used as a high- and low-frequency decomposition tool. The orthogonality of Haar wavelets and binary sparsity naturally match the discrete activation mode of SNNs, and its multi-level subband structure (LL / HL / LH / HH) corresponds to the hierarchical frequency organization of the visual system. In this embodiment, the SFAT coding block, through two-level frequency-aware structured sparse processing, can compress visual sensor data into frequency-related weight parameters and explicitly simulate the coarse-to-fine processing strategy of biological vision. While utilizing the structured redundancy characteristics of the frequency domain to improve the compression rate, it effectively improves classification accuracy.

[0047] Optionally, in this embodiment, the spiking neurons in the SFAT coding block are specifically LIF (Leaky Integrate-and-Fire) neuron models.

[0048] Specifically, such as Figure 2 As shown, in this embodiment, the SFAT encoding block includes: a sparse frequency-aware token mixing layer and a dual-path MLP routing layer; The Token mixing layer includes a hierarchical sparse wavelet layer, a spatial learner, a channel mixer, and a first residual connection layer. The hierarchical sparse wavelet layer is used to perform two-level frequency-aware structured sparse processing on the input pulse features. The spatial learner is used to model the local spatial context of the input pulse features. The channel mixer is used to perform channel information interaction on the input pulse features. The first residual connection layer is used to add the features output by the hierarchical sparse wavelet layer, the spatial learner, and the channel mixer to the input pulse features. The dual-path MLP routing layer is used to further extract features from the token mixing layer and add the extracted features to the input impulse features; for example... Figure 4 As shown, in this embodiment, the dual-path MLP routing layer includes: a second residual connection layer, and a frequency compensation layer, a fully connected layer, and a batch normalization layer connected in sequence; like Figure 5 As shown, the frequency compensation layer includes: a feature segmentation layer, four compensation branches, a global gating network, and a first fusion layer; The feature partitioning layer is used to partition the input features It is divided into four groups of features corresponding to the low-frequency approximate subband feature LL, the vertical detail subband feature HL, the horizontal detail subband feature LH, and the diagonal detail subband feature HH; The four compensation branches take the four groups of features obtained from the partitioning as inputs. Each compensation branch performs a lightweight 1×1 convolution operation on the corresponding group of features to transform them. The transformed features are h. k = BN(Conv_1×1(x k W_e k ), where k represents the feature group number, k=1,2,3,4; x k Let W_e represent the k-th feature group. k Represents the weight of the k-th compensation branch, Conv_1×1 represents 1×1 convolution, and BN represents batch normalization; Global gating networks are used to determine features Generate compensation weights for four sets of features; The first fusion layer is used to multiply each group of features by the corresponding compensation weights and then concatenate them; the features output by the first fusion layer will be processed sequentially by the subsequent fully connected layers and batch normalization layers; The second residual connection layer is used to combine the features output by the batch normalization layer with the features... Add them together.

[0049] Optionally, such as Figure 5 As shown, the global gating network includes a global average pooling layer, a 1×1 convolutional layer, and a batch normalization layer; in this embodiment, the first fusion layer also performs 1×1 convolution and batch normalization operations on the spliced ​​features.

[0050] This embodiment introduces a spatial learner and a channel mixer within each SFAT coding block in the coding module. These are used for local spatial context modeling of the input pulse features and channel information interaction, respectively. This complements the features after two-level frequency-aware structured sparse processing. Local spatial context modeling compensates for the loss of fine positional information in wavelet global decomposition, while channel information interaction enables channel feature recalibration, enhancing channel expressiveness. Ultimately, this embodiment improves the spatial discriminativeness and channel expressiveness of the compressed features, thereby further improving classification accuracy. In practical applications, the spatial learner can be selected from units that can achieve local spatial context modeling, such as depthwise separable convolutional units and local window self-attention units, while the channel mixer can be selected from units that can achieve channel information interaction, such as 1×1 convolutional units and channel attention units.

[0051] In this embodiment, a frequency compensation layer is introduced into the dual-path MLP routing layer within the SFAT coding block. This frequency compensation layer divides the input features into K=4 groups along the channel dimension, with each group corresponding to the four sub-band features of wavelet decomposition (LL / HL / LH / HH). After each group undergoes a lightweight 1x1 convolutional transformation, a global gating network generates compensation weights for each sub-band based on global spatial information. These weighted weights are then concatenated and pulsed for output. This selectively recovers high-frequency detail information lost due to sparsity, further improving classification accuracy. Besides this frequency compensation layer, only fully connected layers and batch normalization layers are included, excluding the large-scale fully connected layers found in MLPs. This reduces the number of parameters to approximately 55% of the traditional parallel MLP-based model, significantly reducing energy consumption.

[0052] In practical applications, the number L of SFAT encoding blocks cascaded in the encoding module can be flexibly set according to specific compression ratio requirements and classification accuracy requirements.

[0053] Optionally, in this embodiment, the classification head is specifically a multilayer perceptron.

[0054] The sparse frequency-aware visual sensor data classification system provided in this embodiment requires the following parameters to be learned during training: the soft threshold tau_1 in the two-level frequency-aware structured sparse processing, and the threshold of the spiking neuron in the first-level coefficient-level sparse processing. Threshold of spiking neurons in the second-level channel-level sparse region The system comprises the weights of the global gating network in the dual-path MLP routing layer, the convolutional weights within each compensation branch, the convolutional or attention weights in the spatial learner and channel mixer, and the convolutional and fully connected weights in the pulse segmentation module and the classification head. Compared to traditional spiking neural networks, its parameter count is significantly reduced. This effectively reduces storage and computational resource consumption when performing visual sensor data classification tasks using this system. It can effectively perform image / event stream classification tasks even in scenarios with limited storage and computational resources, such as edge computing, autonomous driving, and robotic disaster relief. Furthermore, this embodiment compresses data in the frequency domain, avoiding blind pruning in the temporal feature map, thus preventing the loss of high-frequency information and effectively improving classification accuracy.

[0055] Example 2: A sparse frequency-aware visual sensor data classification system. This embodiment is similar to Embodiment 1 above, except that the dual-path MLP routing layer structure within the SFAT coding block is different in this embodiment. For example... Figure 6 As shown, in some optional embodiments, the dual-path MLP routing layer includes: a frequency compensation layer, a multilayer perceptron, a second fusion layer, and a third residual connection layer. The frequency compensation layer consists of: a feature segmentation layer, four compensation branches, a global gating network, and a first fusion layer; The feature partitioning layer is used to partition the input features It is divided into four groups of features corresponding to the low-frequency approximate subband feature LL, the vertical detail subband feature HL, the horizontal detail subband feature LH, and the diagonal detail subband feature HH; The four compensation branches take the four groups of features obtained from the division as input, and each compensation branch is used to perform a 1×1 convolution operation on the corresponding group of features. Global gating networks are used to determine features Generate compensation weights for four sets of features; The first fusion layer is used to multiply each group of features by the corresponding compensation weights and then concatenate them; Multilayer perceptrons use features For input; The second fusion layer is set between the hidden layer and the output layer of the multilayer perceptron. It is used to add the features output by the hidden layer of the multilayer perceptron to the features output by the frequency compensation layer and then input them into the output layer of the multilayer perceptron. The third residual connection layer is used to combine the features output by the multilayer perceptron with the features of the output layer of the multilayer perceptron. Add them together.

[0056] This embodiment introduces a frequency compensation layer between the hidden layer and the output layer of the standard multilayer perceptron as a residual branch to compensate for high-frequency information. Compared with the traditional multilayer perceptron approach, it can effectively improve classification accuracy and is suitable for high-precision reasoning scenarios.

[0057] In this embodiment, the specific implementation of the remaining modules of the sparse frequency-aware visual sensor data classification system can be referred to the description in Embodiment 1 above.

[0058] Example 3: A training method for a sparse frequency-aware visual sensor data classification system, used to train the sparse frequency-aware visual sensor data classification system provided in Embodiment 1 or 2 above.

[0059] This embodiment includes: Obtain labeled visual sensor data and construct a training dataset; the visual sensor data can be images or event streams. An end-to-end training method for a visual sensor data classification system with sparse frequency perception is used to train the system. After training, a well-trained visual sensor data classification system is obtained.

[0060] When training a sparse frequency-aware visual sensor data classification system, the loss function value is calculated based on the predicted output and the true class label, and the network parameters are updated through gradient backpropagation until the network converges. Optionally, in this embodiment, the loss function value is calculated using the cross-entropy loss function, gradient backpropagation is performed using the STE gradient estimator, and the learnable parameters are jointly optimized using the AdamW optimizer and the cosine annealing learning rate scheduling strategy. Some early spiking neural network training methods use ANN-to-SNN conversion, which has a long time delay; this embodiment performs end-to-end direct training based on gradient substitution methods (such as STE), consistent with the current Spiking Transformer training paradigm, so that frequency-related parameters such as hierarchical sparse threshold and sub-band sensing pulse threshold are adaptively optimized during training, without the need for a pre-trained model, and the training process converges stably.

[0061] In this embodiment, for the image classification task, the CIFAR-100 (32×32 pixels, 100 classes) and Tiny-ImageNet (64×64 pixels, 200 classes) datasets are specifically selected for constructing the training datasets; for the event stream classification task, the CIFAR10-DVS (128×128 pixels) and DVS-Gesture (128×128 pixels, 11 classes) datasets are selected for constructing the training datasets. RandAugment and Mixup data augmentation are applied to the training samples in the selected training datasets.

[0062] For the CIFAR-100 dataset, we trained for 400 epochs using Transformer-4-384 (i.e., the encoding module contains L=4 concatenated SFAT encoding blocks, and the hidden layer dimension is 384), with a batch size of 128 and an initial learning rate of 1e. -3 The weight decay is set to 0.05; For the Tiny-ImageNet dataset, the training was conducted for 200 epochs, and RandAugment and Mixup data augmentation were used. For the DVS-Gesture and CIFAR10-DVS neuromorphic datasets, train for 100 epochs with a time step T of 4 or 10.

[0063] Traditional spiking neural network (ANN) training methods often involve first training an ANN and then converting it into a spiking neural network (SNN), a process known as ANN-to-SNN conversion, which incurs a significant time delay. This embodiment employs an end-to-end approach to train the proposed sparse frequency-aware visual sensor data classification system, eliminating the need for pre-trained models or ANN-to-SNN conversions and overcoming the long time delay constraint of ANN-to-SNN conversions. The training process then achieves stable convergence.

[0064] Example 4: A method for classifying visual sensor data, comprising: The visual sensor data to be classified is input into a trained visual sensor data classification system to obtain the classification result; the visual sensor data to be classified is an image or an event stream. The trained visual sensor classification system is obtained by training the method provided in Example 3 above.

[0065] It is easy to understand that this embodiment is specifically for an image classification task or an event stream classification task. If the task has higher classification accuracy requirements, the system trained by the sparse frequency-aware visual sensor data classification system provided in Embodiment 2 above shall be selected; if the task has higher resource consumption and energy consumption requirements, the sparse frequency-aware visual sensor data classification system provided in Embodiment 1 above shall be selected.

[0066] Example 5: A computer-readable storage medium includes a stored computer program; when the computer program is executed by a processor, it implements the training method provided in Embodiment 3 above, or the visual sensor data classification method provided in Embodiment 4 above.

[0067] Example 6: An electronic device, comprising: A computer-readable storage medium for storing computer programs; And a processor for reading a computer program stored in a computer-readable storage medium to implement the training method provided in Embodiment 3 above, or the visual sensor data classification method provided in Embodiment 4 above.

[0068] The following explanation, based on specific comparative and ablation experiments, further illustrates the beneficial effects of this invention.

[0069] In the experiment, existing Spikformer, SDformer, and SWformer were selected as existing comparison methods. For ease of description, in the following analysis, the sparse frequency-aware visual sensor data classification system provided by this invention is abbreviated as SFAT. SFAT(P) represents a system in which the dual-path MLP routing layer adopts a parallel fusion mode of basic MLP path and frequency compensation path (i.e., the system corresponding to Example 2). This system is a system in a parallel mode. SFAT(L) represents a lightweight system in which the dual-path MLP routing layer removes the basic MLP path and retains only the frequency compensation path (i.e., the system corresponding to Example 1). This system is a system in a lightweight mode.

[0070] Evaluation metrics include Top-1 accuracy, number of parameters, number of synaptic operands (SOPs), and energy consumption (based on 45nm process, MAC energy consumption 4.6pJ, AC energy consumption 0.9pJ).

[0071] Comparison of accuracy of different methods on the CIFAR-100 dataset, for example Figure 7 As shown, the accuracy of different methods on the Tiny-ImageNet dataset is compared. Figure 8 As shown, the accuracy of different methods on different neuromorphic datasets (DVS-Gesture, CIFAR01-DVS) is compared. Figure 9 As shown, the ablation experiment results of the parallel mode and lightweight mode in this invention on the CIFAR-100 dataset are as follows: Figure 10 , Figure 11 As shown, the number of parameters and computational cost of different methods on the CIFAR-100 dataset are compared as follows: Figure 12 As shown, the energy efficiency Pareto fronts of different methods on the CIFAR-100 dataset are compared. Figure 13 As shown.

[0072] Figure 7 The results show that on the CIFAR-100 dataset, SFAT(P) achieves 79.91% performance with a Transformer-4-384 configuration, surpassing Spikformer (77.86%), SDformer (78.4%), and SWformer (79.3%). Simultaneously, SOPs and energy consumption are reduced to 54.3% and 54.6% of SWformer, respectively. SFAT(L) achieves comparable performance (77.84%) with approximately 55% of the parameter count, and further reduces SOPs and energy consumption to approximately 37% of the baseline.

[0073] Figure 8 The results show that on the Tiny-ImageNet dataset, SFAT(P) achieves 66.44%, surpassing all comparison methods, with energy consumption only 34.1% of Spikformer and 63.0% of SWformer. SFAT(L) achieves 62.68% with approximately 55% of the parameters, reducing energy consumption to 20.0% of Spikformer and 37.0% of SWformer.

[0074] like Figure 9As shown, experiments on neuromorphic event stream datasets demonstrate that the method of this invention is generalizable to event-driven data. On the CIFAR10-DVS dataset (128×128 pixels, 10 classes, time step T=16), SFAT(P) achieves a classification accuracy of 78.7%, and SFAT(L) achieves a classification accuracy of 78.8%. On the DVS-Gesture dataset (128×128 pixels, 11 gesture classes, time step T=16), SFAT(P) achieves a classification accuracy of 97.9%, and SFAT(L) achieves a classification accuracy of 97.6%. These results indicate that although the high-frequency dynamic characteristics of event stream data differ from the low-frequency dominance assumption of static images, resulting in slightly lower performance than the best performance on static image datasets, the architecture of this invention still maintains comparable accuracy on event stream classification tasks, verifying the adaptability of sparse frequency-aware structures to multimodal visual sensor data. In the diagram, Trans-2-256 indicates that the encoding module contains L=2 concatenated SFAT encoding blocks, and the hidden layer dimension is 256.

[0075] Figure 10 In the diagram, the baseline represents SFAT(P). Figure 11 In the diagram, the baseline represents SFAT(L); Figure 10 and Figure 11 In this context, "w / o L1" means removing the first level of coefficient sparsity from the baseline, "w / o L2" means removing the second level of sparsity from the baseline, "w / MLP" means that the dual-path MLP routing layer directly adopts the classic MLP, and "w / o GN" means removing the global gated network in the frequency compensation layer from the baseline. Figure 10 , Figure 11 The ablation experiment results show that removing the first-level coefficient sparsity leads to a 0.25% decrease in accuracy in parallel mode and a 0.55% decrease in accuracy in lightweight mode. Removing the second-level channel sparsity has a slight impact on parallel mode but causes a significant increase in SOPs by 23.4%, while in lightweight mode, accuracy decreases by 0.47% and SOPs increase by 17.2%. After replacing DPMR with the classic MLP, the accuracy of lightweight mode actually improves by 1.67%, but energy consumption surges by 44.7%, verifying that the present invention achieves comparable accuracy with extremely low energy consumption by replacing the basic MLP with FCP, achieving a better energy efficiency trade-off.

[0076] Figure 12 The results show that SFAT(P) significantly reduces computational energy consumption through structured sparsity, with a parameter count comparable to the baseline; SFAT(L) reduces the parameter count to about 55% by removing large-scale convolutions from the basic MLP, achieving a flexible trade-off between accuracy and parameter count.

[0077] Figure 13The energy efficiency Pareto fronts of various methods on the CIFAR-100 dataset are compared. The horizontal axis represents Top-1 accuracy (%), and the vertical axis represents energy consumption (mJ). Methods closer to the upper right corner (high accuracy, low energy consumption) are better. It can be observed that SFAT(P) occupies the optimal Pareto point. Its accuracy reaches 79.91%, the highest among all methods; at the same time, its energy consumption is approximately 0.514 mJ, significantly lower than Spikformer (approximately 1.318 mJ) and SWformer (approximately 0.942 mJ). This indicates that SFAT(P) achieves a significant reduction in energy consumption without sacrificing accuracy, breaking the traditional trade-off bottleneck of "high accuracy necessarily means high energy consumption." SFAT(L) occupies the extreme energy efficiency point. Its energy consumption is approximately 0.355 mJ, the lowest among all methods; its accuracy still reaches 77.84%, basically on par with Spikformer (77.86%). This point is located at the uppermost end of the Pareto front, indicating that acceptable classification accuracy is maintained under extremely low energy consumption constraints. It is suitable for edge deployment scenarios with strictly limited computing resources and battery capacity (such as embedded vision sensors and mobile terminals).

[0078] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A sparse frequency-aware visual sensor data classification system, characterized in that, include: The pulse segmentation module is used to convert visual sensor data into pulse features; The encoding module is used to perform structural compression on the pulse features to obtain encoded features; And a classification head, used to predict the category corresponding to the visual sensor data based on the encoded features; the visual sensor data is an image or an event stream; The encoding module includes L cascaded SFAT encoding blocks; the SFAT encoding blocks are used to perform structural compression on the input pulse features through two-stage frequency-aware structured sparse processing. L is a positive integer; The two-level frequency-sensing structured sparse processing includes: After performing Haar wavelet decomposition on the input pulse features along the width dimension, low-frequency sub-band features L and high-frequency sub-band features H are separated, and the two are stacked along the channel dimension to form feature S_1; according to S_tilde = sign(S_1) ReLU(|S_1| - tau_1) sparsifies feature S_1 to obtain feature S_tilde; spiking neurons are used to convert feature S_tilde into spiking feature S_hat_1; sign represents the sign function, ReLU represents the linear rectified function; tau_1 is a learnable soft threshold used to represent the weight coefficient of each pixel in feature S_1, and the weight coefficient of pixels in the low-frequency sub-band feature L is less than the weight coefficient of pixels in the high-frequency sub-band feature H; || represents the absolute value; After performing Haar wavelet decomposition on the pulse feature S_hat_1 along the height dimension, low-frequency approximate sub-band features LL, vertical detail sub-band features HL, horizontal detail sub-band features LH, and diagonal detail sub-band features HH are separated. The energy of each channel within each sub-band feature is mapped to the corresponding channel's gating weight using the Sigmoid function and multiplied by the corresponding channel to achieve channel-level sparsity. The four channel-level sparsified sub-band features are stacked to form feature S_2. The pulse neuron is used to convert feature S_2 into pulse feature S_hat_2. For a channel within a sub-band feature, its energy is the absolute mean of that channel in the time, batch, and spatial dimensions. Perform inverse wavelet transform on the pulse feature S_hat_2.

2. The sparse frequency-aware visual sensor data classification system as described in claim 1, characterized in that, The SFAT encoding block includes: a sparse frequency-aware token mixing layer and a dual-path MLP routing layer; The Token mixing layer includes a hierarchical sparse wavelet layer, a spatial learner, a channel mixer, and a first residual connection layer. The hierarchical sparse wavelet layer is used to perform two-level frequency-aware structured sparse processing on the input pulse features. The spatial learner is used to perform local spatial context modeling on the input pulse features. The channel mixer is used to perform channel information interaction on the input pulse features. The first residual connection layer is used to add the features output by the hierarchical sparse wavelet layer, the spatial learner, and the channel mixer to the input pulse features. The dual-path MLP routing layer is used to further extract features from the token mixing layer and add the extracted features to the input pulse features.

3. The sparse frequency-aware visual sensor data classification system as described in claim 2, characterized in that, The dual-path MLP routing layer includes: a second residual connection layer, and a frequency compensation layer, a fully connected layer, and a batch normalization layer connected in sequence; The frequency compensation layer includes: a feature segmentation layer, four compensation branches, a global gating network, and a first fusion layer; The feature segmentation layer is used to divide the input features It is divided into four groups of features corresponding to the low-frequency approximate subband feature LL, the vertical detail subband feature HL, the horizontal detail subband feature LH, and the diagonal detail subband feature HH; The four compensation branches take the four groups of features obtained from the division as input, and each compensation branch is used to perform a 1×1 convolution operation on the corresponding group of features. The global gating network is used to determine features Generate compensation weights for four sets of features; The first fusion layer is used to multiply each group of features by the corresponding compensation weights and then concatenate them; The second residual connection layer is used to connect the features output by the batch normalization layer with the features Add; Wherein, feature F is the feature output by the Token mixing layer.

4. The sparse frequency-aware visual sensor data classification system as described in claim 2, characterized in that, The dual-path MLP routing layer includes: a frequency compensation layer, a multilayer perceptron, a second fusion layer, and a third residual connection layer; The frequency compensation layer includes: a feature segmentation layer, four compensation branches, a global gating network, and a first fusion layer; The feature segmentation layer is used to divide the input features It is divided into four groups of features corresponding to the low-frequency approximate subband feature LL, the vertical detail subband feature HL, the horizontal detail subband feature LH, and the diagonal detail subband feature HH; The four compensation branches take the four groups of features obtained from the division as input, and each compensation branch is used to perform a 1×1 convolution operation on the corresponding group of features. The global gating network is used to determine features Generate compensation weights for four sets of features; The first fusion layer is used to multiply each group of features by the corresponding compensation weights and then concatenate them; The multilayer perceptron uses features For input; The second fusion layer is disposed between the hidden layer and the output layer of the multilayer perceptron, and is used to add the features output by the hidden layer of the multilayer perceptron to the features output by the frequency compensation layer and then input them into the output layer of the multilayer perceptron. The third residual connection layer is used to connect the features output by the output layer of the multilayer perceptron with the features... Add; Wherein, feature F is the feature output by the Token mixing layer.

5. The sparse frequency-aware visual sensor data classification system as described in any one of claims 2 to 4, characterized in that, The spatial learner is a depth-separable convolutional unit or a local window self-attention unit.

6. The sparse frequency-aware visual sensor data classification system as described in any one of claims 2 to 4, characterized in that, The channel mixer is a 1×1 convolutional unit or a channel attention unit.

7. The training method for the sparse frequency-aware visual sensor data classification system as described in any one of claims 1 to 6, characterized in that, include: Obtain labeled visual sensor data and construct a training dataset; The visual sensor data is either an image or an event stream; The sparse frequency-aware visual sensor data classification system according to any one of claims 1 to 6 is trained end-to-end using the training dataset. After training, a trained visual sensor data classification system is obtained.

8. A method for classifying visual sensor data, characterized in that, include: The visual sensor data to be classified is input into a trained visual sensor data classification system to obtain the classification result; The visual sensor data to be classified is either an image or an event stream; The trained visual sensor data classification system is obtained by the training method described in claim 7.

9. A computer-readable storage medium, characterized in that, Includes a stored computer program; when the computer program is executed by a processor, it implements the training method of claim 7 or the visual sensor data classification method of claim 8.

10. An electronic device, characterized in that, include: A computer-readable storage medium for storing computer programs; And a processor for reading a computer program stored in the computer-readable storage medium to implement the training method of claim 7 or the visual sensor data classification method of claim 8.

Images