Image recognition method based on fourier and cross-time constrained pulse neural network

Abstract

The application discloses an image recognition method based on a Fourier and cross-time constraint pulse neural network, and has the characteristics that the method comprises the following steps: constructing an image recognition system based on the Fourier and cross-time constraint pulse neural network; an image acquisition module acquires an image sample set, and a training set is obtained through preprocessing; the training set is used to train the network, a step function is used in forward propagation to make a pulse neuron emit a pulse, and in backward propagation, a derivative of a finite term Fourier series is used as a gradient estimation of a pulse on a membrane potential; a cosine similarity is used to measure the similarity between each time step, a loss of a target function after r times of training of the network is calculated, and network weight parameters are adjusted according to the loss of the target function; the trained network is used to perform an image classification and recognition operation on preprocessed image data to be recognized, and an image classification result is output. The method improves the model performance.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to an image recognition method based on Fourier and time-constrained spiking neural networks. Background Technology

[0002] Spiking neural networks, as well-known brain-inspired computational models, utilize spiking neurons to fire discrete binary pulse sequences to transmit information in a manner similar to biological neurons, thus possessing biological interpretability and high energy efficiency.

[0003] The unique firing mechanism of spiking neurons is a double-edged sword. This all-or-nothing spiking activity inevitably hinders the direct application of the backpropagation algorithm, which has achieved great success in artificial neural networks (ANNs), to train spiking neural networks (SNNs). Two main methods have been developed to mitigate the non-differentiability of spiking activity. One is an indirect conversion method called ANN2SNN, which converts trained ANNs into SNNs of equivalent structure and replaces the nonlinear activations in the ANNs with spiking neurons, thus matching the spiking frequency of the SNNs with the activations in the ANNs. While this method can reduce the performance gap between SNNs and ANNs, it requires a considerable time step to reduce the error between activation values ​​and spiking frequency. Furthermore, this method is only suitable for static datasets and not for neuromorphic datasets. Moreover, the conversion method cannot fully utilize the unique spatiotemporal information processing capabilities of SNNs. The other method is to directly train SNNs through pseudo-gradient learning, specifically by using smooth curves to replace the original all-or-nothing spiking activity during backpropagation. While pseudo-gradients can be used to directly train SNNs and achieve good performance, the variety of pseudo-gradient functions available today, all approximating the Heaviside function with a differentiable function, still suffers from vanishing and exploding gradients, leading to performance degradation in trained SNNs. Although many works have made significant efforts to address this issue, the pseudo-gradient functions they employ may disrupt the main direction of the true gradient.

[0004] The drawbacks of existing techniques: Since the gradient of the impulse firing activity is almost zero everywhere, several attempts have been made to use surrogate gradient (SG) functions to mitigate this and optimize SNNs using the backpropagation algorithm. However, these surrogate gradient functions may disrupt the principal direction of the gradient for optimizing SNNs, thus limiting the performance of directly training SNNs. Summary of the Invention

[0005] This invention provides an image recognition method based on Fourier transform and cross-time constraints of spiking neural networks, which can effectively improve the performance of spiking neural network models.

[0006] To achieve the above objectives, this invention provides an image recognition method based on Fourier transform and time-constrained spiking neural networks, which, crucially, includes the following steps:

[0007] Step 1: Construct an image recognition system based on Fourier transform and time-constrained spiking neural networks. The image recognition system is configured with an image acquisition module, a preprocessing module, and a Fourier transform and time-constrained spiking neural network connected in sequence.

[0008] Step 2: The image acquisition module acquires an image sample set and passes the image sample set to the preprocessing module for preprocessing to obtain a training set;

[0009] Step 3: Train the Fourier-based and time-constrained spiking neural network using the training set. In the forward propagation, a step function is used to make the spiking neurons fire pulses. In the backward propagation, the derivative of a finite-term Fourier series is used as the gradient estimate of the pulse on the membrane potential.

[0010] Step 4: Use cosine similarity (CS) to measure the similarity between each time step, calculate the loss of the objective function of the Fourier-based and cross-time-constrained spiking neural network after the rth training iteration, and adjust the network weight parameters based on the loss of the objective function.

[0011] Step 5: Repeat steps 3-4 until the set number of training iterations is reached, then end the training to obtain a trained spiking neural network based on Fourier transform and cross-time constraints.

[0012] Step 6: The image acquisition module acquires the image data to be identified, and then passes the image data to the preprocessing module for preprocessing before passing it to the trained spiking neural network based on Fourier transform and cross-time constraints.

[0013] Step 7: The trained Fourier-based and time-constrained spiking neural network performs image classification and recognition operations on the preprocessed image data to be identified, and outputs the image classification results.

[0014] The above design effectively approximates the gradient of the impulse firing activity by using Fourier series in the frequency domain. This provides a new perspective for mitigating the non-differentiability of impulse activity. Utilizing Fourier series preserves the principal direction of the true gradient of the impulse activity, enabling the trained spiking neural network (SNN) model to converge faster and more accurately. Furthermore, by using cosine similarity to constrain the output distribution of the SNN's output layer to be more similar at each time step, the information carried by the output at each time step is better utilized to guide the training of the SNN, effectively improving the performance of the SNN model.

[0015] Preferably, in step 3, the step function is a Heaviside function, and the calculation expression of the Heaviside function is:

[0016]

[0017] Where S represents the pulse sequence, t represents time, and V represents the membrane potential at both ends of the neuron. th This represents the membrane potential threshold, and Headviside() represents the Heaviside function;

[0018] The expression for calculating the derivative of the finite-term Fourier series is as follows:

[0019]

[0020] Where n represents the number of Fourier series; w represents the angular frequency. i represents an index variable and is a positive integer.

[0021] While pseudo-gradient methods allow spiking neural networks (SNNs) to be trained using backpropagation algorithms, currently proposed pseudo-gradient functions approximate the step function in the spatial domain. These functions vary in shape and exhibit different performance characteristics across different tasks. This invention approximates the step function using Fourier series in the frequency domain, thus better capturing the direction of the true gradient in SNNs.

[0022] Preferably: the derivative h of the finite-term Fourier series n ′(t) is constructed through the following process:

[0023] Any periodic function can be expressed as a linear combination of a Fourier series consisting of trigonometric functions, namely sine and cosine functions, as follows:

[0024]

[0025] Where f(t) is a function with a period of T = 2π. It is the DC component, a n and b n These are Fourier coefficients;

[0026] a0、a n and b n The calculation expression is as follows:

[0027]

[0028]

[0029] The aperiodic step function is extended periodically to become a periodic function. The coefficients of the Fourier series and the DC component of this step function are calculated using the following expressions:

[0030] a0 = 1

[0031] a n =0

[0032]

[0033] The Fourier series expression for the periodic Heaviside function is as follows:

[0034]

[0035] The derivative expression of the Fourier series is:

[0036]

[0037] Using an infinite-term Fourier series to approximate the Heaviside function is a lossless representation, but an infinite-term Fourier series incurs huge computational overhead. This invention uses a finite-term Fourier series, ignoring the relatively low-energy high-frequency components. Therefore, the expression for the finite-term Fourier series is:

[0038]

[0039] The derivative expression of a finite-term Fourier series is:

[0040]

[0041] The derivative of a finite-term Fourier series is used as the derivative of the step function for backpropagation.

[0042] Using a finite-term Fourier series to approximate the Heaviside function has two advantages:

[0043] 1) Avoid the computational overhead of infinite trigonometric function combinations by using a finite number of Fourier series.

[0044] 2) Finite-term Fourier series are more conducive to approximating the Heaviside function, as they do not affect the low-frequency information of the Heaviside function, which often accounts for the majority of the energy.

[0045] Preferably, in step 4, the loss function expression for the cosine similarity CS is as follows:

[0046]

[0047] Where T represents the total time step, and CS(V[l],V[t]) represents the cosine similarity of the membrane potential at time l and time t.

[0048] This invention adds a cosine similarity constraint to the objective function of SNNs, making their outputs as consistent as possible at each time step and reducing the differences between different time steps. The cosine similarity constraint fully utilizes the temporal information contained in the outputs of SNNs at different time steps to improve the generalization ability of SNNs on the test set.

[0049] Preferably, in step 4, the objective function is calculated as follows:

[0050] L = L CE +λL CS

[0051]

[0052] Where L represents the total loss value, L CE L represents the cross-entropy loss. CS λ represents the cosine similarity loss; λ is a hyperparameter used to constrain the size of the cosine similarity loss term. The sum of the average membrane potentials of all neurons, k represents the k-th image category, K is the total number of image categories, and y p This represents the true label of the p-th neuron; This represents the average membrane potential of the p-th neuron.

[0053] L CE Used to ensure that the average membrane potential approximates the actual label, L CS This is used to ensure that the similarity of the outputs is more consistent across each time step. The final loss function L can better optimize the direction of weight updates, thereby preventing abnormal total output caused by abnormal outputs at specific time steps and reducing the overconfidence of SNNs predictions at specific time points.

[0054] Preferably, when the image sample set is a static dataset, the deep residual learning framework SEW-ResNet19 for deep spiking neural networks is used as a spiking neural network based on Fourier and cross-time constraints for training.

[0055] When the image sample set is a neuromorphic dataset, the VGGSNN model, which combines the VGG network and the spiking neural network, is used as a spiking neural network based on Fourier transform and cross-time constraints for training.

[0056] Preferably, the dynamic equation of the LIF neuron in the Fourier-based and time-constrained spiking neural network is:

[0057]

[0058] Where, τ m It is the membrane time constant, and τ m =RC, where R represents resistance and C represents capacitance; V represents the membrane potential across the neuron; I represents the input current, which is determined by the product of the input pulse sequence and the synaptic weights, as expressed below:

[0059]

[0060] Among them, W ij represents the weight between presynaptic neuron i and postsynaptic neuron j, and S represents the pulse sequence;

[0061] The continuous differential equation of the LIF neuron is transformed into a discrete difference equation, as shown in the following expression:

[0062]

[0063] Preferably, in step 7, the average membrane potential of the output layer of the Fourier-based and time-constrained spiking neural network is used as a classification index, and the expression for updating the membrane potential state of the neurons in the output layer of the Fourier-based and time-constrained spiking neural network is:

[0064]

[0065] Each neuron in the output layer corresponds to an image category, and the image category corresponding to the neuron with the highest average membrane potential emitted by the output layer is output as the image classification result.

[0066] For example, when the image categories include five types of images: airplane, car, ship, bicycle, and train, the output layer contains five neurons, which correspond one-to-one with the five image categories. If the neuron corresponding to the airplane has the highest average membrane potential, then the image classification result of the image data to be identified is an airplane image.

[0067] In the Fourier-based and time-constrained spiking neural network, the input image X is directly encoded to obtain a pulse sequence input, which is then passed through LIF neurons to obtain a pulse output. In the last layer of the network, the LIF neurons do not fire pulses (assuming the threshold is infinite), but only accumulate membrane potential and use the membrane potential as the classification criterion.

[0068] The beneficial effects of this invention are:

[0069] 1. By utilizing a combination of finite terms of Fourier series, the function can be consistent with the Heaviside function in the low-frequency part, with errors only existing in the high-frequency part. It preserves the principal direction of the true gradient, which can enable spiking neural networks (SNNs) to converge faster and achieve high performance.

[0070] 2. By utilizing the temporal information contained in the output distribution of each time step of the output layer of SNNs, a cosine similarity constraint was designed to make the output distribution of each time step more consistent. Attached Figure Description

[0071] Figure 1 This is a flowchart illustrating the overall workflow of the present invention.

[0072] Figure 2 The figure shows the curves of several pseudo-gradient functions after fast Fourier transform in the embodiment, and the difference between them and the Heaviside curve in the frequency domain.

[0073] Figure 3 The following is an accuracy curve for different Fourier series terms n in the embodiments;

[0074] Figure 4 The output distribution diagrams of the baseline model and FTC-SNN in the embodiment are shown below after the input is DVS-CIFAR10 samples;

[0075] Figure 5 The output distribution diagrams of the baseline model and FTC-SNN in the embodiment are shown below when the input is CIFAR10 samples;

[0076] Figure 6 This is a schematic diagram of the network structure of SEW-ResNet19 and VGGSNN in the embodiment. Detailed Implementation

[0077] The present invention will be further described in detail below with reference to the accompanying drawings and specific examples. The following embodiments or drawings are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0078] like Figure 1 As shown: An image recognition method based on Fourier transform and cross-time-constrained spiking neural networks includes the following steps:

[0079] Step 1: Construct an image recognition system based on Fourier transform and time-constrained spiking neural networks. The image recognition system is configured with an image acquisition module, a preprocessing module, and a Fourier transform and time-constrained spiking neural network connected in sequence.

[0080] Step 2: The image acquisition module acquires an image sample set and passes the image sample set to the preprocessing module for preprocessing to obtain a training set;

[0081] Step 3: Train the Fourier-based and time-constrained spiking neural network using the training set. In the forward propagation, a step function is used to make the spiking neurons fire pulses. In the backward propagation, the derivative of a finite-term Fourier series is used instead of the step function as the gradient estimate of the pulse on the membrane potential.

[0082] Step 4: Use cosine similarity (CS) to measure the similarity between each time step, calculate the loss of the objective function of the Fourier-based and cross-time-constrained spiking neural network after the rth training iteration, and adjust the network weight parameters based on the loss of the objective function.

[0083] Step 5: Repeat steps 3-4 until the set number of training iterations is reached, then end the training to obtain a trained spiking neural network based on Fourier transform and cross-time constraints.

[0084] Step 6: The image acquisition module acquires the image data to be identified, and then passes the image data to the preprocessing module for preprocessing before passing it to the trained spiking neural network based on Fourier transform and cross-time constraints.

[0085] Step 7: The trained Fourier-based and time-constrained spiking neural network performs image classification and recognition operations on the preprocessed image data to be identified, and outputs the image classification results.

[0086] In step 2, the preprocessing operations include image cropping and image enhancement.

[0087] In step 3, the step function is the Heaviside function, and the expression for calculating the Heaviside function is:

[0088]

[0089] Where S represents the pulse sequence, t represents time, and V represents the membrane potential at both ends of the neuron. th Represents the membrane potential threshold; Heaviside() represents the Heaviside function.

[0090] The expression for calculating the derivative of the finite-term Fourier series is as follows:

[0091]

[0092] Where n represents the number of Fourier series; w represents the angular frequency. i represents an index variable and is a positive integer.

[0093] The derivative h of the finite-term Fourier series n ′(t) is constructed through the following process:

[0094] Any periodic function can be expressed as a linear combination of a Fourier series consisting of trigonometric functions, namely sine and cosine functions, as follows:

[0095]

[0096] Where f(t) is a function with a period of T = 2π. It is the DC component, a n and b n These are Fourier coefficients;

[0097] a0、a n and b n The calculation expression is as follows:

[0098]

[0099] The aperiodic step function is extended periodically to become a periodic function. The coefficients of the Fourier series and the DC component of this step function are calculated using the following expressions:

[0100] a0 = 1

[0101] a n =0

[0102]

[0103] The Fourier series expression for the periodic Heaviside function is as follows:

[0104]

[0105] The derivative expression of the Fourier series is:

[0106]

[0107] Therefore, the expression for the finite-term Fourier series is:

[0108]

[0109] The derivative expression of a finite-term Fourier series is:

[0110]

[0111] Using a finite-term Fourier series to approximate the Heaviside function has two advantages:

[0112] 1) Avoid the computational overhead of infinite trigonometric function combinations by using a finite number of Fourier series.

[0113] 2) Finite-term Fourier series are more conducive to approximating the Heaviside function, as they do not affect the low-frequency information of the Heaviside function, which often accounts for the majority of the energy.

[0114] To illustrate the second advantage more intuitively, this embodiment will discuss h separately. n Perform a Fast Fourier Transform (FFT) on the Heaviside function (t), the differentiable impulse function Dspike(t), and the asymptotic evolution function EvAF(t), and plot the differences between the FFT-processed Heaviside function and these three functions, such as... Figure 2 As shown. From Figure 2 The third line shows that h n The function (t) approximates the Heaviside function using a combination of 20 trigonometric functions. In the frequency domain, its low-frequency portion is consistent with the Heaviside function, while Dspike(x) and EvAF(x) differ from the Heaviside(x) function in both the low- and high-frequency portions. Using a finite-term Fourier series can more accurately preserve the principal gradient direction of the original Heaviside function, which is the low-frequency portion with the most energy.

[0115] In step 4, the loss function expression for the cosine similarity CS is as follows:

[0116]

[0117] Where T represents the total time step, and CS(V[l],V[t]) represents the cosine similarity of the membrane potential at time l and time t.

[0118] In step 4, the objective function is calculated as follows:

[0119] L = L CE +λL CS

[0120]

[0121] Where L represents the total loss value, L CE L represents the cross-entropy loss. CS λ represents the cosine similarity loss; λ is a hyperparameter used to constrain the size of the cosine similarity loss term. The sum of the average membrane potentials of all neurons, k represents the k-th image category, K is the total number of image categories, and y p This represents the true label of the p-th neuron; This represents the average membrane potential of the p-th neuron.

[0122] When the image sample set is a static dataset, the deep residual learning framework SEW-ResNet19 for deep spiking neural networks is used as a spiking neural network based on Fourier and cross-time constraints for training.

[0123] When the image sample set is a neuromorphic dataset, the VGGSNN model, which combines the VGG network and the spiking neural network, is used as a spiking neural network based on Fourier transform and cross-time constraints for training.

[0124] The dynamic equations of the LIF neurons in the Fourier-based and time-constrained spiking neural network are as follows:

[0125]

[0126] Where, τ m It is the membrane time constant, and τ m =RC, where R represents resistance and C represents capacitance; V represents the membrane potential across the neuron; I represents the input current, which is determined by the product of the input pulse sequence and the synaptic weights, as expressed below:

[0127]

[0128] Among them, W ij represents the weight between presynaptic neuron i and postsynaptic neuron j, and S represents the pulse sequence;

[0129] The continuous differential equation of the LIF neuron is transformed into a discrete difference equation, as shown in the following expression:

[0130]

[0131] In step 7, the average membrane potential of the output layer of the Fourier-based and time-constrained spiking neural network is used as a classification index. The expression for updating the membrane potential state of the neurons in the output layer of the Fourier-based and time-constrained spiking neural network is as follows:

[0132]

[0133] Each neuron in the output layer corresponds to an image category, and the image category corresponding to the neuron with the highest average membrane potential emitted by the output layer is output as the image classification result.

[0134] Next, through ablation experiments, spiking neural networks (SNNs) trained solely using Fourier series (FS) as pseudo-gradient functions are compared with benchmarks. Then, cosine similarity constraints are imposed on their objective functions to verify the effectiveness of the invention. The invention is then compared with other state-of-the-art methods on two datasets. Finally, the impact of the number of Fourier series (FS) is analyzed, and visualizations of temporal similarity are provided.

[0135] To provide more convincing results, this embodiment demonstrates the superiority of the invention for training SNNs on two challenging datasets: the static CIFAR-10 and CIFAR-100 datasets, and the neuromorphic datasets DVS-CIFAR10 and N-Caltech101.

[0136] The CIFAR-10 dataset contains 60,000 color images across 10 classes, with 50,000 used for training and the remaining 10,000 for testing. The number of images per class is equal in both the training and testing sets, and each image is 32x32 pixels. The CIFAR-100 dataset contains 60,000 images across 100 classes, with 600 32x32 color images per class. 500 images are used for training, and 100 for testing. The DVS-CIFAR10 dataset converts 10,000 images from the CIFAR-10 dataset into 10,000 event streams using a dynamic vision sensor. 9,000 of these event streams are used as the training set, and the remaining 1,000 as the test set. The N-Caltech101 dataset contains 101 event streams, with the training and testing sets divided in a 9:1 ratio.

[0137] For the CIFAR-10 / 100 training set, data augmentation was performed using random cropping, random horizontal flipping, and cutout (randomly cutting out portions of samples and padding with zero pixels). For the DVS-CIFAR10 training set, since the original sensor resolution was 128*128, the original images were first converted to 48*48 resolution, and then random cropping and random horizontal flipping were applied. For the N-Caltech101 training set, the resolution was similarly converted to 48*48, and the same data augmentation methods were applied.

[0138] For the CIFAR-10 / 100 dataset, this embodiment uses the SEW-ResNet19 network architecture, and for the neuromorphic dataset, it uses the VGGSNN (64C3-128C3-AP2-256C3-256C3-AP2-512C3-512C3-AP2-512C3-512C3-AP2-10FC) network architecture. Here, C3 indicates a 3x3 kernel size, AP2 indicates average pooling with a 2x2 kernel size, and FC represents a fully connected layer. The network architectures of SEW-ResNet19 and VGGSNN are as follows: Figure 6 As shown.

[0139] The hyperparameters used in this experiment differ slightly for the static dataset for ANNs and the neuromorphic dataset for SNNs. Table 1 lists the detailed hyperparameter settings, including neuron leak terms, firing thresholds, encoding time windows, and learning rates.

[0140] For CIFAR-10 / 100, direct coding is used, meaning static images are directly input into the SNNs, and their first layer encodes the images into binary pulse sequences. For the remaining two neuromorphic datasets, since each event stream contains a lot of event information, directly inputting it into the SNNs would result in massive memory consumption and increased training time. Therefore, the original event stream information is integrated into frame data. Furthermore, to avoid losing information contained in the membrane potential, neurons in the output layer only integrate input from the previous layer; the membrane potential does not decay and no pulses are fired. For CIFAR-10 / 100, an SGD optimizer with a momentum of 0.9 and a weight decay of 5e-4 is used, and the learning rate decays to 0 with cosine decay during training. For DVS-CIFAR10 and N-Caltech101, an Adam optimizer with a weight decay of 1e-4 is used, and the learning rate also decays to 0 with cosine decay during training. Additionally, to save GPU memory and accelerate model training time, mixed precision is used to train the SNNs. The following accuracy rates are the results of three replicate experiments with different random seeds, and are presented as mean and standard deviation.

[0141] Table 1

[0142]

[0143] In the ablation experiments, VGGSNN was used as the backbone, and multiple ablation experiments were conducted on the DVS-CIFAR10 and N-Caltech101 datasets to verify the effectiveness of the two modules proposed in this invention. For fair comparison, the widely used triangular pseudo-gradient function [ref] was used to calculate the gradient of the spiking behavior of the spiking neuron during backpropagation. The loss function used was the cross-entropy between the mean membrane potential of the output layer and the target label, i.e., the cross-entropy loss function L. CE The trained SNNs models serve as the baseline. As shown in Table 2, for the DVS-CIFAR10 dataset, the first row represents the baseline, and the second row represents replacing the pseudo-gradient function with the Fourier series (FS). It can be seen that using FS benefits the trained model, increasing the accuracy from 81.10% to 81.90%. As shown in the third row, combining FS with cosine similarity constraints achieves the best performance, increasing accuracy from 81.10% to 83.70%, demonstrating the superiority of using both FS and cosine similarity constraints simultaneously during training. Similarly, in the N-Caltech101 dataset, it can be seen that using FS alone increases accuracy from 79.87% to 81.62%, while using both modules simultaneously increases accuracy to 82.06%. In summary, the experiments in Table 2 demonstrate that the two modules proposed in this invention help improve the performance of SNNs.

[0144] Table 2

[0145]

[0146] In Table 2, Among the many pseudo-gradient functions, the most popular is the triangular pseudo-gradient function, which is expressed as:

[0147]

[0148] Here, γ is a constraint factor that controls the range and magnitude of the gradient.

[0149] To demonstrate the superiority of the proposed method, a comparison with several existing works on the challenging CIFAR-10 / 100 dataset, which contains more complex information, is shown in Table 3. For the CIFAR-10 dataset, the table shows that, in methods for training SNNs from scratch, the Fourier-based and cross-time-constrained spiking neural network FTC-SNN achieves 0.17% higher accuracy using only 2 time steps than RecDis-SNN trained with 6 time steps and using membrane potential distribution loss. The TET method encourages the membrane potential distribution of the SNN's output layer to approximate the true label at each time step and also utilizes the temporal information processing capability of SNNs, achieving 94.50% accuracy at a time step of 6. In contrast, FTC-SNN achieves 95.72% accuracy at a time step of 2, which is 1.22% higher than TET. The Joint A-SNN, trained using a hybrid method, achieves a recognition accuracy of 95.45% at a time step of 4, while FTC-SNN achieves 0.36% higher accuracy at the same time step. The SRP, trained using a transformation method, achieves 95.60% accuracy at a time step of 8, but FTC-SNN achieves higher accuracy and lower latency at a time step of 2. Furthermore, on the CIFAR-100 dataset, the SRP method using ANN2SNN with a VGG16 network and a time step of 32 achieves 76.45% accuracy, while the SGS-SNN, trained directly from scratch, achieves 76.74% accuracy at a time step of 6. FTC-SNN achieves comparable or even better performance at a time step of only 2. Comparative experiments with a series of recent works on these two static datasets fully demonstrate that FTC-SNN can achieve high performance with low latency.

[0150] Table 3

[0151]

[0152] Among them, PLIF is parameterized leak integral distribution, Dspike is differentiable pulse, Joint A-SNN is joint training of artificial neural network and spiking neural network, SRP is optimization strategy based on residual membrane potential, TET is time-efficient training, RecDis-SNN is correcting membrane potential distribution to directly train spiking neural network, SGS-SNN is alternative gradient scaling for direct training of spiking neural network, LIAF is leak integral digital distribution, TCJA-SNN is temporal channel joint attention of spiking neural network, NDA is neuromorphic data augmentation, Eventmix is ​​an efficient data augmentation strategy based on event learning, and FTC-SNN is spiking neural network based on Fourier and cross-temporal constraints.

[0153] Compared to static datasets designed for ANNs, neuromorphic datasets inherently contain rich temporal information and exhibit high temporal correlation, making them more suitable for SNNs to leverage their temporal information processing capabilities. Both datasets use VGGSNN networks to train SNNs. Table 4 compares the proposed method with existing work, showing that FTC-SNN achieves state-of-the-art performance on both datasets. On the DVS-CIFAR10 dataset, the proposed method achieves a best-in-class recognition accuracy of 83.70%, outperforming TCJA-SNN (which incorporates an attention mechanism) by approximately 3% and the TET method by approximately 0.53%. On the N-Caltech101 dataset, the proposed method achieves a best-in-class performance of 82.06%, exceeding TCJA-SNN by approximately 3.56%. Two other neuromorphic data augmentation methods, NDA and EventMix, using ResNet-19 / 18 network structures, achieved accuracies of 78.60% and 79.50%, respectively. The method of this invention also employs data augmentation techniques, outperforming the NDA and EventMix methods by approximately 3.46% and 2.56% respectively in the VGGSNN network structure. In these two neuromorphic datasets, the inputs at different time steps differ, resulting in varying outputs at each time step. By applying cosine similarity constraints, the outputs at each time step can be made more consistent, thereby improving recognition accuracy.

[0154] Table 4

[0155]

[0156] This invention uses a finite number of Fourier series to approximate impulse activity. Different numbers of Fourier series yield significantly different approximation results. A larger number of Fourier series can reduce the gap between the approximation and the Heaviside function, but it also results in more computation, ultimately increasing the training time of SNNs. Here, using ResNet-20 as a benchmark, a series of experiments were conducted on the CIFAR-10 dataset, and the accuracy achieved by SNNs with different numbers of Fourier series as pseudo-gradient functions is presented. Figure 3As shown, the test accuracy of SNNs changes when the Fourier series term n is fixed from 1 to 17 during training. It can be seen that the network performs better when n is fixed between 3 and 12 than other values. Since the SNNs of this invention are trained from scratch, when n is fixed to a value greater than 12, although the difference between the Fourier series term and the Heaviside function is smaller than when n is fixed between 3 and 11, the network performance begins to decline. When the Fourier series term n is fixed to 6, 7, and 8, the performance of the SNNs becomes saturated, so n is allowed to start from a relatively moderate value and vary as training progresses. This setting allows the network to start training with a moderate n in the early stages, and a larger n in the later stages can compensate for the difference between the Fourier series and the impulse activity function. In the experiments in 4.2, 4.3, and 4.4 above, n is gradually increased from a moderate value ns (e.g., 7) to 2ns.

[0157] This invention imposes a constraint on the loss function to ensure that its output at each time step is consistent. This allows SNNs to achieve high performance with lower latency while maintaining consistent output at each step. This embodiment visualizes the output distribution caused by different types of input. First, for visualizing the input of a sample in the DVS-CIFAR10 dataset, the distribution of the output layers of the FTC-SNN and the baseline model at different time steps is shown in the visualization results. Figure 4 As shown, the baseline model exhibits significant differences in output at each time step. In the first time step, the output of each neuron is uniform, while the true label is the tenth category, leading to incorrect identification. Furthermore, at time steps 3, 6, 7, 9, and 10, the output distribution of the baseline model is inconsistent, identifying the input samples as 6, 2, 6, 2, and 6 respectively. In contrast, the FTC-SNN, in the first time step, constrains the output of the originally uniformly distributed ten neurons to maximize the probability of the tenth neuron, thus correcting the identification result to the true category. Moreover, from the second to the tenth time step, the output distribution of the FTC-SNN shows strong consistency, with the tenth neuron consistently having the highest probability, while the probabilities of the other neurons are essentially zero. Similarly, as... Figure 5As shown, given a sample from CIFAR-10 as input to the baseline model and FTC-SNN, the output distribution at different time steps is displayed. Finally, the mean cosine similarity of the baseline model is calculated to be 0.6394 and 0.4245 for input samples of CIFAR-10 and DVS-CIFAR10, respectively, while that of FTC-SNN is 0.9819 and 0.9788. The results show that the cosine similarity constraint of this invention can make the output distribution more consistent at each time step. In summary, FTC-SNN can constrain the output at each time step to be more consistent through cosine similarity, making it more likely to achieve high accuracy performance within a shorter time step.

[0158] This invention aims to alleviate the difficulty of non-differentiability of impulse activations in directly trained SNNs from a novel perspective. Unlike previous methods that approximate the gradient of impulse activity using spatially differentiable functions, this invention provides insights from the frequency domain. Previous dynamic pseudo-gradients, when approximating the gradient of impulse activity, exhibit discrepancies with the original Heaviside function in both low- and high-frequency regions. However, the proposed method, utilizing a combination of finite-term Fourier series, achieves consistency with the Heaviside function in the low-frequency region, with errors only in the high-frequency region. It preserves the principal direction of the true gradient, thus enabling SNNs to converge faster and achieve high performance. Furthermore, this invention utilizes the temporal information inherent in the output distribution of the SNN's output layer at each time step to design a cosine similarity constraint to ensure greater consistency in the output distribution at each time step. Combining these two methods, FTC-SNN achieves competitive performance compared to existing work.

[0159] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An image recognition method based on Fourier transform and time-constrained spiking neural networks, characterized in that, Includes the following steps: Step 1: Construct an image recognition system based on Fourier transform and time-constrained spiking neural networks. The image recognition system is configured with an image acquisition module, a preprocessing module, and a Fourier transform and time-constrained spiking neural network connected in sequence. Step 2: The image acquisition module acquires an image sample set and passes the image sample set to the preprocessing module for preprocessing to obtain a training set; Step 3: Train the Fourier-based and time-constrained spiking neural network using the training set. In the forward propagation, a step function is used to make the spiking neurons fire pulses. In the backward propagation, the derivative of a finite-term Fourier series is used as the gradient estimate of the pulse on the membrane potential. Step 4: Use cosine similarity (CS) to measure the similarity between each time step, calculate the loss of the objective function of the Fourier-based and cross-time-constrained spiking neural network after the rth training iteration, and adjust the network weight parameters based on the loss of the objective function. Step 5: Repeat steps 3-4 until the set number of training iterations is reached, then end the training to obtain a trained spiking neural network based on Fourier transform and cross-time constraints. Step 6: The image acquisition module acquires the image data to be identified, and then passes the image data to the preprocessing module for preprocessing before passing it to the trained spiking neural network based on Fourier transform and cross-time constraints. Step 7: The trained Fourier-based and time-constrained spiking neural network performs image classification and recognition operations on the preprocessed image data to be identified, and outputs the image classification results.

2. The image recognition method based on Fourier and time-constrained spiking neural networks according to claim 1, characterized in that: In step 3, the step function is the Heaviside function, and the expression for calculating the Heaviside function is: Where S represents the pulse sequence, t represents time, and V represents the membrane potential at both ends of the neuron. th Represents the membrane potential threshold; Heaviside() represents the Heaviside function. The expression for calculating the derivative of the finite-term Fourier series is as follows: Where n represents the number of Fourier series; w represents the angular frequency. i represents an index variable and is a positive integer.

3. The image recognition method based on Fourier and time-constrained spiking neural networks according to claim 2, characterized in that: The derivative h of the finite-term Fourier series n ′(t) is constructed through the following process: Any periodic function can be expressed as a linear combination of a Fourier series consisting of trigonometric functions, namely sine and cosine functions, as follows: Where f(t) is a function with a period of T = 2π. It is the DC component, a n and b n These are Fourier coefficients; a0、a n and b n The calculation expression is as follows: The aperiodic step function is extended periodically to become a periodic function. The coefficients of the Fourier series and the DC component of this step function are calculated as follows: a0＝1 a n ＝0 The Fourier series expression for the periodic Heaviside function is as follows: The derivative expression of the Fourier series is: Therefore, the expression for the finite-term Fourier series is: The derivative expression of a finite-term Fourier series is:

4. The image recognition method based on Fourier and time-constrained spiking neural networks according to claim 1, characterized in that: In step 4, the loss function expression for the cosine similarity CS is as follows: Where T represents the total time step, and CS(V[l],V[t]) represents the cosine similarity of the membrane potential at time l and time t.

5. The image recognition method based on Fourier and time-constrained spiking neural networks according to claim 4, characterized in that: In step 4, the objective function is calculated as follows: L=L CE +λL CS Where L represents the total loss value, L CE L represents the cross-entropy loss. CS This represents the cosine similarity loss; λ is a hyperparameter. The sum of the average membrane potentials of all neurons is represented by y, k represents the k-th image category, K is the total number of image categories, and y represents the total number of image categories. p This represents the true label of the p-th neuron; This represents the average membrane potential of the p-th neuron.

6. The image recognition method based on Fourier and time-constrained spiking neural networks according to claim 1, characterized in that: When the image sample set is a static dataset, the deep residual learning framework SEW-ResNet19 for deep spiking neural networks is used as a spiking neural network based on Fourier and cross-time constraints for training. When the image sample set is a neuromorphic dataset, the VGGSNN model, which combines the VGG network and the spiking neural network, is used as a spiking neural network based on Fourier transform and cross-time constraints for training.

7. The image recognition method based on Fourier and time-constrained spiking neural networks according to claim 1, characterized in that: The dynamic equations of the LIF neurons in the Fourier-based and time-constrained spiking neural network are as follows: Where, τ m It is the membrane time constant, and τ m =RC, where R represents resistance and C represents capacitance; V represents the membrane potential across the neuron; I represents the input current, which is determined by the product of the input pulse sequence and the synaptic weights, as expressed below: Among them, W ij represents the weight between presynaptic neuron i and postsynaptic neuron j, and S represents the pulse sequence; The continuous differential equation of the LIF neuron is transformed into a discrete difference equation, as shown in the following expression:

8. The image recognition method based on Fourier and time-constrained spiking neural networks according to claim 1, characterized in that: In step 7, the average membrane potential of the output layer of the Fourier-based and time-constrained spiking neural network is used as a classification index. The expression for updating the membrane potential state of the neurons in the output layer of the Fourier-based and time-constrained spiking neural network is as follows: Each neuron in the output layer corresponds to an image category, and the image category corresponding to the neuron with the highest average membrane potential emitted by the output layer is output as the image classification result.

Images