Image classification method based on cross-level recursive spiking neural network

Abstract

The invention relates to the technical field of image classification, in particular to an image classification method based on a cross-level recursive spiking neural network, which comprises the following steps of: firstly, integrating forward input and reverse input by using a soft gating mechanism to construct a bidirectional spiking neuron, and then constructing a bidirectional pulse driving type pulse Mangban hybrid module based on the bidirectional spiking neuron; thirdly, constructing a multi-time-period encoder by utilizing a bidirectional pulse driving type pulse Mangban hybrid module; meanwhile, a fusion core component and a conversion core component are used for fusing and converting various types of high-level information to construct a repeater; and finally, combining the encoder and the repeater into a cross-level recursive spiking neural network, and training the cross-level recursive spiking neural network model by using a substitution gradient descent method. According to the method, a model can be established for cone neurons, an uplink and downlink bidirectional process in cross-level connection is established, and more complex and advanced feature representation is generated, so that the network can capture richer information.

Description

Technical Field

[0001] The present invention relates to the technical field of image classification, and particularly relates to an image classification method based on a cross-level recursive spiking neural network. Background Art

[0002] With the continuous development of Internet technology, digital information technology and computer technology, a large amount of multimedia data, such as images, audio, and video, is generated every day. Nowadays, in the face of a large amount of image information, automatic image classification has become a new research hotspot. Given the powerful non-linear modeling ability of deep learning models and the convenient end-to-end learning method, deep learning technology has become the mainstream method in image recognition tasks. However, the training of these deep learning models relies on high-performance graphics processing units (GPUs) to improve the computing speed, resulting in a large amount of power and water resource consumption. Compared with deep learning, brain-inspired computing mimics the low-energy consumption characteristics of the brain and can significantly reduce energy consumption when performing complex tasks compared with traditional computing platforms. It has the advantages of low power consumption and low latency, and is particularly suitable for applications in power consumption and latency-sensitive environments such as edge computing.

[0003] The research on brain-inspired computing mainly focuses on spiking neural networks, and its application in image recognition tasks is gradually attracting attention. For example, the team of the Institute of Automation, Chinese Academy of Sciences (Li Qianpeng, Jia Shuncheng, Zhang Tielin, Chen Liang. Efficient Image Classification Based on Adaptive Time-Step Spiking Neural Network [J]. Acta Automatica Sinica, 2024, 50(9): 1724-1735.) proposed an adaptive time selection algorithm, which supports more accurately obtaining the minimum time step required for correct classification of samples by associating the confidence of the output layer and the time steps required for inference, and can effectively reduce the average inference latency and energy consumption. Zhu et al. (Zhu Y, Wang Exploring loss functions for time-based training strategy in spiking neural networks [C]. Proceedings of the 37th Conference on Neural Information Processing Systems, 2024: 1-8.) proposed an enhanced computational loss function to replace the commonly used mean square counting loss function, providing sufficient positive overall gradients for time-based training strategies, thereby improving image classification performance.

[0004] However, current spiking neural networks still lack in-depth exploration of brain-like characteristics. On the one hand, the biological neuron models used can only model standard neurons and cannot model pyramidal neurons, the most abundant type in the cerebral cortex. On the other hand, the model structure lacks an up-and-down bidirectional process with cross-level connections and cannot generate more complex and advanced feature representations, so the network cannot capture more abundant information.

[0005] To solve the above problems, the present invention constructs gated bidirectional spiking neurons, builds a bidirectional spiking-driven spiking mamba hybrid module based on spiking neurons, and further constructs a cross-level recurrent spiking neural network. Summary of the Invention

[0006] The object of the present invention is to solve the deficiencies in the prior art and propose an image classification method based on a cross-level recurrent spiking neural network, which can model pyramidal neurons, establish an up-and-down bidirectional process with cross-level connections, generate more complex and advanced feature representations, so that the network can capture more abundant information.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] An image classification method based on a cross-level recurrent spiking neural network, the specific steps are as follows:

[0009] Step 1, construct bidirectional spiking neurons: For the forward input of the bidirectional spiking neurons and the reverse input from the bidirectional spiking neurons in the higher neuron layer, use a soft gating mechanism to integrate the forward input and the reverse input;

[0010] Step 2, construct a bidirectional spiking-driven spiking mamba hybrid module: Based on the bidirectional spiking neurons, construct a bidirectional spiking-driven spiking mamba hybrid module;

[0011] Step 3, construct an encoder: Use the bidirectional spiking-driven spiking mamba hybrid module to construct a multi-time segment encoder;

[0012] Step 4, construct a relay: Use two core components of fusion and transformation to fuse and transform various types of high-level information;

[0013] Step 5: Construct and train a cross-level recurrent spiking neural network model: Combine the encoder and the relay into a cross-level recurrent spiking neural network, and use the surrogate gradient descent method to complete the training of the cross-level recurrent spiking neural network model.

[0014] Preferably, in step 1, the specific method is as follows:

[0015] S11: At time t, the forward input of the bidirectional spiking neurons from the j-th spiking neuron in the previous layer is represented as The reverse inputs from the bi-polar neurons in the higher neuron layer are respectively denoted as The forward input and the reverse input are integrated by using a soft gating mechanism as follows:

[0016]

[0017] In Equation (1), Sig(·) represents the Sigmoid function, W1 and W2 respectively represent the weight coefficients of the forward input signal and the reverse input signal, and o jt represents the integrated input signal;

[0018] S12: Assume that the membrane potential of the ith bi-polar neuron at time t-1 is denoted as u i(t-1) , then the instantaneous membrane potential of the neuron before firing is expressed as follows:

[0019]

[0020] In Equation (2), τ represents the integration time constant, and the output of the ith bi-polar neuron at time t is denoted as:

[0021]

[0022] In Equation (3), H(·) is the Heaviside function, and υ θ represents the threshold;

[0023] S13: The membrane potential of the ith bi-polar neuron at time t is reset to:

[0024] u it = m it (1 - o it ) + ν rest o it (4)

[0025] In Equation (4), ν rest represents the reset time constant.

[0026] Preferably, in step 2, the specific method is as follows:

[0027] S21: For the output O it of the bi-polar neuron, first perform a normalization process on it. The normalization operation is denoted as LN(·), and its processing formula is:

[0028]

[0029] In Equation (5), represents the output of the jth neuron in the normalization layer;

[0030] S22: Construct a linear layer using spiking neurons. The output of the $i$-th neuron is expressed as:

[0031]

[0032] In Equation (6), $\tau$ represents the integration time constant, $H(\cdot)$ is the Heaviside function, and $\upsilon$ θ represents the threshold;

[0033] S23: Construct coefficient matrices $A$, $B$, and $C$ using a random scanning method to establish the upper branch of the bidirectional pulse-driven spiking Mamba hybrid module. The upper branch calculates the state transition equation for the output $O$ it as follows:

[0034]

[0035] For the output $y$ of the state transition equation it perform activation processing, which is mathematically described as follows:

[0036]

[0037] In Equation (8), SiLU(·) is the activation function;

[0038] S24: Establish the lower branch of the bidirectional pulse-driven spiking Mamba hybrid module. The lower branch performs the following operations on $O$ it as follows:

[0039]

[0040] S25: Perform a product operation on the output of the upper branch and the output of the lower branch while establishing a residual connection, which is mathematically described as follows:

[0041]

[0042] In Equation (10), $z$ it is the output of the bidirectional pulse-driven spiking Mamba hybrid module.

[0043] Preferably, in step 3, the specific method is as follows:

[0044] S31: The total recursive time period of the cross-layer recursive spiking neural network model is $M$. For any input image, at the $m$-th time period, after encoding operation, it is expressed as $o$ m1 , and assume that the output of the image encoder in the previous time period is expressed as $e$ m-1 . Fuse $o$ m1 and $e$ m-1 and input them into the image encoder at the $m$-th time period;

[0045] S32: The image encoder for each time period includes four stages. The basic element of each stage is the bidirectional pulse-driven pulse mamba hybrid module constructed in step 2. The numbers of bidirectional pulse-driven pulse mamba hybrid modules in each stage are represented as N1, N2, N3, and N4 respectively.

[0046] Preferably, in step 4, the specific method is as follows:

[0047] S41: The repeater mainly includes two core components, namely fusion and conversion. In the m-th time period, the fusion component uses the information e h from the highest-level classification stage as the query input and the key input, and uses the output e m of the fourth-stage bidirectional pulse-driven pulse mamba hybrid module in this time period as the value input, and performs fusion using the pulse cross-attention mechanism. The mathematical description is as follows:

[0048]

[0049] In formula (11), F w is the weight of the linear layer constructed by pulse neurons, and e f is the output of the fusion component; d1 represents the feature dimension of e h .

[0050] S42: The conversion component uses the fused signal e f as the query input and the key input, and uses the image input in this time period as the value input, and performs fusion using the pulse cross-attention mechanism. The mathematical description is as follows:

[0051]

[0052] In formula (12), F w is the weight of the linear layer constructed by pulse neurons, o m2 is the output of the fusion component, and d2 represents the feature dimension of e f .

[0053] Preferably, in step 5, the specific method is as follows:

[0054] S51: Connect the encoder constructed in step 3 and the repeater constructed in step 4 together to form a cross-level recursive pulse neural network model;

[0055] S52: Randomly select a batch of N images. Any one image is represented as x n , and its corresponding label value is represented as p n . Input it into the cross-level recursive pulse neural network model to obtain the output g n . For g n and p nCalculate the cross-entropy loss function and use the alternative gradient descent method to complete the training of the cross-level recursive spiking neural network model.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] 1. The present invention can build a model for pyramidal neurons.

[0058] 2. The present invention can establish a two-way process of up and down cross-level connections for the model structure, generate more complex and advanced feature representations, and the network captures richer information. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 is a flowchart of the present invention;

[0060] Figure 2 is a flowchart of the bidirectional pulse-driven spiking mamba hybrid module in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0061] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings, so that those skilled in the art can better understand the advantages and features of the present invention, and thus make a clearer definition of the protection scope of the present invention. The embodiments described in the present invention are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts belong to the protection scope of the present invention.

[0062] Refer to Figure 1-2 , an image classification method based on a cross-level recursive spiking neural network, and the specific steps are as follows:

[0063] Step 1: Construct a bidirectional spiking neuron: Integrate the forward input of the bidirectional spiking neuron and the reverse input from the bidirectional spiking neuron in the high-level neuron layer using a soft gating mechanism.

[0064] Step 2: Construct a bidirectional pulse-driven spiking mamba hybrid module: Based on the bidirectional spiking neuron, construct a bidirectional pulse-driven spiking mamba hybrid module.

[0065] Step 3: Construct an encoder: Use the bidirectional pulse-driven spiking mamba hybrid module to construct a multi-time segment encoder.

[0066] Step 4: Construct a relay: Use two core components of fusion and conversion to fuse and convert various types of high-level information.

[0067] Step 5: Construct and train a cross - layer recurrent spiking neural network model: Combine the encoder and the repeater into a cross - layer recurrent spiking neural network, and use the surrogate gradient descent method to complete the training of the cross - layer recurrent spiking neural network model.

[0068] Specifically, in step 1, the specific method is as follows:

[0069] S11: At time t, the forward input of the bidirectional spiking neuron from the j - th spiking neuron in the upper layer is represented as The backward inputs from the bidirectional spiking neurons in the high - level neuron layer are respectively represented as Use the soft gating mechanism to perform the following integration operation on the forward input and the backward input:

[0070]

[0071] In formula (1), Sig(·) represents the Sigmoid function, W1 and W2 respectively represent the weight coefficients of the forward input signal and the backward input signal, and o jt represents the integrated input signal;

[0072] S12: Assume that the membrane potential of the i - th bidirectional spiking neuron at time t - 1 is represented as u i(t-1) , then the instantaneous membrane potential of this neuron before firing is represented as follows:

[0073]

[0074] In formula (2), τ represents the integration time constant, and the output of the i - th bidirectional spiking neuron at time t is represented as:

[0075]

[0076] In formula (3), H(·) is the Heaviside function, and υ θ represents the threshold;

[0077] S13: The membrane potential of the i - th bidirectional spiking neuron at time t is reset to:

[0078] u it = m it (1 - o it ) + ν rest o it (4)

[0079] In formula (4), v rest represents the reset time constant.

[0080] Specifically, in step 2, the specific method is as follows:

[0081] S21: For the output O of the bidirectional spiking neuronit , first, perform normalization on it. The normalization operation is denoted as LN(·), and its processing formula is:

[0082]

[0083] In Equation (5), represents the output of the j-th neuron in the normalization layer;

[0084] S22: Use spiking neurons to construct a linear layer. Then, the output of the i-th neuron is expressed as:

[0085]

[0086] In Equation (6), τ represents the integration time constant, H(·) is the Heaviside function, and υ θ represents the threshold;

[0087] S23: Use the random scanning method to construct the coefficient matrices A, B, and C, and establish the upper branch of the bidirectional pulse-driven spiking mamba hybrid module. The upper branch calculates the state transition equation for the output O it :

[0088]

[0089] For the output y of the state transition equation it perform activation processing, and the mathematical description is as follows:

[0090]

[0091] In Equation (8), SiLU(·) is the activation function;

[0092] S24: Establish the lower branch of the bidirectional pulse-driven spiking mamba hybrid module. The lower branch performs the following processing on O it :

[0093]

[0094] S25: Perform a product operation on the output of the upper branch and the output of the lower branch , and at the same time establish a residual connection. The mathematical description is as follows:

[0095]

[0096] In Equation (10), z it is the output of the bidirectional pulse-driven spiking mamba hybrid module.

[0097] Specifically, in Step 3, the specific method is as follows:

[0098] S31: The total recursive time period of the cross - hierarchical recursive spiking neural network model is M. For any input image, at the m - th time period, after encoding operation, it is represented as o m1 , and assume that the output of the image encoder in the previous time period is represented as e m-1 . Merge o m1 with e m-1 and input the merged result into the image encoder at the m - th time period;

[0099] S32: The image encoder in each time period includes four stages. The basic element of each stage is the bidirectional pulse - driven spiking mamba hybrid module constructed in step 2. The numbers of bidirectional pulse - driven spiking mamba hybrid modules in each stage are represented as N1, N2, N3, and N4 respectively.

[0100] Specifically, in step 4, the specific method is as follows:

[0101] S41: The repeater mainly includes two core components: fusion and transformation. At the m - th time period, the fusion component takes the information e h from the highest - level classification stage as the query input and key input, takes the output e m of the bidirectional pulse - driven spiking mamba hybrid module in the fourth stage of this time period as the value input, and uses the pulse cross - attention mechanism for fusion. The mathematical description is as follows:

[0102]

[0103] In formula (11), F w is the weight of the linear layer constructed by spiking neurons, e f is the output of the fusion component, and d1 represents the feature dimension of e h .

[0104] S42: The transformation component takes the fused signal e f as the query input and key input, takes the image input of this time period as the value input, and uses the pulse cross - attention mechanism for fusion. The mathematical description is as follows:

[0105]

[0106] In formula (12), F w is the weight of the linear layer constructed by spiking neurons, o m2 is the output of the fusion component, and d2 represents the feature dimension of e f .

[0107] Specifically, in step 5, the specific method is as follows:

[0108] S51: Connect the encoder constructed in step 3 and the relay constructed in step 4 together to form a cross - hierarchical recurrent spiking neural network model;

[0109] S52: Randomly extract a batch of N images. Any one image is represented as x n , and its corresponding label value is represented as p n . Input it into the cross - hierarchical recurrent spiking neural network model, and the output is g n . For g n and p n , calculate the cross - entropy loss function, and use the surrogate gradient descent method to complete the training of the cross - hierarchical recurrent spiking neural network model.

[0110] Example:

[0111] Step 1: At time t, the forward input of the bidirectional spiking neuron from the j - th spiking neuron in the upper layer is represented as The reverse inputs of the bidirectional spiking neurons from the high - level neuron layer are respectively represented as Use the soft gating mechanism to integrate the forward input and the reverse input to obtain Assume that the membrane potential of the i - th bidirectional spiking neuron at time t - 1 is represented as u i(t-1) . Use linear operation to obtain the instantaneous membrane potential m of the neuron before firing it . Use the Heaviside function to obtain the output o of the i - th bidirectional spiking neuron at time t it . At the same time, reset the membrane potential to u it .

[0112] Step 2: For the output O of the bidirectional spiking neuron it , first perform normalization processing on it to obtain Use spiking neurons to construct a linear layer, use the random scanning method to construct coefficient matrices A, B, C, and establish the upper branch of the bidirectional spiking - driven spiking mamba hybrid module. The upper branch calculates the state - transition equation for the output O it to obtain y it and perform activation processing on it to obtain Establish the lower branch of the bidirectional spiking - driven spiking mamba hybrid module. The lower branch performs activation processing on O it . Perform a product operation on the output of the upper branch and the output of the lower branch , and establish a residual connection at the same time.

[0113] Step 3: The total recurrent time period of the cross - hierarchical recurrent spiking neural network model is M. For any input image, in the m - th time period, after encoding operation, it is represented as o m1and assume that the output of the image encoder in the previous time period is represented as e m-1 . Merge o m1 with e m-1 and input it into the image encoder of the m-th time period. The image encoder for each time period includes four stages, and the number of bidirectional pulse-driven pulse mamba hybrid modules in each stage is represented as N1, N2, N3, and N4 respectively.

[0114] Step 4: The repeater mainly includes two core components: fusion and transformation. In the m-th time period, the fusion component uses the information e h of the highest-level classification stage as the query input and the key input, uses the output e m of the fourth-stage bidirectional pulse-driven pulse mamba hybrid module in this time period as the value input, and performs fusion using the pulse cross-attention mechanism. The transformation component uses the fused signal e f as the query input and the key input, uses the image input in this time period as the value input, and performs fusion using the pulse cross-attention mechanism.

[0115] Step 5: Connect the encoder and the repeater together to form a cross-level recursive pulse neural network model. Randomly select a batch of N images, and any one image is represented as x n , and its corresponding label value is represented as p n . Input it into the cross-level recursive pulse neural network model to obtain an output of g n . Calculate the cross-entropy loss function for g n and p n , and use the alternative gradient descent method to complete the training of the cross-level recursive pulse neural network model.

[0116] In summary, the present invention can model pyramidal neurons, establish an up-and-down bidirectional process with cross-level connections, generate more complex and advanced feature representations, so that the network can capture richer information.

[0117] The descriptions and practices disclosed in the present invention are easy to think about and understand for ordinary technical personnel in the technical field. Without departing from the principles of the present invention, several improvements and refinements can also be made. Therefore, modifications or improvements made without departing from the spirit of the present invention should also be regarded as within the protection scope of the present invention.

Claims

1. An image classification method based on a cross - level recursive spiking neural network, characterized in that, The specific steps are as follows: Step 1: Construct a bi-pulse neuron: Integrate the forward input of the bi-pulse neuron and the reverse input from the bi-pulse neurons in the higher neuron layer using a soft gating mechanism for the forward and reverse inputs; Step 2: Construct a bi-pulse-driven pulse mamba hybrid module: Based on the bi-pulse neuron, construct a bi-pulse-driven pulse mamba hybrid module; Step 3: Construct an encoder: Use the bi-pulse-driven pulse mamba hybrid module to construct a multi-time period encoder; Step 4: Construct a relay: Use two core components, fusion and transformation, to fuse and transform multiple types of high-level information; Step 5: Construct and train a cross-level recurrent pulse neural network model: Combine the encoder and the relay into a cross-level recurrent pulse neural network, and use the alternative gradient descent method to complete the training of the cross-level recurrent pulse neural network model.

2. The image classification method based on a cross-level recursive spiking neural network according to claim 1, wherein In Step 1, the specific method is as follows: S11: At time t, the forward input of the two-way spiking neuron from the j-th spiking neuron in the upper layer is represented as The backward inputs of the two-way spiking neurons from the upper neuron layer are respectively represented as The forward input and the backward input are integrated as follows by using a soft gating mechanism: In Equation (1), Sig(·) represents the Sigmoid function, W1 and W2 represent the weight coefficients of the forward input signal and the reverse input signal respectively, and o jt represents the integrated input signal; S12: Assume that the membrane potential of the $i$-th two-way pulse neuron at time $t - 1$ is denoted as $u$. i(t-1) , then the instantaneous membrane potential of this neuron before firing is expressed as follows: In Equation (2), τ represents the integration time constant, and the output of the i-th bi-pulse neuron at time t is expressed as: In Equation (3), H(·) is the Heaviside function, and υ θ represents the threshold value; S13: The membrane potential of the i-th bi-pulse neuron at time t is reset to: u it = m it (1 - o it ) + ν rest o it (4) In Equation (4), v rest represents the reset time constant.

3. A method for image classification based on a cross - level recursive spiking neural network according to claim 1, wherein In Step 2, the specific method is as follows: S21: For the output O of the two-way pulse neuron it , first perform normalization on it. The normalization operation is denoted as LN(·), and its processing formula is: In formula (5), represents the output of the j-th neuron in the i-th normalization layer; S22: Use the pulse neuron to construct a linear layer, and the output of the i-th neuron is expressed as: In Equation (6), τ represents the integration time constant, H(·) is the Heaviside function, and υ θ represents the threshold value; S23: Construct coefficient matrices A, B, and C using a random scanning method, establish the upper branch of the bidirectional pulse-driven pulse Mamba hybrid module, and the upper branch outputs O it Calculate the state transition equation: The output y of the state transition equation it is activated, and the mathematical description is as follows: In Equation (8), SiLU(·) is the activation function; S24: Establish the lower branch of the bidirectional pulse-driven pulse Mamba hybrid module, and the lower branch performs the following processing on O it as follows: S25: Output of the upper branch and the output of the lower branch are multiplied, and at the same time, a residual connection is established. The mathematical description is as follows: In formula (10), z it is the output of the bidirectional pulse-driven pulse Mamba hybrid module.

4. A method for image classification based on a cross-level recursive pulse neural network according to claim 1, characterized in that In Step 3, the specific method is as follows: S31: The total recursive time period of the cross - level recursive pulse neural network model is M. For any input image, in the m - th time period, after encoding operation, it is represented as o m1 , and assume that the output of the image encoder in the previous time period is represented as e m-1 , fuse o m1 and e m-1 and input them into the image encoder in the m - th time period; S32: The image encoder for each time period includes four stages. The basic element of each stage is the bi-pulse-driven pulse mamba hybrid module constructed in Step 2. The numbers of bi-pulse-driven pulse mamba hybrid modules in each stage are represented as N1, N2, N3, and N4 respectively.

5. A method for image classification based on a cross - level recursive spiking neural network according to claim 1, characterized in that, In Step 4, the specific method is as follows: S41: The repeater includes two core components: fusion and conversion. In the m-th time period, the fusion component uses the information e of the highest-level classification stage h as the query input and the key input, and uses the output e of the fourth-stage bidirectional pulse-driven pulse mamba hybrid module in this time period m as the value input, and performs fusion using the pulse cross-attention mechanism. The mathematical description is as follows: In formula (11), F w is the weight of the linear layer constructed by pulsed neurons, e f is the output of the fusion component, and d1 represents the h feature dimension of e; S42: The conversion component uses the fused signal e f as the query input and the key input, uses the image input in this time period as the value input, and performs fusion using the pulse cross-attention mechanism. The mathematical description is as follows: In formula (12), F w is the weight of the linear layer constructed by pulse neurons, o m2 is the output of the fusion component, and d2 represents e f 's feature dimension.

6. The image classification method based on a cross - level recursive spiking neural network according to claim 1, wherein, In Step 5, the specific method is as follows: S51: Connect the encoder constructed in Step 3 and the relay constructed in Step 4 together to form a cross-level recurrent pulse neural network model; S52: Randomly select a batch of N images, and any one image is represented as x n , and its corresponding label value is represented as p n , input it into the cross-level recursive spiking neural network model, and the output is g n , for g n and p n Calculate the cross-entropy loss function, and use the alternative gradient descent method to complete the training of the cross-level recursive spiking neural network model.

Images