Multi-label picture classification method based on spiking neural network

Abstract

The invention discloses a multi-label picture classification method based on a spiking neural network, and the method comprises the steps: firstly designing a gradient return mode RateBP which better fits the average distribution characteristics of spiking neurons, and is used for reducing the video memory occupation of the spiking neural network during training; the method comprises the following steps of: designing a high-performance Neck network comprising two parts, namely an Encoder and a Decoder, by using a pulse neuron based on Rate BP; based on the designed Neck network, in combination with a high-performance Backbone network, establishing a two-stage pulse neural network for multi-label picture classification; designing a high-performance loss function to train a multi-label picture classification model; and finally, a high-performance pulse neural network model for multi-label picture classification, which is suitable for the field of multi-label picture classification, is obtained. In addition, the method is the first exploration of the spiking neural network in the field of multi-label picture classification, the application field of the spiking neural network is expanded, and then picture multi-label detection is achieved.

Description

Technical Field

[0001] This invention relates to the application of spiking neural networks in the field of multi-label image classification, and specifically to a multi-label image classification method based on spiking neural networks. Background Technology

[0002] The concept of spiking neural networks (SNNs) can be traced back to neuroscience research in the 1980s and 1990s. During this period, neuroscientists began to focus on more precise and efficient communication mechanisms between biological neurons, namely, transmitting information in the form of pulses. This method of information transmission laid the theoretical foundation for the birth of spiking neural networks. In 1997, Wolfgang Maass first discussed in detail the neural network model based on spiking neurons in his paper, calling it the third-generation neural network model, pioneering work for subsequent research on spiking neural networks. Compared to the previous two generations of neural networks, spiking neural networks place greater emphasis on bridging the gap between neuroscience and machine learning. The neurons in their networks are closer to real biological neurons, giving spiking neural networks advantages such as information transmission methods closer to biological neurons, rich dynamic characteristics, and high computational efficiency. Therefore, spiking neural networks are more suitable for modern applications requiring low power consumption, high real-time performance, and large-scale data processing. This type of neural network, which is more biologically interpretable, has lower computational costs, and is easier to implement in hardware, will be the future direction of neural network development.

[0003] Despite the numerous advantages of spiking neural networks (SNNs), current SNN techniques still have shortcomings. First, current training methods for gradient updates utilize backpropagation time (BPTT), similar to RNN gradient updates. This method relies on membrane potential states, leading to high memory consumption and computational efficiency for multi-step neuron training. Second, SNNs are currently primarily applied to simple tasks, with limited exploration of complex, real-world tasks such as multi-label image classification. Furthermore, previous multi-label image classification methods require additional manual design to model label correlations and locate regions of interest, lacking simple and effective methods. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies. This invention proposes a multi-label image classification method based on spiking neural networks. By designing a gradient backpropagation method, RateBP, that better reflects the average firing characteristics of spiking neurons, the memory usage during spiking neural network training is reduced, thus improving computational efficiency. Simultaneously, this invention also constructs a Neck network comprising an encoder and a decoder. Its structure is simple, its performance is excellent, and it requires no additional manual design. By combining a high-performance backbone network and a loss function, a state-of-the-art (SOTA) multi-label image classification spiking neural network model is achieved, thereby enhancing the application value of spiking neural networks to a certain extent.

[0005] This invention is achieved through the following technical solution:

[0006] A multi-label image classification method based on spiking neural networks includes the following steps:

[0007] (1) Design a gradient backpropagation method for spiking neurons, RateBP, in which LIF neurons are used to reduce the memory usage of gradients during training of spiking neural networks.

[0008] (2) Based on the RateBP method, a Neck network consisting of an encoder and a decoder is designed to detect the labels contained in the image. The encoder is used to capture fine-grained features of the image, and the decoder is used to associate the image labels and features.

[0009] (3) Based on the designed Neck network and combined with the Backbone network, a full pulsar neural network model for multi-label image classification is built;

[0010] (4) Design a loss function to train the built multi-label image classification all-pulse neural network model;

[0011] (5) Obtain a dataset in the field of multi-label image classification, and classify the dataset to obtain a training set, a validation set and a test set. The training set is used to train a spiking neural network; the validation set is used to select the optimal spiking neural network; finally, the test set is input into the optimal spiking neural network to output the detection labels contained in the image.

[0012] Further, step (1) specifically involves: using a multi-step spiking neuron with RateBP gradient backpropagation, whose forward propagation process is consistent with that of a traditional multi-step spiking neuron; and introducing two intermediate variables r and r during the backward gradient backpropagation process. This is used to calculate the derivative during gradient backpropagation in a multi-step spiking neuron, where r refers to the average firing rate of the spiking neuron. This refers to the average input of the spiking neuron; both variables use the average value at time step T because the spiking neural network uses the average value at time step T when calculating the loss during training; and the gradient backpropagation is not related to the state of the multi-step neuron when calculating the derivative, thus avoiding the memory occupation of O(T) caused by the BPTT time backpropagation method used by the traditional spiking neuron during gradient backpropagation, and reducing its space complexity to O(1).

[0013] Specifically, the RateBP gradient backpropagation method is used to avoid high memory usage during the training of spiking neural networks, and its calculation formula is as follows:

[0014]

[0015] in Let represent the loss function; x(t) represent the input of the spiking neuron at time t; s(t) represent the pulse output of the spiking neuron at time t; and r is the average firing rate of the spiking neuron. It refers to the average input of the spiking neuron; the above formula does not include the spiking neuron membrane potential state u(t), but is only related to the average input and average firing rate of the spiking neuron. Therefore, the memory space complexity during gradient backpropagation is reduced to O(1).

[0016] In the above formula, A gradient proxy method is needed for computation, specifically: by characterizing static... The curve was calculated, and an effective surrogate function was chosen to compute the gradient instead; the surrogate function and surrogate gradient expressions are as follows:

[0017]

[0018] r refers to the average firing rate of the spiking neuron; This refers to the average input of a spiking neuron; u threshold This refers to the membrane potential threshold of a spiking neuron.

[0019] Further, the Neck network in step (2) comprises an encoder and a decoder. Specifically, the encoder receives image features extracted by the Backbone network, enhances the portion of the image features containing detection labels through multi-layer encoder blocks, and its output is used in the Spurious Cross-Attention (SCA) module of the decoder module to associate image features with detection labels. Each encoder block contains a Spurious Self-Attention (SSA) module and a Spurious Multilayer Perceptron (SMLP) module. The decoder is also composed of several stacked decoder blocks, each containing a Spurious Self-Attention (SSA) module, a Spurious Cross-Attention (SCA) module, and a Spurious Multilayer Perceptron (SMLP) module. The input of the decoder is divided into two parts: one part is a learnable label embedding vector used to model the detection labels, which is introduced into the decoder through the Spurious Self-Attention (SSA) module; the other part is the fine-grained image features enhanced by the encoder, which are introduced into the decoder through the SCA Spurious Cross-Attention (SCA) module. The decoder associates image features with detection labels by receiving these two inputs, and then outputs the probability of each detection label appearing in the image through the Sigmoid function.

[0020] Specifically, the encoder is used to capture fine-grained features of an image. The encoder is composed of N stacked encoder blocks, and each block has the following formula:

[0021] Q,K = SN r (BN(X(W q W k ,)))

[0022]

[0023] X ′ =SSA(Q,K)+X

[0024] X o =SMLP(X ′ )+X ′

[0025] Where X is the input to the encoder block, specifically the image features extracted from the Backbone network output; W q W k W SSA , It is a learnable linear matrix; BN is the Batch Normalization operation; SN rIt uses RateBP gradient backpropagation for spiking neurons; SSA is a designed spiking self-attention operation, which, unlike traditional self-attention mechanisms, only uses Q-attention queries and K-attention keys; D is the number of channels for the Q-attention query in dimension i; ⊙ is the Hadamard product operation; SMLP is a multilayer perceptron network composed of spiking neurons; X ′ It is the intermediate output of the encoder block; X o It is the output of the encoder block.

[0026] Specifically, the decoder is used to associate image labels and features. The decoder is composed of N stacked decoder blocks, and the formula for each block is as follows:

[0027]

[0028] X ′ =SSA(Q s ,K s ,V s )+X

[0029]

[0030] X″=SCA(Q c ,K c ,V c )+X ′

[0031]

[0032] X o =SMLP(X″)+X″

[0033] embeddings label These are learnable parameters used to model the labels for detection; features image These are the image features extracted by the encoder; It is a learnable linear matrix; PE is positional encoding; BN is batch normalization; SN r These are spiking neurons using RateBP gradient backpropagation; SSA is a designed spiking self-attention operation, SCA is a designed spiking cross-attention operation, and s is a scaling factor; SMLP is a multilayer perceptron network composed of spiking neurons; X ′ X″ is the intermediate output of the decoder block; X o It is the output of the decoder block.

[0034] Further, step (3) specifically involves the following: the Backbone network uses the QKFormer spiking neural network model. QKFormer is the best performing network in the field of spiking neural networks. It uses a hierarchical feature extraction method to capture multi-scale spatiotemporal features. Furthermore, QKFormer utilizes the spatiotemporal spiking features to design the QK Attention method, which is suitable for spiking neurons, to achieve efficient information interaction and feature extraction.

[0035] Specifically, in step (4), a loss function is designed to train the constructed multi-label image classification all-pulse neural network model. The specific formula of the loss function is as follows:

[0036] p k =Sigmod(W k X o +b k )

[0037]

[0038] Where k is the category; p k It represents the probability that the image output by the model contains that class; y k It determines whether the labeled images in the dataset contain that class of labels; W k and b k These are learnable parameters; It is the loss function; α1 and α2 are two hyperparameters; X o It is the output of the decoder.

[0039] Specifically, the dataset for the multi-label image classification domain in step (5) is any one of MS-COCO, PASCAL VOC, NUS-WIDE, and Visual Genome.

[0040] The beneficial effects of the present invention are as follows:

[0041] 1. A novel gradient backpropagation method, RateBP, is proposed for spiking neurons. Compared with the traditional gradient backpropagation method BPTT, RateBP does not depend on the state of the neuron during gradient backpropagation calculations. This avoids the O(T) memory complexity of traditional BPTT gradient backpropagation, reducing its space complexity to O(1). This method effectively reduces the memory usage of spiking neurons during training and improves computational efficiency.

[0042] 2. Based on the RateBP gradient backpropagation method, a Neck network containing an encoder and a decoder was designed. Compared with previous multi-label image classification methods, this avoids the complex design required to obtain label relevance and locate regions of interest. The detection labels are modeled using learnable label embeddings, and an attention mechanism is combined to implement a simple yet high-performance Neck network. This network associates detection labels with image features, thereby achieving multi-label image detection.

[0043] 3. Using the high-performance QKFormer spiking neural network as the backbone network, combined with a designed Neck network containing encoders and decoders, a fully spiking neural network model for multi-label image classification was constructed, achieving competitive results. This model is an energy-efficient and high-performance spiking neural network model suitable for multi-label image classification. Furthermore, this is the first application of spiking neural networks in the field of multi-label image classification, and its proposal has significantly enhanced the application value of spiking neural networks. Attached Figure Description

[0044] Figure 1 This is the overall flowchart of the present invention;

[0045] Figure 2 This is a schematic diagram of the RateBP gradient backpropagation of a spiking neuron;

[0046] Figure 3 It is a static LIF spiking neuron Schematic diagram of the curve;

[0047] Figure 4 This is a schematic diagram of the spiking neural network structure for multi-label image classification constructed in this invention;

[0048] Figure 5 This is a visualization of the attention of a spiking neural network model for multi-label image classification with a specific label. Detailed Implementation

[0049] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and specific examples. The following drawings are for illustrative purposes only and are not intended to limit the scope of the invention.

[0050] Example 1

[0051] Please see Figures 1 to 5 , Figure 1 This is a flowchart of a multi-label image classification algorithm based on a spiking neural network, provided by an example of the present invention. It includes the following steps:

[0052] Step 1: Design a gradient backpropagation method, RateBP, that better matches the average firing characteristics of spiking neurons;

[0053] The multi-step spiking neuron employing RateBP gradient backpropagation follows the same forward propagation process as the traditional multi-step spiking neuron. During backward gradient backpropagation in training, two intermediate variables, r and , are introduced. This is used to calculate the derivative during gradient backpropagation in a multi-step spiking neuron, where r refers to the average firing rate of the spiking neuron. This refers to the average input of a spiking neuron, the specific process of which is as follows: Figure 2 As shown. Both variables use the average value at time step T because the spiking neural network uses the average value at time step T when calculating the loss during training. In this way, when calculating the derivative during gradient backpropagation, it is no longer related to the state of the multi-step neurons, avoiding the O(T) memory occupation caused by the BPTT time backpropagation method used by traditional spiking neurons during gradient backpropagation, and reducing its space complexity to O(1).

[0054] Traditional spiking neurons use the BPPT method for gradient backpropagation, and its calculation formula is as follows:

[0055]

[0056] in Let represent the loss function; x(t) represent the input of the spiking neuron at time t; s(t) represent the pulse output of the spiking neuron at time t; and u(t) represent the membrane potential of the spiking neuron at time t. Since the spiking neuron uses a step function during the feedforward process, ... It cannot be calculated directly, so gradient surrogate is usually used during training to solve this problem.

[0057] According to the above formula, it can be seen that u(t) is involved in the calculation of gradient backpropagation of spiking neurons. Therefore, the training process depends on the membrane potential state u(t), and its memory space complexity is O(T). When the time step T is large, it will lead to a very high memory usage problem.

[0058] RateBP gradient backpropagation primarily addresses the issue of high memory usage during training of spiking neural networks, while simultaneously improving training efficiency. Its calculation formula is as follows:

[0059]

[0060] in Let represent the loss function; x(t) represent the input of the spiking neuron at time t; s(t) represent the pulse output of the spiking neuron at time t; and r is the average firing rate of the spiking neuron. This refers to the average input of the spiking neuron. The above formula does not include the spiking neuron's membrane potential state u(t), but only relates to the average input and average firing rate of the spiking neuron. Therefore, the memory complexity during gradient backpropagation is reduced to O(1).

[0061] In the above formula, It cannot be calculated directly and requires gradient proxy. This is based on the neuron membrane potential threshold u. threshold Taking a LIF neuron with a membrane potential time constant of 1 and a membrane potential time constant of 0.5 as an example, the static state is characterized... Curves, such as Figure 3 As shown, we have chosen an effective surrogate function to compute the alternative gradient. The surrogate function and the surrogate gradient are as follows:

[0062]

[0063]

[0064] r refers to the average firing rate of the spiking neuron; This refers to the average input of a spiking neuron; u threshold This refers to the membrane potential threshold of a spiking neuron.

[0065] The spiking neuron used in this embodiment is a LIF neuron. The membrane potential threshold of the LIF neuron is u. threshold Except for neurons that act on the output of the attention mechanism, which are set to 0.5, all others are set to 1.0; the time constant tau of the membrane potential of LIF neurons is 0.5; and the time step T of LIF neurons is 4.

[0066] Step 2: Based on RateBP spiking neurons, design a high-performance Neck network containing an encoder and a decoder;

[0067] The Neck network consists of an encoder and a decoder. The encoder receives image features extracted from the backbone network and enhances the portion of the image features containing the detection labels through multiple encoder blocks. Its output is used in the SCA pulse cross-attention module of the decoder module to associate the image features with the detection labels. Each encoder block mainly contains an SSA pulse self-attention module and an SMLP pulse MLP multilayer perceptron module. The decoder is also composed of multiple stacked decoder blocks, each mainly containing an SSA pulse self-attention module, an SCA pulse cross-attention module, and an SMLP pulse multilayer perceptron module. The decoder's input is divided into two parts: a learnable label embedding vector, used to model the detection labels, which is introduced into the decoder through the SSA pulse self-attention module; and fine-grained image features enhanced by the encoder, which are introduced into the decoder through the SCA pulse cross-attention module. The decoder, by accepting these two inputs, associates the image features with the detection labels, and then, through the sigmoid function, finally outputs the probability of each detection label appearing in the image.

[0068] The encoder is used to capture fine-grained features of an image. The encoder consists of N stacked encoder blocks, each with the following formula:

[0069] Q,K = SN r (BN(X(W q W k ,)))

[0070]

[0071] X ′ =SSA(Q,K)+X

[0072] X o =SMLP(X ′ )+X ′

[0073] Where X is the input to the encoder block, specifically the image features extracted from the Backbone network output; W q W k W SSA , It is a learnable linear matrix; BN is batch normalization; SN rIt uses RateBP gradient backpropagation for spiking neurons; SSA is a designed spiking self-attention operation, which, unlike traditional self-attention mechanisms, only uses the attention query Q and the attention key K; D is the number of channels of the Q attention query in dimension i; ⊙ is the Hadamard product operation; SMLP is an MLP multilayer perceptron network composed of spiking neurons; X ′ It is the intermediate output of the encoder block; X o It is the output of the encoder block.

[0074] The decoder is used to associate image labels and features. The decoder is composed of N stacked decoder blocks, and the formula for each block is as follows:

[0075]

[0076] X ′ =SSA(Q s ,K s ,V s )+X

[0077] Q c =SN r (BN(PE(XW q c )))

[0078]

[0079] X″=SCA(Q c ,K c ,V c )+X ′

[0080]

[0081] X o =SMLP(X″)+X″

[0082] embeddings label These are learnable parameters used to model the labels for detection; features image These are the image features extracted by the Encoder; It is a learnable linear matrix; PE is positional encoding; BN is batch normalization; SN r It uses spiking neurons with RateBP gradient backpropagation; SSA is a designed spiking self-attention operation, SCA is a designed spiking cross-attention operation, and s is a scaling factor; SMLP is an MLP multilayer perceptron network composed of spiking neurons; X ′ X″ is the intermediate output of the decoder block; Xo It is the output of the decoder block.

[0083] The SN used in this embodiment r It is a LIF neuron that uses RateBP gradient backpropagation; the scaling factor s is 0.125.

[0084] Step 3: Based on the designed Neck network, combined with the high-performance Backbone network, build a full spiking neural network for multi-label image classification;

[0085] The backbone network uses the QKFormer spiking neural network model. Therefore, combined with the aforementioned Neck network, the designed neural network model is a full-spiking neural network model for multi-label image classification. QKFormer is currently the highest-performing spiking neural network, achieving state-of-the-art (SOTA) results on multiple publicly available datasets. QKFormer employs a hierarchical feature extraction approach, capable of capturing multi-scale spatiotemporal features. Furthermore, unlike traditional self-attention mechanisms, QKFormer utilizes spatiotemporal spiking features to design Q-KAttention, an attention computation method suitable for spiking neurons, achieving efficient information exchange and feature extraction.

[0086] The QKFormer model used in this embodiment contains 10 encoder blocks with a channel dimension of 384.

[0087] Step 4: Design a high-performance loss function to train the built multi-label image classification model;

[0088] To train an optimal multi-label image classification model, a high-performance loss function needs to be designed, the specific formula of which is as follows:

[0089] p k =Sigmod(W k X o +b k )

[0090]

[0091] Where k is the category; p k It represents the probability that the image output by the model contains that class; y k It determines whether the labeled images in the dataset contain that class of labels; W k and b k These are learnable parameters; It is the loss function; α1 and α2 are two hyperparameters; X o It is the output of the decoder.

[0092] In this embodiment, the values ​​of the two parameters α1 and α2 are 0 and 1, respectively.

[0093] Step 5: Obtain a dataset for multi-label image classification, and divide the dataset into a training set, a validation set, and a test set. The training set is used to train a spiking neural network; the validation set is used to select the optimal spiking neural network; the test set is input into the optimal spiking neural network, and the output is the detection labels contained in the image.

[0094] A multi-label image classification spiking neural network model was trained. Training was performed on the MSCOCO 2014 dataset, which contains 82,783 images for training, 40,503 images for validation, and 40,775 images for testing. The provided image detection labels include 80 categories. This invention can also be trained using commonly used datasets in the field of multi-label image classification, such as PASCAL VOC, NUS-WIDE, and Visual Genome.

[0095] The final result of this invention is that, given an input image, it can obtain multiple detection labels contained within that image, as shown in the following figure. Figure 4 As shown in the diagram. A visualization of the attention mechanism in a spiking neural network model for multi-label image classification, for a given label, is shown below. Figure 5 As shown. Given an original image, for different attention queries, the model can output the degree of interest in different regions of the original image. Higher relevance to the query results in brighter areas, and lower relevance results in darker areas, reflecting the model's accuracy to some extent. (See attached image.) Figure 5 Taking the first row as an example, given the original image, when the attention query is "person," the trained model can highlight the person portion of the original image; when the attention query is "motorcycle," the trained model can highlight the motorcycle portion of the original image (the same applies to the airplane in the second row and the cup in the third row). This fully verifies the accuracy of the trained model.

[0096] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-label image classification method based on a spiking neural network, characterized in that, Includes the following steps: (1) Design a gradient backpropagation method for spiking neurons, RateBP, in which LIF neurons are used to reduce the memory usage of gradients during training of spiking neural networks. (2) Based on the RateBP method, a Neck network consisting of an encoder and a decoder is designed to detect the labels contained in the image. The encoder is used to capture fine-grained features of the image, and the decoder is used to associate the image labels and features. (3) Based on the designed Neck network and combined with the Backbone network, a full pulsar neural network model for multi-label image classification is built; (4) Design a loss function to train the built multi-label image classification all-pulse neural network model; (5) Obtain a dataset in the field of multi-label image classification, and classify the dataset to obtain a training set, a validation set and a test set. The training set is used to train a spiking neural network; the validation set is used to select the optimal spiking neural network; finally, the test set is input into the optimal spiking neural network to output the detection labels contained in the image.

2. The multi-label image classification method based on a spiking neural network according to claim 1, characterized in that, Step (1) specifically involves: using a multi-step spiking neuron with RateBP gradient backpropagation. Its forward propagation process is consistent with that of a traditional multi-step spiking neuron. During training, two intermediate variables, r and r, are introduced for backward gradient backpropagation. This is used to calculate the derivative during gradient backpropagation in a multi-step spiking neuron, where r refers to the average firing rate of the spiking neuron. This refers to the average input of a spiking neuron; The reason why both variables use the average value at time step T is that the spiking neural network uses the average value at time step T when calculating the loss during training; and the gradient backpropagation is not related to the state of multi-step neurons when calculating the derivative, thus avoiding the memory occupation of O(T) caused by the BPTT time backpropagation method used by traditional spiking neurons during gradient backpropagation, and reducing its space complexity to O(1).

3. The multi-label image classification method based on a spiking neural network according to claim 2, characterized in that, The RateBP gradient backpropagation method is used to avoid high memory usage during the training of spiking neural networks. Its calculation formula is as follows: in Let x(t) represent the loss function; x(t) represent the input of the spiking neuron at time t; and s(t) represent the pulse output of the spiking neuron at time t. r refers to the average firing rate of the spiking neuron; It refers to the average input of the spiking neuron; the above formula does not include the spiking neuron membrane potential state u(t), but is only related to the average input and average firing rate of the spiking neuron. Therefore, the memory space complexity during gradient backpropagation is reduced to O(1). In the above formula, A gradient proxy method is needed for computation, specifically: by characterizing static... The curve was calculated, and an effective surrogate function was chosen to compute the gradient instead; the surrogate function and surrogate gradient expressions are as follows: r refers to the average firing rate of the spiking neuron; This refers to the average input of a spiking neuron; u threshold This refers to the membrane potential threshold of a spiking neuron.

4. The multi-label image classification method based on a spiking neural network according to claim 1, characterized in that, The Neck network in step (2) consists of two parts: an encoder and a decoder. Specifically, the encoder receives image features extracted by the Backbone network and enhances the part of the image features containing detection labels through multi-layer encoder blocks. Its output is used in the Spurious Cross Attention (SCA) module in the decoder module to associate the image features and the detection labels. Each encoder block contains a Spurious Self Attention (SSA) module and a Spurious Multilayer Perceptron (SMLP) module. The decoder is also composed of several stacked decoder blocks. Each decoder block contains a pulsed self-attention module (SSA), a pulsed cross-attention module (SCA), and a pulsed multilayer perceptron module (SMLP). The input to the decoder is divided into two parts: one part is a learnable label embedding vector, used to model the detected labels, which is introduced into the decoder through the pulsed self-attention module (SSA); the other part is the fine-grained image features enhanced by the encoder, which are introduced into the decoder through the SCA pulsed cross-attention module. The decoder, by accepting these two inputs, associates the image features with the detected labels, and then, through the sigmoid function, finally outputs the probability of each detected label appearing in the image.

5. The multi-label image classification method based on a spiking neural network according to claim 4, characterized in that, The encoder is used to capture fine-grained features of the image. The encoder is composed of N stacked encoder blocks, and the formula for each block is as follows: Q,K=SN r (BN(X(W q ,W k ,))) X ′ =SSA(Q,K)+X X o =SMLP(X ′ )+X ′ Where X is the input to the encoder block, specifically the image features extracted from the Backbone network output; W q W k W SSA , It is a learnable linear matrix; BN is the Batch Normalization operation; SN r It is a spiking neuron that uses RateBP gradient backpropagation; SSA is a designed spiking self-attention operation. Unlike traditional self-attention mechanisms, it only uses Q-attention queries and K-attention keys; D is the number of channels of the Q-attention query in dimension i; ⊙ is the Hadamard product operation; SMLP is a multilayer perceptron network composed of spiking neurons; X ′ It is the intermediate output of the encoder block; X o It is the output of the encoder block.

6. The multi-label image classification method based on a spiking neural network according to claim 4, characterized in that, The decoder is used to associate image labels and features. The decoder is composed of N stacked decoder blocks, and the formula for each block is as follows: X ′ =SSA(Q s ,K s ,V s )+X X″=SCA(Q c ,K c ,V c )+X ′ X o =SMLP(X″)+X″ embeddings label These are learnable parameters used to model the labels for detection; features image These are the image features extracted by the encoder; It is a learnable linear matrix; PE is the positional encoding operation; BN is the batch normalization operation; SN r It is a spiking neuron that uses RateBP gradient backpropagation; SSA is a designed spiking self-attention operation, SCA is a designed spiking cross-attention operation, and s is a scaling factor; SMLP is a multilayer perceptron network composed of spiking neurons; X ′ X″ is the intermediate output of the decoder block; X o It is the output of the decoder block.

7. The multi-label image classification method based on a spiking neural network according to claim 1, characterized in that, The specific steps (3) are as follows: The Backbone network uses the QKFormer spiking neural network model. QKFormer is the best performing network in the field of spiking neural networks. It uses a hierarchical feature extraction method to capture multi-scale spatiotemporal features. Furthermore, QKFormer uses the spatiotemporal spiking features to design the QK Attention method, which is suitable for spiking neurons, to achieve efficient information interaction and feature extraction.

8. The multi-label image classification method based on a spiking neural network according to claim 1, characterized in that, In step (4), a loss function is designed to train the constructed multi-label image classification all-pulse neural network model. The specific formula of the loss function is as follows: p k =Sigmod(W k X o +b k ) Where k is the category; p k It represents the probability that the image output by the model contains that class; y k It determines whether the labeled images in the dataset contain that class of labels; W k and b k These are learnable parameters; It is the loss function; α1 and α2 are two hyperparameters; X o It is the output of the decoder.

9. The multi-label image classification method based on a spiking neural network according to claim 1, characterized in that, The dataset for the multi-label image classification domain in step (5) can be any one of MS-COCO, PASCAL VOC, NUS-WIDE, or VisualGenome.

Images