An image classification method and system based on a hybrid attention pulse neural network
By using a hybrid attention spiking neural network, local and global information of images are extracted in a more refined manner, which solves the problem of the lagging performance of the spiking Transformer in image classification and achieves efficient and energy-saving image classification results, making it suitable for resource-constrained devices.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2024-05-30
- Publication Date
- 2026-06-05
Smart Images
Figure CN118587486B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image classification, specifically relating to an image classification method and system based on a hybrid attention spiking neural network. Background Technology
[0002] Compared to artificial neural networks, spiking neural networks achieve high efficiency and energy savings by mimicking the computation and information transmission mechanisms of biological neurons, making them more suitable for deployment and operation on resource-constrained edge devices. The Transformer architecture, due to its powerful ability to acquire contextual information, exhibits excellent performance in artificial neural networks. Currently, breakthroughs have been achieved in image classification tasks using the spiking Transformer architecture by modifying the attention mechanism and its upstream and downstream modules.
[0003] However, current methods lag significantly behind artificial neural networks in performance. This is because current methods ignore the differences between the spiking Transformer and the artificial neural network Transformer. First, under current conditions, there is a lack of large-scale event camera datasets more suitable for training spiking models. Furthermore, because spiking networks need to expand over time steps, training on GPUs requires a significant increase in GPU memory. These dual limitations of datasets and hardware prevent spiking networks from being pre-trained. Second, the sparsity of spiking signals makes information extraction in the attention mechanism of ordinary Transformer structures insufficient. This necessitates that the design of spiking Transformers consider fine-grained feature extraction, including global and local aspects. These differences make it difficult for spiking Transformers to capture inductive bias from insufficient training data, resulting in a significant performance gap between spiking Transformers and artificial neural network Transformers. Summary of the Invention
[0004] To overcome the shortcomings of existing methods and make full use of global and local information in limited data, enabling the pulse Transformer to acquire the most effective features in pulse sparse signals, this invention provides an image classification method and system based on a hybrid attention spiking neural network.
[0005] An image classification method based on a hybrid attention spiking neural network, the method comprising:
[0006] Image encoding: Perform affine transformation and convolutional downsampling on the input image to obtain impulse tokens;
[0007] Hybrid attention encoding involves inputting a pulse token into several transformer blocks. Each transformer block performs layer normalization on the pulse token and calculates attention. The calculation result is connected to the input token to form a residual connection, resulting in a first pulse token output. This output is then normalized again, and local information extraction and global channel information weighting are performed on the resulting output to obtain a second pulse token output. The second pulse token output and the first pulse token output are connected to form a residual connection, and the processed token is input to the downstream transformer block or classification head network.
[0008] The tokens encoded with hybrid attention are classified to obtain classification results for RGB images and event camera images.
[0009] Furthermore, the image encoding involves the input image sequentially entering an affine transformation layer, k convolutional + spiking neurons, and another affine transformation layer to achieve the conversion of pulse signals and image segmentation.
[0010] Furthermore, the calculation process for the first pulse token output includes:
[0011] Input pulse token X∈R N×C First, a composite token is generated, where N represents the token length and C represents the channel dimension. Then, using the number of attention heads n in a self-attention mechanism, the input impulse token is divided into n parts along the channel dimension, resulting in X∈R. N×d×n ,in
[0012] Average the pulse tokens over N dimensions, where X∈R n×d ;
[0013] The impulse token passes through a linear layer and is transformed back into a C-dimensional X∈R. n×C Vectors as channel mixing tokens;
[0014] The channel blending token is pulse-coded by a spiking neuron, then added to the position embedding vector to provide a channel blending token with position information, and concatenated with the original input token in N dimensions to form a composite token;
[0015] The composite token is processed by self-attention calculation. The composite token passes through a linear layer and is segmented. Then, it is pulse-coded by a spiking neuron to obtain three 0-1 vectors, which are the pulse signals Q, K, and V, respectively.
[0016] Q, K, and V are calculated as a matrix according to the self-attention mechanism, i.e., SA = QK T V;
[0017] The token X∈R obtained after self-attention computation (N+n)×C;
[0018] Let X∈R (N+n)×C Tokens are categorized into batch tokens, category tokens, and channel tokens based on their length.
[0019] The channel token is averaged and pooled, added to the category token, and finally concatenated with the batch token to obtain the output of the self-attention submodule of the composite token, whose dimension is consistent with the input.
[0020] Furthermore, the calculation process for the second pulse token output is as follows:
[0021] The first pulse token output is divided into batch tokens and category tokens. Local information is extracted from the batch tokens. The global channel information vector is obtained from the batch tokens after local information extraction. Then, the category tokens are weighted by the global channel information. The batch tokens after local information extraction and the category tokens after global channel information weighting are concatenated to obtain the output token of the channel attention feedforward module.
[0022] Furthermore, the local information extraction includes:
[0023] The batch token extracts local information using two convolutional + batch normalization structures. Each convolutional + batch normalization structure forms a residual connection, which is then pulse-coded by a spiking neuron.
[0024] Furthermore, the weighted global channel information includes:
[0025] The obtained global information vector is multiplied by the category token in the channel attention feedforward submodule to obtain the channel-weighted category token.
[0026] Furthermore, the classification head network consists of fully connected layers.
[0027] According to a second aspect of this specification, an image classification system based on a hybrid attention spiking neural network is also provided, the system comprising: an image encoding module, a hybrid attention calculation module, and a classification module;
[0028] The image encoding module performs affine transformation and convolutional downsampling on the input image through an affine transformation layer and a convolutional + spiking neuron to obtain a spiking token.
[0029] The hybrid attention encoding module consists of several transformer blocks. Each transformer block includes a composite token self-attention submodule and a channel attention feedforward submodule. The transformer block normalizes the pulse tokens and then performs attention calculations via the composite token self-attention submodule. The calculation result forms a residual connection with the input token. Then, it is also normalized by the layer and output by the channel attention feedforward submodule to form a residual connection. The processed token is then input to the downstream transformer block or classification module.
[0030] The classification module is used to classify the tokens encoded by the hybrid attention encoding module to obtain classification results for RGB images and event camera images.
[0031] According to a third aspect of this specification, an image classification apparatus based on a hybrid attention spiking neural network is also provided, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the image classification method based on a hybrid attention spiking neural network.
[0032] According to a fourth aspect of this specification, a computer-readable storage medium is also provided, on which a program is stored, which, when executed by a processor, implements the image classification method based on a hybrid attention spiking neural network.
[0033] Compared with existing methods, this invention has the following advantages: it extracts local and global information and channel attention from image tokens with fine precision; it designs and implements a more efficient self-attention calculation method to obtain image features of sparse pulse signals; the hybrid attention spiking neural network uses pulse signal addition and multiplication as the main calculation method, which has very low floating-point operation count and energy consumption, and can run on neuromorphic computing chips; under the same data settings, it achieves state-of-the-art performance of spiking neural networks on the CIFAR dataset, achieving performance close to that of artificial neural networks with significant energy-saving advantages, and achieves excellent performance on the event camera dataset. Attached Figure Description
[0034] Figure 1 This is a structural diagram of an image classification system based on a hybrid attention spiking neural network, provided as an embodiment of the present invention.
[0035] Figure 2 This is a structural diagram of the composite token self-attention module according to an embodiment of the present invention.
[0036] Figure 3 This is a structural diagram of the channel attention feedforward module according to an embodiment of the present invention.
[0037] Figure 4 This is a schematic diagram of an image classification device based on a hybrid attention spiking neural network provided in an embodiment of the present invention. Detailed Implementation
[0038] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
[0039] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.
[0040] An image classification method based on a hybrid attention spiking neural network, the method comprising:
[0041] Image encoding: Perform affine transformation and convolutional downsampling on the input image to obtain impulse tokens;
[0042] Hybrid attention encoding involves inputting a pulse token into several transformer blocks. Each transformer block performs layer normalization on the pulse token and calculates attention. The calculation result is connected to the input token to form a residual connection, resulting in a first pulse token output. This output is then normalized again, and local information extraction and global channel information weighting are performed on the resulting output to obtain a second pulse token output. The second pulse token output and the first pulse token output are connected to form a residual connection, and the processed token is input to the downstream transformer block or classification head network.
[0043] The tokens encoded with hybrid attention are classified to obtain classification results for RGB images and event camera images.
[0044] Furthermore, the image encoding involves the input image sequentially entering an affine transformation layer, k convolutional + spiking neurons, and another affine transformation layer to achieve the conversion of pulse signals and image segmentation.
[0045] Furthermore, the calculation process for the first pulse token output includes:
[0046] Input pulse token X∈R N×C First, a composite token is generated, where N represents the token length and C represents the channel dimension. Then, using the number of attention heads n in a self-attention mechanism, the input impulse token is divided into n parts along the channel dimension, resulting in X∈R. N×d×n ,in
[0047] Average the pulse tokens over N dimensions, where X∈R n×d ;
[0048] The impulse token passes through a linear layer and is transformed back into a C-dimensional X∈R. n×C Vectors as channel mixing tokens;
[0049] The channel blending token is pulse-coded by a spiking neuron, then added to the position embedding vector to provide a channel blending token with position information, and concatenated with the original input token in N dimensions to form a composite token;
[0050] The composite token is processed by self-attention calculation. The composite token passes through a linear layer and is segmented. Then, it is pulse-coded by a spiking neuron to obtain three 0-1 vectors, which are the pulse signals Q, K, and V, respectively.
[0051] Q, K, and V are calculated as a matrix according to the self-attention mechanism, i.e., SA = QK T V;
[0052] The token X∈R obtained after self-attention computation (N+n)×C ;
[0053] Let X∈R (N+n)×C Tokens are categorized into batch tokens, category tokens, and channel tokens based on their length.
[0054] The channel token is averaged and pooled, added to the category token, and finally concatenated with the batch token to obtain the output of the self-attention submodule of the composite token, whose dimension is consistent with the input.
[0055] Furthermore, the calculation process for the second pulse token output is as follows:
[0056] The first pulse token output is divided into batch tokens and category tokens. Local information is extracted from the batch tokens. The global channel information vector is obtained from the batch tokens after local information extraction. Then, the category tokens are weighted by the global channel information. The batch tokens after local information extraction and the category tokens after global channel information weighting are concatenated to obtain the output token of the channel attention feedforward module.
[0057] Furthermore, the local information extraction includes:
[0058] The batch token extracts local information using two convolutional + batch normalization structures. Each convolutional + batch normalization structure forms a residual connection, which is then pulse-coded by a spiking neuron.
[0059] Furthermore, the weighted global channel information includes:
[0060] The obtained global information vector is multiplied by the category token in the channel attention feedforward submodule to obtain the channel-weighted category token.
[0061] Corresponding to the embodiments of the above methods, this invention proposes an image classification system based on a hybrid attention spiking neural network, such as... Figure 1 As shown, the system includes: an image encoding module, a hybrid attention calculation module, and a classification module;
[0062] The image encoding module is an image encoding network that performs affine transformation and convolutional downsampling on the input image through an affine transformation layer and convolutional + spiking neurons to obtain pulse tokens; the image encoding network plays a dual role in converting pulse signals and dividing the image into blocks.
[0063] The hybrid attention encoding module and the hybrid attention network consist of N transformer blocks (the value of N is determined by the size of the transformer model to be constructed). Each transformer block includes a composite token self-attention submodule and a channel attention feedforward submodule. The processing of the upstream input token in a transformer block is as follows: the impulse token is normalized, and then attention is calculated through the composite token self-attention submodule. The calculation result forms a residual connection with the input token. Then, it is also normalized through the layer and output by the channel attention feedforward submodule to form a residual connection. The processed token is then input to the downstream transformer block or classification module.
[0064] The classification module is used to classify the tokens encoded by the hybrid attention encoding module to obtain classification results for RGB images and event camera images.
[0065] The specific structure of the image coding network is an affine transformation layer, k convolutional + spiking neurons, and an affine transformation layer.
[0066] like Figure 2 As shown, the specific processing procedure of the composite token self-attention submodule is as follows:
[0067] (1) Let the input token X∈R N×C (For clarity, the first two dimensions of the token—time and batch size—are omitted in the following description), where N represents the token length and C represents the channel dimension. The input token is first used to generate a composite token. Using the number of attention heads n in a self-attention mechanism, the input token is divided into n parts along the channel dimension, resulting in X∈R. N×d×n ,in
[0068] (2) Average the tokens over N dimensions to obtain X∈Rd×n In order to integrate the characteristics of the token in the channel dimension, its shape is changed to X∈R n×d .
[0069] (3) The token passes through a linear layer and is transformed back into a C-dimensional X∈R n×C Vector. Through processes (1), (2), and (3), the input tokens interact with the features of different channels to obtain channel-mixed tokens.
[0070] (4) The channel blending token is pulse-coded by a spiking neuron and then added to the position embedding vector to provide a channel blending token with positional information. The channel blending token is concatenated with the original input token in N dimensions to form a composite token.
[0071] (5) The next step is the self-attention calculation of the composite token. The composite token passes through a linear layer and is segmented, and then spiked by spiking neurons to obtain three 0-1 vectors Q, K and V.
[0072] (6) Q, K, and V are calculated as a matrix according to the self-attention mechanism, i.e., SA = QK T V, Q, K, and V are all pulse signals, ensuring the computational efficiency and energy-saving advantages of self-attention computing.
[0073] (7) The token X∈R after self-attention computation (N+n)×C To ensure consistency of token dimensions passed within the transformer block, X∈R (N+n)×C In terms of token length, tokens are divided into batch tokens, category tokens, and channel tokens (corresponding to channel hybrid tokens before self-attention calculation).
[0074] (8) The channel token is averaged and pooled, added to the category token, and finally concatenated with the batch token to obtain the output of the self-attention submodule of the composite token. Its dimension is consistent with the input, which facilitates subsequent processing.
[0075] like Figure 3 As shown, the specific processing procedure of the channel attention feedforward submodule is as follows:
[0076] (1) The input tokens are divided into batch tokens and category tokens, which are processed separately. Batch tokens are processed for local information extraction, while category tokens are processed for global channel information weighting.
[0077] (2) The batch token extracts local information by two convolutional and batch normalization structures. Each convolutional and batch normalization structure forms a residual connection, which is then pulse-coded by a spiking neuron.
[0078] (3) After the local information of the batch tokens is extracted, it passes through a pooling layer, a fully connected layer, a spiking neuron, and another fully connected layer in sequence to obtain a global information vector. This step is called squeezing and excitation to obtain global information channel attention.
[0079] (4) Multiply the global information channel attention obtained in the previous step with the category token in step (1) to obtain the channel-weighted category token.
[0080] (5) Concatenate the batch token processed in step (2) and the category token processed in step (4) to obtain the output token of the channel attention feedforward submodule for downstream module processing.
[0081] The classification module is a classification head network, which consists of fully connected layers. It classifies the tokens encoded by the hybrid attention coding network to obtain the classification results.
[0082] To demonstrate the advancements of this invention, experiments were conducted on the RGB dataset CIFAR and the event camera datasets CIFAR10DVS and DVS128Gesture to verify and compare the classification performance of the algorithm. The task evaluation metric used was the top-1 accuracy.
[0083] Table 1
[0084]
[0085]
[0086] Table 1 shows a comparative experiment of the proposed HAST spiking neural network and similar spiking neural network methods such as Spikingformer on the CIFAR dataset. It can be seen that under the same experimental settings, HAST of this invention achieves the highest initial accuracy, demonstrating the superiority of this invention.
[0087] Table 2
[0088] Network CIFAR10DVS DVS128Gesture SEW ResNet 74.4％ 97.9％ PLIF 74.8％ 97.6％ Spikformer 80.9％ 98.3％ Spikingformer 81.3％ 98.3％ HAST(ours) 81.9％ 98.3％
[0089] Table 2 shows a comparative experiment between the proposed HAST spiking neural network and similar spiking neural network methods such as Spikingformer on the CIFAR10DVS and DVS128Gesture event camera datasets. Under the same experimental settings, HAST achieved state-of-the-art performance on CIFAR10DVS, and its performance on the DVS128Gesture dataset was comparable to other methods. Tables 1 and 2 demonstrate the superiority of this invention on different datasets.
[0090] Corresponding to the aforementioned embodiment of an image classification method based on a hybrid attention spiking neural network, the present invention also provides an embodiment of an image classification device based on a hybrid attention spiking neural network.
[0091] See Figure 4 The present invention provides an image classification device based on a hybrid attention spiking neural network, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it implements an image classification method based on a hybrid attention spiking neural network as described in the above embodiment.
[0092] The image classification device based on a hybrid attention spiking neural network provided by this invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 4 The diagram shown is a hardware structure diagram of any device with data processing capabilities, including the image classification device based on a hybrid attention spiking neural network provided by the present invention. (Except for...) Figure 4 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.
[0093] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.
[0094] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0095] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements an image classification method based on a hybrid attention spiking neural network as described in the above embodiments.
[0096] The computer-readable storage medium can be an internal storage unit of any data processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.
[0097] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the image classification method based on a hybrid attention spiking neural network.
[0098] The above description is merely a preferred embodiment of the present invention. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make many possible variations and modifications to the technical solutions of the present invention using the methods and techniques disclosed above, or modify them into equivalent embodiments with equivalent changes, without departing from the scope of the technical solutions of the present invention. Therefore, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall still fall within the protection scope of the technical solutions of the present invention.
Claims
1. An image classification method based on a hybrid attention spiking neural network, characterized in that, The method includes: Image encoding: Perform affine transformation and convolutional downsampling on the input image to obtain impulse tokens; Hybrid attention encoding involves inputting a pulse token into several transformer blocks. Each transformer block performs layer normalization on the pulse token and calculates attention. The calculation result is connected to the input token to form a residual connection, resulting in a first pulse token output. This output is then normalized again, and local information extraction and global channel information weighting are performed on the resulting output to obtain a second pulse token output. The second pulse token output and the first pulse token output are connected to form a residual connection, and the processed token is input to the downstream transformer block or classification head network. The tokens encoded with hybrid attention are classified to obtain classification results for RGB images and event camera images.
2. The image classification method based on a hybrid attention spiking neural network according to claim 1, characterized in that, The image encoding process involves sequentially passing the input image through an affine transformation layer, k convolutional + spiking neurons, and another affine transformation layer to convert the pulse signal and segment the image into blocks.
3. The image classification method based on a hybrid attention spiking neural network according to claim 1, characterized in that, The calculation process for the first pulse token output includes: Input pulse token X∈R N×C First, a composite token is generated, where N represents the token length and C represents the channel dimension. Then, using the number of attention heads n in a self-attention mechanism, the input impulse token is divided into n parts along the channel dimension, resulting in X∈R. N ×d×n ,in Average the pulse tokens over N dimensions, where X∈R n×d ; The impulse token passes through a linear layer and is transformed back into a C-dimensional X∈R. n×C Vectors as channel mixing tokens; The channel blending token is pulse-coded by a spiking neuron, then added to the position embedding vector to provide a channel blending token with position information, and concatenated with the original input token in N dimensions to form a composite token; The composite token is processed by self-attention calculation. The composite token passes through a linear layer and is segmented. Then, it is pulse-coded by a spiking neuron to obtain three 0-1 vectors, which are the pulse signals Q, K, and V, respectively. Q, K, and V are calculated as a matrix according to the self-attention mechanism, i.e., SA = QK T V; The token X∈R obtained after self-attention computation (N+n)×C ; Let X∈R (N+n)×C In terms of token length, tokens are categorized into batch tokens, category tokens, and channel tokens. The channel token is averaged and pooled, added to the category token, and finally concatenated with the batch token to obtain the output of the self-attention submodule of the composite token, whose dimension is consistent with the input.
4. The image classification method based on a hybrid attention spiking neural network according to claim 1, characterized in that, The calculation process for the second pulse token output is as follows: The first pulse token output is divided into batch tokens and category tokens. Local information is extracted from the batch tokens. The global channel information vector is obtained from the batch tokens after local information extraction. Then, the category tokens are weighted by the global channel information. The batch tokens after local information extraction and the category tokens after global channel information weighting are concatenated to obtain the output token of the channel attention feedforward module.
5. The image classification method based on a hybrid attention spiking neural network according to claim 4, characterized in that, The local information extraction includes: The batch token extracts local information using two convolutional + batch normalization structures. Each convolutional + batch normalization structure forms a residual connection, which is then pulse-coded by a spiking neuron.
6. The image classification method based on a hybrid attention spiking neural network according to claim 4, characterized in that, The weighted global channel information includes: The obtained global information vector is multiplied by the category token in the channel attention feedforward submodule to obtain the channel-weighted category token.
7. The image classification method based on a hybrid attention spiking neural network according to claim 1, characterized in that, The classification head network consists of fully connected layers.
8. An image classification system based on a hybrid attention spiking neural network, characterized in that, The system includes: an image encoding module, a hybrid attention calculation module, and a classification module; The image encoding module performs affine transformation and convolutional downsampling on the input image through an affine transformation layer and a convolutional + spiking neuron to obtain a spiking token. The hybrid attention encoding module consists of several transformer blocks. Each transformer block includes a composite token self-attention submodule and a channel attention feedforward submodule. The transformer block normalizes the pulse tokens and then performs attention calculations via the composite token self-attention submodule. The calculation result forms a residual connection with the input token. Then, it is also normalized by the layer and output by the channel attention feedforward submodule to form a residual connection. The processed token is then input to the downstream transformer block or classification module. The classification module is used to classify the tokens encoded by the hybrid attention encoding module to obtain classification results for RGB images and event camera images.
9. An image classification device based on a hybrid attention spiking neural network, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements an image classification method based on a hybrid attention spiking neural network as described in any one of claims 1-8.
10. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements an image classification method based on a hybrid attention spiking neural network as described in any one of claims 1-8.