An image classification method and device based on time reversible pulse transformer
By employing a multi-scale computational architecture and self-attention module of a time-reversible pulse Transformer neural network, the problems of single-scale output and high training memory in existing models are solved, achieving efficient image classification and feature extraction, which is suitable for low-power computing chips.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2024-05-29
- Publication Date
- 2026-06-09
AI Technical Summary
Existing Transformer spiking neural network models can only output features at a single scale, making them unsuitable for downstream vision tasks. Furthermore, their high training memory requirements limit their application over long time steps.
A time-reversible pulse Transformer neural network with a multi-scale computational architecture is used to extract multi-scale image features through reversible pulse feature mapping and self-attention modules, and saves intermediate variable memory during training, thus reducing training memory requirements.
It achieves high accuracy and low training memory consumption in image classification tasks, is suitable for low-power computing chips, and improves the training efficiency and classification accuracy of the model.
Smart Images
Figure CN118506094B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of neural network and image classification technology, specifically relating to an image classification method and apparatus based on a time-reversible pulse Transformer. Background Technology
[0002] Spiking Neural Networks (SNNs) are neural network models inspired by biological neurons, designed to simulate the pulse transmission mechanism between them. Unlike traditional artificial neural networks, SNNs use discrete pulse signals to encode and transmit information. This event-driven processing approach makes SNNs excellent at handling dynamic information and time-series data, while maintaining high computational efficiency. Therefore, SNNs are suitable for low-power neuromorphic computing chips, such as edge computing scenarios like drones and mobile robots.
[0003] With the widespread application of Transformers in deep learning, Transformers based on spiking neural networks have also received considerable attention. These structures utilize spiking neurons to convert query, key, and value matrices into pulse signals, ensuring the spiking nature of matrix dot product operations. However, these SNN Transformers also have some limitations. First, they can only output features at a single scale, making them unsuitable for downstream tasks such as object detection and semantic segmentation. Second, the complexity of the attention mechanism and the extended time dimension of spiking neurons result in these models requiring extremely high training memory, severely limiting their model capacity and making it impossible to directly train SNN Transformers with long step sizes. Summary of the Invention
[0004] This invention addresses the shortcomings of existing technologies by providing an image classification method and apparatus based on a time-reversible pulse Transformer. The proposed time-reversible pulse Transformer neural network employs a multi-scale computational architecture, capable of outputting image features at four different resolutions for downstream visual tasks, or directly using the features from the last layer for image classification. The main network of the reversible pulse Transformer adopts a time-reversible structure at the macroscopic level, allowing pulse features at adjacent time steps to be mutually calculated, thus saving memory consumption of intermediate variables and reducing training memory. Compared to non-reversible pulse Transformers with similar structures, this invention significantly reduces training memory. Compared to other directly trained pulse Transformer models, it significantly improves accuracy in image classification tasks.
[0005] This invention is achieved through the following technical solution: an image classification method based on a time-reversible pulse Transformer, the method comprising the following steps:
[0006] (1) Perform pulse coding on the images that need to be classified;
[0007] (2) The high-dimensional features of the pulse encoded image are extracted by the reversible pulse Transformer main network composed of multiple computation stages connected in series. The pulse features of adjacent time steps can be calculated from each other. Each computation stage includes a pulse feature mapper and several pulse self-attention modules.
[0008] The pulse feature mapper upsamples the pulse features of the previous time step in the next calculation stage and performs feature transformation, and then downsamples the pulse features of the current time step in the previous calculation stage after performing feature transformation. The sum of the two results is used as the final output, and the pulse features are extracted through several pulse self-attention modules.
[0009] (3) Decode the high-dimensional features of the image pulses into their respective categories and corresponding probabilities to obtain the image classification results.
[0010] Furthermore, the reversible pulse Transformer main network adopts a time-reversible structure at the macroscopic level, comprising four computational stages; assuming Let be the transformation function for the i-th computation stage at time step t. Given its output features, the forward propagation formula for the main network is as follows:
[0011]
[0012] During forward propagation, the characteristics of time t are obtained using the characteristics of time t-1; during backward propagation, the characteristics of time t-1 are obtained using the characteristics of time t, as shown in the following formula:
[0013]
[0014]
[0015] Furthermore, the pulse feature mapper consists of upsampling and downsampling modules, spiking neurons, convolutional neural network layers, and batch normalization layers, and includes two branches, specifically:
[0016] (1) The left branch upsamples the pulse features of the previous time step in the next calculation stage using nearest neighbor interpolation, and then performs feature transformation through the spiking neuron, convolutional neural network layer and batch normalization layer respectively;
[0017] (2) The right branch first uses spiking neurons, convolutional neural network layers and batch normalization layers to transform the input features of the current time step in the previous calculation stage, and then uses max pooling layers to downsample the features;
[0018] (3) Finally, the sum of the left and right branches is used as the final output.
[0019] Furthermore, the spiking self-attention module comprises spiking neurons (SN), convolutional neural network layers (Conv), and batch normalization layers (BN), and the operation process is as follows:
[0020] Q = SN Q (BN Q (Conv Q (SN Q (X)))) (9)
[0021] K=SN K (BN K (Conv K (SN K (X)))) (10)
[0022] V=SN V (BN V (Conv V (SN V (X)))) (11)
[0023] SSA(X)=Conv(BN(QK T V·s)) (12)
[0024] Where X represents the pulse feature input to the pulse self-attention module; Q, K, and V represent the Query, Key, and Value of the self-attention mechanism; s represents the scale mapping value; and SN represents the scale mapping value. Q SN K and SN V For the spiking neurons corresponding to Q, K, and V, BN Q BN K BN V For the batch normalization layers corresponding to Q, K, and V, Conv Q Conv K Conv V These are the convolutional neural network layers corresponding to Q, K, and V.
[0025] Furthermore, during the training of the reversible pulse Transformer main network, some intermediate variables in the forward propagation process are removed to save training memory; in the backward propagation, the variables are recalculated, the gradients of each parameter are derived and calculated, and then the parameters are updated.
[0026] Furthermore, depending on the number of channels and modules in the pulse self-attention module, the reversible pulse Transformer main network capacity has three configurations: Small, Base, and Large, where:
[0027] (1) The number of channels and modules of Small are (56, 112, 224, 384) and (1, 2, 4, 2) respectively;
[0028] (2) The number of channels and modules of Base are (80, 160, 240, 384) and (1, 2, 3, 2) respectively;
[0029] (3) The number of channels and modules of Large are (120,240,384,512) and (1,2,6,2), respectively.
[0030] Furthermore, in step (3), the pulse high-dimensional features of the image are decoded into their respective categories and corresponding probabilities based on the feature decoder, which consists of a batch normalization layer and a fully connected layer.
[0031] Secondly, the present invention also provides an image classification device based on a time-reversible pulse Transformer, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the image classification method based on a time-reversible pulse Transformer.
[0032] Thirdly, the present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the aforementioned image classification method based on a time-reversible pulse Transformer.
[0033] Fourthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned image classification method based on a time-reversible pulse Transformer.
[0034] Compared to existing technologies, this invention has the following advantages: It presents a directly trainable reversible pulse Transformer with very low floating-point operation count and energy consumption, enabling it to run on neuromorphic computing chips; it proposes a multi-scale computational architecture for the pulse Transformer, capable of outputting image features at different resolutions; and it proposes a time-reversible pulse structure, allowing pulse features at adjacent time steps to be mutually calculated, thereby saving memory consumption of intermediate variables and reducing training memory. This invention can significantly improve the accuracy of image classification and reduce the model's training memory. Attached Figure Description
[0035] Figure 1 A schematic diagram of the time-reversible pulse Transformer computation structure provided in an embodiment of the present invention;
[0036] Figure 2This is a schematic diagram of the pulse feature mapper network structure provided in an embodiment of the present invention;
[0037] Figure 3 This is a schematic diagram illustrating the change of training memory over time steps, as provided in an embodiment of the present invention.
[0038] Figure 4 This is a schematic diagram of an image classification device based on a time-reversible pulse Transformer, provided as an embodiment of the present invention. Detailed Implementation
[0039] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
[0040] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.
[0041] This invention proposes an image classification method based on a time-reversible pulse Transformer, which inputs an RGB image into a pulse Transformer neural network based on a time-reversible architecture to obtain the image category;
[0042] The time-reversible pulse Transformer neural network includes a pulse encoder, a reversible pulse Transformer main network, and a feature decoder.
[0043] The pulse encoder consists of a convolutional neural network layer and a batch normalization layer. After performing an initial transformation on the input image, it expands the image along the time dimension to achieve pulse coding.
[0044] The reversible pulse Transformer main network is responsible for extracting high-dimensional pulse features from the image. It adopts a multi-scale computational architecture, consisting of four computational stages connected in series. Each computational stage includes a pulse feature mapper and several pulse self-attention modules.
[0045] The pulse Transformer main network adopts a time-reversible structure at the macroscopic level, such as... Figure 1 As shown. Assume Let be the transformation function for the i-th computation stage at time step t, where t∈(1,T), and T is the total number of time steps. Let the output feature of the i-th computation stage be denoted as , then the forward propagation formula of the pulse body network is as follows:
[0046]
[0047] Where α1 to α4 are the learnable mapping parameters for the corresponding computation stages. This is the output of the pulse encoder.
[0048] During forward propagation, the characteristics of time t are obtained using the characteristics of time t-1. Since this propagation is reversible, it can be reversed to obtain the characteristics of time t-1 using the characteristics of time t. The propagation formula is as follows:
[0049]
[0050] The pulse feature mapper consists of upsampling / downsampling modules, spiking neurons, convolutional neural network layers, and batch normalization layers, such as... Figure 2 As shown, it contains two branches, left and right, specifically:
[0051] (1) Left branch pairs with input features Upsampling is performed using nearest neighbor interpolation, followed by feature transformation through spiking neurons, convolutional neural network layers, and batch normalization layers.
[0052] (2) Right branch pairs with input features First, the input features are transformed using spiking neurons, convolutional neural network layers, and batch normalization layers, and then feature downsampling is performed using max pooling layers.
[0053] (3) The pulse feature mapper ultimately uses the sum of the left and right branches as the final output.
[0054] The spiking self-attention module (SSA) consists of spiking neurons (SN), convolutional neural network layers (Conv), and batch normalization layers (BN). Its main computational process is shown in the following formula:
[0055] Q = SN Q (BN Q (Conv Q (SN Q (X)))) (9)
[0056] K=SN K (BN K (Conv K (SN K (X)))) (10)
[0057] V=SN V (BN V (Conv V (SN V (X)))) (11)
[0058] SSA(X)=Conv(BN(QKT V·s)) (12)
[0059] Where X represents the pulse feature input to the pulse self-attention module; Q, K, and V represent the Query, Key, and Value of the self-attention mechanism; s is the scale mapping value, set to a constant of 0.125; and SN... Q SN K and SN V For the spiking neurons corresponding to Q, K, and V, BN Q BN K BN V For the batch normalization layers corresponding to Q, K, and V, Conv Q Conv K Conv V These are the convolutional neural network layers corresponding to Q, K, and V.
[0060] The time-reversible spiking Transformer network, during network training, manually removes some intermediate variables (including all intermediate convolutional neural network layers, batch normalization layers, and the input and output of spiking neurons) from the forward propagation process to save training memory; during backpropagation, the variables are recalculated using formulas (5)-(8), the gradients of each parameter are derived and calculated, and then the parameters are updated. Assume... The output of the i-th computation stage at time step t, i.e. Let represent the gradient. The gradient for each computational stage is calculated using the following formula.
[0061]
[0062]
[0063] Subsequently, the result can be obtained through an automatic differentiation mechanism. The gradient of the learnable parameters is used to update the network parameters.
[0064] Depending on the number of channels and modules in the pulse self-attention module, the reversible pulse Transformer main network capacity has three configurations: Small, Base, and Large.
[0065] (1) The number of channels and modules of Small are (56,112,224,384) and (1,2,4,2) respectively.
[0066] (2) The number of channels and modules of Base are (80, 160, 240, 384) and (1, 2, 3, 2), respectively.
[0067] (3) The number of channels and modules of Large are (120,240,384,512) and (1,2,6,2), respectively.
[0068] The feature decoder decodes the high-dimensional features of an image into its category and corresponding probability, and consists of a batch normalization layer and a fully connected layer.
[0069] The pulse encoder and pulse Transformer main network structure of the time-reversible pulse Transformer can be used as a general feature extractor for feature extraction in downstream vision tasks.
[0070] To demonstrate the advancements of the proposed method, a comparative experiment was first conducted on the CIFAR100 dataset, comparing the proposed time-reversible pulse Transformer structure with the Spikingformer, an approximate parametric spiking neural network. Then, the training memory usage over time was compared between the proposed method using the time-reversible structure and the method without it. Simultaneously, the training memory of the Spikingformer was also used as a reference. Figure 3 As shown.
[0071] CIFAR100 and CIFAR10 are commonly used datasets for image classification tasks, containing a total of 60,000 RGB images of size 32×32. These images cover a variety of categories, including animals, fruits, people, vehicles, flowers, etc. The CIFAR100 dataset contains 100 categories, while the CIFAR10 dataset contains 10 categories, which can be used to validate the image classification capabilities of algorithms.
[0072] Table 1
[0073]
[0074]
[0075] Table 1 compares the parameter count, FLOPS, and accuracy of the Base model with the proposed time-reversible pulse Transformer structure and the Spikingformer model with an approximate parameter count spiking neural network. It can be seen that the method of this invention significantly improves the image classification accuracy of the CIFAR10 and CIFAR100 datasets with similar parameter counts.
[0076] Corresponding to the aforementioned embodiment of an image classification method based on a time-reversible pulse Transformer, the present invention also provides an embodiment of an image classification device based on a time-reversible pulse Transformer.
[0077] See Figure 4The present invention provides an image classification device based on a time-reversible pulse Transformer, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it implements an image classification method based on a time-reversible pulse Transformer as described in the above embodiment.
[0078] The embodiment of the image classification device based on time-reversible pulse Transformer provided by this invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 4 The diagram shown is a hardware structure diagram of any device with data processing capabilities, including the image classification device based on a time-reversible pulse Transformer provided by the present invention. (Except for...) Figure 4 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.
[0079] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.
[0080] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0081] This invention also provides a computer-readable storage medium storing a program that, when executed by a processor, implements an image classification method based on a time-reversible pulse Transformer as described in the above embodiments.
[0082] The computer-readable storage medium can be an internal storage unit of any data processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.
[0083] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned image classification method based on a time-reversible pulse Transformer.
[0084] The above description is merely a preferred embodiment of the present invention. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make many possible variations and modifications to the technical solutions of the present invention using the methods and techniques disclosed above, or modify them into equivalent embodiments with equivalent changes, without departing from the scope of the technical solutions of the present invention. Therefore, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall still fall within the protection scope of the technical solutions of the present invention.
Claims
1. An image classification method based on a time-reversible pulse Transformer, characterized in that, The method includes the following steps: (1) Perform pulse coding on the images that need to be classified; (2) The high-dimensional features of the pulse in the image after pulse coding are extracted by the reversible pulse Transformer main network composed of multiple computation stages. The pulse features of adjacent time steps can be calculated from each other. Each computation stage includes a pulse feature mapper and several pulse self-attention modules. The pulse feature mapper upsamples the pulse features of the previous time step in the next calculation stage and performs feature transformation, and then downsamples the pulse features of the current time step in the previous calculation stage after performing feature transformation. The sum of the two results is used as the final output, and the pulse features are extracted through several pulse self-attention modules. The reversible pulse Transformer main network adopts a time-reversible structure at the macroscopic level and includes four computational stages; assuming For the first Each computational stage at time step Transformation function, Given its output features, the forward propagation formula for the main network is as follows: During forward propagation, time is utilized. Feature Calculation Time Characteristics; during reverse propagation, time is utilized Feature Calculation Time The characteristics are defined by the following formula: (3) Decode the high-dimensional features of the image pulses into their respective categories and corresponding probabilities to obtain the image classification results.
2. The image classification method based on time-reversible pulse Transformer according to claim 1, characterized in that, The pulse feature mapper consists of upsampling and downsampling modules, spiking neurons, convolutional neural network layers, and batch normalization layers, and includes two branches, specifically: (1) The left branch upsamples the pulse features of the previous time step in the next calculation stage using nearest neighbor interpolation, and then performs feature transformation through the spiking neuron, the convolutional neural network layer and the batch normalization layer respectively; (2) The right branch first uses spiking neurons, convolutional neural network layers and batch normalization layers to transform the input features of the current time step in the previous calculation stage, and then uses max pooling layers to downsample the features; (3) Finally, the sum of the left and right branches is used as the final output.
3. The image classification method based on time-reversible pulse Transformer according to claim 1, characterized in that, The pulse self-attention module includes spiking neurons. Convolutional Neural Network Layer Normalization layer The composition and calculation process are as follows: in, The pulse characteristics input to the pulse self-attention module; For the Query, Key, and Value of a self-attention mechanism; The scale-mapped value, , and for The corresponding spiking neuron, , , for The corresponding batch normalization layer, , , for The corresponding convolutional neural network layer.
4. The image classification method based on a time-reversible pulse Transformer according to claim 1, characterized in that, During the training of the reversible pulse Transformer main network, some intermediate variables in the forward propagation process are removed to save training memory; in the backward propagation, the variables are recalculated, the gradients of each parameter are derived and calculated, and then the parameters are updated.
5. The image classification method based on a time-reversible pulse Transformer according to claim 1, characterized in that, Depending on the number of channels and modules in the pulse self-attention module, the reversible pulse Transformer main network capacity has three configurations: Small, Base, and Large. (1) The number of channels and modules of Small are (56, 112, 224, 384) and (1, 2, 4, 2) respectively; (2) The number of channels and modules of Base are (80, 160, 240, 384) and (1, 2, 3, 2) respectively; (3) The number of channels and modules of Large are (120, 240, 384, 512) and (1, 2, 6, 2) respectively.
6. The image classification method based on a time-reversible pulse Transformer according to claim 1, characterized in that, In step (3), the pulse high-dimensional features of the image are decoded into their respective categories and corresponding probabilities based on the feature decoder, which consists of a batch normalization layer and a fully connected layer.
7. An image classification device based on a time-reversible pulse Transformer, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that... When the processor executes the executable code, it implements an image classification method based on a time-reversible pulse Transformer as described in any one of claims 1-6.
8. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements an image classification method based on a time-reversible pulse Transformer as described in any one of claims 1-6.
9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements an image classification method based on a time-reversible pulse Transformer as described in any one of claims 1-6.