Classification method of full-addition pulse self-attention transformer

Abstract

The application provides a full-addition pulse self-attention Transformer classification method, which comprises the following steps: obtaining an image to be classified; performing feature extraction on the image based on a preprocessing module of a classification model to obtain a basic feature map, and performing relative position coding on the basic feature map to obtain a position coding result; performing pulse sequence conversion on the position coding result based on a block mixing module of the classification model to obtain a pulse feature map, and performing pulse-driven self-attention operation on a first pulse matrix of the pulse feature map in the block dimension to obtain a membrane potential feature map; performing pulse-driven self-attention operation on a second pulse matrix of the membrane potential feature map in the channel dimension based on a channel mixing module of the classification model to obtain an output membrane potential; and applying the output membrane potential to perform image classification based on a decoding classification module of the classification model. The method provided by the application can reduce the power consumption of image classification processing while ensuring the accuracy of image classification processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a full-addition pulse self-attention Transformer classification method. BACKGROUND

[0002] Image recognition and image classification are basic research problems in the field of computer vision. With the advent of the big data era, large-scale image classification problems have gradually emerged, and the research on accurate and efficient image classification algorithms and network architectures has become a hot topic for technical personnel.

[0003] Most of the current mainstream image classification models use traditional convolutional neural network architectures. However, due to the need for a large number of multi-layer stacks to achieve an observable receptive field, the improvement of the quality of the convolutional neural network will cause an increase in the operation cost and bring unacceptable power consumption. In recent years, the self-attention model proposed in the field of natural language processing has achieved breakthrough application in the field of computer vision through its global receptive field and high recognition efficiency. However, the self-attention model needs a large amount of data for training, and its operation cost and power consumption are not very optimistic.

[0004] How to realize an accurate, efficient and low-power image classification method is a technical problem that needs to be solved in the field of computer vision and even the field of edge computing. SUMMARY

[0005] The present application provides a full-addition pulse self-attention Transformer classification method to solve the problem of high power consumption and low efficiency in image classification processing in the prior art.

[0006] The present application provides a full-addition pulse self-attention Transformer classification method, comprising:

[0007] An image to be classified is obtained;

[0008] A feature extraction is performed on the image based on a preprocessing module of a classification model to obtain a basic feature map, and a relative position coding is performed on the basic feature map to obtain a position coding result of the image;

[0009] A pulse sequence transformation is performed on the position coding result based on a block mixing module of the classification model to obtain a pulse feature map, a first pulse matrix of the pulse feature map is extracted, and a pulse-driven self-attention operation is performed on the first pulse matrix in the block dimension to obtain a membrane potential feature map of the image;

[0010] a channel mixing module based on the classification model, extracts a second pulse matrix of the membrane potential feature map, and performs pulse-driven self-attention operation on the second pulse matrix in the channel dimension to obtain an output membrane potential of the image;

[0011] a decoding classification module based on the classification model, applies the output membrane potential to perform image classification to obtain an image category of the image.

[0012] According to the classification method of the full-addition pulse self-attention Transformer provided by the application, the first pulse matrix includes a query matrix Q1, a key matrix K1 and a value matrix V1, and the pulse-driven self-attention operation on the first pulse matrix in the block dimension includes:

[0013] performing element-wise mask operation on the key matrix K1 and the value matrix V1 to obtain a first operation result;

[0014] summing the first operation result by column to obtain a first summation result;

[0015] performing pulse sequence conversion on the first summation result to obtain a first pulse sequence;

[0016] performing row-wise mask operation on the query matrix Q1 and the first pulse sequence to obtain a first pulse output, and the first pulse output is used to determine the membrane potential feature map.

[0017] According to the classification method of the full-addition pulse self-attention Transformer provided by the application, the block mixing module based on the classification model, the position encoding result is converted into a pulse feature map, a first pulse matrix of the pulse feature map is extracted, and pulse-driven self-attention operation is performed on the first pulse matrix in the block dimension to obtain a membrane potential feature map of the image, including:

[0018] based on the main branch of the block mixing module, the position encoding result is converted into a pulse feature map, a first pulse matrix of the pulse feature map is extracted, and pulse-driven self-attention operation is applied to mix the features of different block dimensions of the first pulse matrix to obtain the first pulse output, and the first pulse output is applied to determine the first main branch membrane potential;

[0019] based on the residual branch of the block mixing module, the membrane potential of the position encoding result is extracted to obtain a first residual branch membrane potential, and the first main branch membrane potential and the first residual branch membrane potential are fused to determine the membrane potential feature map.

[0020] According to the full-addition pulse self-attention Transformer classification method provided by the application, the second pulse matrix includes a query matrix Q2, a key matrix K2 and a value matrix V2, the pulse-driven self-attention operation is performed on the second pulse matrix in the channel dimension, and the operation includes:

[0021] The matrix and the matrix are subjected to element-by-element mask operation to obtain a second operation result;

[0022] The second operation result is summed by column to obtain a second summation result;

[0023] The second summation result is subjected to pulse sequence conversion to obtain a second pulse sequence;

[0024] The matrix and the second pulse sequence are subjected to row-by-row mask operation to obtain a second pulse output, and the second pulse output is used for determining the output membrane potential after being transposed;

[0025] The matrix matrix and matrix are transpositions of the query matrix Q2, the key matrix K2 and the value matrix V2 respectively.

[0026] According to the full-addition pulse self-attention Transformer classification method provided by the application, the channel mixing module based on the classification model extracts a second pulse matrix of the membrane potential feature map, and performs pulse-driven self-attention operation on the second pulse matrix in the channel dimension to obtain an output membrane potential of the image, and the operation includes:

[0027] Based on the main branch of the channel mixing module, a second pulse matrix of the membrane potential feature map is extracted, and pulse-driven self-attention operation is applied to mix the features of different channel dimensions of the second pulse matrix to obtain a second pulse output, and the second pulse output is applied to determine a second main branch membrane potential;

[0028] Based on the residual branch of the channel mixing module, the membrane potential of the membrane potential feature map is extracted to obtain a second residual branch membrane potential, and the second main branch membrane potential and the second residual branch membrane potential are fused to determine the output membrane potential.

[0029] According to the full-addition pulse self-attention Transformer classification method provided by the application, the preprocessing module based on the classification model extracts features of the image to obtain a basic feature map, and performs relative position coding on the basic feature map to obtain a position coding result of the image, and the operation includes:

[0030] A pulse block decomposition module based on the preprocessing module splits the image into blocks to obtain image blocks, and performs feature extraction on the image blocks to obtain a basic feature map.

[0031] A position encoding module based on the preprocessing module sequentially performs pulse sequence conversion, convolution operation and batch normalization processing on the basic feature map to obtain a position feature, and fuses the basic feature map and the position feature to determine the position encoding result.

[0032] According to the classification method of the full-addition pulse self-attention Transformer provided by the application, there are multiple encoding modules in the classification model, and the encoding module is composed of the block mixing module and the channel mixing unit module in series.

[0033] The multiple encoding modules are sequentially connected in series, and the output membrane potential determined by the previous encoding module is the position encoding result corresponding to the next encoding module.

[0034] The application also provides a classification device of a full-addition pulse self-attention Transformer, comprising:

[0035] An acquisition unit is configured to acquire an image to be classified.

[0036] A processing unit is configured to perform feature extraction on the image based on a preprocessing module of a classification model to obtain a basic feature map, and perform relative position encoding on the basic feature map to obtain a position encoding result of the image.

[0037] A block mixing unit is configured to perform pulse sequence conversion on the position encoding result based on a block mixing module of the classification model to obtain a pulse feature map, extract a first pulse matrix of the pulse feature map, and perform pulse-driven self-attention operation on the first pulse matrix in the block dimension to obtain a membrane potential feature map of the image.

[0038] A channel mixing unit is configured to extract a second pulse matrix of the membrane potential feature map based on a channel mixing module of the classification model, and perform pulse-driven self-attention operation on the second pulse matrix in the channel dimension to obtain an output membrane potential of the image.

[0039] A decoding and classification unit is configured to apply the output membrane potential to perform image classification based on a decoding and classification module of the classification model to obtain an image category of the image.

[0040] The application further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the classification method of the full-addition pulse self-attention Transformer according to any one of the above when executing the program.

[0041] The application further provides a non-transitory computer-readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the classification method of the full-addition pulse self-attention Transformer according to any one of the above.

[0042] The application further provides a computer program product, which includes a computer program, wherein the computer program is executed by a processor to implement the classification method of the full-addition pulse self-attention Transformer according to any one of the above.

[0043] The application provides a classification method of a full-addition pulse self-attention Transformer, which extracts features and encodes relative positions of an image to be classified to obtain a position encoding result, and sequentially operates the position encoding result through a block mixing module and a channel mixing module with a pulse-driven self-attention mechanism, so as to achieve the purpose of using full-addition pulse-driven self-attention operation to complete feature fusion, fully utilize the operation characteristics of a pulse neural network, greatly reduce the model operation cost and running energy consumption, and maintain good performance of the model, so as to realize the image classification with reduced power consumption and maintained accuracy, and facilitate the edge deployment of a neuromorphic chip. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the application or prior art, the following will briefly introduce the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0045] Figure 1 is a flowchart of the classification method of the full-addition pulse self-attention Transformer provided by the application;

[0046] Figure 2 is one of the flowcharts of the pulse-driven self-attention operation provided by the application;

[0047] Figure 3 is a flowchart of step 130 in the classification method of the full-addition pulse self-attention Transformer provided by the application;

[0048] Figure 4Figure 2 is a flowchart of the second process of the pulse-driven self-attention operation provided by the present application;

[0049] Figure 5 Figure 4 is a flowchart of step 140 in the classification method of the full-addition pulse self-attention Transformer provided by the present application;

[0050] Figure 6 Figure 5 is a flowchart of step 120 in the classification method of the full-addition pulse self-attention Transformer provided by the present application;

[0051] Figure 7 Figure 6 is a structural diagram of the classification model provided by the present application;

[0052] Figure 8 Figure 7 is a structural diagram of the classification device of the full-addition pulse self-attention Transformer provided by the present application;

[0053] Figure 9 Figure 8 is a structural diagram of the electronic device provided by the present application. DETAILED DESCRIPTION

[0054] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0055] In the field of image recognition and classification, most of the models currently used are convolutional neural networks and self-attention models based on traditional neural networks. However, these models face a trade-off between power consumption and accuracy in practical applications. High accuracy often has unacceptable power consumption, while models with relatively low power consumption often do not have high enough accuracy.

[0056] Due to the increasing depth and complexity of convolutional neural networks, a large amount of computing cost and storage demand is required, resulting in high power consumption of convolutional neural networks in processing image problems, which hinders the development of convolutional neural networks in practical applications. Although many studies have improved the optimization algorithm of the network to improve the parameter optimization ability, it is still unable to reduce the network processing power consumption without losing a large amount of accuracy. How to ensure the accuracy of image classification while reducing the processing power consumption and improving the processing efficiency of image classification is a problem to be solved in the field of image classification.

[0057] To this end, the embodiment of the present application provides a classification method of full-addition pulse self-attention Transformer, which can further reduce the energy consumption of the model running and maintain good performance by implementing all feature mixing units in the classification model in a full-addition pulse driven self-attention manner, thereby overcoming the above problems.

[0058] Figure 1 is a flowchart of the classification method of full-addition pulse self-attention Transformer provided by the present application, as Figure 1 shown, the method comprises:

[0059] Step 110, obtaining an image to be classified;

[0060] Specifically, the classification method provided by the embodiment of the present application is applicable to an electronic device, which can be a server, a desktop computer or other terminal device, such as a mobile phone, a tablet computer, etc., and can also be a cloud or a remote server. The embodiment of the present application does not limit the specific form of the electronic device.

[0061] The electronic device obtains an image to be classified, which can be reading an image to be classified on its own storage device, or receiving an image to be classified sent by other devices. Here, the image to be classified can be an image or a video. When the input is an image, a single image can be directly obtained as the image to be classified. The above input image can include a face image, a human body image, an animal image, and other object images. The embodiment of the present application does not make specific limitations. When the input is a video, the electronic device can sample at least one video frame from the video stream according to a preset interval, such as an interval of 10 frames, and collect one frame. At least one video frame is used as an image to be classified for subsequent processing. It can be understood that the classification process of each video frame of the video is the same as that of a single image.

[0062] Step 120, based on the preprocessing module of the classification model, performing feature extraction on the image to obtain a basic feature map, and performing relative position coding on the basic feature map to obtain a position coding result of the image;

[0063] Specifically, the classification model is a neural network model obtained by applying a spiking neural network (SNN) to a Transformer model, which can be used for classification processing of the image. After obtaining the image to be classified, the image is input into the classification model to obtain a predicted classification probability map output by the classification model, and based on the predicted classification probability map, the image class of the image can be determined.

[0064] The classification model comprises a preprocessing module configured to perform feature extraction and relative position embedding (RPE) on an image to be classified. In this regard, the feature extraction on the image can be performed by using an existing image feature extraction method, such as edge feature extraction, local binary pattern feature extraction, SIFT (Scale-invariant feature transform) feature extraction, HOG (Histogram of Oriented Gradient) feature extraction, or the like. Alternatively, the image feature extraction method in the Vision Transformer (ViT) model can be used, in which the image is first divided into a plurality of image blocks, and then the feature extraction is performed on the plurality of image blocks respectively, so as to obtain a basic feature map.

[0065] After the basic feature map is extracted, the relative position embedding can be performed on the basic feature map to obtain a position feature, and then the basic feature map and the position feature are added, so as to obtain a position encoding result of the image. In this regard, the relative position embedding is a method for encoding position information, in which the position information is converted into a fixed vector, and then the vector is added to an original feature vector, so that each position in the feature vector contains distance or relative position information with other features, and the relationship between different features can be more accurately represented, so as to improve the representation ability of the classification model and reduce the complexity of the classification model.

[0066] In step 130, the position encoding result is subjected to pulse sequence conversion by a block mixing module of the classification model, so as to obtain a pulse feature map, a first pulse matrix of the pulse feature map is extracted, and the first pulse matrix is subjected to pulse-driven self-attention operation in the block dimension, so as to obtain a membrane potential feature map of the image.

[0067] Specifically, the classification model comprises a block mixing module connected with the preprocessing module. The block mixing module can comprise a pulse neuron layer. After the position encoding result is obtained, the position encoding result can be input to the block mixing module, the pulse sequence conversion of the position encoding result is performed by the pulse neuron layer, so as to obtain the pulse feature map. The subsequent calculation process conforms to the 01 calculation characteristics of the SNN, so as to improve the calculation speed and reduce the calculation energy consumption.

[0068] Here, the pulse neuron layer can be a LIF (Leakage Integrate-and-fire) neuron layer, and the pulse feature map is a feature map used to represent global pulse information of an image, and the pulse feature refers to pixels in the image that have mutation properties, such as boundaries, corners, etc. When converting the position encoding result into a pulse sequence, the pulse response at the pixel point in the position encoding result can be calculated first, and then the pulse response value is mapped to the position encoding result at the corresponding position, so that the pulse feature map is obtained. The data in the pulse feature map is in the form of pulses and only contains 0 and 1, so it meets the 01 calculation characteristics of the SNN. Unlike traditional deep learning models that use continuous decimal values to pass information, the SNN uses discrete pulse sequences to calculate and transmit information, so the above pulse sequence conversion refers to the process of the pulse neuron receiving continuous values and converting them into pulse sequences.

[0069] After obtaining the pulse feature map, the block mixing module can extract a first pulse matrix of the pulse feature map, and fuse the features of different block dimensions of the first pulse matrix through pulse-driven self-attention operation, so as to obtain a membrane potential feature map. Here, the membrane potential feature map is a representation form used to describe local features of an image, which uses the concept of pulses to describe image features.

[0070] Step 140, the channel mixing module based on the classification model extracts a second pulse matrix of the membrane potential feature map, and performs pulse-driven self-attention operation on the second pulse matrix in the channel dimension to obtain the output membrane potential of the image.

[0071] Specifically, the classification model further includes a channel mixing module connected to the block mixing processing module. After obtaining the membrane potential feature map, the membrane potential feature map can be input to the channel mixing module, a second pulse matrix of the membrane potential feature map is extracted, and the features of different channel dimensions of the second pulse matrix are mixed through pulse-driven self-attention operation, so as to obtain the output membrane potential of the image. Here, the membrane potential refers to the signal transmission state of information exchange and calculation between neurons, which is analogous to the membrane potential of biological neurons; the output membrane potential refers to the final output membrane potential value of the membrane potential feature map after the channel mixing module performs operation processing.

[0072] In the embodiment of the application, by performing pulse-driven self-attention operation on the pulse matrix in the block dimension and the channel dimension respectively, more information can be extracted by the model, so that the model can maintain good performance while reducing the energy consumption of the model.

[0073] Step 150, the decoding classification module based on the classification model applies the output membrane potential to classify the image to obtain the image category of the image.

[0074] Specifically, the classification model further comprises a decoding classification module configured to apply the output membrane potential to image classification to obtain an image category. Here, the classification module can be a fully connected layer, a classifier configured to classify features using a support vector machine, or a classifier configured to use a full convolution and a recurrent connection structure, and the embodiments of the present application do not make specific limitations thereon. For example, when the classification module is configured to classify using a fully connected layer, each neuron in the classification module receives the output membrane potential of all neurons in the previous layer of the encoding module, and performs weighted summation thereon, and then processes the same through an activation function to obtain a final predicted classification probability map, based on which the image category of the image can be determined.

[0075] It can be understood that the preprocessing module, the block mixing module, the channel mixing module and the decoding classification module of the classification model are obtained by training an initial model based on sample data. Specifically, the classification model can be trained in the following manner: a large number of sample images are collected, the sample images are classified to obtain the image category of each image in the sample images. Here, the sample images can be classified by manual classification, by using existing image classification methods, or by collecting sample images based on existing image classification data sets (such as Cifar10, Cifar100, Imagenet, etc.) to obtain the image category of each image based on the image classification data set. The initial model is trained based on the sample images and the image category of each image in the sample images, thereby obtaining the classification model.

[0076] The method provided by the embodiments of the present application achieves the purpose of completing feature fusion using full-addition pulse-driven self-attention operation by performing feature extraction and relative position encoding on the image to be classified, obtaining a position encoding result, and sequentially operating the position encoding result through the block mixing module and the channel mixing module of the pulse-driven self-attention mechanism. The pulse neural network operation characteristics can be fully utilized to greatly reduce the model operation cost and running energy consumption while maintaining good performance of the model, thereby achieving the purpose of reducing the image classification processing power consumption while maintaining the accuracy of image classification, which is conducive to the edge deployment of neuromorphic chips.

[0077] Based on the above embodiments, Figure 2 is one of the flowcharts of the pulse-driven self-attention operation provided by the present application, as Figure 2 shown, the first pulse matrix comprises a query matrix Q1, a key matrix K1 and a value matrix V1. In step 130, pulse-driven self-attention operation is performed on the first pulse matrix in the block dimension, including:

[0078] performing element-wise masking operation on the key matrix K1 and the value matrix V1 to obtain a first operation result;

[0079] sum the first operation result by column to obtain a first summation result;

[0080] perform pulse sequence conversion on the first summation result to obtain a first pulse sequence;

[0081] perform row-by-row mask operation on the query matrix Q1 and the first pulse sequence to obtain a first pulse output, the first pulse output being used to determine a membrane potential feature map.

[0082] It should be noted that after obtaining the pulse feature map, the block mixing module can extract a first pulse matrix of the pulse feature map and perform pulse-driven self-attention operation on the first pulse matrix. Here, the first pulse matrix includes a matrix Q1 (representing query), a matrix K1 (representing key), and a matrix V1 (representing value), and the query matrix Q1, the key matrix K1, and the value matrix V1 are all feature representations of the image to be classified in pulse form. For example, the pulse feature map can be subjected to three linear mappings respectively to obtain the corresponding three matrices. In order to ensure that the three matrices obtained are all pulse matrices, batch normalization and pulse sequence conversion processing can be performed on the three matrices respectively, so that the query matrix Q1, the key matrix K1, and the value matrix V1 obtained finally are all matrices in pulse form.

[0083] Since the key matrix K1 and the value matrix V1 are both in pulse form and only contain 0 and 1, the element-by-element mask operation performed on the key matrix K1 and the value matrix V1 is an element-by-element 01 mask operation, which is equivalent to the dot product operation of the matrix, and in essence is a logical AND operation without multiplication operation. The first operation result obtained by performing element-by-element mask operation on the key matrix K1 and the value matrix V1 is also a matrix in pulse form. Summing the first operation result by column, i.e., adding and summing the elements of each column of the matrix, this operation step is a full addition operation, which also does not involve multiplication operation. The first pulse sequence and the query matrix Q1 obtained are both in pulse form and only contain 0 and 1, so the column-by-column mask operation performed on the query matrix Q1 and the first pulse sequence is a column-by-column 01 mask operation, which is equivalent to the broadcast element-wise product operation, and in essence is also a logical AND operation without multiplication operation. Therefore, the above pulse-driven self-attention operation is an operation containing only logical AND and full addition, which can greatly improve the calculation speed and greatly exert the advantage of SNN in reducing the calculation energy consumption.

[0084] It can be understood that although the first operation result is a matrix in the form of pulses, after summing the operation result by column, the first sum result obtained is not necessarily a matrix in the form of pulses, and elements other than 0 and 1 can appear in the first sum result. In order to ensure that the subsequent operation process conforms to the 01 calculation characteristics of the SNN and the calculation characteristics of not performing multiplication operation, the first sum result can be subjected to pulse sequence transformation, so as to obtain the first pulse sequence containing only 0 and 1 elements, thereby accelerating the subsequent calculation speed.

[0085] Specifically, the position encoding result of the image is subjected to pulse sequence transformation to obtain a pulse feature map S1 in the form of binary spike input. The pulse feature map S1 is input into a block mixing module of the classification model, sequentially passes through a linear mapping (Linear) layer and a spiking neuron (Spiking neuron) layer, and is subjected to linear mapping and pulse sequence transformation processing to obtain a first pulse matrix, i.e., a query matrix Q1, a key matrix K1 and a value matrix V1. The query matrix Q1, the key matrix K1 and the value matrix V1 can all be nxd matrices.

[0086] The key matrix K1 and the value matrix V1 are subjected to element-wise mask operation to obtain a first operation result, which is also an nxd matrix and can be represented as K1ΘV1, wherein Θ represents element-wise mask operation. It should be understood that when the key matrix K1 and the value matrix V1 are subjected to element-wise mask operation, a plurality of independent attention heads can be introduced. The outputs of the plurality of attention heads are spliced and linearly transformed to obtain the final representation, so as to improve the representation ability and processing ability of the model.

[0087] The first operation result is summed by column (corresponding to the block dimension) to obtain a first sum result, which is a 1xd matrix and can be represented as SUM c (K1ΘV1), wherein SUM c represents summing by column. It can be understood that after the element-wise 01 mask operation is performed on the key matrix K1 and the value matrix V1, the part containing 1 in the first operation result is relatively small, and therefore the full addition operation performed on the first operation result is sparse, which can further reduce the calculation energy consumption.

[0088] The first sum result is input into the spiking neuron layer for pulse sequence transformation to obtain a first pulse sequence, which is a 1xd matrix sequence and can be represented as SN(SUM c (K1ΘV1)), wherein SN() represents LIF neuron operation.

[0089] The first pulse sequence and the query matrix Q1 are subjected to column-by-column mask operation to obtain a first pulse output, which is an nxd matrix and can be represented as Q1⊙SN(SUM c (K1⊙V1)). The first pulse output is denoted as The above pulse-driven self-attention operation process can be represented as:

[0090]

[0091] Wherein, SDSA C () represents the pulse-driven self-attention operation in the block dimension. After obtaining the first pulse output, the membrane potential feature map of the image can be determined by applying the first pulse output. The pulse-driven self-attention operation introduced in the embodiment of the present application enables the classification model to have the event-driven characteristics of pulse operation, which can greatly reduce the energy consumption of the classification model running and is conducive to its operation deployment in a neuromorphic chip. Moreover, the pulse-driven attention operation applies the self-attention mechanism, which can improve the efficiency and accuracy of image classification by the classification model.

[0092] Based on any of the above embodiments, Figure 3 is a flowchart of step 130 in the classification method of the full-addition pulse self-attention Transformer provided by the present application, as Figure 3 shown, step 130 specifically includes:

[0093] Step 131, based on the main branch of the block mixing module, performing pulse sequence conversion on the position encoding result to obtain a pulse feature map, extracting a first pulse matrix of the pulse feature map, and applying pulse-driven self-attention operation to mix the features of different block dimensions of the first pulse matrix to obtain a first pulse output, and applying the first pulse output to determine a first main branch membrane potential.

[0094] Step 132, based on the residual branch of the block mixing module, extracting the membrane potential of the position encoding result to obtain a first residual branch membrane potential, and fusing the first main branch membrane potential and the first residual branch membrane potential to determine a membrane potential feature map.

[0095] It should be noted that the traditional deep neural network has problems such as gradient disappearance and gradient explosion, which can easily lead to a decrease in accuracy. In order to solve this problem, the block mixing module in the embodiment of the present application uses residual connection, including a main branch and a residual branch, which correspond to two paths in the block mixing module. The main branch is used to extract features, and the residual branch is used to learn features by adding the output of the main branch to the input through residual connection. This not only alleviates the problems of gradient disappearance and gradient explosion in the deep neural network, but also improves the accuracy of the classification model.

[0096] Specifically, after obtaining the position encoding result of the image, the position encoding result is input into the block mixing module, and the position encoding result will enter the main branch and the residual branch respectively. Based on the main branch, the first main branch membrane potential can be obtained, wherein the first main branch membrane potential refers to the membrane potential value of the main branch neuron finally output after the main branch performs operation and processing on the pulse feature map. After obtaining the first pulse output, the first pulse output can be sequentially subjected to pulse sequence conversion, linear mapping and batch normalization processing, so as to obtain the first main branch membrane potential.

[0097] After the position encoding result enters the residual branch, the membrane potential of the position encoding result is extracted through the residual branch, and the first residual branch membrane potential can be obtained. Here, the first residual branch membrane potential refers to the membrane potential value of the neuron output by the residual branch. Exemplarily, the residual branch can include a linear mapping layer, a batch normalization layer and a pulse neuron layer, wherein the pulse neuron layer is optional. After the pulse feature map is input, it can be sequentially subjected to linear mapping, batch normalization and pulse sequence conversion processing, so as to obtain the first residual branch membrane potential.

[0098] After obtaining the first residual branch membrane potential, the first main branch membrane potential and the first residual branch membrane potential are added by using the residual connection, so as to obtain the membrane potential feature map, thereby effectively alleviating the problems of gradient disappearance and gradient explosion in the deep neural network, and further improving the accuracy of the classification model in the image classification task. It should be understood that after the pre-processing module performs feature extraction and relative position encoding on the image to be classified, the obtained position encoding result is essentially a membrane potential. Directly inputting the position encoding result into the residual branch realizes the residual connection based on the membrane potential. Compared with the pulse sequence, the membrane potential can deliver more abundant information, thereby being conducive to improving the accuracy of the classification model in the image classification.

[0099] Based on any one of the above embodiments, Figure 4 is a second flowchart of the pulse-driven self-attention operation provided by the present application, as shown in Figure 4 The second pulse matrix includes a query matrix Q2, a key matrix K2 and a value matrix V2. In step 140, pulse-driven self-attention operation is performed on the second pulse matrix in the channel dimension, including:

[0100] Element-wise mask operation is performed on the matrix and the matrix to obtain a second operation result;

[0101] The second operation result is summed by column to obtain a second summation result;

[0102] The second summation result is subjected to pulse sequence conversion to obtain a second pulse sequence;

[0103] Element-wise mask operation is performed on the matrix and the second pulse sequence to obtain a second pulse output, and the second pulse output is transposed to determine the output membrane potential;

[0104] matrix matrix and matrix The transposed matrix of the query matrix Q2, the key matrix K2 and the value matrix V2 respectively.

[0105] It should be noted that after obtaining the membrane potential feature map, the channel mixing module can extract the second pulse matrix of the membrane potential feature map, and perform pulse-driven self-attention operation on the second pulse matrix. Here, the second pulse matrix includes the query matrix Q2 (representing Query), the key matrix K2 (representing Key) and the value matrix V2 (representing Value). For example, the membrane potential feature map can be subjected to three linear mappings respectively to obtain the corresponding three matrices. In order to ensure that the three matrices obtained are pulse matrices, batch normalization and pulse sequence conversion processing can be performed on the three matrices respectively, so that the query matrix Q2, the key matrix K2 and the value matrix V2 obtained finally are all in the form of pulse matrices.

[0106] Specifically, the membrane potential feature map is subjected to pulse sequence conversion to obtain a binary pulse input S2, which is input into the channel mixing module of the classification model, and sequentially passes through a linear mapping layer and a pulse neuron layer to perform linear mapping and pulse sequence conversion processing on the binary pulse input S2, thereby obtaining the second pulse matrix, i.e., the query matrix Q2, the key matrix K2 and the value matrix V2. The query matrix Q2, the key matrix K2 and the value matrix V2 can all be nxd matrices. In order to mix the features of different channel dimensions of the second pulse matrix, the query matrix Q2, the key matrix K2 and the value matrix V2 can be transposed respectively to obtain the transposed matrix matrix and matrix Thus, based on the transposed matrix, the feature fusion is completed using pulse-driven self-attention operation.

[0107] In the embodiment of the application, the implementation process of the pulse-driven self-attention operation is consistent with the implementation process of the pulse-driven self-attention operation in the channel dimension in the above-mentioned embodiment, and will not be described here. The pulse-driven self-attention operation process of the embodiment of the application can be represented as:

[0108]

[0109] wherein, SDSA R () represents the pulse-driven self-attention operation in the channel dimension. After obtaining the second pulse output, the second pulse output is transposed to determine the output membrane potential of the image.

[0110] In the embodiment of the present application, by transposing the second pulse matrix, the pulse-driven self-attention operation can be performed again on the matrix from the channel dimension, and the self-attention operations in the block dimension and the channel dimension are combined and superimposed, so that the performance of the model can be maintained while reducing the model size and the running energy consumption.

[0111] Based on any of the above embodiments, Figure 5 is a flowchart of step 140 in the classification method of the full-addition pulse self-attention Transformer provided by the present application, as Figure 5 shown, step 140 specifically includes:

[0112] Step 141, based on the main branch of the channel mixing module, extracts the second pulse matrix of the membrane potential feature map, and applies the pulse-driven self-attention operation to mix the features in different channel dimensions of the second pulse matrix, to obtain the second pulse output, and applies the second pulse output to determine the second main branch membrane potential;

[0113] Step 142, based on the residual branch of the channel mixing module, extracts the membrane potential of the membrane potential feature map to obtain the second residual branch membrane potential, and fuses the second main branch membrane potential and the second residual branch membrane potential to determine the output membrane potential.

[0114] It should be noted that the residual connection is also used in the channel mixing module, including the main branch and the residual branch, and the main branch and the residual branch correspond to two paths in the channel mixing module. The main branch is used to extract features, and the residual branch is used to learn features by adding the output of the main branch and the input through the residual connection. By using the residual connection in the channel mixing module, the present embodiment can effectively alleviate the problems of gradient disappearance and gradient explosion in the deep neural network, and further improve the accuracy of image classification by the classification model.

[0115] Specifically, after obtaining the membrane potential feature map of the image, the membrane potential feature map can be input to the channel mixing module, and the membrane potential feature map will enter the main branch and the residual branch respectively. Based on the main branch, the second main branch membrane potential can be obtained, where the second main branch membrane potential refers to the membrane potential value of the main branch neuron finally output by the main branch after operating and processing the membrane potential feature map. After obtaining the second pulse output, the second pulse output can be sequentially subjected to pulse sequence conversion, linear mapping and batch normalization processing, so as to obtain the second main branch membrane potential.

[0116] After the membrane potential feature map enters the residual branch, the membrane potential of the membrane potential feature map is extracted through the residual branch, and a second residual branch membrane potential can be obtained. Here, the second residual branch membrane potential refers to the membrane potential value of the neuron output by the residual branch. Illustratively, the residual branch can include a linear mapping layer, a batch normalization layer, and a spiking neuron layer, wherein the spiking neuron layer is optional. After the membrane potential feature map is input, it can be sequentially subjected to linear mapping, batch normalization, and pulse sequence conversion processing, thereby obtaining the second residual branch membrane potential.

[0117] After obtaining the second residual branch membrane potential, the second main branch membrane potential and the second residual branch membrane potential are added using the residual connection, and the output membrane potential is obtained, thereby effectively alleviating problems such as gradient vanishing and gradient explosion in the deep neural network, and further improving the accuracy of the classification model in image classification tasks.

[0118] Based on any of the above embodiments, Figure 6 is a flowchart of step 120 in the classification method of the full-addition pulse self-attention Transformer provided by the present application, as Figure 6 shown, step 120 specifically includes:

[0119] Step 121, based on the pulse block decomposition module of the preprocessing module, the image is split into blocks to obtain image blocks, and the image blocks are subjected to feature extraction to obtain a basic feature map;

[0120] Step 122, based on the position encoding module of the preprocessing module, the basic feature map is sequentially subjected to pulse sequence conversion, convolution operation, and batch normalization processing to obtain a position feature, and the position feature and the basic feature map are added to obtain a position encoding result.

[0121] Specifically, the preprocessing module includes a pulse block decomposition module and a position encoding module. The pulse block decomposition module is connected to the position encoding module. The pulse block decomposition module is used to split the image into blocks to obtain image blocks (patches) and perform feature extraction on the image blocks to obtain a basic feature map. The position encoding module is used to perform relative position encoding on the basic feature map to obtain a position feature, and add the position feature and the basic feature map to obtain a position encoding result.

[0122] The pulse block decomposition module can include a first decomposition block and a second decomposition block. The first decomposition block can have one or more. Each first decomposition block can include a convolution layer, a batch normalization layer, a spiking neuron layer, and a max pooling layer. The second decomposition block can include a convolution layer, a batch normalization layer, and a max pooling layer. After the image to be classified is input into the pulse block decomposition module, the pulse block decomposition module linearly projects it onto a pulse form feature vector and divides it into a plurality of pulse form patch sequences, thereby obtaining a basic feature map of the image.

[0123] The position coding module can include a pulse neuron layer, a convolution layer and a batch normalization layer, the basic feature map is input to the position coding module, and the position coding module is sequentially subjected to pulse sequence conversion, convolution operation and batch normalization processing, so that the position feature is obtained. The position feature and the basic feature map are added, and the position coding result of the image is obtained.

[0124] In the embodiment of the application, the image is split into blocks by the pulse block decomposition module to obtain a plurality of image blocks, and the local and global features of the image can be extracted based on the plurality of image blocks, so that the processing of the entire image is finally realized, and the classification model can process each image block in parallel, which is beneficial to improve the calculation speed of the classification model.

[0125] Based on any of the above embodiments, there are a plurality of encoding modules in the classification model, and the encoding module is composed of a block mixing module and a channel mixing unit module in series;

[0126] The plurality of encoding modules are sequentially connected in series, and the output membrane potential determined by the previous encoding module is the position coding result corresponding to the next encoding module.

[0127] Specifically, the block mixing module and the channel mixing module are both feature mixing units in the classification model, and can be connected in series to form the encoding module of the classification model. In order to further improve the accuracy of the model, a plurality of encoding modules can be provided, and the plurality of encoding modules are sequentially connected in series. For the first encoding module, the input is the output of the pre-processing module, i.e. the position coding result, and the position coding result is essentially a membrane potential. After the position coding result is input into the first encoding module, the output membrane potential output by the first encoding module can be obtained. For the second encoding module, the input is the output membrane potential determined by the first encoding module. By analogy, the output membrane potential determined by the previous encoding module is the input of the next encoding module, i.e. the corresponding position coding result.

[0128] In the embodiment of the application, the block mixing module and the channel mixing module are connected in series to form the encoding module, so that all the feature mixing units in the classification model can be realized in the full-addition pulse-driven self-attention manner, and the pulse-driven self-attention operation is performed from two different dimensions, and the two operations are combined and superimposed, which can further reduce the energy consumption of the model running and maintain good performance. In addition, by sequentially connecting a plurality of encoding modules in series, a classification model with a deeper depth can be formed, which is beneficial to improve the representation ability and learning ability of the classification model, so that the classification model can better process more complex and diverse features.

[0129] Based on any of the above embodiments, Figure 7 is a structural schematic diagram of the classification model provided by the application, such as Figure 7As shown, the classification model includes a preprocessing module, an encoding module, and a decoding classification module. The preprocessing module includes a pulse block decomposition module and a position encoding module. The pulse block decomposition module includes a first decomposition block and a second decomposition block. There can be multiple first decomposition blocks, for example, in an embodiment of the present application, there are three first decomposition blocks. Each first decomposition block can include a convolution layer (Conv 2D), a batch normalization layer (Batch Normalization, BN), a spike neuron layer (Spike Neuron Layer), and a max pooling layer (MaxPooling, MP). The second decomposition block can include a convolution layer, a batch normalization layer, and a max pooling layer. The position encoding module can include a spike neuron layer, a convolution layer, and a batch normalization layer. It should be noted that the above-mentioned spike neuron layers can all be LIF neuron layers.

[0130] After the image to be classified is input into the preprocessing module, the image is split into blocks by the pulse block decomposition module to obtain a plurality of image blocks, and feature extraction is performed on the image blocks to obtain a basic feature map. The basic feature map is sequentially subjected to pulse sequence conversion, convolution operation, and batch normalization processing by the position encoding module to obtain a position feature. The basic feature map and the position feature are added through a residual connection to obtain a position encoding result of the image.

[0131] The processing process of the preprocessing module on the image to be classified can be represented by the following formula:

[0132] u=PSM(I),I∈R T×C×H×W ,u∈R T×N×D

[0133] s=SN(u),s∈R T×N×D

[0134] RPE=BN(Conv2d(s)),RPE∈R T×N×D

[0135] U0=u+RPE,U0∈R T×N×D

[0136] Wherein, I represents the image to be classified, PSM represents the pulse block decomposition module, u represents the basic feature map, SN is the spike neuron layer, s is the output of the spike neuron layer, RPE represents the position feature, and U0 is the position encoding result finally output by the preprocessing module; T represents the time dimension of the image, C represents the channel number of the image, H represents the height of the image, W represents the width of the image, N represents the number of image blocks, and D represents the feature channel dimension of the image.

[0137] The encoding module includes a block mixing module and a channel mixing module, both of which use residual connection. The block mixing module includes a main branch and a residual branch, the main branch includes a block mixing pre-processing layer, a block mixing operation layer and a block mixing post-processing layer, the block mixing pre-processing layer is used to process the position encoding result to obtain a first pulse matrix, the block mixing pre-processing layer can include a pulse neuron layer and three processing blocks, the three processing blocks correspond to obtaining a query matrix Q1, a key matrix K1 and a value matrix V1 respectively, each processing block can include a linear mapping layer, a batch normalization layer and a pulse neuron layer; the block mixing post-processing layer can include a pulse neuron layer, a linear mapping layer and a batch normalization layer. The channel mixing module includes a main branch and a residual branch, the main branch includes a channel mixing pre-processing layer, a channel mixing operation layer and a channel mixing post-processing layer, the channel mixing pre-processing layer is used to process the membrane potential feature map to obtain a second pulse matrix, the channel mixing pre-processing layer can include a pulse neuron layer and three processing blocks, the three processing blocks correspond to obtaining a query matrix Q2, a key matrix K2 and a value matrix V2 respectively, each processing block can include a linear mapping layer, a batch normalization layer and a pulse neuron layer; the channel mixing post-processing layer can include a pulse neuron layer, a linear mapping layer and a batch normalization layer. It should be understood that the above pulse neuron layers can be LIF neuron layers.

[0138] The position encoding result is essentially a membrane potential. After the pre-processing module outputs the position encoding result, the position encoding result is converted into a pulse sequence by the pulse neuron layer of the main branch of the block mixing module to obtain a pulse feature map. The pulse feature map is input into three processing blocks respectively, and each processing block is sequentially subjected to linear mapping, batch normalization and pulse sequence conversion processing, and three pulse matrices, i.e., a query matrix Q1, a key matrix K1 and a value matrix V1, can be obtained. The query matrix Q1, the key matrix K1 and the value matrix V1 are input into the block mixing operation layer to perform pulse-driven self-attention operation, and a first pulse output is obtained. The first pulse output is input into the block mixing post-processing layer to sequentially perform pulse sequence conversion, linear mapping and batch normalization processing, and a first main branch membrane potential output can be obtained. It can be understood that the pulse-driven self-attention operation on the first pulse matrix in the block dimension avoids complex matrix multiplication operation, reduces the complexity of attention operation to linear level, and is beneficial to speed up the calculation and reduce the calculation power consumption.

[0139] The position encoding result is connected to the output end of the main branch through the residual branch of the block mixing module. The position encoding result can be added to the first residual branch membrane potential output by the residual branch as the first main branch membrane potential, and the membrane potential feature map finally output by the block mixing module is obtained.

[0140] The membrane potential feature map is input to a channel mixing module. The membrane potential feature map is subjected to pulse sequence conversion by a pulse neuron layer of the channel mixing pre-processing layer, and is subjected to linear mapping, batch normalization and pulse sequence conversion in sequence by three processing blocks to obtain a second pulse matrix, i.e., a query matrix Q2, a key matrix K2 and a value matrix V2. The query matrix Q2, the key matrix K2 and the value matrix V2 are input to a channel mixing operation layer. The features of different channel dimensions of the second pulse matrix are mixed to obtain a second pulse output. The second pulse output is input to a channel mixing post-processing layer. Pulse sequence conversion, linear mapping and batch normalization are performed in sequence to obtain the final output membrane potential. It should be understood that the pulse-driven self-attention operation on the channel dimension of the second pulse matrix can not only avoid complex matrix multiplication and reduce computing power consumption, but also can superimpose and fuse the features in the block dimension, thereby reducing the energy consumption of the model while maintaining the good performance of the model.

[0141] The encoding module can be multiple, for example, there are L encoding modules in the embodiment of the application. The multiple encoding modules are connected in sequence. The output membrane potential determined by a previous encoding module is the position encoding result corresponding to a subsequent encoding module. After processing by multiple encoding modules, the final output membrane potential is input to a decoding classification module to obtain a final prediction classification probability map. Based on the prediction classification probability map, the image category of the image can be determined.

[0142] The processing process of the encoding module can be represented by the following formula:

[0143] S l-1 =SN(U l-1 ),S l-1 ∈R T×N×D ,l=1,…,L

[0144] U′ l =SDSA C (S l-1 )+U l-1 ,U′ l ∈R T×N×D ,l=1,…,L

[0145] S′ l =SN(U′ l ),S′ l ∈R T×N×D ,l=1,…,L

[0146] U l =SN(SDSA R (S′ l )+U′ l ),U l ∈R T×N×D ,l=1,…,L

[0147] wherein, U l-1 is the position encoding result input by the current encoding module, and is the output membrane potential determined by the previous encoding module, S l-1 is the pulse feature map, U′ l is the membrane potential feature map, S′ l is the output of the first pulse neuron layer in the channel mixing module, U l is the output membrane potential determined by the current encoding module, and is the position encoding result corresponding to the subsequent encoding module.

[0148] After the final output membrane potential is obtained through the encoding module, the output membrane potential is input to the decoding classification module for global average pooling, and the image category of the image can be obtained through the fully connected classification head. The decoding classification module can be expressed as follows:

[0149] Y=CH(GAP(S L ))

[0150] wherein, CH represents the processing of the fully connected classification head, GAP represents the processing of the global average pooling, S L represents the output membrane potential of the final obtained image.

[0151] For the existing image classification datasets CIFAR-10 and CIFAR-100, the classification model of the embodiment of the present application and the existing image classification model are respectively used for image classification, and the image classification results can be obtained, which are shown in the following table:

[0152]

[0153] It should be noted that the first column in the table represents the model name, for example, Spike-driven Transformer is a pulse-driven neural network model obtained by applying SNN to Transformer, which is an existing image classification model. All Self-attention Spike-driven Transformer is the classification model of the embodiment of the present application, which is a pulse-driven neural network model obtained by applying the Self-attention mechanism using SNN to all feature mixing units of Transformer.

[0154] The second column in the table represents the network structure of the model, for example, Spiking Transformer-2-256 represents a neural network architecture based on the model in the first column, 2 means that the model contains 2 SpikingTransformer encoding modules, and 256 means that each encoding module has 256 neurons.

[0155] The third column of the table represents the dataset name; the fourth column of the table represents whether the event-driven characteristic of the spike operation is present, where "√" represents "yes" and "×" represents "no"; the fifth column of the table represents the parameter amount of the model, in M (mega); the sixth column of the table represents the time step; and the seventh column of the table represents the accuracy of the image classification result.

[0156] It can be understood that the existing Spike-driven Transformer model adopts a general Transformer model structure, and the classification model All Self-attention Spike-driven Transformer of the embodiment of the present application replaces the existing image classification model based on the pulse MLP (multi-layer perceptron) mixing module with a pulse self-attention based channel mixing module. Obviously, since the energy consumption of the pulse self-attention is lower than that of the pulse MLP, the classification method of the embodiment of the present application can further reduce the network energy consumption.

[0157] For example, assuming that the shape of the input tensor x is [4, 96, 384, 14, 14], in the order from left to right, each value respectively represents the time step, batch normalization value, channel, height, and width of x. For the existing pulse MLP based mixing module, the calculation amount includes full connection with channel change from C→4C (C represents the number of channels) and full connection of 4C→C, and the total FLOPs (Floating Point Operations) is 231M. In contrast, the pulse self-attention based channel mixing module used in the embodiment of the present application includes three channel invariant full connections, a module for self-attention calculation, and a channel number invariant full connection for mapping the weighted attention score result, and the FLOPs of the three parts are 86.7M, 0.009M, and 28.9M, respectively. The total FLOPs of the three parts is 115.6M. As can be seen, the pulse self-attention based channel mixing module proposed in the embodiment of the present application reduces the calculation amount by 49.9% compared with the existing pulse MLP based mixing module, and therefore the classification method provided in the embodiment of the present application has an energy consumption advantage.

[0158] Based on the experimental results recorded in the above table, compared with the existing image classification model Spike-driven Transformer, the classification model of the embodiment of the application has improved performance while reducing the model parameter quantity and size. The existing image classification model Spike-driven Transformer does not use pulse-driven self-attention operation for all feature mixing units, while the classification model of the embodiment of the application introduces pulse-driven self-attention operation for each dimension feature of the image feature map, which can maintain the performance of the model while reducing the model size and running energy consumption.

[0159] Based on any of the above embodiments, Figure 8 is a structural schematic diagram of a full-addition pulse self-attention Transformer classification device provided by the application, as Figure 8 shown, the device comprises:

[0160] The acquisition unit 810 is configured to acquire an image to be classified.

[0161] The processing unit 820 is configured to perform feature extraction on the image based on a preprocessing module of the classification model, obtain a basic feature map, and perform relative position coding on the basic feature map to obtain a position coding result of the image.

[0162] The block mixing unit 830 is configured to perform pulse sequence conversion on the position coding result based on a block mixing module of the classification model, obtain a pulse feature map, extract a first pulse matrix of the pulse feature map, and perform pulse-driven self-attention operation on the first pulse matrix in the block dimension to obtain a membrane potential feature map of the image.

[0163] The channel mixing unit 840 is configured to extract a second pulse matrix of the membrane potential feature map based on a channel mixing module of the classification model, and perform pulse-driven self-attention operation on the second pulse matrix in the channel dimension to obtain an output membrane potential of the image.

[0164] The decoding and classification unit 850 is configured to apply the output membrane potential to perform image classification based on a decoding and classification module of the classification model to obtain an image category of the image.

[0165] The device provided by the embodiment of the present application achieves the purpose of completing feature fusion by using full-addition pulse-driven self-attention operation by performing feature extraction and relative position coding on the image to be classified, sequentially passing the position coding result through the block mixing module and the channel mixing module of the pulse-driven self-attention mechanism, and can fully utilize the pulse neural network operation characteristics, greatly reduce the model operation cost and running energy consumption, and at the same time, make the model maintain good performance, so as to realize the reduction of image classification processing power consumption while maintaining the accuracy of image classification, which is beneficial to the edge deployment of neuromorphic chips.

[0166] Based on any of the above embodiments, the first pulse matrix includes a query matrix Q1, a key matrix K1 and a value matrix V1, and the pulse-driven self-attention operation is performed on the first pulse matrix in the block dimension, including:

[0167] Element-wise mask operation is performed on the key matrix K1 and the value matrix V1 to obtain a first operation result;

[0168] The first operation result is summed by column to obtain a first summation result;

[0169] The first summation result is pulse sequence transformed to obtain a first pulse sequence;

[0170] Row-wise mask operation is performed on the query matrix Q1 and the first pulse sequence to obtain a first pulse output, and the first pulse output is used to determine the membrane potential feature map.

[0171] Based on any of the above embodiments, the block mixing unit 830 specifically includes:

[0172] The first main branch unit is configured to perform pulse sequence transformation on the position coding result based on the main branch of the block mixing module to obtain a pulse feature map, extract a first pulse matrix of the pulse feature map, and apply pulse-driven self-attention operation to mix the features of different block dimensions of the first pulse matrix to obtain a first pulse output, and apply the first pulse output to determine a first main branch membrane potential.

[0173] The first residual branch unit is configured to extract the membrane potential of the position coding result based on the residual branch of the block mixing module to obtain a first residual branch membrane potential, and fuse the first main branch membrane potential and the first residual branch membrane potential to determine the membrane potential feature map.

[0174] Based on any of the above embodiments, the second pulse matrix includes a query matrix Q2, a key matrix K2 and a value matrix V2, and the pulse-driven self-attention operation is performed on the second pulse matrix in the channel dimension, including:

[0175] Element-wise mask operation is performed on the matrix and the matrix to obtain a second operation result;

[0176] summing the second operation result by column to obtain a second summation result;

[0177] performing pulse sequence conversion on the second summation result to obtain a second pulse sequence;

[0178] performing row-by-row mask operation on the matrix and the second pulse sequence to obtain a second pulse output, and the second pulse output is used for determining the output membrane potential after transposition;

[0179] the matrix the matrix and the matrix are respectively transpositions of the query matrix Q2, the key matrix K2 and the value matrix V2.

[0180] Based on any one of the above embodiments, the channel mixing unit 840 specifically includes:

[0181] a second main branch unit, configured to extract a second pulse matrix of a membrane potential feature map based on a main branch of the channel mixing module, and apply pulse-driven self-attention operation to mix features of different channel dimensions of the second pulse matrix to obtain a second pulse output, and apply the second pulse output to determine a second main branch membrane potential;

[0182] a second residual branch unit, configured to extract a membrane potential of the membrane potential feature map based on a residual branch of the channel mixing module to obtain a second residual branch membrane potential, and fuse the second main branch membrane potential and the second residual branch membrane potential to determine the output membrane potential.

[0183] Based on any one of the above embodiments, the processing unit 820 specifically includes:

[0184] a pulse block decomposition unit, configured to decompose the image into blocks based on a pulse block decomposition module of the preprocessing module to obtain image blocks, and extract basic feature maps from the image blocks;

[0185] a position encoding unit, configured to sequentially perform pulse sequence conversion, convolution operation and batch normalization processing on the basic feature maps based on a position encoding module of the preprocessing module to obtain position features, and fuse the basic feature maps and the position features to determine a position encoding result.

[0186] Based on any one of the above embodiments, there are multiple encoding modules in the classification model, and the encoding modules are composed of the block mixing module and the channel mixing unit module in series;

[0187] The multiple encoding modules are sequentially connected in series, and the output membrane potential determined by a previous encoding module is the position encoding result corresponding to a subsequent encoding module.

[0188] Figure 9An example of a schematic diagram of a physical structure of an electronic device is shown in Figure 9 As shown, the electronic device can include a processor 910, a communications interface 920, a memory 930, and a communications bus 940, wherein the processor 910, the communications interface 920, and the memory 930 complete mutual communication through the communications bus 940. The processor 910 can invoke a logic instruction in the memory 930 to execute a full-addition pulse self-attention Transformer classification method, which includes: obtaining an image to be classified; performing feature extraction on the image based on a preprocessing module of a classification model to obtain a basic feature map, and performing relative position coding on the basic feature map to obtain a position coding result of the image; performing pulse sequence conversion on the position coding result based on a block mixing module of the classification model to obtain a pulse feature map, extracting a first pulse matrix of the pulse feature map, and performing pulse-driven self-attention operation on the first pulse matrix in the block dimension to obtain a membrane potential feature map of the image; extracting a second pulse matrix of the membrane potential feature map based on a channel mixing module of the classification model, and performing pulse-driven self-attention operation on the second pulse matrix in the channel dimension to obtain an output membrane potential of the image; and applying the output membrane potential to perform image classification based on a decoding classification module of the classification model to obtain an image category of the image.

[0189] In addition, the logic instruction in the memory 930 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0190] In another aspect, the present application also provides a computer program product comprising a computer program, which can be stored on a non-transitory computer-readable storage medium, and the computer program can be executed by a processor to enable a computer to perform the classification method of the full-addition pulse self-attention Transformer provided by the above method, which comprises: obtaining an image to be classified; performing feature extraction on the image based on a pre-processing module of the classification model to obtain a basic feature map, and performing relative position coding on the basic feature map to obtain a position coding result of the image; performing pulse sequence conversion on the position coding result based on a block mixing module of the classification model to obtain a pulse feature map, extracting a first pulse matrix of the pulse feature map, and performing pulse-driven self-attention operation on the first pulse matrix in the block dimension to obtain a membrane potential feature map of the image; extracting a second pulse matrix of the membrane potential feature map based on a channel mixing module of the classification model, and performing pulse-driven self-attention operation on the second pulse matrix in the channel dimension to obtain an output membrane potential of the image; and performing image classification by applying the output membrane potential based on a decoding classification module of the classification model to obtain an image category of the image.

[0191] In another aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and the computer program can be executed by a processor to implement the classification method of the full-addition pulse self-attention Transformer provided by the above method, which comprises: obtaining an image to be classified; performing feature extraction on the image based on a pre-processing module of the classification model to obtain a basic feature map, and performing relative position coding on the basic feature map to obtain a position coding result of the image; performing pulse sequence conversion on the position coding result based on a block mixing module of the classification model to obtain a pulse feature map, extracting a first pulse matrix of the pulse feature map, and performing pulse-driven self-attention operation on the first pulse matrix in the block dimension to obtain a membrane potential feature map of the image; extracting a second pulse matrix of the membrane potential feature map based on a channel mixing module of the classification model, and performing pulse-driven self-attention operation on the second pulse matrix in the channel dimension to obtain an output membrane potential of the image; and performing image classification by applying the output membrane potential based on a decoding classification module of the classification model to obtain an image category of the image.

[0192] The device embodiments described above are only schematic, wherein the units shown as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment scheme. Those skilled in the art can understand and implement without creative labor.

[0193] Those skilled in the art can clearly understand the implementation of the various embodiments by means of software and necessary general hardware platforms through the description of the above embodiments, and of course, the implementation can also be through hardware. Based on this understanding, the above technical solutions, essentially or in other words, the part that contributes to the prior art can be embodied in the form of a software product. The computer software product can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the method described in each embodiment or some part of the embodiment.

[0194] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solution deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A classification method for a fully additive impulse self-attention Transformer, characterized in that, include: Obtain the image to be classified; The preprocessing module based on the classification model extracts features from the image to obtain a basic feature map, and performs relative position encoding on the basic feature map to obtain the position encoding result of the image. Based on the block mixing module of the classification model, the position encoding result is transformed into a pulse sequence to obtain a pulse feature map. The first pulse matrix of the pulse feature map is extracted, and pulse-driven self-attention operation is performed on the first pulse matrix in the block dimension to obtain the membrane potential feature map of the image. Based on the channel mixing module of the classification model, the second pulse matrix of the membrane potential feature map is extracted, and pulse-driven self-attention operation is performed on the second pulse matrix in the channel dimension to obtain the output membrane potential of the image; The decoding and classification module based on the classification model applies the output membrane potential to classify the image and obtain the image category of the image; The first pulse matrix includes a query matrix Q1, a key matrix K1, and a value matrix V1. The pulse-driven self-attention operation on the first pulse matrix at the block dimension includes: Perform element-wise masking operations on the key matrix K1 and the value matrix V1 to obtain the first operation result; The first calculation result is summed column by column to obtain the first summation result; The first summation result is converted into a pulse sequence to obtain the first pulse sequence; A row-by-row masking operation is performed on the query matrix Q1 and the first pulse sequence to obtain the first pulse output, which is used to determine the membrane potential feature map. The second pulse matrix includes a query matrix Q2, a key matrix K2, and a value matrix V2. The pulse-driven self-attention operation on the second pulse matrix in the channel dimension includes: For the matrix and the matrix Perform element-by-element masking to obtain the second result; The results of the second operation are summed column by column to obtain the second summation result; The second summation result is converted into a pulse sequence to obtain a second pulse sequence; For the matrix The second pulse sequence is subjected to a line-by-line masking operation to obtain the second pulse output, which is then transposed to determine the output membrane potential. The matrix ,matrix sum matrix These are the transposes of the query matrix Q2, key matrix K2, and value matrix V2, respectively.

2. The classification method for the fully additive impulse self-attention Transformer according to claim 1, characterized in that, The block mixing module based on the classification model performs pulse sequence transformation on the position encoding result to obtain a pulse feature map, extracts the first pulse matrix from the pulse feature map, and performs pulse-driven self-attention operation on the first pulse matrix in the block dimension to obtain the membrane potential feature map of the image, including: Based on the main branch of the block mixing module, the position encoding result is converted into a pulse sequence to obtain a pulse feature map. The first pulse matrix of the pulse feature map is extracted, and pulse-driven self-attention operation is applied to mix the features of different block dimensions of the first pulse matrix to obtain the first pulse output. The first pulse output is then used to determine the membrane potential of the first main branch. Based on the residual branches of the block mixing module, the membrane potential of the position encoding result is extracted to obtain the first residual branch membrane potential. The first main branch membrane potential and the first residual branch membrane potential are fused to determine the membrane potential feature map.

3. The classification method for the fully additive impulse self-attention Transformer according to claim 1, characterized in that, The channel mixing module based on the classification model extracts the second pulse matrix of the membrane potential feature map and performs pulse-driven self-attention operation on the second pulse matrix in the channel dimension to obtain the output membrane potential of the image, including: Based on the main branch of the channel mixing module, the second pulse matrix of the membrane potential feature map is extracted, and pulse-driven self-attention operation is applied to mix the features of different channel dimensions of the second pulse matrix to obtain the second pulse output. The second pulse output is then used to determine the membrane potential of the second main branch. Based on the residual branch of the channel mixing module, the membrane potential of the membrane potential feature map is extracted to obtain the second residual branch membrane potential. The output membrane potential is determined by fusing the second main branch membrane potential and the second residual branch membrane potential.

4. The classification method for a fully additive impulse self-attention Transformer according to any one of claims 1 to 3, characterized in that, The preprocessing module based on the classification model extracts features from the image to obtain a basic feature map, and performs relative position encoding on the basic feature map to obtain the position encoding result of the image, including: Based on the pulse block decomposition module of the preprocessing module, the image is divided into blocks to obtain image blocks, and features are extracted from the image blocks to obtain basic feature maps; Based on the position encoding module of the preprocessing module, the basic feature map is sequentially subjected to pulse sequence transformation, convolution operation and batch normalization to obtain position features, and the position encoding result is determined by fusing the basic feature map and the position features.

5. The classification method for a fully additive impulse self-attention Transformer according to any one of claims 1 to 3, characterized in that, The classification model contains multiple encoding modules, which are composed of sequentially connected block mixing modules and channel mixing modules. Multiple encoding modules are connected in series, and the output membrane potential determined by the previous encoding module is the position encoding result corresponding to the next encoding module.

6. A classification device using a fully additive pulse self-attention Transformer, characterized in that, include: The acquisition unit is used to acquire the image to be classified. The processing unit is used by the preprocessing module based on the classification model to extract features from the image to obtain a basic feature map, and to encode the basic feature map by relative position to obtain the position encoding result of the image. A block mixing unit is used to perform pulse sequence transformation on the location encoding result based on the block mixing module of the classification model to obtain a pulse feature map, extract the first pulse matrix of the pulse feature map, and perform pulse-driven self-attention operation on the first pulse matrix in the block dimension to obtain the membrane potential feature map of the image. The channel mixing unit is used to extract the second pulse matrix of the membrane potential feature map based on the channel mixing module of the classification model, and perform pulse-driven self-attention operation on the second pulse matrix in the channel dimension to obtain the output membrane potential of the image; A decoding and classification unit is used to classify images based on the classification model and apply the output membrane potential to obtain the image category of the image. The first pulse matrix includes a query matrix Q1, a key matrix K1, and a value matrix V1. The pulse-driven self-attention operation on the first pulse matrix at the block dimension includes: Perform element-wise masking operations on the key matrix K1 and the value matrix V1 to obtain the first operation result; The first calculation result is summed column by column to obtain the first summation result; The first summation result is converted into a pulse sequence to obtain the first pulse sequence; A row-by-row masking operation is performed on the query matrix Q1 and the first pulse sequence to obtain the first pulse output, which is used to determine the membrane potential feature map. The second pulse matrix includes a query matrix Q2, a key matrix K2, and a value matrix V2. The pulse-driven self-attention operation on the second pulse matrix in the channel dimension includes: For the matrix and the matrix Perform element-by-element masking to obtain the second result; The results of the second operation are summed column by column to obtain the second summation result; The second summation result is converted into a pulse sequence to obtain a second pulse sequence; For the matrix The second pulse sequence is subjected to a line-by-line masking operation to obtain the second pulse output, which is then transposed to determine the output membrane potential. The matrix ,matrix sum matrix These are the transposes of the query matrix Q2, key matrix K2, and value matrix V2, respectively.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the classification method of the full-addition impulse self-attention Transformer as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the classification method of the full additive impulse self-attention Transformer as described in any one of claims 1 to 5.

Images