Image classification processing method and electronic equipment

Abstract

The invention provides an image classification processing method and electronic equipment, and the method comprises the steps: obtaining an original Swin Transform model; training is carried out based on a preset QCFS activation function, and a target Swin Transform model is obtained after training is finished; according to a preset integral discharge neuron, an activation function in the target Swin Transform model is replaced, and a target image classification model is obtained; and inputting the to-be-classified image data into the target image classification model, and classifying the to-be-classified image data by the target image classification model to obtain a classification result of the to-be-classified image data. According to the method and the device, the high performance of the Transform and the low power consumption advantage of the SNN can be combined, so that the energy consumption is remarkably reduced while the classification accuracy of the static and neuromorphic data sets is improved, and the application scene of the SNN in low-power-consumption equipment is expanded.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to an image classification processing method and electronic device. Background Technology

[0002] Spiking Neural Networks (SNNs), with their event-driven, low-energy characteristics and biological plausibility, offer a promising solution for edge computing; however, their performance has long been limited by complex tasks. Meanwhile, Transformers, with their powerful self-attention mechanism, have achieved unprecedented high performance in fields such as computer vision.

[0003] Therefore, how to combine the high performance of Transformer with the low power consumption of SNN to build a new neural network architecture that combines the advantages of both has become a research hotspot in the current field of brain-inspired computing and artificial intelligence. Summary of the Invention

[0004] The purpose of this application is to provide an image classification processing method and electronic device to address the shortcomings of the prior art, thereby solving the problem that the prior art cannot combine the high performance of Transformer with the low power consumption advantage of SNN in the field of image processing.

[0005] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, one embodiment of this application provides an image classification processing method, the method comprising: Obtain the original Swing Transformer model; The original Swing Transformer model is trained based on a preset QCFS activation function, and a target Swing Transformer model is obtained after training. The QCFS activation function is different from the activation function of the original Swing Transformer model, and the weight parameters of the target Swing Transformer model are different from the weight parameters of the original Swing Transformer model. Based on the preset integral firing neurons, the activation function in the target Swing Transformer model is replaced to obtain the target image classification model. The weight parameters of the target image classification model are the same as the weight parameters of the target Swing Transformer model. The image data to be classified is input into the target image classification model, which then classifies the image data to be classified to obtain the classification result.

[0006] Secondly, another embodiment of this application provides an image classification processing apparatus, the apparatus comprising: The acquisition module is used to acquire the original Swing Transformer model; The training module is used to train the original Swing Transformer model based on a preset QCFS activation function, and obtain a target Swing Transformer model after training. The QCFS activation function is different from the activation function of the original Swing Transformer model, and the weight parameters of the target Swing Transformer model are different from the weight parameters of the original Swing Transformer model. The replacement module is used to replace the activation function in the target Swin Transformer model according to the preset integral firing neurons to obtain the target image classification model. The weight parameters of the target image classification model are the same as the weight parameters of the target Swin Transformer model. The inference module is used to input the image data to be classified into the target image classification model, and the target image classification model classifies the image data to be classified to obtain the classification result of the image data to be classified.

[0007] Thirdly, another embodiment of this application provides an electronic device, including: a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of any of the methods described in the first aspect above.

[0008] Fourthly, another embodiment of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of any of the methods described in the first aspect above.

[0009] The beneficial effects of this application are as follows: The original Swing Transformer model is trained using a pre-defined QCFS activation function, and a target Swing Transformer model is obtained after training. This allows the target Swing Transformer model to learn to express itself using impulses in advance. Based on pre-defined integral firing neurons, the activation function in the target Swing Transformer model is replaced to obtain a target image classification model. The output of the target image classification model becomes a discrete integer, corresponding to the number of impulse firings within T time steps in the SNN, thereby reducing information loss during subsequent transformations. This combines the high performance of the Transformer with the low power consumption advantage of the SNN, resulting in a target image classification model that combines the strengths of both. Furthermore, the image data to be classified is input into the target image classification model, which classifies the image data to obtain the classification result. This significantly reduces energy consumption while improving the classification accuracy of static and neuromorphic datasets, expanding the application scenarios of SNN in low-power devices. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic flowchart of an image classification processing method provided in an embodiment of this application; Figure 2 This is a schematic flowchart illustrating the process of obtaining a target image classification model in the image classification processing method provided in the embodiments of this application. Figure 3 This is a schematic diagram of the structure of a target image classification model in the image classification processing method provided in the embodiments of this application; Figure 4 This is a schematic flowchart illustrating the process of obtaining the classification result of image data to be classified in the image classification processing method provided in the embodiments of this application. Figure 5 This is a schematic diagram of another structure of the target image classification model in the image classification processing method provided in the embodiments of this application; Figure 6 A schematic flowchart illustrating the process of obtaining an output pulse in the image classification processing method provided in this application embodiment; Figure 7This is a schematic diagram of another structure of the target image classification model in the image classification processing method provided in the embodiments of this application; Figure 8 A schematic flowchart illustrating the process of obtaining a self-attention output pulse in the image classification processing method provided in this application embodiment; Figure 9 This is a schematic diagram of another structure of the target image classification model in the image classification processing method provided in the embodiments of this application; Figure 10 This is a schematic flowchart illustrating the process of obtaining a sensing output pulse in the image classification processing method provided in this embodiment of the application. Figure 11 This is a schematic diagram of another structure of the target image classification model in the image classification processing method provided in the embodiments of this application; Figure 12 This is a schematic flowchart illustrating the process of generating an input feature vector sequence in the image classification processing method provided in this application embodiment. Figure 13 This is a schematic diagram of the electronic device structure provided in an embodiment of this application. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0013] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0014] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0015] Spiking Neural Networks (SNNs), with their event-driven, low-energy characteristics and biological plausibility, offer a promising solution for edge computing; however, their performance has long been limited by complex tasks. Meanwhile, Transformers, with their powerful self-attention mechanism, have achieved unprecedented high performance in fields such as computer vision.

[0016] However, in existing technologies, when converting pre-trained Artificial Neural Networks (ANNs) into SNNs, they are limited to CNN architectures and cannot successfully convert the complete Transformer architecture (especially the self-attention mechanism) into SNNs while maintaining high performance.

[0017] Therefore, combining the high performance of Transformers with the low power consumption of SNNs to construct a novel neural network architecture that combines the strengths of both has become a research hotspot in the interdisciplinary field of neuromorphic computing and artificial intelligence. In other words, how to enable spiking neural networks (SNNs) to achieve top-tier performance like Transformers while maintaining the low power consumption advantage of SNNs themselves is a problem that urgently needs to be solved.

[0018] Based on the aforementioned problems, this application proposes an image classification processing method. The method trains the original Swing Transformer model using a preset QCFS activation function, and obtains a target Swing Transformer model after training. Then, based on preset integral firing neurons, the activation function in the target Swing Transformer model is replaced to obtain a target image classification model. This method combines the high performance of the Transformer with the low power consumption advantages of SNNs, resulting in a target image classification model that possesses the strengths of both. The image data to be classified is then input into the target image classification model, which classifies the image data to obtain the classification result. This significantly reduces energy consumption while improving the classification accuracy of static and neuromorphic datasets, expanding the application scenarios of SNNs in low-power devices.

[0019] It is understood that the image classification processing method provided in this application embodiment can be applied to any electronic device with processing capabilities, such as a low-power device. Specifically, a low-power device may include neuromorphic chips and edge computing devices. Edge computing devices may include, for example, mobile platforms such as mobile phones, smart cameras, wearable devices, IoT sensor nodes, drones, and robots.

[0020] The image classification processing method provided in this application will be described in detail below with reference to several embodiments.

[0021] Figure 1 This is a schematic flowchart of an image classification processing method provided in an embodiment of this application, referring to... Figure 1 As shown, the execution subject of this method is the aforementioned low-power device, and the method includes: S101. Obtain the original Swing Transformer model.

[0022] Alternatively, the original Swing Transformer model can be obtained.

[0023] The original Swing Transformer model refers to the Swing Transformer model that has not been trained.

[0024] Specifically, the Swing Transformer model is a computer vision model based on the Transformer architecture. Through a hierarchical construction method and a moving window attention mechanism, the Swing Transformer model addresses the problems of excessive computation and lack of local information modeling in traditional VisionTransformer models when processing high-resolution images. The activation function of the original Swing Transformer model is a standard activation function (e.g., GELU).

[0025] S102. Train the original Swing Transformer model based on the preset QCFS activation function, and obtain the target Swing Transformer model after training.

[0026] Optionally, after obtaining the original Swing Transformer model, the original Swing Transformer model is trained using a preset QCFS activation function, and the target Swing Transformer model is obtained after training.

[0027] Specifically, training the original Swing Transformer model using the preset QCFS activation function means replacing all activation functions of the original Swing Transformer model with QCFS activation functions, and reconstructing the weight parameters of the entire network of the original Swing Transformer model (including attention weights, MLP parameters, and positional encoding) through training, and obtaining the target Swing Transformer model after training.

[0028] The QCFS activation function differs from the activation function of the original Swing Transformer model, and the weight parameters of the target Swing Transformer model also differ from those of the original Swing Transformer.

[0029] Specifically, the quantization clip-floor-shift (QCFS) activation function is a differentiable impulse surrogate function used to simulate the discreteness of impulse firing during training (such as threshold triggering) while preserving gradient continuity during backpropagation. It can be understood using the following formula:

[0030] Where λ is the scaling factor, used for the quantization step size. L is the quantization level, used to maximize the output value. clip(·,0,L) is limited to the range of 0 to L.

[0031] The original Swing Transformer model is trained using a preset QCFS activation function, and a target Swing Transformer model is obtained after training. This allows the trained target Swing Transformer model to express itself using "discrete integers", so that when the integral firing neurons are replaced, the pulse firing rate of the SNN can accurately approximate the output of the ANN.

[0032] S103. Based on the preset integral firing neurons, replace the activation function in the target Swing Transformer model to obtain the target image classification model.

[0033] Optionally, after training the target Swing Transformer model, all QCFS activation functions in the target Swing Transformer model are replaced by preset integral firing neurons to obtain the target image classification model.

[0034] The weight parameters of the target image classification model are the same as those of the target Swin Transformer model. That is, when replacing all QCFS activation functions in the target Swin Transformer model, no attention weight matrix (Q / K / V), MLP linear layer weights, or positional encoding in the target Swin Transformer model are modified.

[0035] The pre-defined integral-and-fire neuron is the IF neuron, which is the basic computational unit of the spiking neural network, used to simulate biological neurons to accumulate input and fire pulses. The integral-and-fire neuron transforms all continuous numerical calculations within the Transformer into sparse pulse sequences consisting only of 0s and 1s through an "integration-firing-reset" mechanism.

[0036] Each integral firing neuron includes multiple sub-neurons, and the number of sub-neurons in an integral firing neuron is equal to the total number of all elements in the input matrix.

[0037] Specifically, the pre-defined integral firing neurons act in the self-attention mechanism to convert the Q, K, and V matrices into pulses and the attention scores into pulses. The pre-defined integral firing neurons also act in the MLP to convert the upgraded and downgraded data into pulses.

[0038] By replacing the activation function in the target Swin Transformer model with a pre-defined integral firing neuron, a target image classification model is obtained. This allows the real-valued weights of the ANN to regain semantic interpretability in the pulse-time domain, realizing a cross-domain mapping of ANN to SNN using QCFS as a bridge, weight freezing as a constraint, and integral firing as the physical carrier.

[0039] S104. Input the image data to be classified into the target image classification model. The target image classification model classifies the image data to be classified and obtains the classification result of the image data to be classified.

[0040] Optionally, after obtaining the target image classification model, the image data to be classified is input into the target image classification model, and the target image classification model classifies the image data to be classified to obtain the classification result of the image data to be classified.

[0041] The image data to be classified may include: image data from ordinary images and image data from event cameras.

[0042] For example, the input processing module in the target image classification model processes the image data to be classified according to the data source type, generates an input feature vector sequence, and performs classification based on the self-attention module in the target image classification model to obtain the classification result of the image data to be classified.

[0043] In this embodiment, the original Swing Transformer model is trained using a preset QCFS activation function, and a target Swing Transformer model is obtained after training. This allows the target Swing Transformer model to learn to express itself using impulses in advance. Based on preset integral firing neurons, the activation function in the target Swing Transformer model is replaced to obtain a target image classification model. The output of the target image classification model becomes a discrete integer, corresponding to the number of impulse firings within T time steps in the SNN, thereby reducing information loss during subsequent transformations. This combines the high performance of the Transformer with the low power consumption advantage of the SNN to obtain a target image classification model that combines the advantages of both. Then, the image data to be classified is input into the target image classification model, which classifies the image data to be classified and obtains the classification result. This improves the classification accuracy of static and neuromorphic datasets while significantly reducing energy consumption, expanding the application scenarios of SNN in low-power devices.

[0044] In one possible implementation, step S102 above trains the original SwinTransformer model based on a preset QCFS activation function, and obtains the target SwinTransformer model after training, including: The activation function of the original Swing Transformer model is replaced with the QCFS activation function, and the normalization layer of the original Swing Transformer model is replaced with a batch normalization layer to obtain an intermediate Swing Transformer model; the intermediate Swing Transformer model is then trained to obtain the target Swing Transformer model.

[0045] Optionally, the activation function of the original Swing Transformer model can be replaced with the QCFS activation function, and the normalization layer (LayerNorm, LN) of the original Swing Transformer model can be replaced with the batch normalization layer (BN) to obtain an intermediate Swing Transformer model.

[0046] Batch normalization is used to normalize the activation values ​​of each image block within the same batch, making them follow a similar distribution, which greatly improves the firing synchronization rate and stability of subsequent integral firing neurons.

[0047] Optionally, the intermediate Swin Transformer model is trained to obtain the target Swin Transformer model. Specifically, the weight parameters of the trained target Swin Transformer model are implicitly encoded with a new knowledge structure under the joint constraints of QCFS-BN.

[0048] By replacing the activation function of the original Swing Transformer model with the QCFS activation function, gradient compatibility can be resolved. Replacing the normalization layer of the original Swing Transformer model with a batch normalization layer can solve the stability of impulse input. Retraining solves the problem of knowledge transferability.

[0049] In one possible implementation, Figure 2 This is a schematic flowchart illustrating the process of obtaining a target image classification model in the image classification processing method provided in this application embodiment, with reference to... Figure 2 As shown, in S103 above, the activation function in the target Swin Transformer model is replaced according to the preset integral firing neurons to obtain the target image classification model, including: S201. Replace the QCFS activation function in the target Swing Transformer model with a preset integral firing neuron.

[0050] Optionally, all QCFS activation functions in the target Swing Transformer model can be replaced with preset integral firing neurons.

[0051] Specifically, when updating the state of an integrally firing neuron, the following formula can be used as a reference:

[0052] in, The membrane potential at the current moment. This represents the membrane potential at the previous moment. For newly received input, This refers to the membrane potential consumed during this release.

[0053] Specifically, when the integral firing neuron generates pulses, it can refer to the following formula:

[0054] in, This represents the current membrane potential. The preset threshold, It is a step function. If the current membrane potential Subtract threshold If the value is greater than 0, a pulse is emitted (s=1).

[0055] S202. After at least one integral firing neuron in the target Swing Transformer model, a preset random mask pulse processing unit is added to obtain the target image classification model.

[0056] Optionally, a preset random mask pulse processing unit is added after at least one integral firing neuron in the target Swing Transformer model to obtain the target image classification model.

[0057] The number and placement of random mask pulse processing units can be determined by at least one of the following: the computational bottleneck location in the target Swing Transformer model, the degree of information redundancy in the target Swing Transformer model, the hierarchical position of the feature processing module where the mask pulse operation module is located in the target Swing Transformer model, and the overall power consumption target of the target image classification model.

[0058] In one example, if the Q×K matrix multiplication of the self-attention module in the target Swing Transformer model is the bottleneck position, then a random masked impulse processing unit can be added after Q, K, and A of the self-attention module.

[0059] In another example, a preset random mask pulse processing unit is added after all the integral firing neurons of the self-attention module and all the integral firing neurons of the MLP module in the target Swing Transformer model to obtain the target image classification model, thereby achieving the lowest power consumption.

[0060] In another example, a preset random mask pulse processing unit is added after at least one integral firing neuron of the self-attention module and at least one integral firing neuron of the MLP module in the target Swing Transformer model to obtain the target image classification model, thereby achieving appropriate energy saving.

[0061] In another example, different mask rates can be configured for the random mask pulse processing unit according to the hierarchical position of the mask pulse operation module in the feature processing module in the target Swing Transformer model.

[0062] The preset random mask pulse processing unit is used to randomly discard pulses during inference, thereby reducing the power consumption of the SNN while maintaining the model performance.

[0063] For example, the random masked pulse processing unit can be implemented with reference to the following formula:

[0064] Where S is the original impulse matrix (0 / 1), M is a randomly generated binary mask matrix, each element in the binary mask matrix is ​​independently 1 with probability p, and ⊙ represents element-wise multiplication.

[0065] By replacing the QCFS activation function in the target Swing Transformer model with a preset integral firing neuron, the trained and powerful Swing Transformer model can be transformed into a spiking neural network (SNN) that can run on a neuromorphic chip. This inherits the high performance of the Swing Transformer model and achieves full pulsed processing. Furthermore, a preset random masked pulse processing unit is added after each integral firing neuron in the target Swing Transformer model to obtain the target image classification model. This approach can significantly reduce power consumption while maintaining performance, and simultaneously preserve the accuracy of the target image classification model while increasing sparsity.

[0066] The above provides an illustrative example of the process of obtaining a target image classification model. After obtaining the target image classification model, inference can be performed on the image data to be classified using the target image classification model to obtain the classification result of the image data to be classified. The following is a detailed explanation.

[0067] In one possible implementation, Figure 3 This is a schematic diagram of the structure of a target image classification model in the image classification processing method provided in the embodiments of this application. Figure 4 This is a flowchart illustrating the process of obtaining the classification result of the image data to be classified in the image classification processing method provided in this application embodiment, with reference to... Figure 3 as well as Figure 4 As shown, the target image classification model includes: an input processing module, multiple feature processing modules, and an output module connected in sequence. Each feature processing module includes: multiple mask pulse operation modules connected in sequence, and each mask pulse operation module includes multiple integral firing neurons and multiple random mask pulse processing units. In S104 above, the target image classification model classifies the image data to be classified, obtaining the classification result of the image data to be classified, including: S401. The input processing module processes the image data to be classified to generate an input feature vector sequence.

[0068] Optionally, the input processing module performs feature processing on the image data to be classified, processing the image data to be classified into an input feature vector sequence, thus obtaining the input feature vector sequence.

[0069] For example, the input processing module divides the image data to be classified into image blocks and performs linear layer embedding to obtain an input feature vector sequence.

[0070] The input processing module processes the image data to be classified, generating a sequence of input feature vectors. This can unify different data sources and reuse pre-trained weights.

[0071] S402. Each mask pulse operation module in each feature processing module performs feature transformation on the input feature vector sequence based on each integral firing neuron and each random mask pulse processing unit to obtain the output pulse.

[0072] Optionally, each mask pulse operation module in each feature processing module performs feature transformation on the input feature vector sequence in sequence based on each integral firing neuron and each random mask pulse processing unit to obtain the output pulse.

[0073] The target image classification model includes multiple feature processing modules, each consisting of stacked mask pulse operation modules. Downsampling layers connect these modules. These downsampling layers reduce resolution and increase the number of channels between each feature processing module, enabling hierarchical feature extraction.

[0074] Among them, the mask pulse operation module serves as the core computing unit, which is used to perform pulsed feature transformation on the input feature vector sequence based on integral firing neurons and random mask pulse processing units. The mask pulse operation module is implemented based on pulse self-attention and MLP.

[0075] By using the mask pulse operation modules in each feature processing module, based on each integral firing neuron and each random mask pulse processing unit, the input feature vector sequence is transformed to obtain the output pulse. This enables hierarchical feature learning, allowing the model to gradually build a global understanding from local details, while also gradually expanding the receptive field.

[0076] S403. The output module obtains the classification result of the image data to be classified based on the output pulses corresponding to each mask pulse operation module.

[0077] Optionally, the output module includes a classification head, which is used to convert the output pulses corresponding to each mask pulse operation module into category predictions to obtain the classification results of the image data to be classified.

[0078] Specifically, the classification head includes a global average pooling layer, a fully connected layer, multiple IF neurons, and a decision layer. The global pooling layer averages the output pulses from each mask pulse operation module into a vector. The fully connected layer maps this vector to the category score space. Each category corresponds to an IF neuron. Pulses are accumulated within a preset time step. The decision layer takes the category with the most pulses as the classification result.

[0079] The classification result of the image data to be classified is obtained by the output module based on the output pulses corresponding to each mask pulse operation module. It can take advantage of the temporal characteristics of SNN to trade time for accuracy, and also improve the noise resistance.

[0080] In one possible implementation, Figure 5 This is a schematic diagram of another structure of the target image classification model in the image classification processing method provided in the embodiments of this application. Figure 6 This is a schematic flowchart illustrating the process of obtaining output pulses in the image classification processing method provided in this application embodiment, with reference to... Figure 5 as well as Figure 6 As shown, each mask pulse operation module includes: a first batch of normalization layers, a pulse self-attention module, a second batch of normalization layers, and a pulse perceptron module connected in sequence; the pulse self-attention module includes: an integral firing neuron and a random mask pulse processing unit. In S402 above, each mask pulse operation module in each feature processing module performs feature transformation on the input feature vector sequence based on each integral firing neuron and each random mask pulse processing unit to obtain the output pulse, including: S601. The first batch of normalization layers adjusts the distribution of the input pulses to obtain normalized input pulses.

[0081] It can be understood that the input to the first mask pulse operation module is the input feature vector sequence, and the input to each subsequent mask pulse operation module is the output pulse of the previous mask pulse operation module. In the first mask pulse operation module, an integral firing neuron can be added before the first batch of normalization layers to convert the input feature vector sequence into input pulses.

[0082] Optionally, the first batch of normalization layers adjusts the distribution of the input pulses to obtain normalized input pulses. Specifically, the first batch of normalization layers normalizes the input pulses to obtain normalized input pulses, thereby ensuring that the network maintains distribution alignment after transformation, achieving stable training and transformation.

[0083] The numerical range of the input pulses is a floating-point range suitable for processing by IF neurons.

[0084] S602. The integral firing neurons and random mask pulse processing unit in the pulse self-attention module perform pulse self-attention operation and random mask processing on the normalized input pulse to obtain the self-attention output pulse.

[0085] Optionally, the pulse self-attention module uses an integral firing neuron and a random mask pulse processing unit to perform pulse self-attention operation and random mask processing on the normalized input pulse to obtain the self-attention output pulse.

[0086] Pulse self-attention refers to enabling information exchange between tokens, allowing each token to pay attention to other tokens in the sequence. Random masking involves randomly discarding pulses to reduce power consumption.

[0087] S603. The normalized input pulse is residually connected with the self-attention output pulse to obtain the intermediate pulse.

[0088] Optionally, the normalized input pulse and the self-attention output pulse are residually connected to obtain an intermediate pulse.

[0089] S604. The distribution of the intermediate pulses is adjusted by the second batch of normalization layers to obtain the normalized intermediate pulses.

[0090] Optionally, the intermediate pulses are normalized by the second batch of normalization layers to obtain normalized intermediate pulses, thereby ensuring that the network maintains distribution alignment after transformation and achieving stable training and transformation.

[0091] The normalized intermediate pulse values ​​are within the range of floating-point numbers suitable for processing by IF neurons.

[0092] S605. The pulse sensor module performs pulse characteristic transformation on the normalized intermediate pulse to obtain the sensing output pulse.

[0093] Optionally, the pulse sensor module (MLP) performs an intra-token nonlinear transformation on the normalized intermediate pulse to obtain the sensing output pulse, thereby enhancing the model's expressive power.

[0094] S606. Perform residual connection between the self-attention output pulse and the perception output pulse to obtain the output pulse.

[0095] Optionally, the self-attention output pulse and the perception output pulse are residually connected to obtain the output pulse.

[0096] By sequentially connecting the first batch of normalization layers, the impulse self-attention module, the second batch of normalization layers, and the impulse perceptron module within the masked impulse operation module, information fusion can be achieved through information interaction between tokens via the self-attention module, and feature transformation within tokens via the impulse perceptron module, enhancing expressive power. The combination of the self-attention module and the impulse perceptron module considers both the relationships between tokens and the characteristics of each token itself. Simultaneously, it achieves a balance in computational complexity, resulting in a more even distribution of computational load across the entire network.

[0097] Furthermore, residual connections allow gradients to flow directly through deep networks, mitigating gradient vanishing. In addition, the sparsity of pulses in spiking networks can lead to information loss. Through residual connections, even if no pulses are generated at a certain time step by self-attention or MLP, the original information can still be passed through the residuals, ensuring the stability and robustness of the model.

[0098] In one possible implementation, Figure 7 This is a schematic diagram of another structure of the target image classification model in the image classification processing method provided in the embodiments of this application. Figure 8 This is a schematic flowchart illustrating the process of obtaining a self-attention output pulse in the image classification processing method provided in this application embodiment, with reference to... Figure 7 as well as Figure 8 As shown, the pulse self-attention module includes: a first linear layer, a first integral firing neuron, a first random mask pulse processing unit, an attention calculation unit, a second integral firing neuron, a second random mask pulse processing unit, and a third integral firing neuron connected in sequence. In S602 above, the integral firing neurons and random mask pulse processing units in the pulse self-attention module perform pulse self-attention operations and random mask processing on the normalized input pulse to obtain a self-attention output pulse, including: S801. The normalized input pulses are projected by the first linear layer onto the query space, key space, and value space to obtain the query matrix, key matrix, and value matrix.

[0099] Optionally, the normalized input pulses are projected into the query space, key space, and value space by the first linear layer to obtain the query matrix, key matrix, and value matrix.

[0100] S802, The first integral discharge neuron converts the query matrix, key matrix, and value matrix into a query pulse matrix, a key pulse matrix, and a value pulse matrix.

[0101] Optionally, the first integral discharge neuron performs pulse conversion on the query matrix, key matrix, and value matrix, converting floating-point numbers into pulses to obtain the query pulse matrix, key pulse matrix, and value pulse matrix.

[0102] S803. The first random mask pulse processing unit performs random mask processing on the query pulse matrix and the key pulse matrix to obtain the trimmed query pulse matrix and the trimmed key pulse matrix.

[0103] Optionally, the first random mask pulse processing unit performs random masking processing on the query pulse matrix and the key pulse matrix, randomly discarding pulses in the query pulse matrix and the key pulse matrix to obtain the trimmed query pulse matrix and the trimmed key pulse matrix.

[0104] S804. The attention calculation unit calculates the attention score based on the pruned query impulse matrix and the pruned key impulse matrix.

[0105] Optionally, the attention calculation unit calculates the dot product of the pruned query impulse matrix and the pruned key impulse matrix to obtain the attention score. The attention score can be a floating-point number or an integer.

[0106] For example, the dot product of the pruned query impulse matrix and the pruned key impulse matrix can be calculated, and the resulting dot product can be scaled according to a preset feature dimension to obtain an attention score.

[0107] S805: The second integral firing neuron converts the attention score into an attention pulse matrix.

[0108] Optionally, the attention score is converted into pulses by the second integral firing neuron to obtain the attention pulse matrix.

[0109] S806. The second random mask pulse processing unit performs random mask processing on the attention pulse matrix to obtain the trimmed attention pulse matrix.

[0110] Optionally, the second random mask pulse processing unit performs random masking processing on the attention pulse matrix, randomly discarding pulses in the attention scores to obtain the trimmed attention pulse matrix.

[0111] S807, the product of the calculated value impulse matrix and the pruned attention impulse matrix, yields the attention weighting result.

[0112] Optionally, the attention weighting result is obtained by multiplying the calculated value impulse matrix and the pruned attention impulse matrix according to preset weight coefficients. The attention weighting result is either a floating-point number or an integer.

[0113] S808: The third integral firing neuron converts the attention-weighted result into a self-attention output pulse.

[0114] Optionally, the attention-weighted result is converted into pulses by the third integral firing neuron to obtain self-attention output pulses.

[0115] Optionally, the attention weighting result is converted into pulses by the third integral firing neuron. After obtaining the attention pulse result, the attention pulse result can be randomly masked by the third random mask pulse processing unit, and the pulses in the attention pulse result can be randomly discarded to obtain the self-attention output pulse.

[0116] By sequentially connecting a first linear layer, a first integral firing neuron, a first random masked pulse processing unit, an attention calculation unit, a second integral firing neuron, and a second random masked pulse processing unit within the pulsed self-attention module, the entire self-attention module's input is a pulse, each step maintains a pulse form, and the output is also a pulse, achieving true full-pulse self-attention that can run directly on neuromorphic chips. Furthermore, the sparsity of pulses can significantly reduce computational load. Simultaneously, the first and second random masked pulse processing units also achieve lossless energy savings.

[0117] In one possible implementation, Figure 9 This is a schematic diagram of another structure of the target image classification model in the image classification processing method provided in the embodiments of this application. Figure 10 This is a schematic flowchart illustrating the process of obtaining the sensing output pulse in the image classification processing method provided in this application embodiment, with reference to... Figure 9 as well as Figure 10 As shown, the pulse sensor module includes: a third linear layer, a third batch normalization layer, a fourth integral firing neuron, a fourth random mask pulse processing unit, a fourth linear layer, a fourth batch normalization layer, a fifth integral firing neuron, and a fifth random mask pulse processing unit connected in sequence; in S605 above, the pulse sensor module performs pulse feature transformation on the normalized intermediate pulse to obtain the sensing output pulse, including: S1001. The normalized intermediate pulses are upgraded by the third linear layer to obtain the upgraded matrix.

[0118] Optionally, the normalized intermediate pulses are up-dimensioned by the third linear layer to obtain an up-dimension matrix, thereby increasing the representation space. The up-dimension matrix can be a floating-point matrix.

[0119] For example, the normalized intermediate pulses are upgraded by the third linear layer, from dimension D to 4D, to obtain the upgraded matrix.

[0120] S1002. The third batch of normalization layers normalizes the upgraded matrix to obtain the normalized upgraded matrix.

[0121] Optionally, the third batch of normalization layers normalizes the upgraded matrix to obtain a normalized upgraded matrix, thereby ensuring a stable distribution.

[0122] S1003. The normalized up-dimensional matrix is ​​converted into an up-dimensional pulse matrix by the fourth integral firing neuron.

[0123] Optionally, the normalized up-dimensional matrix is ​​converted into pulses by the fourth integral firing neuron to obtain an up-dimensional pulse matrix.

[0124] S1004. The fourth random mask pulse processing unit performs random mask processing on the upgraded pulse matrix to obtain the trimmed upgraded pulse matrix.

[0125] Optionally, the fourth random mask pulse processing unit performs random masking processing on the upgraded pulse matrix to obtain a trimmed upgraded pulse matrix, thereby randomly discarding pulses and reducing the amount of subsequent calculations.

[0126] S1005. The fourth linear layer performs dimensionality reduction on the pruned up-dimensional pulse matrix to obtain the dimensionality-reduced matrix.

[0127] Optionally, the pruned up-dimensional impulse matrix can be reduced in dimension by the fourth linear layer to obtain a reduced-dimensional matrix.

[0128] For example, the pruned, higher-dimensional pulse matrix is ​​reduced in dimension from 4D to D to obtain a reduced-dimensional matrix.

[0129] S1006. The fourth batch of normalization layers normalizes the dimension reduction matrix to obtain the normalized dimension reduction matrix.

[0130] Optionally, the fourth batch of normalization layers normalizes the dimensionality reduction matrix to obtain a normalized dimensionality reduction matrix, thereby ensuring a stable distribution.

[0131] S1007. The normalized dimensionality-reduced matrix is ​​converted into a dimensionality-reduced pulse matrix by the fifth integral firing neuron.

[0132] Optionally, the normalized dimensionality-reduced matrix is ​​converted into pulses by the fifth integral firing neuron to obtain a dimensionality-reduced pulse matrix.

[0133] S1008. The fifth random mask pulse processing unit performs random mask processing on the dimension-reduced pulse matrix to obtain the trimmed dimension-reduced pulse matrix, which is used as the sensing output pulse.

[0134] Optionally, the fifth random mask pulse processing unit performs random mask processing on the dimension-reduced pulse matrix to obtain the trimmed dimension-reduced pulse matrix, which is used as the sensing output pulse, thereby randomly discarding pulses and reducing the subsequent computational load.

[0135] By sequentially connecting a third linear layer, a third batch normalization layer, a fourth integral firing neuron, a fourth random masked pulse processing unit, a fourth linear layer, a fourth batch normalization layer, a fifth integral firing neuron, and a fifth random masked pulse processing unit in the pulse perceptron module, linearly separable features are more easily found in the 4D high-dimensional space, providing greater expressive freedom. This allows the model to learn more complex transformations and ultimately reduce the dimensionality back to D, maintaining consistency with the self-attention module's dimensionality and facilitating residual connections. Furthermore, the information in the reduced D-dimensional space is more compact. The higher-dimensional space (4D) has more redundancy, making it more suitable for the random masked pulse processing unit to perform energy-saving functions. In addition, the random masked pulse processing unit not only saves computation in the current pulse perceptron module but also indirectly saves energy for the next layer of the pulse perceptron module by making the input sparser.

[0136] In one possible implementation, step S401 above involves the input processing module processing the image data to be classified to generate an input feature vector sequence, including: The input processing module processes the image data to be classified according to the data source type, and generates an input feature vector sequence.

[0137] In one example, when the data source is ordinary image data to be classified, the input processing module includes an image patch partitioning layer and a linear embedding layer. The image patch partitioning layer divides the ordinary image into fixed-size blocks to obtain an image patch sequence, and the linear embedding layer maps each image patch in the image patch sequence to the feature space to obtain an input feature vector sequence.

[0138] In one possible implementation, Figure 11 This is a schematic diagram of another structure of the target image classification model in the image classification processing method provided in the embodiments of this application. Figure 12 This is a flowchart illustrating the process of generating an input feature vector sequence in the image classification processing method provided in this application embodiment, with reference to... Figure 11 as well as Figure 12 As shown, the input processing module includes: a reduction layer, an image patching layer, and a linear embedding layer connected in sequence; the above steps are performed by the input processing module according to the data source type of the image data to be classified, processing the image data to be classified to generate an input feature vector sequence, including: S1201. If the data source type of the image data to be classified is an event camera, the reduction layer generates a target tensor based on the image data to be classified.

[0139] Optionally, if the data source type of the image data to be classified is an event camera, then the image data to be classified is a raw event stream. In this case, the raw event stream can be converted into a frame sequence and then converted into synchronous frame data. The reduction layer can then convert the synchronous frame data into a three-channel target tensor by converting the number of channels.

[0140] The reduced layer can be a 1×1 convolutional layer. By using the reduced layer, only the number of channels is changed, without changing the spatial resolution.

[0141] S1202. Divide the target tensor into layers by dividing the image blocks to obtain the image block sequence.

[0142] Optionally, the target tensor is divided into fixed-size blocks by dividing the image blocks into layers, resulting in an image block sequence.

[0143] S1203. The image patch sequence is projected by the linear embedding layer to obtain the input feature vector sequence.

[0144] Optionally, a linear embedding layer maps each image patch in the image patch sequence to a feature space to obtain an input feature vector sequence.

[0145] For image data to be classified from event cameras, a reduction layer is used to transform the input tensor with n×2 channels into an output tensor with 3 channels, making the image data to be classified from event cameras consistent with the input format of ordinary images. This allows for the reuse of large-scale pre-trained weights, accelerates training convergence, and significantly improves the final classification accuracy.

[0146] Based on the same inventive concept, this application also provides an image classification processing device corresponding to the image classification processing method. Since the principle of the device in this application is similar to the image classification processing method described above in this application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0147] The image classification processing apparatus provided in this application includes: an acquisition module, a training module, a replacement module, and an inference module; wherein... The acquisition module is used to acquire the original Swing Transformer model; The training module is used to train the original Swing Transformer model based on the preset QCFS activation function and obtain the target Swing Transformer model after training. The QCFS activation function is different from the activation function of the original Swing Transformer model, and the weight parameters of the target Swing Transformer model are different from the weight parameters of the original Swing Transformer. The replacement module is used to replace the activation function in the target Swing Transformer model according to the preset integral firing neurons, so as to obtain the target image classification model. The weight parameters of the target image classification model are the same as the weight parameters of the target Swing Transformer model. The inference module is used to input the image data to be classified into the target image classification model, which then classifies the image data to be classified and obtains the classification result.

[0148] Optionally, the training module is specifically used for: The activation function of the original Swing Transformer model is replaced with the QCFS activation function, and the normalization layer of the original Swing Transformer model is replaced with a batch normalization layer to obtain an intermediate Swing Transformer model. The intermediate Swing Transformer model is trained to obtain the target Swing Transformer model.

[0149] Optionally, the replacement module is used specifically for: Replace the QCFS activation function in the target Swing Transformer model with a preset integral firing neuron; After at least one integral firing neuron in the target Swing Transformer model, a preset random mask pulse processing unit is added to obtain the target image classification model.

[0150] Optionally, the target image classification model includes: an input processing module, multiple feature processing modules, and an output module connected in sequence. Each feature processing module includes: multiple mask pulse operation modules connected in sequence, each mask pulse operation module including multiple integral firing neurons and multiple random mask pulse processing units; and an inference module, specifically used for: The input processing module processes the image data to be classified, generating an input feature vector sequence; Each mask pulse operation module in each feature processing module performs feature transformation on the input feature vector sequence based on each integral discharge nerve and each random mask pulse processing unit to obtain the output pulse. The output module obtains the classification result of the image data to be classified based on the output pulses corresponding to each mask pulse operation module.

[0151] Optionally, each masked pulse processing module includes: a first batch of normalization layers, a pulse self-attention module, a second batch of normalization layers, and a pulse perceptron module connected in sequence; the pulse self-attention module includes: an integral firing neuron and a random masked pulse processing unit; the inference module is specifically used to: adjust the distribution of the input pulse by the first batch of normalization layers to obtain the normalized input pulse; The pulse self-attention module uses integral firing neurons and random mask pulse processing units to perform pulse self-attention operations and random mask processing on the normalized input pulse to obtain the self-attention output pulse. The normalized input pulse is residually connected to the self-attention output pulse to obtain the intermediate pulse; The distribution of the intermediate pulses is adjusted by the second batch of normalization layers to obtain the normalized intermediate pulses; The pulse sensor module performs pulse characteristic transformation on the normalized intermediate pulse to obtain the sensing output pulse; The output pulse is obtained by residually connecting the self-attention output pulse and the perception output pulse.

[0152] Optionally, the pulse self-attention module includes: a first linear layer, a first integral firing neuron, a first random mask pulse processing unit, an attention calculation unit, a second integral firing neuron, a second random mask pulse processing unit, and a third integral firing neuron connected in sequence; the inference module is specifically used for: The normalized input pulses are projected onto the query space, key space, and value space by the first linear layer to obtain the query matrix, key matrix, and value matrix. The first integral firing neuron converts the query matrix and the key matrix into a query pulse matrix, a key pulse matrix, and a value pulse matrix; The first random mask pulse processing unit performs random mask processing on the query pulse matrix and the key pulse matrix to obtain the trimmed query pulse matrix and the trimmed key pulse matrix. The attention calculation unit calculates the attention score based on the pruned query impulse matrix and the pruned key impulse matrix. The second integral firing neuron converts the attention score into an attention impulse matrix; The second random mask pulse processing unit performs random mask processing on the attention pulse matrix to obtain the trimmed attention pulse matrix; The product of the calculated value impulse matrix and the pruned attention impulse matrix is ​​used to obtain the attention weighting result; The third integral firing neuron converts the attention-weighted result into a self-attention output pulse.

[0153] Optionally, the pulse perceptron module includes: a third linear layer, a third batch normalization layer, a fourth integral firing neuron, a fourth random mask pulse processing unit, a fourth linear layer, a fourth batch normalization layer, a fifth integral firing neuron, and a fifth random mask pulse processing unit connected in sequence; the inference module is specifically used for: The normalized intermediate pulses are upgraded by the third linear layer to obtain the upgraded matrix; The third batch of normalization layers normalizes the upgraded matrix, resulting in a normalized upgraded matrix. The normalized up-dimensional matrix is ​​converted into an up-dimensional pulse matrix by the fourth integral firing neuron; The fourth random mask pulse processing unit performs random masking processing on the upgraded pulse matrix to obtain the trimmed upgraded pulse matrix. The fourth linear layer performs dimensionality reduction on the pruned, increased-dimensional impulse matrix to obtain the reduced-dimensional matrix. The fourth batch of normalization layers normalizes the dimension reduction matrix to obtain the normalized dimension reduction matrix; The normalized dimensionality-reduced matrix is ​​converted into a dimensionality-reduced pulse matrix by the fifth integral firing neuron; The fifth random mask pulse processing unit performs random masking on the dimension-reduced pulse matrix to obtain the trimmed dimension-reduced pulse matrix, which is used as the sensing output pulse.

[0154] Optionally, the inference module is specifically used to: process the image data to be classified according to the data source type of the image data to be classified by the input processing module, and generate an input feature vector sequence.

[0155] Optionally, the input processing module includes: a reduction layer, an image patching layer, and a linear embedding layer connected in sequence; the inference module is specifically used for: If the data source of the image data to be classified is an event camera, the reduction layer generates a target tensor based on the image data to be classified; The target tensor is divided into image patch sequences by dividing the image patch layer; The input feature vector sequence is obtained by projecting the image patch sequence onto the linear embedding layer.

[0156] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0157] This application also provides an electronic device, such as... Figure 13 As shown, Figure 13The schematic diagram of the electronic device structure provided in this application embodiment includes: a processor 1301 and a memory 1302, and optionally, a bus 1303. The memory 1302 stores machine-readable instructions executable by the processor 1301. When the electronic device is running, the processor 1301 and the memory 1302 communicate via the bus 1303, and the processor 1301 executes the machine-readable instructions to perform the steps of the above-described image classification processing method.

[0158] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the above-described image classification processing method.

[0159] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0160] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0161] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. An image classification processing method, characterized in that, include: Obtain the original Swing Transformer model; The original Swing Transformer model is trained based on a preset QCFS activation function, and a target Swing Transformer model is obtained after training. The QCFS activation function is different from the activation function of the original Swing Transformer model, and the weight parameters of the target Swing Transformer model are different from the weight parameters of the original Swing Transformer model. Based on the preset integral firing neurons, the activation function in the target Swing Transformer model is replaced to obtain the target image classification model. The weight parameters of the target image classification model are the same as the weight parameters of the target Swing Transformer model. The image data to be classified is input into the target image classification model, which then classifies the image data to be classified to obtain the classification result.

2. The image classification processing method according to claim 1, characterized in that, The process of training the original Swing Transformer model based on the preset QCFS activation function, and obtaining the target Swing Transformer model after training, includes: The activation function of the original Swing Transformer model is replaced with the QCFS activation function, and the normalization layer of the original Swing Transformer model is replaced with a batch normalization layer to obtain an intermediate Swing Transformer model. The intermediate Swing Transformer model is trained to obtain the target Swing Transformer model.

3. The image classification processing method according to claim 1, characterized in that, The step of replacing the activation function in the target Swin Transformer model according to the preset integral firing neurons to obtain the target image classification model includes: Replace the QCFS activation function in the target Swing Transformer model with a preset integral firing neuron; After at least one integral firing neuron in the target Swing Transformer model, a preset random mask pulse processing unit is added to obtain the target image classification model.

4. The image classification processing method according to claim 3, characterized in that, The target image classification model includes: an input processing module, multiple feature processing modules, and an output module connected in sequence. Each feature processing module includes: multiple mask pulse operation modules connected in sequence. Each mask pulse operation module includes multiple integral firing neurons and multiple random mask pulse processing units. The process of classifying the image data to be classified by the target image classification model to obtain the classification result of the image data to be classified includes: The input processing module processes the image data to be classified to generate an input feature vector sequence; Each of the mask pulse operation modules in each feature processing module performs feature transformation on the input feature vector sequence based on each of the integral discharge neural networks and each of the random mask pulse processing units to obtain the output pulse; The output module obtains the classification result of the image data to be classified based on the output pulses corresponding to each of the mask pulse operation modules.

5. The image classification processing method according to claim 4, characterized in that, Each of the masked pulse processing modules includes: a first batch of normalization layers, a pulse self-attention module, a second batch of normalization layers, and a pulse perceptron module connected in sequence; the pulse self-attention module includes: the integral firing neuron and the random masked pulse processing unit; The step of performing feature transformation on the input feature vector sequence by each of the mask pulse operation modules in each feature processing module, based on each of the integral discharge neural network and each of the random mask pulse processing units, to obtain the output pulse includes: The first batch of normalization layers adjusts the distribution of the input pulses to obtain normalized input pulses; The integral firing neurons and the random mask pulse processing unit in the pulse self-attention module perform pulse self-attention operation and random mask processing on the normalized input pulse to obtain the self-attention output pulse. The normalized input pulse is residually connected to the self-attention output pulse to obtain an intermediate pulse; The distribution of the intermediate pulses is adjusted by the second batch of normalization layers to obtain the normalized intermediate pulses; The pulse sensor module performs pulse feature transformation on the normalized intermediate pulse to obtain the sensing output pulse. The self-attention output pulse and the perception output pulse are residually connected to obtain the output pulse.

6. The image classification processing method according to claim 5, characterized in that, The pulse self-attention module includes: a first linear layer, a first integral firing neuron, a first random mask pulse processing unit, an attention calculation unit, a second integral firing neuron, a second random mask pulse processing unit, and a third integral firing neuron connected in sequence. The process of performing pulse self-attention operation and random masking processing on the normalized input pulse by the integral firing neurons and the random mask pulse processing unit in the pulse self-attention module to obtain a self-attention output pulse includes: The normalized input pulse is projected onto the query space, key space, and value space by the first linear layer to obtain the query matrix, key matrix, and value matrix. The first integral firing neuron converts the query matrix and the key matrix into a query pulse matrix, a key pulse matrix, and a value pulse matrix; The first random mask pulse processing unit performs random mask processing on the query pulse matrix and the key pulse matrix to obtain the trimmed query pulse matrix and the trimmed key pulse matrix. The attention calculation unit calculates the attention score based on the pruned query impulse matrix and the pruned key impulse matrix. The second integral firing neuron converts the attention score into an attention impulse matrix; The second random mask pulse processing unit performs random mask processing on the attention pulse matrix to obtain the trimmed attention pulse matrix; The product of the value impulse matrix and the pruned attention impulse matrix is ​​calculated to obtain the attention weighting result; The third integral firing neuron converts the attention weighting result into a self-attention output pulse.

7. The image classification processing method according to claim 5, characterized in that, The pulse sensor module includes: a third linear layer, a third batch normalization layer, a fourth integral firing neuron, a fourth random mask pulse processing unit, a fourth linear layer, a fourth batch normalization layer, a fifth integral firing neuron, and a fifth random mask pulse processing unit connected in sequence. The step of performing pulse feature transformation on the normalized intermediate pulse by the pulse sensor module to obtain the sensing output pulse includes: The normalized intermediate pulses are subjected to dimensionality-upgrading processing by the third linear layer to obtain an upgraded matrix; The third batch of normalization layers is used to normalize the increased dimension matrix to obtain the normalized increased dimension matrix; The normalized up-dimensional matrix is ​​converted into an up-dimensional pulse matrix by the fourth integral firing neuron; The fourth random mask pulse processing unit performs random mask processing on the upgraded pulse matrix to obtain the trimmed upgraded pulse matrix; The fourth linear layer performs dimensionality reduction processing on the pruned, higher-dimensional pulse matrix to obtain a lower-dimensional matrix; The fourth batch of normalization layers is used to normalize the dimensionality reduction matrix to obtain the normalized dimensionality reduction matrix; The fifth integral firing neuron converts the normalized dimensionality-reduced matrix into a dimensionality-reduced pulse matrix; The fifth random mask pulse processing unit performs random mask processing on the dimensionality-reduced pulse matrix to obtain a trimmed dimensionality-reduced pulse matrix, which is used as the sensing output pulse.

8. The image classification processing method according to claim 4, characterized in that, The step of processing the image data to be classified by the input processing module to generate an input feature vector sequence includes: The input processing module processes the image data to be classified according to the data source type of the image data to be classified, and generates an input feature vector sequence.

9. The image classification processing method according to claim 8, characterized in that, The input processing module includes: a reduction layer, an image block segmentation layer, and a linear embedding layer connected in sequence; The step of the input processing module processing the image data to be classified according to the data source type of the image data to be classified, and generating an input feature vector sequence, includes: If the data source type of the image data to be classified is an event camera, the reduction layer generates a target tensor based on the image data to be classified; The target tensor is divided into image patch sequences by the image patch partitioning layer. The input feature vector sequence is obtained by projecting features onto the image patch sequence using the linear embedding layer.

10. An electronic device, characterized in that, include: The processor and memory, the memory storing machine-readable instructions executable by the processor, which, when the electronic device is running, are executed by the processor to perform the steps of the image classification processing method as described in any one of claims 1 to 9.

Images