Training method of image classification model, image classification method, device and electronic equipment

Abstract

The present disclosure provides a training method of an image classification model, an image classification method, an image classification device and an electronic device, which can be applied to the field of image classification and the field of object recognition. The training method comprises: obtaining a sample image and label data; inputting the sample image into a convolution unit to generate an image feature matrix; converting elements in the image feature matrix into a pulse matrix corresponding to the image feature matrix by using a pulse unit, wherein the pulse convolution layer comprises the convolution unit and the pulse unit; inputting the pulse matrix and a position encoding matrix into a pulse self-attention layer to output a down-sampled image corresponding to the sample image; performing image classification according to the down-sampled image to obtain an image classification result; and training the image classification model according to the image classification result and the label data to obtain a trained image classification model.

Description

Technical Field

[0001] This disclosure relates to the fields of image classification and object recognition, and specifically to a training method for an image classification model, an image classification method, an apparatus, and an electronic device. Background Technology

[0002] With the rapid development of computing, network models based on self-attention mechanisms have achieved remarkable success in the field of deep learning. Models such as Transformer and ViT (Vision Transformer) have successfully handled the dependencies between different positions in the input sequence by introducing self-attention, and maintained the positional information of each token through positional encoding. The inventors discovered that although ViT has achieved great success in image classification, its large number of parameters and high computational complexity make it relatively resource-intensive during inference, thus making deployment on edge devices challenging. Summary of the Invention

[0003] In view of the above problems, this disclosure provides a training method for an image classification model, an image classification method, an apparatus, and an electronic device.

[0004] According to a first aspect of this disclosure, a method for training an image classification model is provided, wherein the image classification model includes at least an image downsampling network, the image downsampling network including a pulse convolutional layer, a position encoding matrix, and a pulse self-attention layer, comprising:

[0005] Acquire sample images and label data;

[0006] The above sample images are input into the convolution unit to generate an image feature matrix;

[0007] The elements in the image feature matrix are pulse-transformed using pulse units to generate a pulse matrix corresponding to the image feature matrix. The pulse convolutional layer includes the convolutional unit and the pulse unit.

[0008] The pulse matrix and the position encoding matrix are input into the pulse self-attention layer, and the output is a downsampled image corresponding to the sample image.

[0009] Based on the downsampled images described above, image classification is performed to obtain the image classification results;

[0010] Based on the above image classification results and the above label data, an image classification model is trained to obtain the trained image classification model.

[0011] According to embodiments of this disclosure, pulse units are used to pulse transform the elements in the image feature matrix to generate a pulse matrix corresponding to the convolution matrix, including:

[0012] The elements of the image feature matrix obtained in stage t are updated based on the intermediate pulse matrix of stage t-1 to obtain the membrane potential matrix of stage t. The membrane potential matrix of stage t includes the membrane potential elements.

[0013] The t-th membrane potential matrix and the preset membrane potential threshold are processed by the step function to obtain the t-th pulse update parameters.

[0014] The intermediate pulse matrix of the t-th pulse is determined based on the above-mentioned pulse-t update parameters and the above-mentioned membrane potential matrix of the t-th pulse, where t > 1 and t is an integer; and

[0015] When T = t, the Tth intermediate pulse matrix is ​​determined as the above pulse matrix, where T ≥ t.

[0016] According to embodiments of this disclosure, the pulse matrix and the position encoding matrix are input into a pulse self-attention layer, and a downsampled image corresponding to the sample image is output, including:

[0017] By fusing the above pulse matrix and the above position encoding matrix, the position pulse matrix is ​​obtained;

[0018] The above position pulse matrix is ​​input into the pulse self-attention unit to obtain the pulse attention matrix;

[0019] The position pulse matrix and the pulse attention matrix are fused to obtain the first fusion matrix;

[0020] The first fusion matrix is ​​input to the impulse perceptron unit to obtain the impulse perceptron matrix, wherein the impulse self-attention layer includes the impulse self-attention unit and the impulse perceptron unit.

[0021] The downsampled image is obtained based on the pulse sensor matrix and the first fusion matrix described above.

[0022] According to embodiments of this disclosure, the above-mentioned position pulse matrix is ​​input to a pulse self-attention unit to obtain a pulse attention matrix, including:

[0023] The query matrix, key matrix, and value matrix are determined based on the aforementioned position pulse matrix.

[0024] The query matrix, key matrix, and value matrix are fused based on the attention mechanism to obtain the attention fusion matrix;

[0025] The attention fusion matrix described above is input into the pulse sub-unit to obtain the pulse fusion matrix;

[0026] The pulse fusion matrix is ​​input into the linear pulse unit to obtain the pulse attention matrix, wherein the pulse self-attention unit includes the linear pulse unit and the pulse sub-unit.

[0027] According to embodiments of this disclosure, the first fusion matrix described above is input to the pulse sensor unit to obtain a pulse sensor matrix, including:

[0028] The first fusion matrix is ​​input into the first linear pulse unit to obtain the first linear fusion matrix;

[0029] The first linear fusion matrix is ​​input into the second linear pulse unit to obtain the pulse sensor matrix, wherein the pulse sensor unit includes multiple linear pulse sub-units.

[0030] According to embodiments of this disclosure, the image downsampling network comprises N:

[0031] The image classification results obtained by performing image classification based on the downsampled images mentioned above include:

[0032] The (n-1)th downsampled image output from the (n-1)th image downsampling network in the N image downsampling networks described above is input into the nth image downsampling network, which outputs the nth downsampled image; and

[0033] When N=n, image classification is performed based on the Nth downsampled image to obtain the above image classification result.

[0034] A second aspect of this disclosure provides an image classification method, comprising:

[0035] Acquire the image to be detected;

[0036] The above-mentioned image to be detected is input into the trained image classification model to obtain the image classification result of the above-mentioned image to be detected. The trained image classification model is obtained according to the above method.

[0037] A third aspect of this disclosure provides a method for training an image classification model, comprising:

[0038] The data acquisition module is used to acquire sample images and label data;

[0039] The feature matrix generation module is used to input the above sample images into the convolution unit to generate an image feature matrix;

[0040] The pulse matrix generation module is used to perform pulse transformation on the elements in the image feature matrix using pulse units to generate a pulse matrix corresponding to the image feature matrix. The pulse convolution layer includes the convolution unit and the pulse unit.

[0041] The image generation module is used to input the above pulse matrix and the above position encoding matrix into the pulse self-attention layer and output a downsampled image corresponding to the above sample image;

[0042] The classification result acquisition module is used to classify the image based on the downsampled image above and obtain the image classification result.

[0043] The training module is used to train the image classification model based on the above image classification results and the above label data, and obtain the trained image classification model.

[0044] The fourth aspect of this disclosure provides an application apparatus for an image classification model, comprising:

[0045] The image acquisition module is used to acquire the image to be detected.

[0046] The image classification module is used to input the image to be detected into the trained image classification model to obtain the image classification result of the image to be detected. The trained image classification model is obtained according to the above method.

[0047] A fifth aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the methods described above.

[0048] According to the training method, image classification method, apparatus, and electronic device of the image classification model provided in this disclosure, sample images and label data are acquired. The sample images are input into convolutional units to generate an image feature matrix. Pulse units are used to pulse transform the elements in the feature matrix to generate a pulse matrix corresponding to the image feature matrix. The pulse matrix and position encoding matrix are input into a pulse self-attention layer, which outputs a downsampled image corresponding to the sample image. Image classification is performed based on the downsampled image to obtain the classification result. The image classification model is trained based on the image classification result and label data to obtain the trained image classification model. This model adopts a pyramid structure, in which the number of tokens in each image downsampling network gradually decreases while the dimensionality gradually increases to better adapt to features at different levels. This optimization not only reduces the number of network parameters and computational complexity, further reducing energy consumption, but also enhances the network's performance in processing multi-scale information. Attached Figure Description

[0049] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0050] Figure 1The illustration schematically depicts an application scenario of an image classification model training method, an image classification method, and an apparatus according to embodiments of the present disclosure.

[0051] Figure 2 A flowchart illustrating a training method for an image classification model according to an embodiment of the present disclosure is shown schematically.

[0052] Figures 3a-3b A flowchart illustrating a training method for an image classification model according to an embodiment of the present disclosure is shown schematically.

[0053] Figure 4 A flowchart illustrating a pulsed self-attention unit in an image classification model according to an embodiment of the present disclosure is shown.

[0054] Figure 5 A flowchart illustrating an image classification method according to an embodiment of the present disclosure is shown schematically;

[0055] Figure 6 This schematic diagram illustrates a structural block diagram of a training apparatus for an image classification model according to an embodiment of the present disclosure;

[0056] Figure 7 A schematic diagram illustrating the structure of an application apparatus for an image classification model according to embodiments of the present disclosure; and

[0057] Figure 8 A block diagram schematically illustrates a training method for implementing an image classification model and an electronic device for implementing the image classification method according to embodiments of the present disclosure. Detailed Implementation

[0058] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0059] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0060] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0061] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0062] With the rapid development of computers, network models based on self-attention mechanisms have achieved significant success in the field of deep learning. Models such as Transformer and ViT, by introducing self-attention, have successfully handled the dependencies between different positions in the input sequence and maintained the positional information of each token through positional encoding. The inventors discovered that while ViT has achieved great success in image classification, its large number of parameters and high computational complexity make it relatively resource-intensive during inference, thus making deployment on edge devices challenging. Spiking Neural Networks (SNNs), as a third-generation neural network, have the advantage of low power consumption. They use a pulse coding method to process neurons, transforming complex multiplication and accumulation calculations into efficient accumulation calculations through a series of pulse inputs. This makes SNNs perform exceptionally well in energy-sensitive scenarios (such as mobile devices and embedded systems). Inspired by biological neural networks, their design more naturally simulates the behavior of biological neurons, providing a unique advantage for neuroscience research. Despite the advantages of SNNs in several aspects, such as higher energy efficiency and a more natural design, they still face some challenges, such as the non-differentiability of the step function at zeros and some performance gaps compared to traditional artificial neural networks. To overcome some limitations of SNNs, substitution functions are often used to address the non-differentiability of the step function. Furthermore, SNN networks are an effective way to improve performance and reduce power consumption. Spikformer, a pulsed version of Transformer networks, successfully combines the advantages of Transformer and SNN networks, achieving good performance and low power consumption in image classification tasks. However, the inventors found that because Spikformer is designed based on ViT, its fixed in-module token size and number somewhat limit its flexibility.

[0063] In view of this, embodiments of the present disclosure provide a method for training an image classification model. The method includes: acquiring sample images and label data; inputting the sample images into a convolutional unit to generate an image feature matrix; using a pulse unit to pulse-transform the elements in the image feature matrix to generate a pulse matrix corresponding to the image feature matrix, wherein the pulse convolutional layer includes convolutional units and pulse units; inputting the pulse matrix and a position encoding matrix into a pulse self-attention layer to output a downsampled image corresponding to the sample images; performing image classification based on the downsampled image to obtain an image classification result; and training an image classification model based on the image classification result and label data to obtain the trained image classification model.

[0064] In the technical solution disclosed herein, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of the relevant data all comply with the relevant laws, regulations, and standards of the relevant countries and regions, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding operation entry points are provided for users to choose to authorize or refuse.

[0065] Figure 1 The illustration schematically depicts an application scenario of an image classification model training method, an image classification method, and an apparatus according to embodiments of the present disclosure.

[0066] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the terminal devices 101, 102, 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0067] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0068] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0069] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0070] It should be noted that the training method and image classification method of the image classification model provided in this embodiment can generally be executed by server 105. Correspondingly, the training device and application device of the image classification model provided in this embodiment can generally be located in server 105. The training method and image classification method of the image classification model provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the training device and application device of the image classification model provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105.

[0071] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0072] The following will be based on Figure 1 The described scene, through Figures 2-4 The training method of the image classification model of the disclosed embodiments is described in detail.

[0073] Figure 2 A flowchart illustrating a training method for an image classification model according to an embodiment of the present disclosure is shown.

[0074] like Figure 2 As shown, the method 200 includes operations S210 to S250.

[0075] In operation S210, sample images and label data are acquired.

[0076] In operation S220, the sample image is input into the convolution unit to generate the image feature matrix.

[0077] In operation S230, the pulse unit is used to perform pulse transformation on the elements in the image feature matrix to generate a pulse matrix corresponding to the image feature matrix.

[0078] In operation S240, the pulse matrix and position encoding matrix are input into the pulse self-attention layer, and the output is a downsampled image corresponding to the sample image.

[0079] In operation S250, image classification is performed based on the downsampled image to obtain the image classification result.

[0080] In operation S260, an image classification model is trained based on the image classification results and label data to obtain the trained image classification model.

[0081] According to embodiments of this disclosure, sample images may include various types of pictures, and the resolution of each picture may be set to 32×32, but is not limited thereto. Embodiments of this disclosure do not limit the specific type or resolution of the pictures. Label data corresponds to sample images, and multiple sample images may correspond to one type of label data.

[0082] According to embodiments of this disclosure, a convolutional unit may include a convolutional layer and a normalization layer. By inputting a sample image into the convolutional unit, an image feature matrix can be obtained.

[0083] According to embodiments of this disclosure, a spiking unit can represent a spiking neuron. Traditional artificial neural networks use continuous values ​​to transmit information between layers, while spiking neural networks use discrete pulse sequences. In a spiking neural network, a spiking neuron can convert continuous values ​​into discrete pulses. Common spiking neurons include IF (Integrated-and-Fire) neurons and LIF (Leaky Integrated-and-Fire) neurons, but are not limited to these. Embodiments of this disclosure do not limit the specific type of neuron.

[0084] According to embodiments of this disclosure, spiking neurons can be used to pulse-convert elements in an image feature matrix to generate a pulse matrix corresponding to the image feature matrix.

[0085] According to embodiments of this disclosure, positional encoding can be used to record the positional relationships between different tokens. Common positional encoding methods include absolute positional encoding, relative positional encoding, and conditional positional encoding. Here, conditional positional encoding can be used to generate a positional encoding matrix. Conditional positional encoding can not only improve the accuracy of image classification models, but also dynamically generate positional codes based on local input information, enabling image classification models to better capture local features.

[0086] According to embodiments of this disclosure, the impulsive self-attention layer uses an impulsive self-attention mechanism that differs from traditional self-attention mechanisms. This eliminates the need for a softmax function when calculating attention scores, effectively handling the correlation of input data and reducing computational complexity.

[0087] According to embodiments of this disclosure, a pulse matrix and a position encoding matrix can be input into a pulse self-attention layer to output a downsampled image corresponding to a sample image.

[0088] According to embodiments of this disclosure, a downsampled image represents a reduced image, generating a thumbnail corresponding to the sample image. Image classification is performed based on the generated downsampled image to obtain the image classification result. Then, the image classification model is trained using the labeled data to obtain the trained image classification model.

[0089] According to embodiments of this disclosure, sample images and label data are acquired. The sample images are input into convolutional units to generate an image feature matrix. Pulse units are used to pulse-transform the elements in the feature matrix to generate a pulse matrix corresponding to the image feature matrix. The pulse matrix and position encoding matrix are input into a pulse self-attention layer, which outputs a downsampled image corresponding to the sample images. Image classification is performed based on the downsampled images to obtain classification results. An image classification model is trained based on the image classification results and label data to obtain the trained image classification model. This model adopts a pyramid structure, where the number of tokens in each module (i.e., an image downsampling network) gradually decreases while the dimensionality gradually increases to better adapt to features at different levels. This optimization not only reduces the number of network parameters and computational complexity, further reducing energy consumption, but also enhances the network's performance in processing multi-scale information.

[0090] Figures 3a-3b A flowchart illustrating a training method for an image classification model according to an embodiment of the present disclosure is shown.

[0091] like Figure 3a As shown, sample images and label data are acquired. The sample images are input into a first image downsampling network to generate a second downsampling image. The second downsampling image is input into another second image downsampling network to generate a third downsampling image. The three downsampling images are then input into a third image downsampling network to generate a fourth downsampling image. Image classification is performed based on the fourth downsampling image to obtain the image classification result. The gradient backpropagation algorithm is used to process the image classification result and label data to generate a loss value. Based on the loss value, an image classification model is trained to obtain the trained image classification model. Each image downsampling network includes a pulse convolutional layer, a position encoding matrix, and a pulse self-attention layer.

[0092] like Figure 3bAs shown, the image downsampling network includes convolutional units, spiking units, a position encoding matrix, a spiking self-attention unit, and a spiking perceptron unit. The convolutional unit can include linear layers and normalization layers. After inputting the sample image into the convolutional unit, an image feature matrix is ​​generated. The spiking unit (i.e., IF neurons and max-pooling layers) performs spiking transformations on the elements of the image feature matrix, generating a spiking matrix corresponding to the image feature matrix. The spiking matrix and the position encoding matrix are fused to obtain the position spiking matrix; the position spiking matrix is ​​then input into the spiking self-attention unit to obtain the spiking attention matrix; the position spiking matrix and the spiking attention matrix are fused to obtain a first fused matrix; the first fused matrix is ​​then input into the first linear spiking unit to obtain a first linear fused matrix; the first linear fused matrix is ​​then input into the second linear spiking unit to obtain the spiking perceptron matrix. Both the first and second linear spiking units include linear layers, normalization layers, and IF neurons. The downsampled image can then be obtained based on the spiking perceptron matrix and the first fused matrix.

[0093] According to embodiments of this disclosure, pulse units are used to pulse-convert elements in an image feature matrix to generate a pulse matrix corresponding to the convolution matrix. This includes: updating the elements of the t-th image feature matrix obtained in the t-th stage based on the (t-1)-th intermediate pulse matrix of the t-th stage to obtain the t-th membrane potential matrix of the t-th stage, the t-th membrane potential matrix including t-th membrane potential elements; processing the t-th membrane potential matrix and a preset membrane potential threshold using a step function to obtain t-th pulse update parameters; determining the t-th intermediate pulse matrix based on the t-th pulse update parameters and the t-th membrane potential matrix, where t > 1 and t is an integer; and determining the T-th intermediate pulse matrix as a pulse matrix when T = t, where T ≥ t.

[0094] According to an embodiment of this disclosure, the elements of the image feature matrix obtained in the t-th stage can be updated according to the intermediate pulse matrix of the t-1 stage to obtain the membrane potential matrix of the t-th stage. The membrane potential matrix of the t-th stage includes the membrane potential elements as shown in formula (1).

[0095] H[t]=V[t-1]+X[t] (1)

[0096] Where H[t] represents the t-th membrane potential matrix, V[t-1] represents the (t-1)-th intermediate pulse matrix, and X[t] represents the element of the t-th image feature matrix.

[0097] According to embodiments of this disclosure, the t-th membrane potential matrix and a preset membrane potential threshold can be processed by a step function to obtain the t-th pulse update parameters, as shown in formula (2).

[0098] S[t]=Θ(H[t]-V th (2)

[0099] Among them, V th S[t] represents the preset membrane potential threshold, S[t] represents the update parameter for the t-th pulse, and Θ(h) is the Heaviside step function. When h≥0, the output is 1, otherwise it is 0.

[0100] According to embodiments of this disclosure, the t-th intermediate pulse matrix can be determined based on the t-th pulse update parameter and the t-th membrane potential matrix, where t > 1 and t is an integer, as shown in formula (3).

[0101] V[t]=H[t](1-S[t])+V rst S[t] (3)

[0102] Where V[t] represents the t-th intermediate pulse matrix.

[0103] According to embodiments of this disclosure, the pulse unit can be a LIF neuron, mainly comprising three key steps: integration, leakage, and firing. The LIF neuron receives input from other neurons, and the membrane potential accumulates with the input, gradually leaking over time. When the membrane potential exceeds a threshold, the neuron fires a pulse, resetting the membrane potential to a specific value, and then continues to receive input and accumulate membrane potential. LIF neurons more closely resemble the real behavior of biological neurons, but their simulation process is relatively complex and not well-suited for hardware implementation.

[0104] According to embodiments of this disclosure, the pulse unit can be an IF neuron. Compared to LIF neurons, IF neurons focus only on the integration and firing of the membrane potential, ignoring the leakage process of the membrane potential. Using IF neurons in large-scale networks can not only effectively reduce computational complexity and memory requirements, but also benefit the limited storage of edge devices.

[0105] According to embodiments of this disclosure, inputting the pulse matrix and the position encoding matrix into a pulse self-attention layer and outputting a downsampled image corresponding to a sample image includes: fusing the pulse matrix and the position encoding matrix to obtain a position pulse matrix; inputting the position pulse matrix into a pulse self-attention unit to obtain a pulse attention matrix; fusing the position pulse matrix and the pulse attention matrix to obtain a first fusion matrix; inputting the first fusion matrix into a pulse perceptron unit to obtain a pulse perceptron matrix, wherein the pulse self-attention layer includes a pulse self-attention unit and a pulse perceptron unit; and obtaining a downsampled image based on the pulse perceptron matrix and the first fusion matrix.

[0106] According to embodiments of this disclosure, the pulse matrix and the position encoding matrix can be added element-wise to obtain a position pulse matrix. The position pulse matrix is ​​then input to a pulse self-attention unit to obtain a pulse attention matrix. The position pulse matrix and the pulse attention matrix are then fused to obtain a first fused matrix. The first fused matrix is ​​then input to a pulse perceptron unit to obtain a pulse perceptron matrix.

[0107] Figure 4 A flowchart illustrating a pulsed self-attention unit in an image classification model according to an embodiment of the present disclosure is shown.

[0108] like Figure 4 As shown, the position pulse matrix is ​​input into the third, fourth, and fifth linear pulse units respectively to determine the query matrix Q. s Key matrix K s Sum matrix V s The query matrix, key matrix, and value matrix are fused using a self-attention fusion unit to obtain an attention fusion matrix. The attention fusion matrix is ​​then input into a pulse subunit to obtain a pulse fusion matrix. Finally, the pulse fusion matrix is ​​input into the sixth linear pulse unit to obtain a pulse attention matrix. The third, fourth, fifth, and sixth linear pulse units each include a linear layer, a normalized layer, and an IF neuron.

[0109] According to embodiments of this disclosure, a position pulse matrix is ​​input to a pulse self-attention unit to obtain a pulse attention matrix, including: determining a query matrix, a key matrix, and a value matrix based on the position pulse matrix; and applying an attention mechanism to the query matrix Q. s Key matrix K s Sum matrix V s The attention fusion matrix is ​​obtained by performing fusion; the attention fusion matrix is ​​input into the pulse sub-unit to obtain the pulse fusion matrix; the pulse fusion matrix is ​​input into the linear pulse unit to obtain the pulse attention matrix, wherein the pulse self-attention unit includes the linear pulse unit and the pulse sub-unit.

[0110] According to embodiments of this disclosure, the position pulse matrix can be encoded into pulse form to generate a query matrix, a key matrix, and a value matrix, as shown in formulas (4), (5), and (6).

[0111] Q s =SN(BN(LinearQ(X))) (4)

[0112] K s =SN(BN(LinearK(×))) (5)

[0113] V s=SN(BN(LinearV(×))) (6)

[0114] Where X is the input of the spiking self-attention unit, SN represents a spiking neuron, BN represents a normalization layer, and Linear represents a linear layer.

[0115] According to embodiments of this disclosure, the query matrix Q can be based on an attention mechanism. s Key matrix K s Sum matrix V s To merge, that is Here, scale is a scaling factor, thus ultimately yielding the attention fusion matrix.

[0116] According to embodiments of this disclosure, the pulse subunit can be an IF neuron. An attention fusion matrix is ​​input to the IF neuron to obtain a pulse fusion matrix. A linear pulse unit can include a linear layer, a normalized layer, and an IF neuron. The pulse fusion matrix can be input to the linear pulse unit to obtain a pulse attention matrix, as shown in formula (7).

[0117]

[0118] Wherein, OUT(SSA) represents the output of the pulse self-attention unit.

[0119] According to an embodiment of this disclosure, inputting a first fusion matrix into a pulse sensor unit to obtain a pulse sensor matrix includes: inputting the first fusion matrix into a first linear pulse unit to obtain a first linear fusion matrix; and inputting the first linear fusion matrix into a second linear pulse unit to obtain a pulse sensor matrix, wherein the pulse sensor unit includes multiple linear pulse sub-units.

[0120] According to embodiments of this disclosure, the first linear pulse unit, the second linear pulse unit, and the linear pulse subunit may each include a linear layer, a normalization layer, and an IF neuron.

[0121] According to embodiments of this disclosure, a first fusion matrix can be input into a first linear pulse unit to obtain a first linear fusion matrix. The obtained first linear fusion matrix is ​​then input into a second linear pulse unit to obtain a pulse sensor matrix.

[0122] According to embodiments of this disclosure, the image downsampling network includes N networks. The image classification based on the downsampled image to obtain the image classification result includes: inputting the (n-1)th downsampled image output by the (n-1)th image downsampling network into the nth image downsampling network to output the nth downsampled image; and when N=n, performing image classification based on the Nth downsampled image to obtain the image classification result.

[0123] According to embodiments of this disclosure, the image downsampling network includes at least one, for example, four image downsampling networks. The first downsampled image output by the first image downsampling network is input into the second image downsampling network to output a second downsampled image. The output second downsampled image is input into the third image downsampling network to output a third downsampled image. The output third downsampled image is input into the fourth image downsampling network to output a fourth downsampled image. When the fourth downsampled image is output, image classification is performed based on the fourth downsampled image to obtain the image classification result.

[0124] According to embodiments of this disclosure, the performance of an image classification model was evaluated on two static datasets, CIFAR-10 and CIFAR-100. The model was compared with state-of-the-art SNN models, as detailed in the comparative analysis below. Furthermore, the power consumption of the model on the CPU was estimated.

[0125] The CIFAR dataset includes 50,000 training images and 10,000 test images, each with a resolution of 32x32. To ensure fair comparison, training was performed on the training set, and the accuracy on the test set was reported using only a single image downsampling network (i.e., top-1). The image classification model can be divided into four stages. Considering the 32x32 image resolution of the CIFAR dataset, no downsampling was performed in the first stage, while the patch size for the latter three stages was 2x2. The specific configuration of the image classification model is shown in Table 1. Direct encoding with a time step of 4 was used to speed up the training process. Experiments were conducted on an NVIDIA GeForce RTX 4090 GPU for a total of 300 epochs. Table 1 shows the C... s and L s These represent the channel dimension of the output token and the number of impulse self-attention layers, respectively.

[0126] During training, the AdamW optimizer with a momentum of 0.9 and a cosine learning rate scheduling strategy were used, with 20 warmup epochs and 10 cooldown epochs. Additionally, data augmentation strategies, such as mixup, were introduced to improve the model's generalization ability.

[0127] Table 1. Configuration details of the image classification model

[0128] Output Size SPVT 4-384 Stage 1 H x W <![CDATA[C1=48,L1=1]]> Stage 2 H / 2 x W / 2 <![CDATA[C2=96,L2=1]]> Stage 3 H / 4xW / 4 <![CDATA[C3=192,L3=1]]> Stage 4 H / 8xW / 8 <![CDATA[C4=384,L1=1]]>

[0129] Experimental results:

[0130] The experimental results on the CIFAR dataset are shown in Table 2. The image classification model has 5.01M parameters, but its top-1 accuracy is 95.53% and 78.39%, respectively, which is better than the Spiked Transformer (* represents the result of the self-implementation of the image classification model).

[0131] Table 2. Experimental results of the image classification model on the CIFAR dataset.

[0132] Network Parameter(M) Time steps CIFAR 10ACC (%) CIFAR100ACC(%) Power(uJ) TSC ResNet.20

[10] 0.27 1024 91.42 68.18 TSC VGG-16

[10] 138.36 1024 93.63 70.97 TET Resnet-19

[11] 12.63 4 94.44 74.47 Spikformer 4-384* 9.32 4 94.71 77.35 255.51 SPVT 4-384 5.01 4 95.53 78.39 238.54

[0133] Energy consumption assessment:

[0134] In SNNs, floating-point operations are replaced by associated synaptic operations. The formula for calculating the synaptic operation is shown in (8):

[0135] SOP s =fr*T*FLOP s (8)

[0136] Here, s represents stage s, fr is the pulse firing rate of stage s, T is the time step of the IF neuron, and FLOP is the time step of the IF neuron. s This represents the number of floating-point operations in stage s, i.e., the number of MACs. SOP s This is based on the number of AC operations based on pulses. Assume a network implemented on a 45nm CPU hardware. The energy consumption of one MAC and AC operation is E respectively. MAC =4.6pJ and E AC =0.9pJ. Therefore, the theoretical energy consumption of the image classification model is:

[0137]

[0138] here, Phase 1 encodes the RGB image into a floating-point operation in the form of pulses. It is the synaptic operation amount of the patch embedding module in stage s. It is the synaptic operation quantity of the SSA module in stage s. It is the synaptic operation quantity of the MLP module in stage s.

[0139] Furthermore, the image classification model contains 4.42M IF neurons, while the Spiked Transformer contains 5.8M LIF neurons, as shown in equation (10) for LIF neurons:

[0140] H[t]=V[t-1]+(1 / τ)*(X[t]-(V[t-1]-Vrst)) (10)

[0141] Therefore, LIF neurons have at least one more multiplication operation than IF neurons, and the simplification of neurons further reduces the energy consumption of image classification models.

[0142] Figure 5 A flowchart illustrating an image classification method according to an embodiment of the present disclosure is shown schematically.

[0143] like Figure 5 As shown, the image classification method includes operations S510 to S520.

[0144] The S510 is used to acquire the image to be detected.

[0145] When operating the S520, the image to be detected is input into the trained image classification model to obtain the image classification result of the image to be detected. The trained image classification model is obtained according to the training method of the image classification model.

[0146] According to embodiments of this disclosure, an image to be detected can be acquired first, and then the acquired image to be detected can be input into a trained image classification model to obtain the image classification result of the image to be detected.

[0147] Based on the above-described image classification model training method, this disclosure also provides an image classification model training apparatus. The following will combine... Figure 6 The device is described in detail.

[0148] Figure 6 A schematic block diagram of a training apparatus for an image classification model according to an embodiment of the present disclosure is shown.

[0149] like Figure 6 As shown, the training device 600 for the image classification model in this embodiment includes a data acquisition module 610, a feature matrix generation module 620, an impulse matrix generation module 630, an image generation module 640, a classification result acquisition module 650, and a training module 660.

[0150] The data acquisition module 610 is used to acquire sample images and label data. In one embodiment, the data acquisition module 610 can be used to perform the operation S210 described above, which will not be repeated here.

[0151] The feature matrix generation module 620 is used to input sample images into the convolution unit to generate an image feature matrix. In one embodiment, the feature matrix generation module 620 can be used to perform the operation S220 described above, which will not be repeated here.

[0152] The pulse matrix generation module 630 is used to perform pulse transformation on the elements in the image feature matrix using pulse units to generate a pulse matrix corresponding to the image feature matrix. In one embodiment, the pulse matrix generation module 630 can be used to perform the operation S230 described above, which will not be repeated here.

[0153] The image generation module 640 is used to input the pulse matrix and the position encoding matrix into the pulse self-attention layer and output a downsampled image corresponding to the sample image. In one embodiment, the image generation module 640 can be used to perform the operation S240 described above, which will not be repeated here.

[0154] The classification result obtaining module 650 is used to classify the image based on the downsampled image and obtain the image classification result. In one embodiment, the classification result obtaining module 650 can be used to perform the operation S250 described above, which will not be repeated here.

[0155] The training module 660 is used to train an image classification model based on the image classification results and label data, thereby obtaining the trained image classification model. In one embodiment, the training module 660 can be used to perform the operation S260 described above, which will not be repeated here.

[0156] According to embodiments of this disclosure, sample images and label data are acquired. The sample images are input into a convolutional unit to generate an image feature matrix. A pulse unit is used to pulse-transform the elements in the feature matrix, generating a pulse matrix corresponding to the image feature matrix. The pulse matrix and position encoding matrix are input into a pulse self-attention layer, outputting a downsampled image corresponding to the sample image. Image classification is performed based on the downsampled image to obtain the classification result. An image classification model is trained based on the image classification result and label data to obtain the trained image classification model. This model adopts a pyramid structure, where the number of tokens in each module gradually decreases while the dimensionality gradually increases to better adapt to features at different levels. This optimization not only reduces the number of network parameters and computational complexity, further reducing energy consumption, but also enhances the network's performance in processing multi-scale information.

[0157] According to embodiments of this disclosure, the pulse matrix generation module includes: an update unit, a parameter update unit, an intermediate matrix determination unit, and a pulse matrix acquisition unit.

[0158] The update unit is used to update the elements of the image feature matrix obtained in stage t based on the intermediate pulse matrix of stage t-1, so as to obtain the membrane potential matrix of stage t, which includes the membrane potential elements.

[0159] The parameter update unit is used to process the t-th membrane potential matrix and the preset membrane potential threshold according to the step function to obtain the t-th pulse update parameters.

[0160] The intermediate matrix determination unit is used to determine the intermediate pulse matrix of the tth pulse based on the pulse update parameters of the tth pulse and the membrane potential matrix of the tth pulse, where t > 1 and t is an integer.

[0161] The pulse matrix obtaining unit is used to determine the Tth intermediate pulse matrix as the pulse matrix when T = t, and T ≥ t.

[0162] According to embodiments of this disclosure, the image generation module includes:

[0163] A position pulse unit is used to fuse the pulse matrix and the position encoding matrix to obtain a position pulse matrix.

[0164] The pulse attention unit is used to input the position pulse matrix into the pulse self-attention unit to obtain the pulse attention matrix.

[0165] The first fusion unit is used to fuse the position pulse matrix and the pulse attention matrix to obtain the first fusion matrix.

[0166] The impulse perceptron unit is used to input the first fusion matrix into the impulse perceptron unit to obtain the impulse perceptron matrix. The impulse self-attention layer includes an impulse self-attention unit and an impulse perceptron unit.

[0167] The downsampled image unit is used to obtain a downsampled image based on the pulse sensor matrix and the first fusion matrix.

[0168] According to embodiments of this disclosure, the pulse attention unit includes: a matrix determination subunit, an attention fusion subunit, a pulse fusion subunit, and a pulse attention subunit.

[0169] The matrix determination sub-unit is used to determine the query matrix, key matrix, and value matrix based on the position pulse matrix.

[0170] The attention fusion subunit is used to fuse the query matrix, key matrix, and value matrix based on the attention mechanism to obtain the attention fusion matrix.

[0171] The pulse fusion subunit is used to input the attention fusion matrix into the pulse subunit to obtain the pulse fusion matrix.

[0172] The pulse attention subunit is used to input the pulse fusion matrix into the linear pulse unit to obtain the pulse attention matrix. The pulse self-attention unit includes the linear pulse unit and the pulse subunit.

[0173] According to embodiments of this disclosure, the pulse sensing unit includes: a first linear subunit and a second linear subunit.

[0174] The first linear sub-unit is used to input the first fusion matrix into the first linear pulse unit to obtain the first linear fusion matrix.

[0175] The second linear subunit is used to input the first linear fusion matrix into the second linear pulse unit to obtain the pulse sensor matrix, wherein the pulse sensor unit includes multiple linear pulse subunits.

[0176] According to embodiments of this disclosure, the image downsampling network comprises N networks.

[0177] According to embodiments of this disclosure, the classification result obtaining module includes: an nth downsampling unit and a classification unit.

[0178] The nth downsampling unit is used to input the (n-1)th downsampled image output from the (n-1)th image downsampling network in the N image downsampling networks into the nth image downsampling network and output the nth downsampled image.

[0179] The classification unit is used to classify the image based on the Nth downsampled image when N=n, and obtain the image classification result.

[0180] Based on the above-described image classification model application method, this disclosure also provides an image classification model application device. The following will be combined with... Figure 7 The device is described in detail.

[0181] Figure 7 A schematic diagram illustrating the structural block diagram of an image classification model application according to an embodiment of the present disclosure is shown.

[0182] like Figure 7 As shown, the image classification model application 700 of this embodiment includes a detection image acquisition module 710 and a detection image classification module 720.

[0183] The image acquisition module 710 is used to acquire the image to be detected. In one embodiment, the image acquisition module 710 can be used to perform the operation S510 described above, which will not be repeated here.

[0184] The image classification module 720 is used to input the image to be detected into a trained image classification model to obtain the image classification result of the image to be detected. The trained image classification model is obtained according to an image classification model training method. In one embodiment, the image classification module 720 can be used to perform the operation S520 described above, which will not be repeated here.

[0185] Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be implemented by dividing them into multiple modules. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as hardware circuitry, such as Field Programmable Gate Arrays (FPGAs), Programmable Logic Arrays (PLAs), Systems-on-Chip, Systems-on-Substrate, Systems-on-Package, Application-Specific Integrated Circuits (ASICs), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0186] For example, any plurality of training devices for image classification models can be combined into one module / unit / subunit, or any one of such modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least a portion of the functionality of one or more of these modules / units / subunits can be combined with at least a portion of the functionality of other modules / units / subunits and implemented in one module / unit / subunit. According to embodiments of this disclosure, at least one of the training devices for image classification models can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the training devices for image classification models can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0187] It should be noted that the training device part of the image classification model in the embodiments of this disclosure corresponds to the training method part of the image classification model in the embodiments of this disclosure, and the application device part of the image classification model in the embodiments of this disclosure corresponds to the application method part of the image classification model in the embodiments of this disclosure. The description of the device part is specifically referred to in the method part, and will not be repeated here.

[0188] Figure 8A block diagram schematically illustrates an electronic device suitable for implementing a training method for an image classification model and an application method for an image classification model, according to embodiments of the present disclosure.

[0189] like Figure 8 As shown, an electronic device 800 according to an embodiment of this disclosure includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage portion 808 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this disclosure.

[0190] RAM 803 stores various programs and data required for the operation of electronic device 800. Processor 801, ROM 802, and RAM 803 are interconnected via bus 804. Processor 801 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 802 and / or RAM 803. It should be noted that programs may also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in one or more memories.

[0191] According to embodiments of this disclosure, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to a bus 804. The electronic device 800 may also include one or more of the following components connected to the I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 810 as needed so that computer programs read from it can be installed into the storage section 808 as needed.

[0192] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0193] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 802 and / or RAM 803 and / or one or more memories other than ROM 802 and RAM 803 described above.

[0194] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the training method or application method of the image classification model provided in the embodiments of this disclosure.

[0195] When the computer program is executed by the processor 801, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0196] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 809, and / or installed from a removable medium 811. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0197] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by processor 801, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0198] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0199] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0200] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0201] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A training method for an image classification model, wherein, The image classification model includes at least one image downsampling network, which includes a pulsed convolutional layer, a position encoding matrix, and a pulsed self-attention layer. The method includes: Acquire sample images and label data; The sample image is input into a convolutional unit to generate an image feature matrix; The elements in the image feature matrix are pulse-transformed using pulse units to generate a pulse matrix corresponding to the image feature matrix. The pulse convolutional layer includes the convolutional unit and the pulse unit. The pulse matrix and the position encoding matrix are input into the pulse self-attention layer, and the downsampled image corresponding to the sample image is output. The (n-1)th downsampled image output by the (n-1)th image downsampled network in the N image downsampled networks is input into the nth image downsampled network, which outputs the nth downsampled image. When N=n, image classification is performed based on the Nth downsampled image to obtain the image classification result; and Based on the image classification results and the label data, an image classification model is trained to obtain the trained image classification model. The step of using pulse units to perform pulse transformation on the elements in the image feature matrix to generate a pulse matrix corresponding to the image feature matrix includes: The elements of the image feature matrix obtained in stage t are updated based on the intermediate pulse matrix of stage t-1 to obtain the membrane potential matrix of stage t, which includes the membrane potential elements of stage t. The t-th membrane potential matrix and the preset membrane potential threshold are processed according to the step function to obtain the t-th pulse update parameters; The intermediate pulse matrix t is determined based on the pulse t update parameters and the membrane potential matrix t, where t > 1 and t is an integer; and When T=t, the Tth intermediate pulse matrix is ​​determined as the pulse matrix, and T≥t.

2. The training method according to claim 1, wherein, The step of inputting the pulse matrix and the position encoding matrix into the pulse self-attention layer and outputting a downsampled image corresponding to the sample image includes: By fusing the pulse matrix and the position encoding matrix, a position pulse matrix is ​​obtained; The position pulse matrix is ​​input into the pulse self-attention unit to obtain the pulse attention matrix; The position pulse matrix and the pulse attention matrix are fused to obtain a first fusion matrix; The first fusion matrix is ​​input to the impulse sensing machine unit to obtain the impulse sensing machine matrix, wherein the impulse self-attention layer includes the impulse self-attention unit and the impulse sensing machine unit; and The downsampled image is obtained based on the pulse sensor matrix and the first fusion matrix.

3. The training method according to claim 2, wherein, The step of inputting the position pulse matrix into the pulse self-attention unit to obtain the pulse attention matrix includes: The query matrix, key matrix, and value matrix are determined based on the position pulse matrix. The query matrix, the key matrix, and the value matrix are fused based on an attention mechanism to obtain an attention fusion matrix; The attention fusion matrix is ​​input into the pulse subunit to obtain the pulse fusion matrix; and The pulse fusion matrix is ​​input to the linear pulse unit to obtain the pulse attention matrix, wherein the pulse self-attention unit includes the linear pulse unit and the pulse sub-unit.

4. The training method according to claim 2, wherein, The step of inputting the first fusion matrix into the pulse sensor unit to obtain the pulse sensor matrix includes: The first fusion matrix is ​​input into the first linear pulse unit to obtain the first linear fusion matrix; and The first linear fusion matrix is ​​input into the second linear pulse unit to obtain the pulse sensor matrix, wherein the pulse sensor unit includes multiple layers of linear pulse sub-units.

5. An image classification method, comprising: Acquire the image to be detected; The image to be detected is input into the trained image classification model to obtain the image classification result of the image to be detected, wherein the trained image classification model is obtained by the method according to any one of claims 1 to 4.

6. A training device for an image classification model, comprising: The data acquisition module is used to acquire sample images and label data; The feature matrix generation module is used to input the sample image into the convolution unit to generate an image feature matrix; The pulse matrix generation module is used to perform pulse transformation on the elements in the image feature matrix using pulse units to generate a pulse matrix corresponding to the image feature matrix, wherein the pulse convolution layer includes the convolution unit and the pulse unit; The image generation module is used to input the pulse matrix and the position encoding matrix into the pulse self-attention layer and output a downsampled image corresponding to the sample image; The classification result acquisition module is used to perform image classification based on the downsampled image and obtain the image classification result; The training module is used to train an image classification model based on the image classification results and the label data to obtain a trained image classification model, wherein the training device for the image classification model is constructed according to any one of claims 1 to 4.

7. An application device for an image classification model, comprising: The image acquisition module is used to acquire the image to be detected. The image classification module is used to input the image to be detected into a trained image classification model to obtain the image classification result of the image to be detected, wherein the trained image classification model is obtained by the method according to any one of claims 1 to 4.

8. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 4.

Images