Image classification method based on multistage complementary spiking neural network model

Abstract

The invention discloses an image classification method based on a multistage complementary pulse neural network model. The method comprises the steps of obtaining an image classification data set; constructing a multistage complementary spiking neural network model, which comprises the following steps: constructing an SNN network model by using a traditional LIF neuron structure; reconstructing an LIF neuron structure in the SNN network model, introducing a multi-stage activation design, and constructing a new multi-stage neuron structure by using a plurality of small neuron structures in the multi-stage activation design; after multi-stage activation design is introduced, complementary potential energy is introduced, complementary activation design is achieved, and a multi-stage complementary neuron structure is obtained; training the multi-stage complementary spiking neural network model by using an image classification data set, wherein the trained multi-stage complementary spiking neural network model is used for image classification; according to the invention, the accuracy of image classification is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to an image classification method based on a multi-level complementary pulse neural network model. Background Art

[0002] Since the release of ChatGPT, it has demonstrated unprecedented capabilities and triggered an AI boom. Multimodal large models have gradually become the future development trend, but building and maintaining these large models is a resource-intensive task. Deep neural networks consume a lot of energy, so the use of more energy-efficient hardware, algorithm innovation, and the promotion of green energy use have gradually become an indispensable part of model research and application.

[0003] Recently, researchers have been inspired by the mechanism of biological neurons using pulses to perform calculations and communication, and have built a new neural network based on this. Although traditional artificial neural networks (ANNs) also draw on the principles of biological nervous systems, they are essentially different from the human brain in terms of structural composition, neuronal computing mechanisms, and learning strategies. The human brain can complete complex cognitive tasks such as multi-target recognition, logical reasoning, and motion control under extremely low energy consumption (about 2 watts), while conventional computers consume up to 250 watts of energy to complete object recognition tasks of the same complexity. SNN (Spike Neural Network) simulates the operating mechanism of biological neural networks and replaces continuous value activation with pulsed signal transmission, which is closer to the natural working mode of biological nervous systems, thereby achieving significant optimization in the utilization of computing resources and showing superior generalization performance and time series data processing capabilities.

[0004] The core feature of SNN: pulse communication, not only reflects the dynamic spatiotemporal relationship between neurons, but also promotes the efficient use of computing resources, enhancing the expressiveness and efficiency of the model in complex time-series tasks. This pulse-based activation mechanism, combined with time encoding and asynchronous processing characteristics, together constitutes the unique energy efficiency and adaptability advantages of SNN, making them particularly suitable for processing tasks involving multiple types of information (multimodal). SNN can effectively integrate input signals of different modes, improve the accuracy and response speed of task execution, and effectively alleviate the high computing cost, large storage requirements and insufficient time information processing faced by ANN when processing multimodal data, further expanding the application boundaries and performance potential of neural network technology.

[0005] Although spiking neural networks have many advantages, since the input and output of spiking neurons are both discrete and binary pulse sequences, direct training of spiking neural networks has a relatively serious gradient vanishing problem, which greatly limits the performance of direct training of spiking neural networks and hinders the in-depth research of spiking neural networks. Although binary spikes can achieve faster information transmission, the information they can express during the transmission process is more limited than floating-point numbers. This limitation may cause different values ​​in ANN to have the same pulse performance in SNN, further deepening the gradient vanishing. In addition, since most SNN networks use the LIF (Leaky Integrate-and-Fire) model, this model structure cannot handle time gradients well for larger time steps, which can also cause the gradient to descend too quickly.

[0006] The information loss caused by the above problems has interfered with the accuracy of SNN networks in image classification tasks to a certain extent. As far as the current SNN networks are concerned, their accuracy in image classification tasks is generally not as good as that of ANN networks with the same network structure. Summary of the invention

[0007] In view of the problems and shortcomings of the prior art, the present invention proposes an image classification method based on a multi-level complementary pulse neural network model. The method redesigns the LIF neuron and adopts a new multi-level complementary neuron structure. While ensuring the low energy consumption of the SNN, the performance of the SNN network is improved and the accuracy of image classification is enhanced.

[0008] The object of the present invention is achieved through the following technical scheme: an image classification method based on a multi-level complementary pulse neural network model, comprising:

[0009] Get the image classification dataset;

[0010] Constructing a multi-level complementary spiking neural network model, including: using a traditional LIF neuron structure to construct an SNN network model; reconstructing the LIF neuron structure in the SNN network model, introducing a multi-level activation design, wherein the multi-level activation design uses a plurality of small neuron structures to construct a new multi-level neuron structure; after introducing the multi-level activation design, then introducing complementary potential energy to realize the complementary activation design, that is, obtaining a multi-level complementary neuron structure;

[0011] The multi-level complementary spiking neural network model is trained using an image classification data set, and the trained multi-level complementary spiking neural network model is used for image classification.

[0012] Furthermore, the multi-level activation design uses multiple small neuron structures to construct a new multi-level neuron structure, and uses the membrane voltage accumulation process under the new structure to replace the membrane voltage accumulation process under the original LIF model. The specific formula is as follows:

[0013]

[0014] Where τ is the membrane time constant, u l [t] represents the membrane voltage of the multi-level neuron structure in layer l at time t. represents the membrane voltage of the i-th neuron structure in layer l at time t-1, Represents the current of the i-th small neuron structure in layer l at time t.

[0015] Furthermore, after introducing the multi-stage activation design, complementary potential energy is then introduced to realize the complementary activation design, including:

[0016] The complementary activation design introduces an additional "complementary potential" condition in the LIF structure to slow down the membrane voltage decay after pulse activation occurs. The membrane voltage decay process under the new structure replaces the decay process under the original LIF model. The formula is as follows:

[0017]

[0018] in represents the membrane voltage of the ith neuron in layer l at time t, V th represents the membrane voltage threshold of the neurons in this layer, s l [t] represents whether a pulse is generated at time t in layer l; It represents the size of the "complementary potential" of the i-th neuron in layer l at time t. α represents the learning rate. In the formula, when a neuron in layer l pulses at time t, its corresponding membrane voltage u l [t] will decline, through additional supplementation Slow down the pace of decline.

[0019] Furthermore, the complementary potential energy is dynamically adjusted according to the pulse activation situation, and the specific formula is as follows:

[0020]

[0021] in It represents the size of the "complementary potential" of the i-th neuron in layer l at time t. β is a hyperparameter used to control the growth rate of the complementary potential.

[0022] Furthermore, it also includes: adjusting the back propagation function according to the current multi-level complementary neuron structure, and the specific formula is as follows:

[0023]

[0024] Among them, L represents the loss value, The gradient of the weights of layer l, is the gradient of the bias of the lth layer, T represents the time step, and K represents the number of small neurons in the multi-level activation.

[0025] The present invention also provides an image classification device based on a multi-level complementary pulse neural network model, comprising:

[0026] A data acquisition module, used to acquire image classification data sets;

[0027] A model building module is used to build a multi-level complementary pulse neural network model, including: using a traditional LIF neuron structure to build an SNN network model; reconstructing the LIF neuron structure in the SNN network model and introducing a multi-level activation design, wherein the multi-level activation design uses multiple small neuron structures to build a new multi-level neuron structure; after introducing the multi-level activation design, then introducing complementary potential energy to realize the complementary activation design, that is, obtaining a multi-level complementary neuron structure;

[0028] The model training module is used to train the multi-level complementary spiking neural network model using an image classification data set, and the trained multi-level complementary spiking neural network model is used for image classification.

[0029] The present invention also provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned image classification method based on a multi-level complementary pulse neural network model.

[0030] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the image classification method based on the multi-level complementary pulse neural network model is implemented.

[0031] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned image classification method based on a multi-level complementary pulse neural network model.

[0032] The beneficial effects of the present invention are as follows:

[0033] By modifying the original SNN network, a new SNN network with better performance and higher accuracy can be obtained while ensuring the same network backbone. Specifically, the original SNN network backbone structure is maintained, and only the LIF neuron model is adjusted. By introducing a multi-level activation design, the original SNN binary expression ability is expanded, and the gradient vanishing problem is improved. In addition, the multi-level activation design can also expand the width of the neuron approximate inverse, expand the information expression ability of the spatial gradient, and further reduce the occurrence of the gradient vanishing problem. By introducing a complementary activation design, the complementary potential energy is additionally compensated on the basis of the original membrane voltage, and the decay rate of the time gradient is improved, so that the SNN network can keep the time gradient from vanishing at a longer time step, so that more information can be captured during the training process, reducing information loss, and thus improving the performance of the entire SNN network. For example, for an SNN pulse neural network in an image recognition scenario, this method can be used to change only the LIF structure in the original SNN network. The improved new network structure can perform image recognition more accurately while maintaining the low energy consumption characteristics of the SNN. In summary, this method can improve the SNN network so that the order of magnitude of operations before and after the improvement is basically the same, but with better performance and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0035] Figure 1 It is the overall workflow diagram of the present invention;

[0036] Figure 2 It is the neuron structure after introducing the multi-level activation mechanism of the present invention;

[0037] Figure 3 It is the neuron structure after the pulse activation mechanism is introduced into the present invention;

[0038] Figure 4 Schematic diagrams of the SNN network structure before and after the improvement of the present invention; wherein (a) is a schematic diagram of the SNN network structure before the improvement, and (b) is a schematic diagram of the SNN network structure after the improvement;

[0039] Figure 5 It is a table of the accuracy of the present invention before and after the improvement on various image data sets;

[0040] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The present invention is described in detail below according to the accompanying drawings and preferred embodiments, and the purpose and effect of the present invention will become more clear. The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. In the absence of conflict, the features in the following embodiments and implementations can be combined with each other. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0042] Depend on Figure 1 As shown, the present invention provides an image classification method based on a multi-level complementary pulse neural network model, and the specific steps include:

[0043] Step 1: Use the traditional LIF neuron structure to build the SNN network model;

[0044] In one embodiment, based on the LIF model, the SNN network model to be constructed is first determined. Here, Spike-ResNet (a type of SNN network) is taken as an example, and its structure is as follows: Figure 4 This step is mainly used to determine the network backbone of the SNN, and the backbone structure will not be adjusted in subsequent steps.

[0045] Step 2: Reconstruct the LIF neuron structure in the SNN network model and introduce a multi-level activation design, which uses multiple small neuron structures to construct a new multi-level neuron structure;

[0046] In one embodiment, the LIF neuron structure of the SNN network model is reconstructed and a multi-level activation design is introduced. Figure 2 As shown in the figure, the optimized and improved neuron is composed of multiple small neurons. Compared with the traditional single neuron structure with binary output, this structure has stronger expression ability, can reduce information loss, and improve gradient disappearance. The corresponding formula is as follows:

[0047]

[0048] Where τ is the membrane time constant, u l [t] represents the membrane voltage of the multi-level neuron structure in layer l at time t. represents the membrane voltage of the i-th neuron structure in layer l at time t-1, Represents the current of the i-th small neuron structure in layer l at time t.

[0049] Step 3: After introducing the multi-level activation design, the complementary potential energy is then introduced to realize the complementary activation design, that is, to obtain the multi-level complementary neuron structure;

[0050] In one embodiment, based on step 2, a complementary activation setting is further introduced. Figure 3 As shown in the figure, the improved structure of each small neuron has additionally compensated the complementary potential energy on the basis of the original membrane voltage. In the traditional LIF model, as the time step increases, the membrane voltage decay rate accelerates and the time gradient gradually disappears. The complementary potential energy effectively improves the problem of the disappearance of the time gradient. The corresponding formula is as follows:

[0051]

[0052] in represents the membrane voltage of the ith neuron in layer l at time t, V th represents the membrane voltage threshold of the neurons in this layer, s l [t] represents whether a pulse is generated at time t in layer l. It represents the size of the "complementary potential" of the i-th neuron in layer l at time t. α represents the learning rate. In the formula, when a neuron in layer l pulses at time t, its corresponding membrane voltage u l [t] will decline, through additional supplementation Slow down the pace of decline.

[0053] Among them, the complementary activation setting introduces complementary potential energy. In the propagation process of the SNN network, the complementary potential energy needs to be dynamically adjusted. After the pulse signal is generated, the complementary potential energy needs to be supplemented to a certain extent. If no pulse signal is generated, the complementary potential energy should also be weakened accordingly to ensure the propagation trend of the overall network. At the same time, it can also weaken the membrane voltage decay rate and reduce the gradient vanishing effect under large time steps. The specific formula is as follows:

[0054]

[0055] in, It represents the size of the "complementary potential" of the i-th neuron in layer l at time t, and β is a hyperparameter used to control the growth rate of the complementary potential. l [t]=1 means a pulse signal is generated. That is, the complementary potential energy of the ith neuron in the lth layer at the current time step will increase by a certain amount using the hyperparameter β as the learning rate. If there is no pulse signal, then It will imitate the membrane voltage decay process and reduce it to a certain extent.

[0056] Step 4: Use the image classification data set to train the multi-level complementary spiking neural network model, and use the trained multi-level complementary spiking neural network model for image classification.

[0057] Specifically, during the spiking neural network training process, since the structure of the original LIF model is changed, the entire back propagation process also needs to be updated. The specific formula is as follows:

[0058]

[0059] in The gradient of the weights of layer l, is the gradient of the bias of the lth layer. T represents the time step, and K represents the number of small neurons in the multi-level activation, which is generally set to 3.

[0060] After the above steps, we have basically constructed a backbone of a multi-level complementary pulse neural network, such as Figure 4 As shown, where s l [t] represents the input of layer l at time t, O l [t] represents the output of layer l at time t, Conv is the convolutional layer, BN is the batch-normal layer, LIF is the LIF pulse neuron structure, and our is the multi-level complementary pulse neuron structure, that is, the improved neuron structure of the present invention. Figure 4 In this embodiment, the Spike-ResNet (a SNN network structure) structure is used as an example, where Figure 4 (a) in the figure is Spike-ResNet. It can be seen that it uses the LIF neuron model. After the above steps, we get the improved Spike-ResNet as follows: Figure 4 As shown in (b), our module represents the improved multi-level complementary pulse neuron structure of this embodiment. It can be seen that the overall network skeleton structure has not changed before and after the improvement, so the order of magnitude of operations has not changed before and after the improvement. However, through the design of our multi-level complementary mechanism, the problem of gradient disappearance under the LIF model can be effectively improved. Next, we only need to train the new network structure, and the training process is no different from before the improvement.

[0061] A control test was conducted before and after the improvement of the Spike-ResNet network to test the effectiveness of the multi-level pulse complementary structure. To ensure fair training, the same random seed, architecture, loss function, weight initialization and all hyperparameters were set for both. In addition, to ensure the accuracy and universality of the test, this experiment selected CIFAR-10, CIFAR-100, ImageNet and DVS-Gesture image datasets for testing. The test results are shown in the figure below. Figure 5 As shown, the horizontal axis represents different data sets, and the vertical axis represents the accuracy achieved by the LIF structure (left) and the multi-level pulse complementary structure (right) under the current data set. It can be seen that the improved network structure has certain performance improvements under different image data sets.

[0062] In summary, after all the steps are completed, a multi-level complementary pulse neural network structure with better accuracy and similar energy consumption than the original pulse neural network can be obtained.

[0063] The present invention also provides an image classification device based on a multi-level complementary pulse neural network model, comprising:

[0064] A data acquisition module, used to acquire image classification data sets;

[0065] A model building module is used to build a multi-level complementary pulse neural network model, including: using a traditional LIF neuron structure to build an SNN network model; reconstructing the LIF neuron structure in the SNN network model and introducing a multi-level activation design, wherein the multi-level activation design uses multiple small neuron structures to build a new multi-level neuron structure; after introducing the multi-level activation design, then introducing complementary potential energy to realize the complementary activation design, that is, obtaining a multi-level complementary neuron structure;

[0066] The model training module is used to train the multi-level complementary spiking neural network model using an image classification data set, and the trained multi-level complementary spiking neural network model is used for image classification.

[0067] It should be noted that the device embodiment shown in this embodiment matches the content of the above method embodiment, and the content of the above method embodiment can be referred to, which will not be repeated here.

[0068] Figure 6 This is a schematic diagram of an electronic device structure provided by an embodiment of the present invention. Figure 6 The electronic device provided in this embodiment includes: a memory and a processor, wherein the memory is used to store information including program instructions, and the processor is used to control the execution of the program instructions. When the program instructions are loaded and executed by the processor, an on-chip optical quantum bit feature enhancement extraction method of the present invention is implemented.

[0069] It should be noted that, in addition to Figure 6 In addition to the memory and processor shown, the electronic device may also include other hardware according to its actual functions, which will not be described in detail.

[0070] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the above-mentioned on-chip optical quantum bit feature enhancement extraction method is implemented.

[0071] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of any device with data processing capability and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0072] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned on-chip optical quantum bit feature enhancement extraction method.

[0073] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0074] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0075] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1A function specified in one or more boxes.

[0076] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0077] The above embodiments are only used to illustrate the design ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design ideas disclosed by the present invention are within the protection scope of the present invention.

Claims

1. An image classification method based on a multi-level complementary pulse neural network model, characterized in that: include: Get the image classification dataset; Constructing a multi-level complementary spiking neural network model, including: using a traditional LIF neuron structure to construct an SNN network model; reconstructing the LIF neuron structure in the SNN network model, introducing a multi-level activation design, wherein the multi-level activation design uses a plurality of small neuron structures to construct a new multi-level neuron structure; after introducing the multi-level activation design, then introducing complementary potential energy to realize the complementary activation design, that is, obtaining a multi-level complementary neuron structure; The multi-level complementary spiking neural network model is trained using an image classification data set, and the trained multi-level complementary spiking neural network model is used for image classification.

2. The image classification method based on a multi-level complementary pulse neural network model according to claim 1, characterized in that: The multi-level activation design uses multiple small neuron structures to construct a new multi-level neuron structure, and uses the membrane voltage accumulation process under the new structure to replace the membrane voltage accumulation process under the original LIF model. The specific formula is as follows: Where τ is the membrane time constant, u l [t] represents the membrane voltage of the multi-level neuron structure in layer l at time t. represents the membrane voltage of the i-th neuron structure in layer l at time t-1, Represents the current of the i-th small neuron structure in layer l at time t.

3. The image classification method based on a multi-level complementary pulse neural network model according to claim 1, characterized in that: After introducing the multi-stage activation design, complementary potential energy is then introduced to realize the complementary activation design, including: The complementary activation design introduces an additional "complementary potential" condition in the LIF structure to slow down the membrane voltage decay after pulse activation occurs. The membrane voltage decay process under the new structure replaces the decay process under the original LIF model. The formula is as follows: in represents the membrane voltage of the ith neuron in layer l at time t, V th represents the membrane voltage threshold of the neurons in this layer, s l [t] represents whether a pulse is generated at time t in layer l; It represents the size of the "complementary potential" of the i-th neuron in layer l at time t. α represents the learning rate. In the formula, when a neuron in layer l at time t generates a pulse, its corresponding membrane voltage u l [t] will decline, through additional supplementation Slowing down the pace of decline.

4. The image classification method based on a multi-level complementary pulse neural network model according to claim 1, characterized in that: The complementary potential energy is dynamically adjusted according to the pulse activation situation, and the specific formula is as follows: in It represents the size of the "complementary potential" of the i-th neuron in layer l at time t. β is a hyperparameter used to control the growth rate of the complementary potential.

5. The image classification method based on a multi-level complementary pulse neural network model according to claim 1, characterized in that: Also includes: The back propagation function is adjusted according to the current multi-level complementary neuron structure. The specific formula is as follows: Where L represents the loss value, The gradient of the weights of layer l, is the gradient of the bias of the lth layer, T represents the time step, and K represents the number of small neurons in the multi-level activation.

6. An image classification device based on a multi-level complementary pulse neural network model, characterized in that: include: A data acquisition module, used to acquire image classification data sets; A model building module is used to build a multi-level complementary pulse neural network model, including: using a traditional LIF neuron structure to build an SNN network model; reconstructing the LIF neuron structure in the SNN network model and introducing a multi-level activation design, wherein the multi-level activation design uses multiple small neuron structures to build a new multi-level neuron structure; after introducing the multi-level activation design, then introducing complementary potential energy to realize the complementary activation design, that is, obtaining a multi-level complementary neuron structure; The model training module is used to train the multi-level complementary spiking neural network model using an image classification data set, and the trained multi-level complementary spiking neural network model is used for image classification.

7. An electronic device comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement an image classification method based on a multi-level complementary pulse neural network model as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, an image classification method based on a multi-level complementary pulse neural network model as described in any one of claims 1 to 5 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the image classification method based on a multi-level complementary pulse neural network model described in any one of claims 1 to 5 is implemented.

Images