Image classification method and device based on time domain diversity spiking neural network

Abstract

The invention provides an image classification method and device based on a time domain diversity spiking neural network. The method comprises the steps of obtaining a target image; inputting the target image into an image classification model to obtain a classification result output by the image classification model; wherein the image classification model is obtained by training a sample image with a classification label, the image classification model comprises an encoder and a decoder, the encoder adopts a time domain diversity spiking neural network, and part of spiking neurons in the time domain diversity spiking neural network are spiking neurons with periodic switching time dynamic states. By adopting the above technical scheme, the problem that the inference overhead is large when the spiking neural network processes the image classification task in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image classification method and device based on a time-domain diversity pulse neural network. Background Art

[0002] Spiking Neural Network (SNN) is different from traditional artificial neural network (ANN). SNN processes information through the transmission of pulse signals and has the ability to process time series information. It is widely used in image classification tasks in the image field.

[0003] Spiking neural networks rely on neurons at different time steps to perform reasoning. Neurons at each time step in existing spiking neural networks participate in reasoning when processing image classification tasks, resulting in high reasoning overhead. Summary of the invention

[0004] The present invention provides an image classification method and device based on a time-domain diversity spiking neural network, which are used to solve the problem of high inference overhead of the spiking neural network when processing image classification tasks in the prior art.

[0005] In a first aspect, the present invention provides an image classification method based on a time-domain diversity pulse neural network, comprising: obtaining a target image. Inputting the target image into an image classification model to obtain a classification result output by the image classification model. The image classification model is trained based on sample images with classification labels, and the image classification model includes an encoder and a decoder. The encoder adopts a time-domain diversity pulse neural network, and some of the pulse neurons in the time-domain diversity pulse neural network are pulse neurons with periodic switching time dynamics.

[0006] Optionally, the time-domain diversity spiking neural network includes multiple subnetworks, each subnetwork corresponds to a time step, each subnetwork includes multiple network blocks connected in sequence, and the multiple network blocks are respectively used to extract features of different levels from low to high and input them into the subnetwork of the next time step. Among them, the first layer and the last layer of spiking neurons in each network block are spiking neurons with complete open time dynamics, and the other spiking neurons except the first layer and the last layer of spiking neurons are spiking neurons with periodic switch time dynamics.

[0007] Optionally, the information transfer of the spiking neuron with complete turn-on temporal dynamics is reversible in the time dimension.

[0008] Optionally, in a time-diverse spiking neural network, only the spiking neurons with complete on-time dynamics in the last time step store the membrane potential.

[0009] Optionally, in the time-domain diversity pulse neural network, the features extracted from the nth network block and the n+1th network block in the subnetwork corresponding to the tth time step are fused with the features extracted from the nth network block in the subnetwork corresponding to the t+1th time step, and used as the input of the n+1th network block in the subnetwork corresponding to the t+1th time step. Wherein, t=1, 2, ..., T-1, T is the number of time steps, n=1, 2, ..., N-1, N is the number of network blocks contained in each subnetwork.

[0010] Optionally, the time-domain diversity pulse neural network also includes a coding layer, which is used to encode the input image and divide the encoded features into T groups, and then input each group of features into a sub-network corresponding to a time step, where T is the number of time steps.

[0011] In a second aspect, the present invention provides an image classification device based on a time-domain diversity pulse neural network, comprising: an acquisition module for acquiring a target image. A processing module for inputting the target image into an image classification model to obtain a classification result output by the image classification model. The image classification model is trained based on sample images with classification labels, and the image classification model includes an encoder and a decoder. The encoder adopts a time-domain diversity pulse neural network, and some of the pulse neurons in the time-domain diversity pulse neural network are pulse neurons with periodic switching time dynamics.

[0012] In a third aspect, the present invention also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the image classification method based on the time-domain diversity pulse neural network as described above is implemented.

[0013] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned image classification methods based on time-domain diversity pulse neural networks.

[0014] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-mentioned image classification methods based on time-domain diversity pulse neural networks.

[0015] The image classification method and device based on the time-domain diversity pulse neural network provided by the present invention obtain a target image and input it into an image classification model to obtain a classification result. The image classification model is trained based on sample images with classification labels and consists of an encoder and a decoder. The encoder adopts a time-domain diversity pulse neural network, in which some pulse neurons are pulse neurons with periodic switching time dynamics. By introducing such periodically changing pulse neurons, it is possible to effectively capture and utilize timing information at different time steps, thereby improving the classification accuracy in image classification tasks. At the same time, the pulse neurons with time-domain diversity effectively reduce the energy consumption during reasoning. When processing image classification tasks, the technical effect of reducing reasoning energy consumption is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a flow chart of the image classification method based on the time-domain diversity pulse neural network provided by the present invention.

[0018] Figure 2 It is a schematic diagram of the overall forward propagation of the time-domain diversity pulse neural network provided by the present invention.

[0019] Figure 3 It is a schematic diagram of the propagation of a pulse neuron with complete opening time dynamics provided by the present invention.

[0020] Figure 4 It is a schematic diagram of the propagation of a pulse neuron with periodic switching time dynamics provided by the present invention when the time dynamics are turned off.

[0021] Figure 5 It is a schematic diagram of the propagation of a pulse neuron with periodic switching time dynamics provided by the present invention when the time dynamics are turned on.

[0022] Figure 6 It is a structural schematic diagram of an image classification device based on a time-domain diversity pulse neural network provided by the present invention.

[0023] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] Figure 1 FIG. 1 is a flow chart of an image classification method based on a time-domain diversity pulse neural network provided by the present invention. Figure 1 As shown, the method includes the following steps S101-S102.

[0026] Step S101: Acquire a target image.

[0027] As an optional embodiment, the target image can be an image of any format acquired by a camera or other image acquisition device, and used as input in the image classification task. The present invention does not impose any further restrictions on the format and acquisition method of the image.

[0028] Step S102: input the target image into the image classification model to obtain the classification result output by the image classification model.

[0029] Among them, the image classification model is trained based on sample images with classification labels. The image classification model includes an encoder and a decoder. The encoder adopts a time-domain diversity pulse neural network. Some pulse neurons in the time-domain diversity pulse neural network are pulse neurons with periodic switching time dynamics.

[0030] As an optional embodiment, the neurons in the diversity pulse neural network of the present invention are divided into two categories, one is the pulse neurons with periodic switching time dynamics, and the other is the pulse neurons with complete opening time dynamics. The present invention does not further limit the period of the time dynamics of the pulse neurons with periodic switching time dynamics, and the present invention does not further limit the number of pulse neurons with periodic switching time dynamics.

[0031] As an optional embodiment, the image classification model includes two parts: an encoder and a decoder. The encoder adopts a time-domain diversity pulse neural network. Some of the pulse neurons in the time-domain diversity pulse neural network are pulse neurons with periodic switching time dynamics. By introducing pulse neurons that dynamically adjust the time dynamics, the model can capture timing information in different time steps, thereby enhancing the image classification model's ability to process complex timing features. During the classification process, the encoder encodes the information in the input image into a pulse signal, and outputs the final classification result after processing by the decoder.

[0032] As an optional embodiment, the decoder may be a lightweight ANN decoder, and the present invention does not impose any further restrictions on the type of the decoder.

[0033] Through the above steps S101-S102, by acquiring the target image and inputting it into the image classification model, the classification result is obtained. The image classification model is trained based on sample images with classification labels and consists of an encoder and a decoder. The encoder adopts a time-domain diversity pulse neural network, in which some pulse neurons are pulse neurons with periodic switching time dynamics. By introducing such periodically changing pulse neurons, it is possible to effectively capture and utilize timing information at different time steps, thereby improving the classification accuracy in image classification tasks. At the same time, the pulse neurons with time-domain diversity effectively reduce the energy consumption during reasoning. When processing image classification tasks, the technical effects of low storage overhead and low reasoning energy consumption are achieved.

[0034] In an exemplary embodiment, the time-domain diversity pulse neural network also includes a coding layer, which is used to encode the input image and divide the encoded features into T groups, and then input each group of features into a sub-network corresponding to a time step, where T is the number of time steps.

[0035] As an optional embodiment, Figure 2 As shown, it is a schematic diagram of the overall forward propagation of the time-domain diversity pulse neural network provided by the present invention. The present invention designs the input coding and network structure in blocks. Specifically, the input picture is encoded only once through the coding layer, and the encoded input features are evenly divided into T groups as the input of each sub-network, avoiding the encoding of multiple time steps and repeated input, reducing the number of parameters of the model and reducing the energy consumption of reasoning. Correspondingly, the entire pulse neural network is also divided into T sub-networks, each sub-network is activated and processes the input of one time step, and the sub-networks only transmit time information on the pulse neurons open in the time domain, and other pulse neurons with closed time dynamics only accept spatial input. By encoding the image features in the input stage and dividing them into multiple time steps for processing, the diversity of time series information is fully utilized. The sub-networks of different time steps process different features, so that the time-domain diversity pulse neural network can gradually capture the changes in time domain features, and improve the sensitivity and processing ability of image classification to time series dynamic information. At the same time, the mechanism of dividing the features into multiple time steps reduces the computational burden of a single time step, avoids the centralized processing of too many features in the same time step, and optimizes the overall computational efficiency. In addition, the distributed processing design of time domain features reduces the computational redundancy in the reasoning stage, enabling the time domain diversity pulse neural network to reduce reasoning energy consumption while ensuring high classification accuracy.

[0036] In an exemplary embodiment, the time-domain diversity spiking neural network includes multiple subnetworks, each subnetwork corresponds to a time step, each subnetwork includes multiple network blocks connected in sequence, and the multiple network blocks are respectively used to extract features of different levels from low to high and input them into the subnetwork of the next time step. Among them, the first and last layers of spiking neurons in each network block are spiking neurons with complete open time dynamics, and the other spiking neurons except the first and last layers of spiking neurons are spiking neurons with periodic switch time dynamics.

[0037] As an optional embodiment, Figure 2 In the example, each time step corresponds to a sub-network, and each sub-network contains multiple sequentially connected network blocks ( Figure 2 (represented as each rounded box in the figure).

[0038] As an optional embodiment, Figure 2 As shown in Figure 2, in each network block, the red neurons represent spiking neurons with full on-time dynamics. Except for the first and last layers of spiking neurons, the other spiking neurons are spiking neurons with periodic on-off time dynamics (i.e. Figure 2 The black, yellow and green neurons in each network block in the figure). The switching periods of the time dynamics of the spiking neurons with various periodic switching time dynamics are different, that is, the switching periods of the time dynamics of the black, yellow and green neurons in the figure are different.

[0039] As an optional embodiment, for a pulse neuron with periodic switching time dynamics, the time domain propagation path is only opened at a fixed period. The switching period of the time dynamics of the pulse neuron with periodic switching time dynamics can be multiple, that is, Figure 2 In the structure shown, each network block may only have two types of spiking neurons with red full-on time dynamics and yellow periodic switching time dynamics. The present invention does not impose further restrictions on the number of spiking neurons with each switching period of time dynamics and the frequency of the switching period of time dynamics. Long short-term memory is adaptively captured by setting different cycle sizes for spiking neurons with different periodic switching time dynamics in each layer.

[0040] In the above embodiments, by designing multi-level feature extraction and dynamic neurons, the diversity pulse neural network can more effectively capture complex timing information. In addition, the combination of pulse neurons with complete turn-on time dynamics and pulse neurons with periodic switch time dynamics provides higher flexibility and efficiency, enabling the diversity pulse neural network to achieve efficient information processing and transmission in different time steps, thereby reducing inference energy consumption.

[0041] In an exemplary embodiment, the information transfer of a spiking neuron with complete turn-on temporal dynamics is reversible in the time dimension.

[0042] As an optional embodiment, Figure 3 FIG. 1 is a schematic diagram of the propagation of a pulse neuron with complete turn-on time dynamics provided by the present invention. The forward propagation process of a pulse neuron with complete turn-on time dynamics is shown in FIG. Figure 3 As shown in the left half of , the back propagation process of the spiking neuron with complete time dynamics is as follows Figure 3 As shown in the right half of , the pulse neurons with complete time-dynamic opening in the present application are similar to the neurons in the existing pulse neural network, and transmit information in the time and space dimensions. The weight update is also consistent with the neurons in the existing pulse neural network, which will not be repeated here.

[0043] It should be noted that Figure 3 middle and Respectively Tier The neuron membrane potential and output pulse at the time step, For the network The weight of the layer. , , They are respectively the loss functions about , and The derivative of .

[0044] As an optional embodiment, for a spiking neuron with periodic on-off time dynamics whose closing time dynamics are in a spiking neuron with periodic on-off time dynamics, information is transmitted only in the spatial dimension, the weight update also depends only on the spatial gradient, and the membrane potential is calculated only by the input pulse of the current time step.

[0045] As an optional embodiment, Figure 4 Schematic diagram of the propagation of the pulse neuron with periodic switching time dynamics provided by the present invention when the time dynamics are turned off. Figure 4 As shown in the left half of the figure, the back propagation process of the pulse neuron with periodic switching time dynamics is shown in Figure 4 The right half of .

[0046] As an optional embodiment, for the pulse neurons with periodic switching time dynamics and turn-on time dynamics in the pulse neurons with periodic switching time dynamics, similar to the pulse neurons with complete turn-on time dynamics, information is transmitted in the time and space dimensions, and the weight update is also consistent with the pulse neurons with complete turn-on time dynamics.

[0047] It should be noted that Figure 4 middle and Respectively Tier The neuron membrane potential and output pulse at the time step, For the network The weight of the layer. ， ， They are respectively the loss functions about , and The derivative of .

[0048] As an optional embodiment, Figure 5 FIG. 1 is a schematic diagram of the propagation of a pulse neuron with periodic switching time dynamics provided by the present invention when the time dynamics are turned on. The forward propagation process of a pulse neuron with periodic switching time dynamics when the time dynamics are turned on is as follows: Figure 5 As shown in the left half of the figure, the back propagation process of the pulse neuron with periodic switching time dynamics is shown in Figure 5 The right half of .

[0049] As an optional embodiment, during the forward propagation process, only the complete time transmission of the spiking neurons with complete turn-on time dynamics in the spiking neural network is retained, and other spiking neurons with periodic switch time dynamics perform periodic information transmission in the time domain.

[0050] It should be noted that Figure 5 middle and Respectively Tier The neuron membrane potential and output pulse at the time step, For the network The weight of the layer. ， ， They are respectively the loss functions about , and The derivative of .

[0051] In an exemplary embodiment, in a time-domain diverse spiking neural network, only the spiking neurons with complete on-time dynamics in the last time step store the membrane potential.

[0052] As an optional embodiment, for a spiking neuron with full time dynamics, information is transferred in both time and space dimensions at each time step. Since the information transfer of a spiking neuron with full time dynamics is reversible in the time dimension, the membrane potential of a certain time step can be calculated from the membrane potential of the next time step. Therefore, when calculating the gradient, only the membrane potential of the last time step needs to be stored, and the membrane potential and activation value of all previous time steps can be reversely derived, thereby reducing storage overhead.

[0053] As an optional embodiment, Layer, t The membrane potential at the time step ( ) can be expressed as ,in Indicates the number of network layers, and Respectively represent Layer, t The membrane potential and output pulse at the time step, is the weight, is the attenuation factor.

[0054] In an exemplary embodiment, in a time-domain diversity pulse neural network, the features extracted from the nth network block and the n+1th network block in the subnetwork corresponding to the tth time step are fused with the features extracted from the nth network block in the subnetwork corresponding to the t+1th time step, and then used as the input of the n+1th network block in the subnetwork corresponding to the t+1th time step. Wherein, t=1, 2, ..., T-1, T is the number of time steps, n=1, 2, ..., N-1, N is the number of network blocks contained in each subnetwork.

[0055] As an optional embodiment, for the pulse neurons with fully turned on time dynamics, the effectiveness of the model is improved by multi-level feature fusion, and the high-level features of several subsequent blocks of the previous time step are fused into the information of the current time step, which is still reversible in the time dimension. Specifically, in the subnetwork corresponding to the tth time step, the time-domain diversity pulse neural network extracts features from the nth network block and the n+1th network block respectively, and these features are fused with the features extracted from the nth network block in the subnetwork corresponding to the t+1th time step. The fused features are used as the input of the n+1th network block in the t+1th time step. This feature fusion mechanism can share information between different time steps and effectively capture the dynamic changes of time series information. By fusing the features of the tth and t+1th time steps, not only the important information of the current time step is retained, but also the feature representation of the previous time step is combined to form a more time-series-related input. The performance of the network in time series data processing is improved, especially in the expression and understanding of complex time domain features. In addition, fusing multi-time step information helps to reduce the redundancy of calculations and improve the energy efficiency ratio of the reasoning stage.

[0056] The executor of each step in the above method can be an image classification device based on a time-domain diversity pulse neural network. The device can be implemented through software and / or hardware. The device can be integrated in an electronic device. The electronic device can be a terminal device (such as a smart phone, a personal computer, etc.), or a server (such as a local server or a cloud server, or a server cluster, etc.), or a processor, or a chip, etc.

[0057] Obviously, the above described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. In order to better understand the above method, the above process is described below in conjunction with the embodiments, but the technical solutions of the embodiments of the present invention are not limited.

[0058] The present invention adopts a periodic switch design for the time dynamics of most pulse neurons with incomplete time dynamics, and designs multi-level time-reversible information transmission for pulse neurons with complete time dynamics that retain time dynamics. The input encoding method and the overall network architecture are designed in blocks, which can break through the performance bottleneck of storage and training time of large-scale pulse neural networks without sacrificing performance.

[0059] The present invention provides two types of designs for the forward propagation of spike neurons in the time dimension. Figure 2 As shown by the non-red spiking neurons in the spiking neural network, the temporal dynamics of most spiking neurons in the spiking neural network are periodically opened or closed. Different cycle sizes are set for different neurons in each layer. The black, yellow, and green colors in the figure represent different cycles. Figure 2As shown in the red spiking neurons in the figure, the complete time forward propagation is retained for them, and the forward propagation of these spiking neurons with complete time dynamics is designed to be reversible in the multi-level time dimension to achieve low storage overhead during training. Specifically, the spiking neural network based on time domain diversity is divided into several network blocks, and the corresponding features are from low level to high level. In the forward propagation process of the time domain diversity spiking neural network, only the complete time propagation of the first and last layers of spiking neurons in each block of the network is retained, and the time propagation of other spiking neurons is periodically turned on.

[0060] For spiking neurons with periodic on-off time dynamics, information is transferred only in the spatial dimension. The membrane potential is calculated only from the input spikes at the current time step, and the weight update also depends only on the spatial gradient.

[0061] For the pulse neurons with periodic switching time dynamics, similar to the ordinary pulse neurons in the prior art, information is transmitted in the time and space dimensions, and the weight update is also consistent with the ordinary pulse neurons.

[0062] For spiking neurons with fully turned-on temporal dynamics, similar to ordinary spiking neurons in the prior art, information is transferred in both the temporal and spatial dimensions, and weight updates also depend on both spatial and temporal gradients. Since the complete information transfer is reversible in the temporal dimension, the membrane potential of a certain time step can be calculated from the membrane potential of the next time step. Therefore, when calculating the gradient, only the membrane potential of the last time step needs to be stored, and the membrane potential and activation values ​​of all previous time steps can be reversely derived, thereby reducing storage overhead.

[0063] For spiking neurons with fully turned-on temporal dynamics, a multi-level feature fusion is further designed to improve the effectiveness of the model: the high-level features of the subsequent stages of the previous time step, that is, the deeper membrane potential, are fused into the features of the current time step.

[0064] The forward and backward propagation processes of a spike neuron with full temporal dynamics are shown in Figure 2. Figure 3 As shown ( Figure 3 The left half is the forward propagation. Figure 3 The right half is the back propagation. The forward and back propagation processes of the pulse neuron with periodic switching time dynamics are shown in Figure 2. Figure 4 As shown ( Figure 4 The left half is the forward propagation. Figure 4 The right half is the back propagation), the forward propagation and back propagation process of the pulse neuron with periodic switching time dynamics is shown in Figure 5 As shown ( Figure 5 The left half is the forward propagation. Figure 5 The right half is back propagation).

[0065] The formula for calculating the membrane potential during forward propagation is: ,in, Indicates the number of network layers, and Respectively represent Layer, t The membrane potential and output pulse at the time step, is the weight, is the attenuation factor. In practical applications, In the above embodiment equal.

[0066] The calculation formula of the membrane potential during the back propagation process has been explained in the above embodiment and will not be repeated here.

[0067] The present invention also designs the input coding and network structure in blocks, avoiding the coding of multiple time steps and repeated input, which can reduce the number of model parameters and reduce the energy consumption of reasoning. Specifically, the input image is encoded only once through the coding layer, and the encoded input features are evenly divided into T groups as block inputs of T time steps. Correspondingly, the entire spiking neural network is also divided into T subnetworks, each subnetwork is activated and processes the input of one time step, and the subnetworks only transmit time information on the spiking neurons that are open in the time domain, while other spiking neurons that close the time dynamics only accept spatial input.

[0068] In addition, the present invention adopts a mask self-supervised pre-training plus fine-tuning training method, uses a time-domain diversity pulse neural network as the SNN encoder, and adds a lightweight ANN decoder. The image is randomly masked in units of patches, and the MSE mean square error loss is used as the loss function in pre-training to predict the normalized pixel value of the mask patch. The encoder only operates on visible pixels and avoids information leakage through sparse convolution. The decoder uses a small-scale Transformer to reconstruct the mask image. The decoder will be removed in the fine-tuning stage, and only the pre-trained encoder will be used for supervised training.

[0069] As an optional embodiment, the time-domain diversity pulse neural network in the present invention is used to perform performance tests on image classification under the ImageNet dataset for models with two magnitudes of parameter amounts of 15.2M and 29.8M, and a classification accuracy of 73.2% is achieved under the conditions of 29.8M and a time step of 4. At present, the time-domain diversity pulse neural network adopted by the present invention has achieved the best performance in the SNN model based on CNN (ResNet), while using the least storage, training time and energy consumption. Compared with the current best Transformer-based SNN model spike-driven Transformer, the present invention achieves similar classification accuracy, and at the same time has an 8.6-fold, 2.0-fold and 1.6-fold improvement in training storage overhead, training time and inference energy consumption, respectively.

[0070] Compared with the existing technology, the present invention simultaneously achieves low storage overhead, low energy consumption, large-scale scalability and high performance, and achieves classification accuracy close to the optimal result on the ImageNet dataset, while using lower training storage overhead, less training time and lower inference energy consumption.

[0071] The image classification device based on the time-domain diversity pulse neural network provided by the present invention is described below. The image classification device based on the time-domain diversity pulse neural network described below and the image classification method based on the time-domain diversity pulse neural network described above can be referenced to each other.

[0072] Figure 6 Schematic diagram of the structure of the image classification device based on the time domain diversity pulse neural network provided by the present invention. Figure 6 As shown, the device includes.

[0073] The acquisition module 610 is used to acquire a target image.

[0074] The processing module 620 is used to input the target image into the image classification model to obtain the classification result output by the image classification model.

[0075] Among them, the image classification model is trained based on sample images with classification labels. The image classification model includes an encoder and a decoder. The encoder adopts a time-domain diversity pulse neural network. Some pulse neurons in the time-domain diversity pulse neural network are pulse neurons with periodic switching time dynamics.

[0076] The above-mentioned device obtains the target image and inputs it into the image classification model to obtain the classification result. The image classification model is trained based on sample images with classification labels and consists of an encoder and a decoder. The encoder adopts a time-domain diversity pulse neural network, in which some pulse neurons are pulse neurons with periodic switching time dynamics. By introducing such periodically changing pulse neurons, it is possible to effectively capture and utilize timing information at different time steps, thereby improving the classification accuracy in image classification tasks. At the same time, the pulse neurons with time-domain diversity effectively reduce the energy consumption during reasoning. When processing image classification tasks, the technical effects of low storage overhead and low reasoning energy consumption are achieved.

[0077] In some embodiments, the time-domain diversity spiking neural network includes multiple subnetworks, each subnetwork corresponds to a time step, each subnetwork includes multiple network blocks connected in sequence, and the multiple network blocks are respectively used to extract features of different levels from low to high and input them into the subnetwork of the next time step. Among them, the first layer and the last layer of spiking neurons in each network block are spiking neurons with complete open time dynamics, and the other spiking neurons except the first layer and the last layer of spiking neurons are spiking neurons with periodic switch time dynamics.

[0078] In some embodiments, information transfer of spiking neurons with full turn-on temporal dynamics is reversible in the time dimension.

[0079] In some embodiments, in a time-domain diversity spiking neural network, only the spiking neurons with complete turn-on temporal dynamics in the last time step store the membrane potential.

[0080] In some embodiments, in a time-domain diversity pulse neural network, features extracted from the nth network block and the n+1th network block in the subnetwork corresponding to the tth time step are fused with features extracted from the nth network block in the subnetwork corresponding to the t+1th time step, and used as input to the n+1th network block in the subnetwork corresponding to the t+1th time step. Where t=1, 2, ..., T-1, T is the number of time steps, n=1, 2, ..., N-1, N is the number of network blocks contained in each subnetwork.

[0081] In some embodiments, the time-domain diversity pulse neural network also includes a coding layer, which is used to encode the input image and divide the encoded features into T groups, and then input each group of features into a sub-network corresponding to a time step, where T is the number of time steps.

[0082] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the image classification method based on the time-domain diversity pulse neural network, and the method includes.

[0083] Get the target image.

[0084] The target image is input into the image classification model to obtain the classification result output by the image classification model.

[0085] Among them, the image classification model is trained based on sample images with classification labels. The image classification model includes an encoder and a decoder. The encoder adopts a time-domain diversity pulse neural network. Some pulse neurons in the time-domain diversity pulse neural network are pulse neurons with periodic switching time dynamics.

[0086] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0087] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image classification method based on the time-domain diversity pulse neural network provided by the above-mentioned methods, which includes.

[0088] Get the target image.

[0089] The target image is input into the image classification model to obtain the classification result output by the image classification model.

[0090] Among them, the image classification model is trained based on sample images with classification labels. The image classification model includes an encoder and a decoder. The encoder adopts a time-domain diversity pulse neural network. Some pulse neurons in the time-domain diversity pulse neural network are pulse neurons with periodic switching time dynamics.

[0091] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the image classification method based on the time-domain diversity pulse neural network provided by the above-mentioned methods is implemented, and the method includes.

[0092] Get the target image.

[0093] The target image is input into the image classification model to obtain the classification result output by the image classification model.

[0094] Among them, the image classification model is trained based on sample images with classification labels. The image classification model includes an encoder and a decoder. The encoder adopts a time-domain diversity pulse neural network. Some pulse neurons in the time-domain diversity pulse neural network are pulse neurons with periodic switching time dynamics.

[0095] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0096] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image classification method based on time-domain diversity pulse neural network, characterized in that: include: Get the target image; Inputting the target image into an image classification model to obtain a classification result output by the image classification model; Among them, the image classification model is trained based on sample images with classification labels. The image classification model includes an encoder and a decoder. The encoder adopts a time-domain diversity pulse neural network. Some of the pulse neurons in the time-domain diversity pulse neural network are pulse neurons with periodic switching time dynamics.

2. The image classification method based on time-domain diversity pulse neural network according to claim 1 is characterized in that: The time-domain diversity pulse neural network includes a plurality of sub-networks, each of which corresponds to a time step, and each of which includes a plurality of network blocks connected in sequence, and the plurality of network blocks are respectively used to extract features of different levels from low to high and input them into the sub-network of the next time step; The first and last layers of pulse neurons in each network block are pulse neurons with fully turned-on time dynamics, and the other pulse neurons except the first and last layers of pulse neurons are pulse neurons with periodic switching time dynamics.

3. The image classification method based on time-domain diversity pulse neural network according to claim 2 is characterized in that: The information transmission of the spiking neuron with complete turn-on temporal dynamics is reversible in the time dimension.

4. The image classification method based on time-domain diversity pulse neural network according to claim 3 is characterized in that: In the time-domain diversity pulse neural network, only the pulse neurons with complete turn-on time dynamics in the last time step store membrane potential.

5. The image classification method based on time-domain diversity pulse neural network according to any one of claims 2 to 4, characterized in that: In the time-domain diversity pulse neural network, the features extracted from the nth network block and the n+1th network block in the subnetwork corresponding to the tth time step are fused with the features extracted from the nth network block in the subnetwork corresponding to the t+1th time step, and the fused features are used as the input of the n+1th network block in the subnetwork corresponding to the t+1th time step; wherein t=1, 2, ..., T-1, T is the number of time steps, n=1, 2, ..., N-1, N is the number of network blocks contained in each subnetwork.

6. The image classification method based on time-domain diversity pulse neural network according to claim 2 is characterized in that: The time-domain diversity pulse neural network also includes a coding layer, which is used to encode the input image and divide the encoded features into T groups, and then input each group of features into a sub-network corresponding to a time step, where T is the number of time steps.

7. An image classification device based on time-domain diversity pulse neural network, characterized in that: include: An acquisition module, used for acquiring a target image; A processing module, used for inputting the target image into an image classification model to obtain a classification result output by the image classification model; Among them, the image classification model is trained based on sample images with classification labels. The image classification model includes an encoder and a decoder. The encoder adopts a time-domain diversity pulse neural network. Some of the pulse neurons in the time-domain diversity pulse neural network are pulse neurons with periodic switching time dynamics.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the image classification method based on the time-domain diversity pulse neural network as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image classification method based on the time-domain diversity pulse neural network as claimed in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the image classification method based on the time-domain diversity pulse neural network as claimed in any one of claims 1 to 6 is implemented.

Images