Image classification method and device based on time domain diversity pulse neural network

Abstract

The application provides an image classification method and device based on a time-domain diversity pulse neural network, and the method comprises the following steps: obtaining a target image; inputting the target image into an image classification model to obtain a classification result output by the image classification model; wherein the image classification model is trained based on sample images with classification labels, and the image classification model comprises an encoder and a decoder, the encoder adopts a time-domain diversity pulse neural network, and part of pulse neurons in the time-domain diversity pulse neural network are periodic switch time dynamic pulse neurons. By using the above technical scheme, the problem that the inference cost is high when a pulse neural network processes an image classification task in the prior art is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to an image classification method and device based on a time-domain diversity spiking neural network. BACKGROUND

[0002] A spiking neural network (SNN) is different from a traditional artificial neural network (ANN), and the SNN performs information processing through the transmission of pulse signals and has the ability to process time-series information, and is widely used in image classification tasks in the image field.

[0003] The spiking neural network relies on neurons at different time steps to perform inference work, and each neuron at each time step in the existing spiking neural network participates in the inference work when processing an image classification task, which has a large inference overhead. SUMMARY

[0004] The present application provides an image classification method and device based on a time-domain diversity spiking neural network to solve the problem of a large inference overhead of the existing spiking neural network when processing an image classification task.

[0005] In a first aspect, the present application provides an image classification method based on a time-domain diversity spiking neural network, comprising: obtaining a target image. The target image is input into an image classification model to obtain a classification result output by the image classification model. The image classification model is trained based on sample images with classification labels, and the image classification model includes an encoder and a decoder. The encoder adopts a time-domain diversity spiking neural network, and some spiking neurons in the time-domain diversity spiking neural network are periodically switched time dynamic spiking neurons.

[0006] Optionally, the time-domain diversity spiking neural network includes a plurality of sub-networks, each sub-network corresponding to a time step, and each sub-network including a plurality of network blocks connected in sequence, the plurality of network blocks being respectively used to extract different levels of features from low to high and input into a sub-network at a next time step. The first layer and the last layer of spiking neurons of each network block are complete-on time dynamic spiking neurons, and other spiking neurons except the first layer and the last layer of spiking neurons are periodically switched time dynamic spiking neurons.

[0007] Optionally, the information transmission of the complete-on time dynamic spiking neuron is reversible in the time dimension.

[0008] Optionally, in the time-domain diversity spiking neural network, only the complete-on time dynamic spiking neurons of the last time step store membrane potentials.

[0009] Optionally, in the time-domain diversity spiking neural network, the features extracted by the n th network block and the n+1 th network block in the sub-network corresponding to the t th time step are fused with the features extracted by the n th network block in the sub-network corresponding to the t+1 th time step, and the fused features are used as the input of the n+1 th network block in the sub-network corresponding to the t+1 th time step. Wherein, t=1, 2, …, T-1, T is the number of time steps, n=1, 2, …, N-1, N is the number of network blocks contained in each sub-network.

[0010] Optionally, the time-domain diversity spiking neural network further comprises an encoding layer, the encoding layer is configured to encode the input image, divide the encoded features into T groups, and then input each group of features into a sub-network corresponding to a time step, wherein T is the number of time steps.

[0011] In a second aspect, the present application provides an image classification device based on a time-domain diversity spiking neural network, comprising: an acquisition module configured to acquire a target image; and a processing module configured to input the target image into an image classification model to obtain a classification result output by the image classification model. The image classification model is trained based on sample images with classification labels. The image classification model comprises an encoder and a decoder. The encoder adopts a time-domain diversity spiking neural network. Some spiking neurons in the time-domain diversity spiking neural network are periodically switched time dynamic spiking neurons.

[0012] In a third aspect, the present application further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor implements the image classification method based on the time-domain diversity spiking neural network as described above when executing the computer program.

[0013] In a fourth aspect, the present application further provides a non-transitory computer readable storage medium having a computer program stored thereon. The computer program is executed by a processor to implement the image classification method based on the time-domain diversity spiking neural network as described above.

[0014] In a fifth aspect, the present application further provides a computer program product comprising a computer program. The computer program is executed by a processor to implement the image classification method based on the time-domain diversity spiking neural network as described above.

[0015] The application provides an image classification method and device based on a time-domain diversity spiking neural network. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0017] Figure 1 FIG. 1 is a flowchart of an image classification method based on a time-domain diversity spiking neural network provided by the application.

[0018] Figure 2 FIG. 2 is a schematic diagram of the overall forward propagation of the time-domain diversity spiking neural network provided by the application.

[0019] Figure 3 FIG. 3 is a schematic diagram of the propagation of a complete on-time dynamic spiking neuron provided by the application.

[0020] Figure 4 FIG. 4 is a schematic diagram of the propagation of a periodic on-off time dynamic spiking neuron in the off-time dynamic state provided by the application.

[0021] Figure 5 FIG. 5 is a schematic diagram of the propagation of a periodic on-off time dynamic spiking neuron in the on-time dynamic state provided by the application.

[0022] Figure 6 FIG. 6 is a structural schematic diagram of an image classification device based on a time-domain diversity spiking neural network provided by the application.

[0023] Figure 7 FIG. 7 is a structural schematic diagram of an electronic device provided by the application. DETAILED DESCRIPTION

[0024] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in conjunction with the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0025] Figure 1 is a flowchart of an image classification method based on a time-domain diversity spiking neural network provided by the present application. As shown in Figure 1 , the method comprises the following steps S101-S102.

[0026] Step S101, obtaining a target image.

[0027] As an optional embodiment, the target image can be an image of any format obtained by a camera or other image acquisition device as input in an image classification task. The present application does not make further limitation on the format and acquisition method of the image.

[0028] Step S102, inputting the target image into an image classification model to obtain a classification result output by the image classification model.

[0029] The image classification model is trained based on sample images with classification labels, and the image classification model comprises an encoder and a decoder. The encoder adopts a time-domain diversity spiking neural network, and part of the spiking neurons in the time-domain diversity spiking neural network are periodically switchable time dynamic spiking neurons.

[0030] As an optional embodiment, the neurons in the diversity spiking neural network in the present application are divided into two categories, one category is periodically switchable time dynamic spiking neurons, and the other category is completely on time dynamic spiking neurons. The period of the time dynamic of the periodically switchable time dynamic spiking neurons is not further limited in the present application, and the number of the periodically switchable time dynamic spiking neurons is not further limited in the present application.

[0031] As an optional embodiment, the image classification model comprises two parts of an encoder and a decoder. The encoder adopts a time-domain diversity spiking neural network, part of the spiking neurons in the time-domain diversity spiking neural network are periodically switchable time dynamic spiking neurons, the periodically switchable time dynamic spiking neurons are introduced to make the model capture the time sequence information in different time steps and enhance the processing ability of the image classification model for complex time sequence features. In the classification process, the encoder encodes the information in the input image into a pulse signal, and outputs the final classification result after processing by the decoder.

[0032] As an optional embodiment, the decoder can be a lightweight ANN decoder, and the application does not further limit the type of decoder.

[0033] Through the above steps S101-S102, by acquiring the target image and inputting it into the image classification model, the classification result is obtained. The image classification model is trained based on sample images with classification labels and is composed of an encoder and a decoder. The encoder adopts a time-domain diversity spiking neural network, in which part of the spiking neurons are periodically switched time dynamic spiking neurons. By introducing such periodically changing spiking neurons, time sequence information can be effectively captured and utilized at different time steps, thereby improving the classification accuracy in the image classification task. At the same time, the time-domain diversity spiking neurons effectively reduce the energy consumption during inference. In processing the image classification task, the technical effects of low storage overhead and low inference energy consumption are achieved.

[0034] In an exemplary embodiment, the time-domain diversity spiking neural network further comprises an encoding layer, the encoding layer being configured to encode the input image and divide the encoded features into T groups, and then input each group of features into a sub-network corresponding to a time step, where T is the number of time steps.

[0035] As an optional embodiment, as shown in Figure 2 The present application provides a schematic diagram of the overall forward propagation of the time-domain diversity spiking neural network. The present application divides the input encoding and network structure into blocks, specifically, the input image is encoded only once by the encoding layer, and the encoded input features are divided into T groups as the input of each sub-network, avoiding multiple time-step encoding and repeated input, reducing the parameter amount of the model and reducing the inference energy consumption. Correspondingly, the entire spiking neural network is also divided into T sub-networks, each sub-network is activated and processes the input of one time step, and only the time-domain open spiking neurons transmit time information between sub-networks, and other closed time dynamic spiking neurons only accept spatial input. By encoding the image features at the input stage and dividing them into multiple time steps for processing, the diversity of time sequence information is fully utilized. Different time step sub-networks process different features, so that the time-domain diversity spiking neural network can gradually capture the changes of time-domain features, improving the sensitivity and processing capacity of image classification to time dynamic information. At the same time, the mechanism of dividing features into multiple time steps reduces the computational burden of a single time step and avoids excessive features being processed in the same time step, optimizing the overall computational efficiency. In addition, the distributed processing design of time-domain features reduces the computational redundancy in the inference stage, so that the time-domain diversity spiking neural network can reduce the inference energy consumption while ensuring high classification accuracy.

[0036] In an exemplary embodiment, the time-domain diversity spiking neural network comprises a plurality of sub-networks, each sub-network corresponding to a time step, each sub-network comprising a plurality of network blocks connected in sequence, and the plurality of network blocks are respectively used to extract features of different levels from low to high and input to the sub-network of the next time step. Wherein, the first layer and the last layer of spiking neurons of each network block are full-on time dynamic spiking neurons, and the other spiking neurons except the first layer and the last layer of spiking neurons are periodic switch time dynamic spiking neurons.

[0037] As an optional embodiment, Figure 2 each time step corresponds to a sub-network, and a plurality of network blocks are connected in sequence in each sub-network Figure 2 indicated as each rounded box.

[0038] As an optional embodiment, as Figure 2 indicated, in each network block, the red neurons represent full-on time dynamic spiking neurons. The other spiking neurons except the first layer and the last layer of spiking neurons are periodic switch time dynamic spiking neurons (i.e. Figure 2 indicated, the black, yellow and green neurons in each network block). The time dynamic switch periods of the plurality of periodic switch time dynamic spiking neurons are different, that is, the time dynamic switch periods of the black, yellow and green neurons in the figure are different.

[0039] As an optional embodiment, for the periodic switch time dynamic spiking neurons, only the time-domain propagation path is opened on the fixed period, and the time dynamic switch period of the periodic switch time dynamic spiking neurons can be multiple, that is, as Figure 2 indicated, in the structure, each network block can only have two kinds of red full-on time dynamic spiking neurons and yellow periodic switch time dynamic spiking neurons. The present application does not further limit the number of spiking neurons and the frequency of time dynamic switch period for each time dynamic switch period. By setting different period sizes for different periodic switch time dynamic spiking neurons in each layer, long and short-term memories are adaptively captured.

[0040] In the above embodiment, by designing multi-level feature extraction and dynamic neurons, the diversity spiking neural network can more effectively capture complex time sequence information. In addition, the combination of full-on time dynamic spiking neurons and periodic switch time dynamic spiking neurons provides higher flexibility and efficiency, so that the diversity spiking neural network can realize efficient information processing and transmission in different time steps, and reduces the inference energy consumption.

[0041] In one exemplary embodiment, the information transmission of the full-onset-time dynamic spiking neuron is reversible in the time dimension.

[0042] As an alternative embodiment, Figure 3 is a schematic diagram of the propagation of the full-onset-time dynamic spiking neuron provided by the present application. The forward propagation process of the full-onset-time dynamic spiking neuron is shown in the left half of Figure 3 , and the backward propagation process of the full-onset-time dynamic spiking neuron is shown in the right half of Figure 3 . The full-onset-time dynamic spiking neuron in the present application is similar to the neuron in the existing spiking neural network, and performs information transmission in the time and spatial dimensions, and the weight update is consistent with the neuron in the existing spiking neural network, which will not be described here.

[0043] It should be noted that, Figure 3 and are the membrane potentials and output pulses of the neurons in the first layer at the first time step, respectively. is the weight of the network in the first layer. , , are the derivatives of the loss function with respect to , and .

[0044] As an alternative embodiment, for the off-time dynamic periodic switch-time dynamic spiking neuron in the periodic switch-time dynamic spiking neuron, only information transmission is performed in the spatial dimension, and the weight update also depends only on the spatial gradient, and the membrane potential is only calculated from the input pulse of the current time step.

[0045] As an alternative embodiment, Figure 4 is a schematic diagram of the propagation of the off-time dynamic periodic switch-time dynamic spiking neuron provided by the present application. The forward propagation process of the off-time dynamic periodic switch-time dynamic spiking neuron is shown in the left half of Figure 4 , and the backward propagation process of the off-time dynamic periodic switch-time dynamic spiking neuron is shown in the right half of Figure 4 .

[0046] As an alternative embodiment, for the on-time dynamic periodic switch-time dynamic spiking neuron in the periodic switch-time dynamic spiking neuron, similar to the full-onset-time dynamic spiking neuron, information transmission is performed in the time and spatial dimensions, and the weight update is consistent with the full-onset-time dynamic spiking neuron. ​

[0047] It should be noted that, Figure 4 In the above embodiment, and are the membrane potential and output spike of the neuron in the i-th layer at the t-th time step, respectively, is the weight of the i-th layer in the network. are the derivatives of the loss function with respect to , ， and ， ,

[0048] As an optional embodiment, Figure 5 is a schematic diagram of the propagation of the periodically switched time dynamic pulse neuron in the open time dynamic state. The forward propagation process of the periodically switched time dynamic pulse neuron in the open time dynamic state is shown in the left half of Figure 5 , and the backward propagation process of the periodically switched time dynamic pulse neuron in the open time dynamic state is shown in the right half of Figure 5 .

[0049] As an optional embodiment, only the complete time transfer of the complete open time dynamic pulse neuron in the pulse neural network is retained in the forward propagation process, and the other periodically switched time dynamic pulse neurons perform periodic information transfer in the time domain.

[0050] It should be noted that, Figure 5 In the above embodiment, and are the membrane potential and output spike of the neuron in the i-th layer at the t-th time step, respectively, is the weight of the i-th layer in the network. are the derivatives of the loss function with respect to , ， ， , and

[0051] In an exemplary embodiment, only the complete open time dynamic pulse neuron in the time domain diversity pulse neural network stores the membrane potential at the last time step.

[0052] ​​​​​​As an optional embodiment, for the full opening time dynamic pulse neuron, information transmission is performed in time and space dimensions at each time step. Since the information transmission of the full opening time dynamic pulse neuron is reversible in time dimension, the membrane potential at a certain time step can be calculated from the membrane potential at the next time step, so only the membrane potential at the last time step needs to be stored when calculating the gradient, and the membrane potential and activation value at all previous time steps can be derived reversely, thereby reducing the storage overhead.

[0053] As an optional embodiment, the first layer, the first t time step membrane potential ) can be represented as , wherein represents the number of network layers, and respectively represent the membrane potential and output pulse of the first layer, the first t time step, is the weight, is the decay factor.

[0054] In an exemplary embodiment, in the time domain diversity pulse neural network, the features extracted by the n-th network block and the n+1-th network block in the sub-network corresponding to the t-th time step are fused with the features extracted by the n-th network block in the sub-network corresponding to the t+1-th time step, and the fused features are used as the input of the n+1-th network block in the sub-network corresponding to the t+1-th time step. Wherein, t=1, 2, …, T-1, T is the number of time steps, n=1, 2, …, N-1, N is the number of network blocks contained in each sub-network.

[0055] As an optional embodiment, for the full opening time dynamic pulse neuron, the effectiveness of the model is improved by multi-level feature fusion, the high-level features of the subsequent blocks at the previous time step are fused into the information at the current time step, and the time dimension is still reversible. Specifically, the time domain diversity pulse neural network extracts features from the nth network block and the n+1th network block in the subnetwork corresponding to the tth time step, and these features are fused with the features extracted from the nth network block in the subnetwork corresponding to the t+1th time step. The fused features are used as the input of the n+1th network block in the t+1th time step. This feature fusion mechanism can share information between different time steps and effectively capture the dynamic changes of the time sequence information. By fusing the features of the tth and t+1th time steps, not only the important information of the current time step is retained, but also the feature representation of the previous time step is combined to form an input with more time sequence correlation. The performance of the network in time sequence data processing is improved, especially in the expression and understanding of complex time domain features. In addition, fusing multiple time step information helps to reduce the redundancy of calculation and improve the energy efficiency ratio in the reasoning stage.

[0056] The execution subject of each step in the above method can be an image classification device based on the time domain diversity pulse neural network. The device can be realized by software and / or hardware, and can be integrated in an electronic device. The electronic device can be a terminal device (such as a smart phone, a personal computer, etc.), a server (such as a local server or a cloud server, which can also be a server cluster, etc.), a processor, a chip, etc.

[0057] Obviously, the above-described embodiments are only a part of the embodiments of the present application, not all the embodiments. In order to better understand the above method, the above process is described in combination with the embodiments, but it is not used to limit the technical solutions of the embodiments of the present application, specifically.

[0058] The present application adopts a periodic switching design for the time dynamics of most non-full opening time dynamic pulse neurons, and designs a multi-level time reversible information transmission for the full opening time dynamic pulse neuron that retains the time dynamics. The input encoding method and the overall network architecture are designed in blocks, which can break through the performance bottleneck of storage and training time of large-scale pulse neural networks without sacrificing performance.

[0059] The present application designs two types for the forward propagation of pulse neurons in the time dimension. As shown in the non-red pulse neuron in Figure 2 , the time dynamics of most pulse neurons in the pulse neural network are periodically opened or closed. Different neurons in each layer are set to different cycle sizes, and black, yellow and green in the figure represent different cycles. As shown in Figure 2As shown in the red spiking neurons, their complete temporal forward propagation is preserved. Furthermore, the forward propagation of these fully enabled temporally dynamic spiking neurons is designed to be multi-layered and reversible in terms of temporal dimensions, achieving low storage overhead during training. Specifically, the temporally diverse spiking neural network is divided into several network blocks, with corresponding features ranging from low to high level. During the forward propagation of the temporally diverse spiking neural network, only the complete temporal propagation of the first and last layers of spiking neurons in each block is preserved, while the temporal propagation of other spiking neurons is periodically enabled.

[0060] For spiking neurons that periodically switch on and off time-dynamics, information is transmitted only in the spatial dimension. The membrane potential is calculated solely from the input pulse at the current time step, and weight updates depend only on the spatial gradient.

[0061] For spiking neurons with dynamic on-off cycles, similar to ordinary spiking neurons in the prior art, information is transmitted in the time and space dimensions, and the weight updates are also consistent with those of ordinary spiking neurons.

[0062] For a fully activated temporally dynamic spiking neuron, similar to ordinary spiking neurons in existing technologies, information is transmitted in both temporal and spatial dimensions, and weight updates depend on both spatial and temporal gradients. Since the complete information transmission is reversible in the temporal dimension, the membrane potential at a certain time step can be calculated from the membrane potential at the next time step. Therefore, when calculating gradients, only the membrane potential of the last time step needs to be stored, and the membrane potentials and activation values ​​of all previous time steps can be derived in reverse, thereby reducing storage overhead.

[0063] For spiking neurons with fully activated time dynamics, a multi-level feature fusion model is further designed to improve the effectiveness of the model: high-level features from the previous time step and subsequent stages, i.e., deeper membrane potentials, are fused into the features of the current time step.

[0064] The complete forward and backward propagation process of a time-dynamic spiking neuron is as follows: Figure 3 As shown ( Figure 3 The left half represents forward propagation. Figure 3 The right half represents backward propagation. The forward and backward propagation processes of time-dynamically controlled periodic switching neurons are as follows: Figure 4 As shown ( Figure 4 The left half represents forward propagation. Figure 4 The right half represents backward propagation. The forward and backward propagation processes of a time-dynamically controlled spiking neuron are as follows: (The text then describes the process of turning on and off the spiking neuron using a time-dynamically controlled periodic switch.) Figure 5 As shown ( Figure 5 The left half represents forward propagation. Figure 5 The right half represents backpropagation.

[0065] The calculation formula of the membrane potential in the forward propagation process is , wherein, represents the number of network layers, and respectively represent the membrane potential and output pulse of the first layer, the first t time step, is the weight, is the decay factor. In practical applications, is equal to in the above embodiment.

[0066] The calculation formula of the membrane potential in the backward propagation process has been described in the above embodiment, and will not be repeated here.

[0067] The present application also blocks the input encoding and network structure, avoids multi-time step encoding and repeated input, can reduce the model parameter quantity and reduce the inference energy consumption. Specifically, the input picture is encoded only once through the encoding layer, and the encoded input features are divided into T groups as the block input of T time steps. Correspondingly, the entire pulse neural network is also divided into T sub-networks, each sub-network is activated and processes the input of one time step, and only the time information is transmitted on the open time domain pulse neuron between the sub-networks, and the other closed time dynamic pulse neurons only accept spatial input.

[0068] In addition, the present application adopts a training method of mask self-supervised pre-training and fine-tuning, adopts a time domain diversity pulse neural network as an SNN encoder, and adds a light-weight ANN decoder. The image is randomly masked in units of patches, the MSE mean square error loss is used as the loss function in the pre-training, and the normalized pixel value of the masked patch is predicted. The encoder only operates the visible pixel points and avoids information leakage through sparse convolution, and the decoder uses a small-scale Transformer to reconstruct the masked image. In the fine-tuning stage, the decoder will be removed, and only the pre-trained encoder is used for supervised training.

[0069] As an optional embodiment, the time-domain diversity spiking neural network in the application is used to perform performance test of image classification on an ImageNet dataset for a model with a parameter quantity of 15.2M and 29.8M respectively, and a classification accuracy of 73.2% is reached in the case of 29.8M and 4 time steps. The time-domain diversity spiking neural network used in the application has the optimal performance in the SNN model based on CNN (ResNet), and the least storage, training time and energy consumption are used. Compared with the optimal SNN model based on Transformer, spike-driven Transformer, the application reaches similar classification accuracy, and the training storage overhead, training time and inference energy consumption are improved by 8.6 times, 2.0 times and 1.6 times respectively.

[0070] Compared with the prior art, the application simultaneously realizes low storage overhead, low energy consumption, large-scale expansion and high performance, and reaches similar classification accuracy to the optimal result on an ImageNet dataset, while using lower training time storage overhead, less training time and lower inference energy consumption.

[0071] The image classification device based on the time-domain diversity spiking neural network provided by the application is described below, and the image classification device based on the time-domain diversity spiking neural network described below can be mutually referred to the image classification method based on the time-domain diversity spiking neural network described above.

[0072] Figure 6 is a structural schematic diagram of the image classification device based on the time-domain diversity spiking neural network provided by the application. As shown in Figure 6 , the device includes.

[0073] The acquisition module 610 is configured to acquire a target image.

[0074] The processing module 620 is configured to input the target image into an image classification model to obtain a classification result output by the image classification model.

[0075] The image classification model is trained based on a sample image with a classification label, the image classification model includes an encoder and a decoder, the encoder adopts a time-domain diversity spiking neural network, and part of the spiking neurons in the time-domain diversity spiking neural network are periodic on-off time dynamic spiking neurons.

[0076] The device obtains a target image, inputs the target image into an image classification model, and obtains a classification result. The image classification model is trained based on sample images with classification labels and is composed of an encoder and a decoder. The encoder adopts a time-domain diversity spiking neural network, in which part of the spiking neurons are periodically switched time dynamic spiking neurons. By introducing such periodically changed spiking neurons, time sequence information can be effectively captured and utilized at different time steps, thereby improving the classification accuracy in the image classification task. At the same time, the time-domain diversity spiking neurons effectively reduce the energy consumption during inference. When processing the image classification task, the technical effects of low storage overhead and low inference energy consumption are achieved.

[0077] In some embodiments, the time-domain diversity spiking neural network includes a plurality of sub-networks, each corresponding to a time step, and each sub-network includes a plurality of network blocks connected in sequence, and the plurality of network blocks are respectively used to extract features of different levels from low to high and input to the sub-network of the next time step. Wherein, the first layer and the last layer of spiking neurons of each network block are full-on time dynamic spiking neurons, and other spiking neurons except the first layer and the last layer of spiking neurons are periodically switched time dynamic spiking neurons.

[0078] In some embodiments, the information transmission of the full-on time dynamic spiking neuron is reversible in the time dimension.

[0079] In some embodiments, in the time-domain diversity spiking neural network, only the full-on time dynamic spiking neuron of the last time step stores the membrane potential.

[0080] In some embodiments, in the time-domain diversity spiking neural network, the features extracted by the n th network block and the n+1 th network block in the sub-network corresponding to the t th time step are fused with the features extracted by the n th network block in the sub-network corresponding to the t+1 th time step, and the fusion result is used as the input of the n+1 th network block in the sub-network corresponding to the t+1 th time step. Wherein, t=1, 2, …, T-1, T is the number of time steps, n=1, 2, …, N-1, N is the number of network blocks included in each sub-network.

[0081] In some embodiments, the time-domain diversity spiking neural network further includes an encoding layer, which is used to encode the input image and divide the encoded features into T groups, and then input each group of features into the sub-network corresponding to a time step, wherein T is the number of time steps.

[0082] Figure 7 An example of an entity structure diagram of an electronic device is shown in Figure 7As shown, the electronic device can include a processor 710, a communications interface 720, a memory 730, and a communications bus 740, wherein the processor 710, the communications interface 720, and the memory 730 complete mutual communication through the communications bus 740. The processor 710 can invoke a logic instruction in the memory 730 to execute an image classification method based on a time-domain diversity spiking neural network, the method comprising.

[0083] Obtaining a target image.

[0084] Inputting the target image into an image classification model to obtain a classification result output by the image classification model.

[0085] The image classification model is trained based on a sample image with a classification label, and the image classification model includes an encoder and a decoder, the encoder adopts a time-domain diversity spiking neural network, and part of the spiking neurons in the time-domain diversity spiking neural network are periodically switched time dynamic spiking neurons.

[0086] In addition, the logic instruction in the memory 730 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0087] On the other hand, the present application also provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and when the computer program is executed by a processor, the computer can execute the image classification method based on the time-domain diversity spiking neural network provided by the above-mentioned method, the method comprising.

[0088] Obtaining a target image.

[0089] Inputting the target image into an image classification model to obtain a classification result output by the image classification model.

[0090] The image classification model is trained based on sample images with classification labels, and the image classification model comprises an encoder and a decoder.

[0091] In another aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image classification method based on the time-domain diversity spiking neural network provided by the above method, and the method comprises.

[0092] Obtaining a target image.

[0093] Inputting the target image into the image classification model to obtain a classification result output by the image classification model.

[0094] The image classification model is trained based on sample images with classification labels, and the image classification model comprises an encoder and a decoder.

[0095] The device embodiments described above are only schematic, wherein the units shown as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment. Those skilled in the art can understand and implement it without creative labor.

[0096] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software plus necessary general hardware platforms, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of software products, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and include a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0097] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; and although the present application has been described in detail with reference to the foregoing embodiments, it should be appreciated by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features thereof can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image classification method based on a temporal-domain diversity spiking neural network, characterized in that, include: Acquire the target image; The target image is input into an image classification model to obtain the classification result output by the image classification model; The image classification model is trained based on sample images with classification labels. The image classification model includes an encoder and a decoder. The encoder adopts a temporal-domain multivariable spiking neural network. Some of the spiking neurons in the temporal-domain multivariable spiking neural network are periodically switching time-dynamic spiking neurons. The temporal diversity spiking neural network includes multiple sub-networks, each sub-network corresponding to a time step, and each sub-network including multiple network blocks connected in sequence. The multiple network blocks are used to extract features at different levels from low to high and input them into the sub-network at the next time step. In each network block, the first and last layer spiking neurons are fully time-dynamic spiking neurons, while the other spiking neurons besides the first and last layer are periodically time-dynamic spiking neurons.

2. The image classification method based on temporal diversity spiking neural networks according to claim 1, characterized in that, The information transmission of the fully activated time-dynamic spiking neurons is reversible in the time dimension.

3. The image classification method based on temporal diversity spiking neural networks according to claim 2, characterized in that, In the aforementioned temporal-domain diverse spiking neural network, only the spiking neuron with complete turn-on time dynamics in the last time step stores the membrane potential.

4. The image classification method based on a temporal-domain diversity spiking neural network according to any one of claims 1 to 3, characterized in that, In the temporal-domain diverse spiking neural network, the features extracted from the nth and (n+1)th network blocks in the sub-network corresponding to the tth time step are fused with the features extracted from the nth network block in the sub-network corresponding to the (t+1)th time step, and then used as the input to the (n+1)th network block in the sub-network corresponding to the (t+1)th time step; where t = 1, 2, ..., T-1, T is the number of time steps, n = 1, 2, ..., N-1, and N is the number of network blocks contained in each sub-network.

5. The image classification method based on a temporal-domain diversity spiking neural network according to claim 1, characterized in that, The temporal diversity spiking neural network further includes an encoding layer, which encodes the input image and divides the encoded features into T groups. Each group of features is then input into a subnetwork corresponding to a time step, where T is the number of time steps.

6. An image classification device based on a temporal-domain diversity spiking neural network, characterized in that, include: The acquisition module is used to acquire the target image; The processing module is used to input the target image into an image classification model and obtain the classification result output by the image classification model; The image classification model is trained based on sample images with classification labels. The image classification model includes an encoder and a decoder. The encoder adopts a temporal-domain multivariable spiking neural network. Some of the spiking neurons in the temporal-domain multivariable spiking neural network are periodically switching time-dynamic spiking neurons. The temporal diversity spiking neural network includes multiple sub-networks, each sub-network corresponding to a time step, and each sub-network including multiple network blocks connected in sequence. The multiple network blocks are used to extract features at different levels from low to high and input them into the sub-network at the next time step. In each network block, the first and last layer spiking neurons are fully time-dynamic spiking neurons, while the other spiking neurons besides the first and last layer are periodically time-dynamic spiking neurons.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the image classification method based on a temporal diversity spiking neural network as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the image classification method based on a temporal diversity spiking neural network as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the image classification method based on a temporal diversity spiking neural network as described in any one of claims 1 to 5.

Images