Image recognition method and device based on SNN information supplement, equipment and medium

Abstract

The invention relates to the technical field of neural networks, and discloses an image recognition method, device and equipment based on SNN information supplementation and a medium, and the method comprises the steps: obtaining a target image, inputting a pulse neural network and a convolutional neural network in an image recognition model, and carrying out the feature extraction and forward propagation; in each corresponding processing stage, processing input features respectively corresponding to the current processing stage in the pulse neural network and the convolutional neural network to obtain a first feature and a second feature, combining the first feature and the second feature into a third feature, and respectively outputting the third feature and the second feature to the next processing stage of the pulse neural network and the next processing stage of the convolutional neural network; after the target features are obtained in the last processing stage of the spiking neural network, the classification result of the target image is determined based on the target features through the full connection layer, supplementary features are provided for the features extracted in each stage of the spiking neural network through the convolutional neural network, and therefore the image processing performance of the spiking neural network is improved.

Description

Technical Field

[0001] The present invention relates to the field of neural network technology, and in particular to an image recognition method, device, computer equipment and storage medium based on SNN information supplementation. Background Art

[0002] In the field of artificial intelligence in recent years, artificial neural networks (ANN) have become a mainstream technology, showing excellent performance in many fields such as image processing, natural language processing, and intelligent driving. ANN can learn and simulate complex data patterns and show extraordinary capabilities in processing large-scale and complex data.

[0003] Taking the field of image processing as an example, when ANN is used for application processing, although good processing effects can be obtained, its energy consumption is often relatively high. When applied on some power-constrained devices, excessive energy consumption will limit the application scope and efficiency of ANN.

[0004] Therefore, a spiking neural network (SNN) with relatively low energy consumption is usually used for image processing. However, although SNN has low energy consumption, its performance is often relatively poor, resulting in unsatisfactory image recognition effects. Summary of the invention

[0005] In view of this, the present invention provides an image recognition method, device, computer equipment and storage medium based on SNN information supplement to solve the problem of relatively poor performance when applying pulse neural network for image processing, resulting in poor image recognition effect.

[0006] In a first aspect, the present invention provides an image recognition method based on SNN information supplementation, which is applied to an image recognition model, wherein the image recognition model comprises: a spiking neural network SNN and a convolutional neural network CNN having a plurality of corresponding processing stages, and the method comprises:

[0007] Acquire a target image, and input the target image into the initial processing stages of the spiking neural network and the convolutional neural network in the trained image recognition model, perform feature extraction and forward propagation;

[0008] In each corresponding processing stage, the input features corresponding to the current processing stage in the spiking neural network and the convolutional neural network are processed to obtain a first feature and a second feature, the first feature and the second feature are combined into a third feature, and the third feature and the second feature are output to the next processing stage of the spiking neural network and the convolutional neural network respectively;

[0009] When the last processing stage of the spiking neural network completes the processing of the output features of the previous processing stage to obtain the target features, the classification result of the target image is determined based on the target features through the fully connected layer of the spiking neural network.

[0010] This method combines the features extracted by the convolutional neural network and the spiking neural network when the spiking neural network and the convolutional neural network perform feature extraction in the corresponding processing stages, and sends the features to the next stage of the spiking neural network for corresponding processing to supplement the features extracted by the spiking neural network. This can effectively improve the effect of the features obtained by the spiking neural network in multiple corresponding processing stages, improve the overall performance of the spiking neural network, and ensure the final image recognition effect of the spiking neural network.

[0011] In an optional embodiment, there are complementary paths between corresponding processing stages of the spiking neural network and the convolutional neural network;

[0012] The first and second features are combined into a third feature, including:

[0013] Determine whether there is a supplementary convolution block on the supplementary path between the spiking neural network and the corresponding current processing stage in the convolutional neural network;

[0014] If it exists, the second feature is processed by the supplementary convolution block, and the first feature and the processed second feature are added to obtain the third feature.

[0015] In this embodiment, a supplementary path is established between the corresponding processing stages of the pulse neural network and the convolutional neural network, and a convolution block is set on the supplementary path as needed. When there is a convolution block in the supplementary path, the second feature extracted by the convolutional neural network is further processed and then merged with the first feature extracted by the pulse neural network to further enhance the effect of the third feature finally obtained by the merger, thereby further enhancing the final image recognition effect.

[0016] In an optional embodiment, each processing stage of the spiking neural network and the convolutional neural network includes at least one residual block, and the residual block includes: an activation function and a plurality of convolution blocks;

[0017] The processing of the input features corresponding to the current processing stage in the pulse neural network and the convolutional neural network to obtain the first feature and the second feature includes:

[0018] The input features are processed layer by layer by each residual block in the current processing stage of the spiking neural network until the first feature is obtained after being processed by the terminal convolution block in the terminal residual block;

[0019] The input features are processed layer by layer through each residual block in the current processing stage of the convolutional neural network until the second feature is obtained after the last residual block is processed;

[0020] Outputting the third feature to the next processing stage of the spiking neural network includes:

[0021] The third feature is activated by the activation function of the terminal residual block in the current processing stage of the spiking neural network, and output to the next processing stage of the spiking neural network.

[0022] In this embodiment, a residual block and an activation function are set in each processing stage, and there are multiple convolution blocks in the residual block. After the second feature obtained after the activation function in the processing stage corresponding to each convolutional neural network and the terminal convolution block of the terminal residual block in the processing stage corresponding to the pulse neural network are merged to obtain the third feature, it is sent to the next processing stage of the pulse neural network through the activation function of the residual block, thereby ensuring the accuracy of feature extraction in each processing stage.

[0023] In an optional embodiment, the activation function of the residual block in the spiking neural network is a LIF neuron, and the activation function of the residual block in the convolutional neural network is a RELU function;

[0024] The step of processing the input features layer by layer through each residual block in the current processing stage of the convolutional neural network includes:

[0025] After the input features are extracted by the convolution block in each residual block, they are activated by the RELU activation function and input to the next residual block;

[0026] The activating the third feature through the activation function of the terminal residual block in the current processing stage of the spiking neural network includes:

[0027] The third feature is activated by the LIF neuron in the terminal residual block in the current processing stage of the spiking neural network.

[0028] In this embodiment, by setting the activation functions of the convolutional neural network and the spiking neural network to the RELU function and the LIF neuron respectively, the overall performance of the spiking neural network can be effectively improved.

[0029] In an optional implementation, the training method of the image recognition model includes:

[0030] Get the preprocessed sample images and corresponding sample labels;

[0031] Inputting the sample image into the image recognition model to be trained to obtain a classification result corresponding to the sample image;

[0032] Based on the classification results and sample labels corresponding to the sample images, the image recognition model is iteratively trained to obtain a trained image recognition model.

[0033] In this embodiment, by obtaining the preprocessed sample image and the corresponding label, the sample image is input into an image recognition model of a convolutional neural network and a pulse neural network with multiple corresponding processing stages to obtain the final classification result, and the classification result and the label are compared to update and iterate the parameters of the pulse neural network and the convolutional neural network in the image recognition model to achieve better image recognition effect.

[0034] In an optional implementation, the iterative training of the image recognition model based on the classification result and the sample label corresponding to the sample image includes:

[0035] Constructing a loss function according to the classification result and the sample label corresponding to the sample image;

[0036] Performing back-propagation updates on the spiking neural network and the convolutional neural network in the image recognition model based on the loss function;

[0037] Return to execute to obtain the preprocessed sample image and the corresponding sample label, and input the sample image into the updated image recognition model until the loss between the classification result and the sample label is less than a preset threshold.

[0038] In this embodiment, a loss function is established to back-propagate the pulse neural network and the convolutional neural network to update the network parameters, and a new round of iteration is performed based on the new sample images and corresponding labels until the loss between the classification result and the sample label obtained by the final image recognition model is less than a preset threshold, thereby ensuring the recognition effect of the final image recognition model.

[0039] In an optional implementation, the obtaining of the preprocessed sample image includes:

[0040] A sample image is acquired, and data amplification, size adjustment and normalization processing are performed on the sample image to obtain a preprocessed sample image.

[0041] In this implementation, data amplification, size adjustment and normalization are performed on sample images to ensure subsequent training effects.

[0042] In a second aspect, the present invention provides an image recognition device, applied to an image recognition model, wherein the image recognition model comprises: a spiking neural network and a convolutional neural network having a plurality of corresponding processing stages, wherein the device comprises:

[0043] An image acquisition module is used to acquire a target image, input the target image into the initial processing stages of the pulse neural network and the convolutional neural network in the trained image recognition model, perform feature extraction and forward propagation;

[0044] A feature merging module is used to process the input features corresponding to the current processing stage in the spiking neural network and the convolutional neural network in each corresponding processing stage to obtain a first feature and a second feature, merge the first feature and the second feature into a third feature, and output the third feature and the second feature to the next processing stage of the spiking neural network and the convolutional neural network respectively;

[0045] The result determination module is used to determine the classification result of the target image based on the target feature through the fully connected layer of the pulse neural network after the last processing stage of the pulse neural network completes the processing of the output feature of the previous processing stage to obtain the target feature.

[0046] In a third aspect, the present invention provides a computer device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the image recognition method based on SNN information supplementation of the above-mentioned first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0047] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the image recognition method based on SNN information supplementation of the above-mentioned first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 is a flow chart of an image recognition method based on SNN information supplementation according to an embodiment of the present invention;

[0050] Figure 2 is a flow chart of another image recognition method based on SNN information supplementation according to an embodiment of the present invention;

[0051] Figure 3 is an example diagram of an SNN training process based on information supplementation according to an embodiment of the present invention;

[0052] Figure 4 is a structural schematic diagram of an image recognition model based on SNN information supplementation according to an embodiment of the present invention;

[0053] Figure 5 is an example diagram of a residual block according to an embodiment of the present invention;

[0054] Figure 6 is a structural block diagram of an image recognition device based on SNN information supplementation according to an embodiment of the present invention;

[0055] Figure 7 is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention;

[0056] Figure 8 It is a structural schematic diagram of a computer device and an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0058] In the field of artificial intelligence in recent years, artificial neural networks (ANN) have become a mainstream technology, showing excellent performance in many fields such as image processing, natural language processing, and intelligent driving. ANN can learn and simulate complex data patterns and show extraordinary capabilities in processing large-scale and complex data.

[0059] Taking the field of image processing as an example, when ANN is used for application processing, although good processing effects can be obtained, its energy consumption is often relatively high. When applied on some power-constrained devices, excessive energy consumption will limit the application scope and efficiency of ANN.

[0060] Therefore, a spiking neural network (SNN) with relatively low energy consumption is usually used for image processing. However, although SNN has low energy consumption, its performance is often relatively poor, resulting in unsatisfactory image recognition effects.

[0061] To this end, an embodiment of the present invention provides an image recognition method based on SNN information supplementation, in which an image recognition model is constructed by a pulse neural network and a convolutional neural network with multiple corresponding processing stages. When the pulse neural network and the convolutional neural network perform feature extraction in the corresponding processing stages, the features extracted by the convolutional neural network and the pulse neural network are merged and sent to the next stage of the pulse neural network for corresponding processing to supplement the features extracted by the pulse neural network. This can effectively improve the effect of the features obtained by the pulse neural network in multiple corresponding processing stages, improve the overall performance of the pulse neural network, and ensure the final image recognition effect of the pulse neural network.

[0062] According to an embodiment of the present invention, an embodiment of an image recognition method based on SNN information supplementation is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0063] In this embodiment, an image recognition method based on SNN information supplementation is provided, which is applied to an image recognition model. The image recognition model includes: a spiking neural network and a convolutional neural network with multiple corresponding processing stages, namely, SNN and CNN. Figure 1 is a flow chart of an image recognition method based on SNN information supplementation according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0064] Step S101, obtain a target image, input the target image into the initial processing stages of the pulse neural network and the convolutional neural network in the trained image recognition model, perform feature extraction and forward propagation.

[0065] First, it is necessary to obtain an image containing the object to be tested, that is, the image to be identified. The image of the object to be tested can be all possible images in nature. The image can be obtained from the image taken in practice, or from an image data set or database. After obtaining the image to be identified, the image is pre-processed, such as unifying the format, size, normalization, etc., to obtain a target image that meets the standard.

[0066] Then the target image is input into the trained image recognition model, which mainly includes pulse neural network and convolutional neural network.

[0067] For neural networks, they usually include multiple processing stages. Through multiple stages of layer-by-layer processing, the features of the image to be identified can be effectively extracted. In each processing stage, a round of feature extraction is performed, and the extracted features are transmitted to the next processing stage for a new round of feature extraction, i.e. forward propagation. Finally, after multiple rounds of feature extraction, effective image features can be obtained, and classification can be performed based on the image features.

[0068] The image recognition model used in the present invention includes a pulse neural network and a convolutional neural network, namely, SNN and CNN. These two neural networks also have multiple processing stages, and the multiple processing stages correspond to each other.

[0069] For example, a pulse neural network has four processing stages: stage 1, stage 2, stage 3, stage 4, and a convolutional neural network has three processing stages, namely stage 1, stage 2, stage 3. Stages 1 to 3 of the convolutional neural network and stages 1 to 3 of the pulse neural network can be set as corresponding processing stages.

[0070] In another example, the convolutional neural network can also be set to 4 processing stages, and these processing stages correspond one-to-one to the processing stages of the pulse neural network. The specific number of processing stages and the corresponding situations can be set according to actual conditions. There is no limitation here. It is just an illustrative explanation of the correspondence between the processing stages of the two neural networks.

[0071] In each processing stage of the convolutional neural network and the pulse neural network, the image features input in the previous stage are further processed to obtain new image features, which are then transmitted to the next stage until the final features are obtained in the last processing stage of the pulse neural network.

[0072] Specifically, before the image is input into the first processing stage of the spiking neural network and the convolutional neural network, preliminary feature extraction can be performed on the image through a convolutional block of the same specification to increase the number of channels and ensure the alignment during feature extraction in the subsequent processing stage. The preliminary extracted features are respectively input into the first processing stage of the convolutional neural network and the spiking neural network, and corresponding feature extraction and forward propagation are performed.

[0073] Exemplarily, the target image can also be directly input into the first processing stage of the pulse neural network and the convolutional neural network, and features of the target image can be extracted in the first processing stage corresponding to each of the two neural networks, and the number of channels of the feature image can be increased.

[0074] Step S102, in each corresponding processing stage, the input features corresponding to the current processing stage in the pulse neural network and the convolutional neural network are processed to obtain the first feature and the second feature, the first feature and the second feature are merged into the third feature, and the third feature and the second feature are output to the next processing stage of the pulse neural network and the convolutional neural network respectively.

[0075] At each processing stage of the convolutional neural network, the feature image input at the current stage is further extracted to obtain the second feature. At the same time, at each processing stage of the convolutional neural network, the feature image input at the current stage is also extracted to obtain the first feature.

[0076] In the corresponding processing stages, for example, the pulse neural network includes 4 processing stages, and the convolutional neural network includes 4 processing stages, wherein the processing stages of the convolutional neural network correspond one to one to the processing stages of the pulse neural network.

[0077] Taking the first stage as an example, it is the corresponding processing stage of the two neural networks. At this time, the pulse neural network merges the first feature obtained by it and the second feature obtained by the convolutional neural network, that is, performs an ADD operation to obtain the third feature. The second feature can be understood as a supplementary feature for the first feature, and the third feature can be regarded as the feature after the first feature completes the information supplement, which can better reflect the image characteristics.

[0078] The third feature is input to the next processing stage of the spiking neural network, so that the first feature is further extracted through the spiking neural network based on the third feature. The next processing stage, that is, the second processing stage is still the corresponding processing stage, and a merging process similar to the first stage and back propagation can be performed, which will not be repeated here. At the same time, the second feature obtained by the convolutional neural network in the first stage is transmitted to the next processing stage for subsequent feature extraction.

[0079] In order to take into account both the image recognition effect and performance consumption, for example, the number of processing stages of the pulse neural network is usually set to N, and the number of processing stages of the convolutional neural network is N-1, and the processing stages of the convolutional neural network and the first N-1 processing stages of the pulse neural network correspond to each other in sequence. That is, the first features obtained in the first N-1 processing stages of the pulse neural network need to be supplemented with information, that is, merged with the corresponding second features, and then transmitted to the next processing stage. Specifically, N is usually 4. The above-mentioned exemplary number of processing stages and corresponding situations are only preferred examples. In actual application scenarios, the specific number of processing stages and corresponding situations can be set according to actual conditions, and there is no limitation here.

[0080] The image recognition model can be regarded as an SNN based on information supplementation, and its architecture is a network structure integrating a spiking neural network (Spiking Neural Network, SNN) and a convolutional neural network. The spiking neural network, as a highly biologically simulated neural network model, can simulate the ability of the human brain to process visual information. When processing images, the spiking neural network can maintain extremely low energy consumption, but at the same time there will be losses in the process of information propagation. Therefore, a convolutional neural network is introduced to extract supplementary information to supplement the features extracted by the spiking neural network.

[0081] Among them, the SNN based on information supplementation includes a pulse neural network branch, a convolutional neural network branch and a fully connected layer; specifically, the pulse neural network branch is used to extract preliminary features to be supplemented from the image of the object to be tested, and the convolutional neural network branch is used to extract supplementary features from the image of the object to be tested. After adding the convolutional neural network supplementary features and the pulse network preliminary features to be supplemented at each stage, the output prediction value is calculated by forward propagation through the fully connected layer. Before being actually applied to image recognition, the pulse neural network and the convolutional neural network can be iteratively trained in turn according to sample data, and when the network converges, a usable SNN based on information supplementation can be obtained for image recognition.

[0082] Step S103, when the last processing stage of the spiking neural network completes the processing of the output features of the previous processing stage to obtain the target features, the classification result of the target image is determined based on the target features through the fully connected layer of the spiking neural network.

[0083] After the feature extraction of multiple processing stages mentioned in the above steps, in the last processing stage of the pulse neural network, the target feature can be obtained after the final feature extraction of the output feature of the previous processing stage.

[0084] After obtaining the target features, the fully connected layer in the pulse neural network will process according to the target features to obtain the classification results for the target image. The specific classification process of the fully connected layer can refer to the relevant technology and will not be repeated here. The main function of the embodiment of the present invention is to supplement information through the convolutional neural network on the basis of the low energy consumption of the pulse neural network to improve the accuracy of the target features finally obtained, thereby improving the accuracy of image classification.

[0085] Exemplarily, a processing stage corresponding to the last processing stage of the pulse neural network can also be set in the convolutional neural network to perform feature merging as in the above step S102, and the merged features are used as target features, which is not limited here.

[0086] The image recognition method based on SNN information supplement provided in this embodiment combines the features extracted by the convolutional neural network and the pulse neural network when the pulse neural network and the convolutional neural network perform feature extraction in the corresponding processing stages, and sends them to the next stage of the pulse neural network for corresponding processing to supplement the features extracted by the pulse neural network. This can effectively improve the effect of the features obtained by the pulse neural network in multiple corresponding processing stages, enhance the overall performance of the pulse neural network, and ensure the final image recognition effect of the pulse neural network.

[0087] According to an embodiment of the present invention, another embodiment of an image recognition method based on SNN information supplementation is provided, which is applied to an image recognition model, wherein the image recognition model includes: a spiking neural network and a convolutional neural network having multiple corresponding processing stages, Figure 2 is a flowchart of another image recognition method based on SNN information supplementation according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0088] Step S201, obtain a target image, input the target image into the initial processing stages of the pulse neural network and the convolutional neural network in the trained image recognition model, perform feature extraction and forward propagation. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0089] Step S202: In each corresponding processing stage, the input features corresponding to the current processing stage in the pulse neural network and the convolutional neural network are processed to obtain a first feature and a second feature, the first feature and the second feature are merged into a third feature, and the third feature and the second feature are output to the next processing stage of the pulse neural network and the convolutional neural network, respectively.

[0090] Specifically, each processing stage of the pulse neural network and the convolutional neural network includes at least one residual block, and the residual block includes: an activation function and multiple convolution blocks.

[0091] There are complementary paths between the corresponding processing stages of spiking neural networks and convolutional neural networks.

[0092] Step S202 includes:

[0093] Step S202-1, processing the input features layer by layer through each residual block in the current processing stage of the pulse neural network until the first feature is obtained after being processed by the terminal convolution block in the terminal residual block.

[0094] It can be understood that in each processing stage of the pulse neural network and the convolutional neural network, the feature image input in the processing stage is processed in turn by setting up multiple residual blocks to obtain the first feature or the second feature extracted in this stage. When each residual block processes the feature image, specific feature extraction processing is performed through the internal convolution block and activation function.

[0095] After the last convolution block in the last residual block of the current processing stage of the spiking neural network completes feature extraction, the extracted feature is defined as the first feature. Generally, the more residual blocks there are, the better the image recognition effect is, and the corresponding energy consumption is also greater. The specific number of residual blocks can be set according to the specific application scenario.

[0096] Step S202-2, processing the input features layer by layer through each residual block in the current processing stage of the convolutional neural network until the second feature is obtained after the last residual block is processed.

[0097] Similarly, after the second feature extracted from the convolutional neural network is processed by the last residual block and activated by the activation function in the residual block, it can be input as the second feature to the next stage and supplemented to the corresponding stage of the pulse neural network to merge and obtain the third feature.

[0098] Specifically, the activation function of the residual block in the convolutional neural network is the RELU function, and the activation function of the residual block in the spiking neural network is the LIF neuron;

[0099] In step S202-2, the input features are processed layer by layer through each residual block in the current processing stage of the convolutional neural network, including:

[0100] After the input features are extracted by the convolution block in each residual block, they are activated by the RELU activation function and input to the next residual block.

[0101] Similarly, in step S202-1, the input features are extracted through the convolution block in each residual block, activated through the LIF neurons and input to the next residual block. However, before the LIF neurons of the last residual block are activated, the first feature and the corresponding second feature need to be merged and then activated.

[0102] Step S202-3, determine whether there is a supplementary convolution block on the supplementary path between the corresponding current processing stages in the pulse neural network and the convolutional neural network.

[0103] It can be understood that there is a supplementary path between the corresponding processing stages of the convolutional neural network and the spiking neural network, on which a convolution block can be set to further process the second feature extracted by the convolutional neural network and then merge it with the first feature extracted by the spiking neural network. If there is no convolution block on the path, the second feature extracted by the convolutional neural network is directly input into the corresponding stage of the spiking neural network and merged with the first feature.

[0104] Step S202-4: If it exists, the second feature is processed by a supplementary convolution block, and the first feature and the processed second feature are added to obtain a third feature.

[0105] The second feature can be further processed by the convolution block on the supplementary path, and the processed second feature can be added to the first feature as supplementary information, which can improve the effectiveness of the third feature after the information supplement is completed, thereby improving the overall performance of the pulse neural network.

[0106] Step S202-5, the third feature is activated by the activation function of the terminal residual block in the current processing stage of the spiking neural network, and output to the next processing stage of the spiking neural network.

[0107] After the first feature and the corresponding second feature are merged and input into the next stage through the activation function, overfitting can be avoided and the form conversion of the feature can be completed at the same time, so that the feature extraction can be better performed in the next processing stage.

[0108] Specifically, the third feature is activated by the activation function of the terminal residual block in the current processing stage of the spiking neural network, including:

[0109] The third feature is activated by the LIF neuron in the terminal residual block in the current processing stage of the spiking neural network.

[0110] Step S203, when the last processing stage of the spiking neural network completes the processing of the output features of the previous processing stage to obtain the target features, the classification result of the target image is determined based on the target features through the fully connected layer of the spiking neural network. Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.

[0111] Exemplarily, the pulse neural network and the convolutional neural network are set to include 4 stages and 3 stages respectively, each stage contains two residual blocks, and no convolutional blocks are configured on the supplementary path. Then the input of the last pulse neuron in the first stage of the pulse neural network includes: the output of the last convolutional block in the first stage of the pulse neural network and the output of the first stage of the convolutional neural network; the input feature of the last pulse neuron in the second stage of the pulse neural network includes the output of the last convolutional block in the second stage of the pulse neural network and the output of the second stage of the convolutional neural network, specifically, these two outputs are added to obtain the input of the last pulse neuron in this stage of the pulse neural network; correspondingly, the input of the second stage of the convolutional neural network is the output of the first stage of the convolutional neural network, the input of the third stage of the convolutional neural network is the output of the second stage of the residual network, and so on.

[0112] Furthermore, users can choose to configure each stage of the convolutional neural network as a residual block or a single convolution block. If they choose to configure it as a residual block, users can further customize the number of residual blocks. In addition, users can independently set whether to add convolution blocks on the supplementary path. The specific setting method needs to be set according to the actual situation and is not restricted here.

[0113] When the above-mentioned SNN based on information supplementation, that is, the image recognition model mentioned in the present invention, is used to perform image recognition, the image of the object to be tested is recognized through the SNN based on information supplementation. Specifically, the pulse neural network is used to extract preliminary features from the image, and the convolutional neural network is used to extract supplementary features from the image. Richer image feature information can be obtained, and the iterative training and image classification of the network are realized through the fusion features of the supplementary features of the convolutional neural network and the preliminary features output by the pulse network, thereby improving the accuracy of object recognition.

[0114] The image recognition model mentioned in the above embodiment, that is, the pulse neural network based on information supplementation, is constructed as follows:

[0115] Preprocessing the sample object image to obtain a preprocessed sample image;

[0116] The preprocessed sample image is processed based on the spiking neural network, and the features before the spiking neural network is supplemented are obtained in the last convolution of each stage;

[0117] The preprocessed images are processed based on the convolutional neural network, and the complementary features of the spiking neural network are obtained at each stage;

[0118] The features before being supplemented of the spiking neural network in each stage and the supplemented features of the convolutional neural network are added before the last spiking neuron in each stage of the spiking neural network to obtain the supplemented features;

[0119] The spiking neural network uses the supplemented features to propagate forward until the probability of each category is calculated.

[0120] The loss value corresponding to the category is calculated according to the cross entropy loss function, and the pulse neural network and the convolutional neural network are reversely iterated and updated using the loss value to update the loss value. When the updated loss value is less than the target threshold, an SNN based on information supplementation is obtained.

[0121] Among them, the image preprocessing operations include image expansion, image normalization, and outlier removal.

[0122] For the image recognition model in the above method embodiment, the training process is mainly as follows:

[0123] Step S301, obtaining a preprocessed sample image and a corresponding sample label.

[0124] Specifically, in step S301, obtaining a preprocessed sample image includes:

[0125] A sample image is acquired, data amplification is performed on the sample image, and size adjustment and normalization are performed to obtain a preprocessed sample image.

[0126] Exemplarily, in this embodiment, the number of sample images can be expanded by data augmentation, and data augmentation can be achieved by flipping, rotating, translating, scaling, noise perturbation, brightness contrast transformation, etc., or by using a generative adversarial network to expand the number of samples. The target sample image is related to the data size standard allowed by the network. By performing data augmentation and normalization on the sample object images, the number of samples required for network training is guaranteed, the problem that the network is prone to overfitting is reduced, and the problem of data imbalance between samples is reduced, thereby improving the sample quality and facilitating the robustness of subsequent model training.

[0127] The image normalization is achieved by the following formula:

[0128]

[0129] Among them, znorm is the preprocessed sample image, z is the target sample image, min(z) is the minimum pixel value in z, and max(z) is the maximum pixel value in z.

[0130] Step S302: input the sample image into the image recognition model to be trained to obtain the classification result corresponding to the sample image.

[0131] Each processing stage of the convolutional neural network and the pulse neural network in the image recognition model will extract features from the feature image input in the previous stage and transmit it to the next processing stage, wherein the convolutional neural network will send the extracted features to the corresponding processing stage of the pulse neural network to supplement the extracted features. After each processing process of the pulse neural network is completed, the obtained feature image will be forward propagated. Finally, the target features corresponding to the sample image are obtained in the last processing stage of the pulse neural network, so as to determine the specific classification results through the fully connected layer. The specific implementation method can refer to the above embodiment and will not be repeated here.

[0132] Step S303, iteratively training the image recognition model based on the classification result and the sample label corresponding to the sample image to obtain a trained image recognition model.

[0133] Specifically, in step S303, it includes:

[0134] Step S303-1, constructing a loss function according to the classification result and the sample label corresponding to the sample image.

[0135] Specifically, the loss function may be a cross entropy loss function, which is expressed as follows:

[0136] Loss=-(y·log(p)+(1-y)·log(1-p));

[0137] Among them, y is the corresponding label in the input network of the sample, and p is the probability value of each prediction result obtained after the final model output is classified by the softmax classifier.

[0138] Step S303-2, back-propagation updates the spiking neural network and convolutional neural network in the image recognition model based on the loss function.

[0139] Specifically, the reverse iterative update of the spiking neural network is implemented by the following steps: reverse iterative update of the spiking neural network is performed using a substitute gradient; wherein the substitute gradient includes:

[0140]

[0141] Among them, α is the learning rate and x is the loss value.

[0142] The ReLU activation function is used in the reverse iterative update process of the convolutional neural network to prevent gradient disappearance and gradient explosion during network training.

[0143] Step S303-3, return to execute to obtain the preprocessed sample image and the corresponding sample label, and input the sample image into the updated image recognition model until the loss between the classification result and the sample label is less than a preset threshold.

[0144] Furthermore, for the above-mentioned image recognition model, i.e., the SNN based on information supplementation, its training process can be as follows: Figure 3 As shown, it is an example diagram of an SNN training process based on information supplementation according to an embodiment of the present invention, which specifically includes:

[0145] (1) Initialize the network parameters of the pulse neural network and convolutional neural network (corresponding to the neural network parameters).

[0146] (2) The preprocessed sample image is input into the pulse neural network to extract the preliminary features of the image, and the preprocessed sample image is input into the convolutional neural network to extract the supplementary features of the image. The supplementary features are added to the preliminary features and then the recognition result corresponding to the image is output through forward propagation.

[0147] (3) The loss value (corresponding error) between the recognition result obtained by calculating the loss function and the actual result of the training image.

[0148] (4) Use the loss value to perform back-propagation updates on the pulse neural network and convolutional neural network, that is, to iteratively update the loss value and adjust the parameters of the neural network.

[0149] (5) If the updated loss value is greater than or equal to the target threshold, return to step (2) to continue training; until the updated loss value is less than the target threshold, an SNN based on information supplementation is obtained.

[0150] The image recognition method based on SNN information supplement provided by the embodiment of the present invention constructs an image recognition model through a pulse neural network and a convolutional neural network with multiple corresponding processing stages, and establishes a supplementary path between the corresponding processing stages of the two neural networks, and further sets convolution blocks on the supplementary path according to actual conditions, so as to supplement the features extracted by the pulse neural network based on the features extracted by the convolutional neural network, thereby improving the overall performance of the network, ensuring that the target features finally obtained after multiple rounds of pulse neural network processing stages can effectively reflect the information of the target image, so that the image recognition model has a better image recognition effect.

[0151] The spiking neural network and convolutional neural network mentioned in the above method embodiment may include N and N-1 processing stages respectively, where N is an integer greater than 2. The specific feature extraction process is as follows:

[0152] The preprocessed sample image is input into the first stage of the pulse neural network, and the first pulse feature before supplementation is obtained in the last convolution of the first stage. The preprocessed sample image is input into the first stage of the convolutional neural network to obtain the first convolution output. The first convolution output is input into the convolution block on the first supplementary path to obtain the first supplementary feature (if there is no convolution block on the supplementary path, the first supplementary feature is the first convolution output), and the Add operation is performed on the first pulse feature before supplementation and the first supplementary feature to obtain the first pulse feature after supplementation.

[0153] Then, the feature after supplementation in the first stage is input to the last neuron of the first stage of the pulse neural network to obtain the first pulse output; the first pulse output is input to the second stage of the pulse neural network, and the second pulse feature before supplementation is obtained in the last convolution of the second stage. The first convolution output is input to the second stage of the convolution neural network to obtain the second convolution output, and the second convolution output is input to the convolution block on the second supplementary path to obtain the second supplementary feature (if there is no convolution block on the supplementary path, the second supplementary feature is the second convolution output), and the second pulse feature before supplementation and the second supplementary feature are Added to obtain the second pulse feature after supplementation, and then the second pulse feature after supplementation is input to the last neuron of the second stage of the pulse neural network to obtain the second pulse output, and so on;

[0154] Until the N-2th pulse output is input into the N-1th stage of the pulse neural network, the N-1th pulse pre-supplementation feature is obtained in the last convolution of the N-1th stage, the N-2th convolution output is input into the N-1th stage of the convolutional neural network to obtain the N-1th convolution output, the N-1th convolution output is input into the convolution block on the N-1th supplementary path to obtain the N-1th supplementary feature (if there is no convolution block on the supplementary path, the N-1th supplementary feature is the N-1th convolution output), and the N-1th pulse pre-supplementation feature and the N-1th supplementary feature are Added to obtain the N-1th pulse post-supplementation feature, and then the N-1th pulse post-supplementation feature is input into the last neuron of the N-1th stage of the pulse neural network to obtain the N-1th pulse output, and the feature of the N-1 pulse output is output to the last processing stage of the pulse neural network, that is, the Nth pulse processing stage for the final feature extraction to obtain the target feature, so as to obtain the final classification result by performing full connection layer processing based on the target feature.

[0155] Specifically, due to the characteristics of the spiking neural network itself, each residual block in the spiking neural network can perform image processing operations for multiple time steps. For example, after the residual block performs image processing for 4 time steps, the outputs of the 4 time steps are merged through the Add operation. The convolutional neural network supplements the information of the spiking network, that is, the 1st to N-2 pre-spiking supplement features and the 1st to N-2 supplement features are added and respectively used as the input of the 2nd to N-1st stages of the spiking neural network. The final features are input into the last stage of the spiking neural network and the fully connected layer to obtain the predicted classification category. The number N can be set according to user needs.

[0156] For example, Figure 4 FIG. 1 is a schematic diagram of the structure of an image recognition model according to an embodiment of the present invention. Where N is 4, the process of obtaining the features after multiple rounds of information supplementation is as follows:

[0157] (1) The preprocessed sample image, i.e., the target image, is first subjected to the first convolution block for preliminary feature extraction to increase the number of channels of the image, and then input into the first stage of the spiking neural network. The last convolution in the first stage obtains the first pulse pre-supplement feature;

[0158] Similarly, the preprocessed sample image first passes through the first convolution block for preliminary feature extraction to increase the number of channels of the image, and then is input into the first stage of the convolutional neural network, and then passes through the convolution block on the supplementary path for further feature extraction to obtain the first supplementary feature.

[0159] (2) The first pulse feature before supplementation and the first supplementary feature are added to obtain the first pulse feature after supplementation, and the first pulse feature after supplementation is input into the last pulse neuron of the first stage of the pulse neural network to obtain the first pulse output, and then the first pulse output is input into the second stage of the pulse neural network, and the second pulse feature before supplementation is obtained by the last convolution in the second stage, and the first supplementary feature is input into the second stage of the convolutional neural network to obtain the second supplementary feature.

[0160] (3) The feature before the second pulse supplement and the second supplement feature are added to obtain the feature after the second pulse supplement, and the feature after the second pulse supplement is input to the last pulse neuron of the second stage of the pulse neural network to obtain the second pulse output, and then the second pulse output is input to the third stage of the pulse neural network, and the last convolution in the third stage is performed to obtain the feature before the third pulse supplement, and the second supplement feature is input to the third stage of the convolutional neural network to obtain the third supplement feature.

[0161] (4) The features before the third pulse supplementation and the third supplemented features are added to obtain the features after the third pulse supplementation, and the features after the third pulse supplementation are input into the last pulse neuron of the third stage of the pulse neural network to obtain the third pulse output, and then the third pulse output is input into the fourth stage of the pulse neural network. After the fourth stage of the pulse neural network processes the input features, the final feature vector is obtained, which is used for the fully connected layer to determine the classification result.

[0162] Exemplarily, the pulse neural network includes a first convolution kernel and multiple stages, wherein the first convolution kernel is used to increase the number of channels of the image before the start of multiple processing stages, each of the multiple stages is composed of multiple residual blocks, and the activation layer included in each residual block is set as a LIF neuron; wherein each residual block is determined based on multiple second convolution kernels.

[0163] The first convolution kernel in the convolutional neural network is the same as the first convolution kernel in the pulse neural network. The convolutional neural network contains multiple stages. The residual block (convolution kernel) of each stage is set by the user. The residual block (convolution kernel) is determined based on the second convolution kernel. The convolution kernel on the supplementary path (if any) is determined based on the third convolution kernel. The ReLU activation function is used in the convolutional neural network for data nonlinear operations, and the LIF neuron is used as the activation function in the pulse network for data nonlinear operations.

[0164] Specifically, in the above exemplary embodiment, the current equation of the LIF (Leaky Integrate-and-Fire) neuron model is expressed as:

[0165] I(t)=I R +I C ;

[0166] Where I(t) is the current passing through the LIF neuron model at time t, IR is the current passing through the resistor R, and IC is the current passing through the capacitor C. According to the above formula, we can derive:

[0167]

[0168] Among them, U(t) is the voltage across the LIF neuron model at time t, Ures is the resting potential, U0 is the membrane potential at t = 0, τ m is the membrane time constant, where τ m =RC.

[0169] The sizes of the first convolution kernel, the second convolution kernel, and the third convolution kernel can be selected according to user needs. For example, the first convolution kernel is a 5*5 convolution kernel, the second convolution kernel is at least one of a 3*3 convolution kernel and a 1*1 convolution kernel, and the third convolution kernel is a 1*1 convolution kernel. For these convolution kernels, it can be understood that each convolution kernel plus an activation function can constitute the convolution block mentioned in the above-mentioned embodiment of the invention, that is, the common basic convolution block.

[0170] In this embodiment, the spiking neural network includes a 5*5 convolution kernel and four stages, each stage contains two residual blocks, which can ensure the effective transmission of key features, wherein each residual block is composed of two 3*3 convolution kernels, and the activation function adopts LIF neurons, which can balance biological rationality and practicality. For example, Figure 5 As shown in FIG. 1 , an example diagram of a residual block according to an embodiment of the present invention is shown in FIG. Figure 5 As shown in the figure, each residual block contains two convolution kernels CONV3*3 on the right, each convolution kernel is followed by a batch normalization layer, and there is an activation function between the two convolution kernels, namely the LIF neuron, and a jump connection, namely the separate convolution kernel CONV3*3 on the left, is used to alleviate the gradient disappearance and gradient explosion problems of the network.

[0171] The convolutional neural network includes a 5*5 convolution kernel and three stages. Each stage contains two residual blocks to ensure the effective transmission of key features. Each residual block consists of two 3*3 convolution kernels. ReLU is used as the activation function to effectively solve the problems of gradient disappearance and gradient explosion. The SNN based on information supplementation also includes a fully connected layer, and the number of output neurons in the fully connected layer is the number of categories.

[0172] In the process of training the information supplement-based SNN, the relevant parameters include: using Stochastic Gradient Descent (SGD) as the optimizer, the learning rate is set to 0.1, the sample batch batch_size is 64, and the maximum number of training epochs is 30.

[0173] A spiking neural network is constructed through at least one first convolution kernel and multiple stages, and a convolutional neural network is constructed through at least one first convolution kernel and multiple stages. The activation function in the spiking neural network is constructed using LIF neurons, and the convolution kernel in the convolutional neural network uses a ReLU activation function to perform data nonlinear operations, which ensures the effective transmission of key features while alleviating the gradient vanishing and gradient exploding problems during network training.

[0174] In this embodiment, an image recognition device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0175] This embodiment provides an image recognition device, which is applied to an image recognition model. The image recognition model includes: a pulse neural network and a convolutional neural network with multiple corresponding processing stages, such as Figure 6 As shown, the device comprises:

[0176] The image acquisition module 401 is used to acquire the target image, input the target image into the initial processing stages of the pulse neural network and the convolutional neural network in the trained image recognition model, perform feature extraction and forward propagation.

[0177] The feature merging module 402 is used to process the input features corresponding to the current processing stage in the pulse neural network and the convolutional neural network in each corresponding processing stage to obtain the first feature and the second feature, merge the first feature and the second feature into a third feature, and output the third feature and the second feature to the next processing stage of the pulse neural network and the convolutional neural network respectively.

[0178] The result determination module 403 is used to determine the classification result of the target image based on the target feature through the fully connected layer of the pulse neural network after the last processing stage of the pulse neural network completes the processing of the output feature of the previous processing stage to obtain the target feature.

[0179] In some optional embodiments, there are complementary paths between corresponding processing stages of the spiking neural network and the convolutional neural network. The feature merging module 402, when merging the first feature and the second feature into a third feature, includes:

[0180] Determine whether there is a supplementary convolution block on the supplementary path between the spiking neural network and the corresponding current processing stage in the convolutional neural network;

[0181] If it exists, the second feature is processed by a supplementary convolution block, and the first feature and the processed second feature are added to obtain the third feature.

[0182] In some optional embodiments, each processing stage of the spiking neural network and the convolutional neural network includes at least one residual block, and the residual block includes: an activation function and a plurality of convolutional blocks; the feature merging module 402, when processing the input features corresponding to the current processing stage in the spiking neural network and the convolutional neural network respectively to obtain the first feature and the second feature, includes:

[0183] The input features are processed layer by layer by each residual block in the current processing stage of the spiking neural network until the first feature is obtained after being processed by the terminal convolution block in the terminal residual block;

[0184] The input features are processed layer by layer through each residual block in the current processing stage of the convolutional neural network until the second feature is obtained after the last residual block is processed;

[0185] The third feature is output to the next processing stage of the spiking neural network, including:

[0186] The third feature is activated by the activation function of the terminal residual block in the current processing stage of the spiking neural network, and output to the next processing stage of the spiking neural network.

[0187] In some optional embodiments, the activation function of the residual block in the spiking neural network is a LIF neuron, and the activation function of the residual block in the convolutional neural network is a RELU function;

[0188] The feature merging module 402, when processing the input features layer by layer through each residual block in the current processing stage of the convolutional neural network, includes:

[0189] After the input features are extracted by the convolution block in each residual block, they are activated by the RELU activation function and input to the next residual block;

[0190] When the third feature is activated by the activation function of the terminal residual block in the current processing stage of the spiking neural network, it includes:

[0191] The third feature is activated by the LIF neuron in the terminal residual block in the current processing stage of the spiking neural network.

[0192] In some optional embodiments, the apparatus further includes a model training module for acquiring preprocessed sample images and corresponding sample labels;

[0193] Input the sample image into the image recognition model to be trained to obtain the classification result corresponding to the sample image;

[0194] Based on the classification results and the sample labels corresponding to the sample images, the image recognition model is iteratively trained to obtain a trained image recognition model.

[0195] In some optional implementations, the model training module iteratively trains the image recognition model based on the classification results and the sample labels corresponding to the sample images, including:

[0196] Construct a loss function based on the classification results and the sample labels corresponding to the sample images;

[0197] Back-propagation updates the spiking neural network and convolutional neural network in the image recognition model based on the loss function;

[0198] Return to execute to obtain the preprocessed sample image and the corresponding sample label, and input the sample image into the updated image recognition model until the loss between the classification result and the sample label is less than the preset threshold.

[0199] In some optional implementations, the model training module, when acquiring the preprocessed sample image, includes:

[0200] A sample image is acquired, data amplification is performed on the sample image, and size adjustment and normalization are performed to obtain a preprocessed sample image.

[0201] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0202] The image recognition device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0203] The embodiment of the present invention also provides a computer device having the above Figure 6 The image recognition device shown.

[0204] See also Figure 7 , Figure 7 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 7 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 10 is taken as an example.

[0205] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0206] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0207] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0208] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0209] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0210] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0211] The embodiment of the present invention further provides an electronic device, such as Figure 8As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the image recognition method based on SNN information supplementation as shown in any of the above method embodiments.

[0212] The present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image recognition method based on SNN information supplementation provided by the above-mentioned method embodiments.

[0213] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0214] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. An image recognition method based on SNN information supplementation, applied to an image recognition model, characterized in that: The image recognition model includes: a spiking neural network (SNN) and a convolutional neural network (CNN) having a plurality of corresponding processing stages, and the method includes: Acquire a target image, and input the target image into the initial processing stages of the spiking neural network and the convolutional neural network in the trained image recognition model, perform feature extraction and forward propagation; In each corresponding processing stage, the input features corresponding to the current processing stage in the spiking neural network and the convolutional neural network are processed to obtain a first feature and a second feature, the first feature and the second feature are combined into a third feature, and the third feature and the second feature are output to the next processing stage of the spiking neural network and the convolutional neural network respectively; When the last processing stage of the spiking neural network completes the processing of the output features of the previous processing stage to obtain the target features, the classification result of the target image is determined based on the target features through the fully connected layer of the spiking neural network.

2. The method according to claim 1, characterized in that There are complementary paths between corresponding processing stages of the spiking neural network and the convolutional neural network; The first and second features are combined into a third feature, including: Determine whether there is a supplementary convolution block on the supplementary path between the spiking neural network and the corresponding current processing stage in the convolutional neural network; If it exists, the second feature is processed by the supplementary convolution block, and the first feature and the processed second feature are added to obtain the third feature.

3. The method according to claim 1, characterized in that Each processing stage of the spiking neural network and the convolutional neural network includes at least one residual block, and the residual block includes: an activation function and a plurality of convolution blocks; The processing of the input features corresponding to the current processing stage in the pulse neural network and the convolutional neural network to obtain the first feature and the second feature includes: The input features are processed layer by layer by each residual block in the current processing stage of the spiking neural network until the first feature is obtained after being processed by the terminal convolution block in the terminal residual block; The input features are processed layer by layer through each residual block in the current processing stage of the convolutional neural network until the second feature is obtained after the last residual block is processed; Outputting the third feature to the next processing stage of the spiking neural network includes: The third feature is activated by the activation function of the terminal residual block in the current processing stage of the spiking neural network, and output to the next processing stage of the spiking neural network.

4. The method according to claim 3, characterized in that The activation function of the residual block in the spiking neural network is the LIF neuron, and the activation function of the residual block in the convolutional neural network is the RELU function; The step of processing the input features layer by layer through each residual block in the current processing stage of the convolutional neural network includes: After the input features are extracted by the convolution block in each residual block, they are activated by the RELU activation function and input to the next residual block; The activating the third feature through the activation function of the terminal residual block in the current processing stage of the spiking neural network includes: The third feature is activated by the LIF neuron in the terminal residual block in the current processing stage of the spiking neural network.

5. The method according to claim 1, characterized in that The training method of the image recognition model includes: Get the preprocessed sample images and corresponding sample labels; Inputting the sample image into the image recognition model to be trained to obtain a classification result corresponding to the sample image; Based on the classification results and sample labels corresponding to the sample images, the image recognition model is iteratively trained to obtain a trained image recognition model.

6. The method according to claim 5, characterized in that The iterative training of the image recognition model based on the classification result and the sample label corresponding to the sample image includes: Constructing a loss function according to the classification result and the sample label corresponding to the sample image; Performing back-propagation updates on the spiking neural network and the convolutional neural network in the image recognition model based on the loss function; Return to execute to obtain the preprocessed sample image and the corresponding sample label, and input the sample image into the updated image recognition model until the loss between the classification result and the sample label is less than a preset threshold.

7. The method according to claim 5, characterized in that The step of obtaining the preprocessed sample image comprises: A sample image is acquired, and data amplification, size adjustment and normalization processing are performed on the sample image to obtain a preprocessed sample image.

8. An image recognition device based on SNN information supplementation, applied to an image recognition model, characterized in that: The image recognition model includes: a spiking neural network (SNN) and a convolutional neural network (CNN) having a plurality of corresponding processing stages, and the device includes: An image acquisition module is used to acquire a target image, input the target image into the initial processing stages of the pulse neural network and the convolutional neural network in the trained image recognition model, perform feature extraction and forward propagation; A feature merging module is used to process the input features corresponding to the current processing stage in the spiking neural network and the convolutional neural network in each corresponding processing stage to obtain a first feature and a second feature, merge the first feature and the second feature into a third feature, and output the third feature and the second feature to the next processing stage of the spiking neural network and the convolutional neural network respectively; The result determination module is used to determine the classification result of the target image based on the target feature through the fully connected layer of the pulse neural network after the last processing stage of the pulse neural network completes the processing of the output feature of the previous processing stage to obtain the target feature.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the image recognition method based on SNN information supplementation as described in any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the image recognition method based on SNN information supplementation as described in any one of claims 1 to 7.

Images