A method for constructing deep spiking neural networks based on meta-residuals

Abstract

The present invention provides a method for constructing a deep spiking neural network based on meta-residuals. By constructing a deep spiking neural network XOResNet, images are encoded and learned, which effectively improves the accuracy of image recognition and significantly reduces pulse redundancy and learning redundancy. The innovation of XOResNet lies in the introduction of the concept of meta-residuals. Through the design of specific residual blocks, efficient extraction and processing of information are achieved. The method is applicable to a variety of image processing scenarios, especially in visual task processing in edge computing devices. The present invention provides a stable and efficient solution for the construction and application of deep spiking neural networks, and has broad application prospects.

Description

Technical Field

[0001] The present invention belongs to the technical field of neural networks, and in particular to a method for constructing a deep pulse neural network based on element residuals. Background Art

[0002] Inspired by the working mechanism of the human brain and the working mode of biological neurons, spiking neural networks (SNNs) are expected to achieve machine intelligence as efficient as the brain. At the same time, SNNs are also regarded as the third generation of neural networks due to their energy consumption advantages of asynchronous binary pulse communication and powerful spatiotemporal information processing capabilities. By borrowing and simulating the learning algorithms and network structures of artificial neural networks (ANNs), SNNs have shown performance close to ANNs in some classification tasks, but are still inferior to ANNs in complex tasks. An important reason is that discrete binary pulses and complex spatiotemporal dynamics limit SNNs from directly adopting the deep structure of ANNs. However, deep networks have more advantages than shallow networks in terms of computational cost and representation capabilities.

[0003] In order to improve the representation ability of SNNs, it is necessary to build deep SNNs by borrowing the residual structure of ANNs. However, existing works directly replace the nonlinear activation units in ANNs with pulse neurons. This pulse version of ResNet still suffers from performance degradation caused by gradient problems. In addition, in order to solve the identity mapping problem, some works merge the outputs of the two branches by adding the output pulses / currents of the main branch and the shortcut branch in the residual structure. This indiscriminate treatment of current and pulses will cause pulse redundancy and information loss, destroying the binary characteristics of pulses. More importantly, the main branch in the residual structure performs residual learning, that is, learning knowledge different from the input (shortcut branch), while the residual structure of the current deep pulse network completely adopts the residual structure in ANNs, which will lead to learning redundancy.

[0004] In view of this, it is an urgent need to design a shortcut connection method that can merge output pulses and output currents respectively without causing pulse redundancy and information loss, and reduce the residual building blocks of redundant learning, so as to build a spiking neural network of arbitrary depth without suffering performance degradation. Based on this, the present invention proposes a deep spiking neural network construction method based on meta-residuals to overcome the current method shackles and technical bottlenecks, and bring revolutionary progress to brain-like intelligent learning systems. Summary of the invention

[0005] The present invention provides a method for constructing a deep pulse neural network based on meta-residuals, aiming to solve the problems mentioned in the above background technology.

[0006] The present invention adopts the following technical means to solve the technical problem:

[0007] The present invention provides a method for constructing a deep pulse neural network based on element residuals, and the construction method comprises the following steps:

[0008] S1. Trunk branch meta-residual feature extraction and selection: The trunk branch includes meta-residual extraction and residual feature extraction. The specific steps are:

[0009] S101. The input data of the residual block is obtained. The trunk branch first performs feature extraction with half the number of channels on the input data through a 3×3 convolution kernel and a 1×1 convolution kernel in parallel, wherein the 1×1 convolution kernel is mainly responsible for dimensional transformation to maintain the similarity of input and output features; then the output features extracted by the two branch features are integrated through an XOR operation, and the features to be learned are selected therefrom to provide the residual before learning of the trunk branch to avoid redundant learning; the features generate real residual features to provide the residual before learning, and the residual is the meta-residual;

[0010] The selection method of the XOR residual feature is expressed by the formula:

[0011]

[0012] Among them, MR represents the XOR residual, Indicates XOR operation, f1 l and Represent two feature extraction branches respectively;

[0013] S102. further extract the meta-residual features using a third convolutional layer with a scale of 3×3, keeping the number of channels unchanged, to obtain higher-level feature information;

[0014] S103. The higher-level features are dimensionally changed through a fourth convolutional layer with a scale of 1×1, and the number of feature channels is expanded to twice the number of input channels of the residual block to meet the needs of the subsequent network structure, thereby obtaining the final output features of the trunk branch;

[0015] S2. Construct shortcut connection of residual block: obtain input training data of residual block, determine the scale of input and output feature map of trunk branch and select shortcut connection branch mapping mode, the specific steps include:

[0016] When the feature map scales of the input and output of the trunk branch are the same, the shortcut connection branch implements identity mapping, and combines the output pulses of the trunk branch and the shortcut branch through an OR operation to maintain the binary attribute of the pulse and reduce pulse redundancy;

[0017] When the scales of the feature maps input and output by the trunk branch are inconsistent, the shortcut branch implements dimensional transformation, and the sum of the currents of the trunk branch and the shortcut branch is used as the input current of the pulse neuron to avoid information loss;

[0018] The shortcut connection for constructing the residual block is expressed by the formula:

[0019]

[0020] Among them, S l [t] represents the input pulse of layer l at time t, F l (S l [t]) represents the output of the trunk branch, SN represents the spike neuron, which converts the input into a [0,1] binary pulse output, and f l (S l [t]) represents the output current of the shortcut branch of the non-identity mapping, im represents the identity mapping, nim represents the non-identity mapping, and ∨ represents the OR operation;

[0021] S3. Construct an XOR meta-residual building block, combining the trunk branch and the shortcut branch based on steps S1 and S2 to form a meta-residual building block;

[0022] S4. Constructing a deep spiking neural network XOResNet: Based on the XOR residual building blocks constructed in step S3, a variable depth spiking neural network framework XOResNet is constructed;

[0023] The spiking neural network framework XOResNet consists of three main parts: input layer, intermediate encoding layer and output layer;

[0024] The intermediate coding layer is formed by sequentially stacking a plurality of XOR element residual building blocks, each building block is responsible for further extracting and processing input features;

[0025] The input layer performs preliminary processing on the input data through convolution operations, and then passes it to the encoding layer composed of XOR residual building blocks. Through layer-by-layer residual learning and feature extraction, the information is gradually compressed and converted, and finally a full connection operation is performed in the output layer to complete the classification or recognition task of the input data.

[0026] Furthermore, in the shortcut connection step of constructing the residual block, in the identity mapping stage, when the input and output scales of the trunk branch are consistent, the output features of the trunk branch and the shortcut branch are both pulse sequences, and the pulse sequences of the two branches are merged through an OR operation to maintain the binary attributes of the pulses and avoid pulse redundancy; in the non-identity mapping stage, when the input and output scales of the trunk branch are inconsistent, the shortcut branch scales the input feature map through a 1×1 convolution, and the outputs of the trunk branch and the shortcut branch are both currents. The two branches are merged by current summation, and the current sum is converted into the output pulse of the residual block through pulse neurons.

[0027] Furthermore, in the trunk branch element residual feature extraction and selection step, two parallel branches of different scales extract input features, which specifically includes the following steps:

[0028] f1 l Use convolution with a kernel size of 3×3 and a step size of 1 to convolve the input pulse feature S l [t] performs feature extraction, f1 l The number of channels is S l [t] half the number of channels;

[0029] f2 l Use a convolution kernel with a scale of 1×1 and a step size of 1 to convolve the input pulse feature S l [t] performs feature extraction, f2 l The number of channels is S l [t] half of the number of channels. This step mainly transforms the dimension of the input features, maintains the similarity of the input and output features, and unifies the scale of the output features.

[0030] Integrate f1 through XOR operation l and f2 l The output features of are used to obtain the meta-residuals that the trunk branch needs to learn.

[0031] Furthermore, in the feature extraction stage of the meta-residual feature extraction and selection step, the XOR meta-residual is feature extracted by convolution with a scale of 3×3 and a step size of 1 or 2, and the number of convolution channels is consistent with the number of meta-residual feature channels.

[0032] Furthermore, after feature extraction is performed on the element residual, the extracted features are transformed in dimension by a convolution with a scale of 1×1, a step size of 1, and an output channel number that is 4 times the input channel number, in preparation for more advanced feature extraction in the next stage.

[0033] Furthermore, in the step of constructing the deep pulse neural network XOResNet, the number of channels of each of the residual blocks is [32, 64, 128] or [64, 128, 256], and the number of identity mappings can be set according to demand, usually in a ratio of [1:1:1] or [3:n:3], where n can be set according to the network depth.

[0034] Furthermore, the first layer of the pulse neural network XOResNet is a convolutional layer, the last layer is a fully connected layer, and the intermediate coding layer is constructed by a meta-residual building block.

[0035] Furthermore, the method also includes performing different types of task training based on the XOR meta-residual building block, including but not limited to image classification, target detection and speech recognition.

[0036] Furthermore, the task training includes training using different data sets, wherein the data sets include Fashion-MNIST, CIFAR-10, CIFAR-100 and miniImageNet.

[0037] Furthermore, the convolutional layer includes a convolution operation and a batch normalization operation. In the identity mapping, the convolutional layer also includes a pulse conversion operation.

[0038] The present invention proposes a method for constructing a deep pulse neural network based on meta-residuals. By constructing a deep pulse neural network XOResNet to encode and learn images, the accuracy of image recognition is effectively improved, while pulse redundancy and learning redundancy are significantly reduced. The method of the present invention is applicable to a variety of image processing scenarios, especially in visual task processing in edge computing devices. The present invention provides a stable and efficient solution for the construction and application of deep pulse neural networks, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic diagram of an XOR element residual building block for constructing a deep pulse neural network provided by the present invention;

[0040] Figure 2 It is a schematic diagram of a common building block for constructing a deep spiking neural network provided by the present invention;

[0041] Figure 3 It is a schematic diagram of the training and testing curves of a general deep spiking neural network provided by the present invention;

[0042] Figure 4 It is a schematic diagram of the shortcut connection method provided by the present invention;

[0043] Figure 5 It is a schematic diagram of the XOR element residual building block provided by the present invention;

[0044] Figure 6 It is a schematic diagram of the structure of the deep pulse neural network XOResNet provided by the present invention;

[0045] Figure 7 It is a schematic diagram of the performance changes of different depth networks provided by the present invention on the CIFAR-10 dataset;

[0046] Figure 8 It is a schematic diagram of the changes in training and testing performance of different models of different depths on different data sets provided by the present invention;

[0047] Fig. 9 It is a schematic diagram of the Jaccard similarity coefficient of the input and output pulse characteristics of the trunk branch in the residual block with identity mapping provided by the present invention;

[0048] Fig.10 It is a schematic diagram of evaluating the models of different depths provided by the present invention on the Fashion-MNIST dataset;

[0049] Fig.11 Schematic diagram of the evaluation of models of different depths provided by the present invention on the CIFAR-10 dataset;

[0050] Fig.12 Schematic diagram of the evaluation of models of different depths provided by the present invention on the CIFAR-100 dataset;

[0051] Fig.13 It is a schematic diagram of the evaluation of models of different depths provided by the present invention on the miniImageNet dataset. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0053] like Figure 1 As shown, the present invention provides a method for constructing a deep pulse neural network based on element residuals, and the construction method includes the following steps:

[0054] S1. Trunk branch meta-residual feature extraction and selection: The trunk branch includes meta-residual extraction and residual feature extraction. The specific steps are:

[0055] S101. The input data of the residual block is obtained. The trunk branch first performs feature extraction with half the number of channels on the input data through a 3×3 convolution kernel and a 1×1 convolution kernel in parallel, wherein the 1×1 convolution kernel is mainly responsible for dimensional transformation to maintain the similarity of input and output features; then the output features extracted by the two branch features are integrated through an XOR operation, and the features to be learned are selected from them to provide the residual before learning of the trunk branch to avoid redundant learning; the features generate real residual features and provide residual before learning, so they are called meta-residuals;

[0056] The selection method of the XOR residual feature is expressed by the formula:

[0057]

[0058] Among them, MR represents the XOR residual, Indicates XOR operation, f1 l and Represent two feature extraction branches respectively;

[0059] S102. further extract the meta-residual features using a third convolutional layer with a scale of 3×3, keeping the number of channels unchanged, to obtain higher-level feature information;

[0060] S103. The higher-level features are dimensionally changed through a fourth convolutional layer with a scale of 1×1, and the number of feature channels is expanded to twice the number of input channels of the residual block to meet the needs of the subsequent network structure, thereby obtaining the final output features of the trunk branch;

[0061] S2. Construct shortcut connection of residual block: obtain input training data of residual block, determine the scale of input and output feature map of trunk branch and select shortcut connection branch mapping mode, the specific steps include:

[0062] When the feature map scales of the input and output of the trunk branch are the same, the shortcut connection branch implements identity mapping, and combines the output pulses of the trunk branch and the shortcut branch through an OR operation (i.e., combines the input and output pulses of the trunk branch through an OR operation) to maintain the binary attribute of the pulse and reduce pulse redundancy;

[0063] When the scales of the feature maps input and output by the trunk branch are inconsistent, the shortcut branch implements dimensional transformation, and the sum of the currents of the trunk branch and the shortcut branch is used as the input current of the pulse neuron to avoid information loss;

[0064] The shortcut connection for constructing the residual block is expressed by the formula:

[0065]

[0066] Among them, Sl [t] represents the input pulse of layer l at time t, F l (S l [t]) represents the output of the trunk branch, SN represents the spike neuron, which converts the input into a [0,1] binary pulse output, and f l (S l [t]) represents the output current of the shortcut branch of the non-identity mapping, im represents the identity mapping, nim represents the non-identity mapping, and ∨ represents the OR operation;

[0067] S3. Construct an XOR meta-residual building block, combining the trunk branch and the shortcut branch based on steps S1 and S2 to form a meta-residual building block;

[0068] S4. Constructing a deep spiking neural network XOResNet: Based on the XOR residual building blocks constructed in step S3, a variable depth spiking neural network framework XOResNet is constructed;

[0069] The spiking neural network framework XOResNet consists of three main parts: input layer, intermediate encoding layer and output layer;

[0070] The intermediate coding layer is formed by sequentially stacking a plurality of XOR element residual building blocks, each building block is responsible for further extracting and processing input features;

[0071] The input layer performs preliminary processing on the input data through convolution operations, and then passes it to the encoding layer composed of XOR residual building blocks. Through layer-by-layer residual learning and feature extraction, the information is gradually compressed and converted, and finally a full connection operation is performed in the output layer to complete the classification or recognition task of the input data.

[0072] In the shortcut connection step of constructing the residual block, in the identity mapping stage, when the input and output scales of the trunk branch are consistent, the output features of the trunk branch and the shortcut branch are both pulse sequences, and the pulse sequences of the two branches are merged through an OR operation to maintain the binary attributes of the pulses and avoid pulse redundancy; in the non-identity mapping stage, when the input and output scales of the trunk branch are inconsistent, the shortcut branch scales the input feature map through a 1×1 convolution, and the outputs of the trunk branch and the shortcut branch are both currents. The two branches are merged by current summation, and the current sum is converted into the output pulse of the residual block through a pulse neuron.

[0073] In the step of extracting and selecting residual features of the main branch, two parallel branches of different scales extract input features, which specifically includes the following steps:

[0074] f1 l Use convolution with a kernel size of 3×3 and a step size of 1 to convolve the input pulse feature Sl [t] performs feature extraction, f1 l The number of channels is S l [t] half the number of channels;

[0075] f2 l Use a convolution kernel with a scale of 1×1 and a step size of 1 to convolve the input pulse feature S l [t] performs feature extraction, f2 l The number of channels is S l [t] half of the number of channels. This step mainly transforms the dimension of the input features, maintains the similarity of the input and output features, and unifies the scale of the output features.

[0076] Integrate f1 through XOR operation l and f2 l The output features of are used to obtain the meta-residuals that the trunk branch needs to learn.

[0077] In the feature extraction stage of the meta-residual feature extraction and selection step, the XOR meta-residual is feature extracted by convolution with a scale of 3×3 and a step size of 1 or 2 (the step size is 1 for identity mapping and the step size is 2 for non-identity mapping), and the number of convolution channels is consistent with the number of meta-residual feature channels. After feature extraction of the meta-residual, the extracted features are dimensionally transformed by convolution with a scale of 1×1, a step size of 1, and an output channel number that is 4 times the input channel number, in preparation for more advanced feature extraction in the next stage.

[0078] In the step of constructing the deep pulse neural network XOResNet, the number of channels of each residual building block is [32, 64, 128] or [64, 128, 256], and the number of identity mappings can be set according to demand, usually in a ratio of [1:1:1] or [3:n:3], and n can be set according to the network depth. The first layer of the pulse neural network XOResNet is a convolution layer, the last layer is a fully connected layer, and the intermediate coding layer is constructed by a meta-residual building block; the convolution layer includes a convolution operation and a batch normalization operation. In the identity mapping, the convolution layer also includes a pulse conversion operation.

[0079] The method also includes performing different types of task training based on the XOR meta-residual building block, including but not limited to image classification, object detection, and speech recognition. The task training includes training using different data sets, wherein the data sets include Fashion-MNIST, CIFAR-10, CIFAR-100, and miniImageNet.

[0080] Taking into account the particularity of pulse communication, an "OR-ADD, OA" shortcut connection method is proposed, that is, when the shortcut branch realizes the identity mapping, the output pulses of the shortcut branch and the trunk branch are combined through the OR operation (OR) to avoid pulse redundancy while maintaining the binary characteristics of the pulse; when the shortcut branch is a non-identity mapping with dimensional transformation, the current sum of the shortcut branch and the trunk branch is used as the input current of the pulse neuron to avoid information loss. Further, for the residual learning of the trunk branch, the concept of meta-residual is introduced to provide meta-residual for the residual learning of the trunk branch. The residual learning of the trunk branch needs to learn features that are different from the input. In order to avoid redundant learning, the present invention screens out the residual features that need to be learned for the trunk branch through the exclusive OR operation (XOR) of the two branches.

[0081] In a specific embodiment, the present invention finds that simply increasing the depth of spiking neural networks (SNNs) or ignoring the difference between SNNs and artificial neural networks (ANNs) and completely adopting the residual structure of ANNs will lead to the performance degradation problem of SNNs based on gradient optimization. Figure 2 and Figure 3 As shown in the figure, Plain Network and Spiking ResNet are deep SNNs built using two basic building blocks (a) and (b), respectively. However, the deep networks of both networks are weaker than the shallow networks in performance, indicating that both networks have suffered from performance degradation problems.

[0082] Step 1: The present invention first proposes an OA shortcut connection method

[0083] The present invention takes into account the particularity of pulse communication in SNNs and proposes an OA connection method for the shortcut connection of the residual structure. Specifically, when the pulse realizes the identity mapping (im), the output pulses of the shortcut branch and the trunk branch are combined by OR operation to maintain the binary attribute of the pulse while avoiding pulse redundancy; when the shortcut connection realizes the non-identity mapping (nim) of dimensional transformation, the current of the shortcut branch and the trunk branch is used as the input current of the spiking neural network (SN) to avoid information loss.

[0084] like Figure 4 As shown in (a), the OR connection method is effective for maintaining binary pulse properties and merging pulses in identity mapping. However, for non-identity mapping, this may cause information loss. Combining the dynamic characteristics of SN, the logical operations in Table 1, and the proposed OA connection method ( Figure 4 (b) and Figure 4 (c)), for non-identity mappings:

[0085]

[0086] Therefore, the present invention proposes an OA connection method to combine the output pulses / currents in different connection situations, such as Figure 4 (b) and Figure 4 As shown in (c), the OA connection method is expressed as:

[0087]

[0088] Among them, S l [t] represents the input pulse of layer l at time t, F l (S l [t]) represents the output of the trunk branch, SN represents the spike neuron, and f l (S l [t]) represents the output current of the shortcut branch of the non-identity mapping, im represents the identity mapping, nim represents the non-identity mapping, and ∨ represents the OR operation.

[0089] Table 1. Conversion between logical operations and arithmetic operations

[0090]

[0091] Step 2: Propose XOR residual extraction method

[0092] SNNs rely on asynchronous binary pulses to transmit and express information. Since pulses are binary, additional calculations and redundant pulses are unnecessary. However, the residual blocks currently used to build deep SNNs do not take this into account and fully adopt the residual extraction method of ResNet, resulting in pulse redundancy, that is, learning redundancy. In order to reduce pulse redundancy and promote residual learning of the trunk branch, the present invention proposes to use XOR operations to filter out the pre-learning residuals for the trunk branch. Since the pre-learning residuals produce the expected residuals, the pre-learning residuals are called meta-residuals.

[0093] For residual learning of residual blocks, the present invention introduces the concept of meta-residual to provide meta-residual for residual learning of trunk branches. The residual learning of trunk branches needs to learn features that are different from the input. To avoid redundant learning, the present invention selects the residual features that need to be learned for the trunk branches through the exclusive OR operation (XOR) of the two branches. The selection method of the XOR meta-residual features is expressed as the following equation:

[0094]

[0095] Among them, MR represents the XOR residual, Indicates XOR operation, f1 l and Represent two feature extraction branches respectively.

[0096] Step 3: Propose XOR Meta-Residual Building Block

[0097] The proposed OA connection method and meta-residual extraction method are integrated to construct the XOR meta-residual building block. Figure 5 As shown, Figure 5 (a) is the XOR residual block that implements the identity mapping. Figure 5 (b) is the XOR residual block that implements the non-identity mapping of dimensional transformation.

[0098] Step 4: Building deep SNNs based on XOR residual building blocks

[0099] As shown in Table 2, the proposed XOR meta-residual building block is used as the basic building block to construct deep SNNs. The construction parameter settings of SNNs of different depths are shown in Table 2, where ×K represents the number of times the same building block is repeatedly stacked. Different data sets have different input and output sizes. Taking 28×28 as an example, it means that the output size is 28×28 pixels, and other scales follow the same pattern. Downsampling is performed in Stage 2 and Stage 3 through convolution operations with a stride of 2. In the Stem stage, a 256×256 image is first scaled down to 1 / 4 by a 4×4 convolution with a stride of 4 in order to reduce the amount of computation. The spatiotemporal expansion diagram of XOResNet corresponding to Table 2 is Figure 6 ,like Figure 6 As shown in Figure 1, the network model shares parameters in the time dimension, where ×N corresponds to Stage and ×M corresponds to ×K.

[0100] The different output sizes in Table 2 are only for Fashion-MNIST, CIFAR-10 / 100, and miniImageNet datasets, and are not intended to limit models and datasets.

[0101] Table 2. Structural settings of XOResNet with different depths

[0102]

[0103] In XOResNet, the OA connection method combines the output pulses / currents. The model with this connection method does not suffer from the gradient vanishing / exploding problem during gradient-based training. When the residual is 0, that is, when the model cannot learn new content, F l (S l [t]) = 0, at this time, the input and output are combined through OR connection, and the output of the lth residual block is O L [t] relative to input S l The gradient of [t] is:

[0104]

[0105] Therefore, the OA connection method is able to overcome the gradient vanishing / exploding problem.

[0106] Figure 6 The following is a network architecture diagram of image encoding and recognition using XOResNet of different depths. Its parameter configuration corresponds to Table 2. The following are two embodiments in the network model, including:

[0107] Embodiment 1

[0108] In order to evaluate the provided meta-residual-based deep spike neural network construction method, in Example 1, the present invention takes the image datasets Fashion-MNIST, CIFAR-10, CIFAR-100 and miniImageNet as examples to evaluate the feature extraction performance of the proposed method and model.

[0109] In Example 1, the recognition accuracy of the model is used as the performance evaluation index of the model, that is, the proportion of images correctly recognized by the model to all images. The higher the recognition accuracy, the better the model recognition performance. The experimental settings are: learning rate l = 0.002, training batch batch_size = 64, training round number epoch = 100 / 150, and the optimizer is the Adam optimizer.

[0110] Table 3. Performance comparison of the proposed method and the state-of-the-art work on different datasets

[0111]

[0112]

[0113] In Example 1, the ResNet connected with OR is named OR ResNet, and the ResNet connected with OA is named OAResNet. They differ only in the shortcut connection method, and the other parameters, model connection structure and training method are exactly the same. In the image encoding process, the image encoding step (simulation step) on Fashion-MNIST, CIFAR-10 and CIFAR-10 is 8, and the encoding step of the image on the miniImageNet dataset is 4.

[0114] According to the structure in Table 2, XOResNets of different depths are constructed and compared with OR ResNet, OA ResNet, Spiking ResNet and Plain Network of the same depth. Figure 7As shown in the figure, in Example 1, the mean and standard deviation of the results of 10 times are taken as the recognition accuracy statistics of the model. Spiking ResNet and Plain Network suffer from performance degradation as the network depth increases. However, XOResNet, OR ResNet and OA ResNet do not show this problem. Figure 8 As shown in Figure 2, the training accuracy and test accuracy of these three models are not inferior to those of shallow models. This shows that both OA and OR connection methods can solve the gradient disappearance / explosion problem. Figure 7 As shown in the figure, XOResNet significantly outperforms OR ResNet and OA ResNet models, indicating that the XOR meta-residual proposed in the present invention promotes the learning of deep SNN. At the same time, OA ResNet outperforms OR ResNet at the same depth, indicating that the OR connection method solves the gradient problem but causes information loss during non-identity mapping connection, which is solved by the OA connection method.

[0115] To evaluate the effectiveness of the proposed method in promoting deep SNN learning, the proposed method is compared with the state-of-the-art work. As shown in Table 3, on the Fashion-MNIST dataset, the highest recognition rate of XOResNet50 is 94.50%, which is far superior to other models. On the CIFAR-10 dataset, XOResNet is second only to ANN2SNN, however, the encoding time step of XOResNet is 8, and the encoding time step of ANN2SNN is 2048, which is 256 times that of XOResNet. In addition, the ANN2SNN with encoding time steps of 30, 400 and 2000 is weaker than the ANN2SNN with 2048 encoding time steps in performance, indicating that the encoding time step is crucial to the ANN2SNN method, which also shows the necessity of training deep SNN based on back-propagation. Similarly, on the 100-classification dataset CIFAR-100 dataset, XOResNet is second only to ANN2SNN. The resolution of the images in the above three datasets does not exceed 32×32. In order to evaluate the model on a more complex image dataset, miniImageNet is selected as the evaluation dataset, which contains 100 classes, 600 images per class, and the image resolution is unified to 256×256. As shown in Table 3, the performance of XOResNet44 is significantly better than other models, indicating that the proposed method has excellent learning and encoding capabilities on complex datasets.

[0116] To further demonstrate the effectiveness of the proposed XOR residual structure in residual learning, this example calculates the Jaccard similarity coefficient (intersection-over-union ratio) of the input and output features of the trunk branch in the identity mapping residual block. The smaller the Jaccard similarity coefficient, the smaller the similarity between the two features, and vice versa. Fig. 9As shown in the figure, for the four data sets, the depths of SNN are 56, 50, 50, and 32, respectively, and the corresponding numbers of identity mapping residual blocks are 15, 13, 13, and 7, respectively. On the four data sets, the Jaccard similarity coefficient of the input-output features in the trunk branch with XOR meta-residuals is always smaller than the Jaccard similarity coefficient of the input-output features in the trunk branch without XOR meta-residuals, indicating that XOR meta-residuals can reduce redundant learning in SNNs while promoting residual learning in the trunk branch.

[0117] Embodiment 2

[0118] The effectiveness of the method for constructing a deep pulse neural network based on XOR residuals proposed by the present invention has been demonstrated in Example 1. To further demonstrate the role of each module and branch, in Example 2, the present invention evaluates the function of each component on four data sets in turn.

[0119] The resolution of the images on the Fashion-MNIST dataset is 28×28. Based on Table 2, this embodiment constructs 6 different depths of XOResNet, OA ResNet and OR ResNet as shown in Table 4. As the baseline model, OR ResNet uses the OR connection method to solve the gradient problem in the deep SNN. As shown in Table 4, the test accuracy of the deep OR ResNet is not inferior to that of the shallow OR ResNet, indicating that the OR connection method effectively solves the gradient problem in the deep SNN. However, the present invention proves in formula (1) that the OR connection method is not suitable for non-identity mapping, so the present invention proposes an OA connection method: the output pulses of the two branches are merged by the OR connection method when they are identical, avoiding pulse redundancy while maintaining the binary characteristics of the pulse; the current of the two branches during non-identity mapping is merged by current sum (ADD), and the merged current is used as the input of the pulse neuron to avoid information loss. As shown in Table 4, the test accuracy of the deep OA ResNet is not inferior to that of the shallow OA ResNet, indicating the effectiveness of the OA connection method in solving the gradient problem in the deep SNN. The test accuracy of OA ResNet with the same depth is better than that of OR ResNet, indicating the effectiveness of the OA connection method. The present invention constructs XOResNet based on the OA connection method and the XOR meta-residual structure. The test accuracy of XOResNet with the same depth is better than that of OA ResNet, indicating the effectiveness of the proposed XOR meta-residual structure in promoting deep SNN learning. In addition, Fig.10 It shows that among the three types of models, in each training stage, OR ResNet, OA ResNet and XOResNet always maintain an increasing performance relationship, which also shows that OA is better than OR, and XOR meta-residual promotes deep SNN learning.

[0120] Table 4. Test accuracy of different depth models on the Fashion-MNIST dataset

[0121]

[0122] Based on Table 2, this embodiment constructs three types of networks with 7 different depths as shown in Table 5, and compares their performance on the CIFAR-10 dataset. For the same depth, the test accuracy of OA ResNet is always better than that of OR ResNet, especially when the depth is 11, which further shows that the OA connection method is better than the OR connection method. At the same time, the performance of XOResNet at the same depth is always better than that of OA ResNet, indicating the effectiveness of the proposed XOR residual in promoting deep SNN learning. At each training stage, Fig.11 As shown in the figure, at the same depth, XOResNet is always better than OA ResNet, and OA ResNet is always better than ORResNet, which further demonstrates the effectiveness of the individual components proposed in this paper.

[0123] Table 5. Test accuracy of different depth models on the CIFAR-10 dataset

[0124]

[0125] Compared to the 10-category CIFAR-10, CIFAR-100 contains 100 categories of images, with 600 images in each category, making the training and learning of the model more challenging. Fig.12 As shown in the figure, the model maintains the same conclusions on the Fashion-MNIST and CIFAR-10 datasets at all four depths: OA ResNet is better than the baseline method OR ResNet, and XOResNet is better than OAResNet, which once again proves the effectiveness of the proposed method.

[0126] The image resolutions in the above three datasets do not exceed 32×32. This example evaluates the performance of the model and components on a more complex dataset, miniImageNet. Fig.13 As shown, in each training stage of the same depth, XOResNet is always better than OA ResNet, and OA ResNet is always better than OR ResNet, which further demonstrates that the OA connection method proposed in the present invention is better than the OR connection method, and the proposed XOR meta-residual structure contributes and has advantages in promoting deep SNN learning.

[0127] Example 1 shows that the XOR-based residual building block proposed in the present invention can improve the learning and characterization capabilities of deep SNNs, and Example 2 shows the effectiveness of the OA connection method and the XOR-based residual extraction method proposed in the present invention.

[0128] Description of the innovative features and technical effects of the embodiments

[0129] The structure and design of ResNet in ANNs are very inspiring and instructive for building deep SNNs. Deep SNNs have stronger representation and learning capabilities. However, the current deep SNNs built based on ANN2SNN require thousands of time steps of iteration, and the gradient training method will cause SNNs to fall into the gradient vanishing / explosion problem and lead to performance degradation. The present invention starts from the particularity of pulse communication and the perspective of residual learning, and finds that in residual connection, the identity mapping needs to avoid pulse redundancy and maintain the binary attribute of the pulse. Therefore, it is proposed to use the OR connection method to merge the output pulses of the shortcut branch and the trunk branch during the identity mapping; for non-identity mapping, the use of the OR connection method will lead to information loss, so it is proposed to use the current sum (ADD) of the shortcut branch and the trunk branch as the input current of the pulse neuron, and maintain the binary attribute of the pulse by merging the current, while avoiding information loss. The present invention names this method of merging outputs in different situations as the OR-ADD (OA) connection method. For the residual learning of the trunk branch, it is found that it needs to learn information different from the input, while the traditional residual structure directly extracts features from the input. Only after learning can the learned residual be determined. This residual is a post-hoc residual and will cause redundant learning. Therefore, the present invention provides the trunk branch with features that need to be learned before residual learning. Because this feature will produce real residuals, this feature is called meta-residual. Specifically, the present invention uses two branches to extract features from the input characteristics, one of which is only responsible for dimensional transformation, and the outputs of the two branches are XORed, meta-residuals are proposed, and then sent to the trunk branch for feature extraction. The present invention integrates the proposed OA connection method and the XOR meta-residual extraction method, proposes an XOR meta-residual building block, and then constructs an XOResNet of arbitrary depth based on the XOR meta-residual building block. The present invention conducts a large number of experiments on image data sets through two examples. The experimental results show that the proposed XOResNet greatly improves the encoding and recognition capabilities of the image. At the same time, it is superior to the most advanced deep SNNs based on gradient optimization in image recognition accuracy. Existing deep SNNs construction methods either have too long encoding time steps, or have defects such as pulse redundancy, destruction of pulse binary characteristics, and redundant learning. The method proposed in the present invention avoids or reduces pulse redundancy, maintains the binary properties of pulses, avoids information loss, and avoids redundant learning. This makes the constructed SNNs more powerful in representation, more energy-efficient, and more suitable for deployment on hardware.

[0130] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for constructing a deep pulse neural network based on meta-residuals, characterized in that: The construction method comprises the following steps: S1. Trunk branch meta-residual feature extraction and selection: The trunk branch includes meta-residual extraction and residual feature extraction. The specific steps are: S101. Obtain input data of the residual block. The trunk branch first passes A convolution The kernel and a convolution kernel are used in parallel to extract features from the input data with half the number of channels. The convolution kernel is mainly responsible for dimensional transformation to maintain the similarity of input and output features; then the output features extracted by the two branch features are integrated through an XOR operation, and the features to be learned are selected from them to provide the residual before the main branch learning to avoid redundant learning; the features generate real residual features to provide the residual before learning, and the residual is the meta-residual; The selection method of XOR residual features is expressed by the formula: ； in, represents the XOR residual, represents the XOR operation, and Represent two feature extraction branches respectively; S102. Use scale The third convolutional layer further extracts the meta-residual features, keeping the number of channels unchanged, to obtain higher-level feature information; S103. By scale The fourth convolutional layer changes the dimension of the higher-level features, expands the number of feature channels to twice the number of input channels of the residual block to meet the needs of the subsequent network structure, and obtains the final output features of the trunk branch; S2. Construct shortcut connection of residual block: obtain input training data of residual block, determine the scale of input and output feature map of trunk branch and select shortcut connection branch mapping mode, the specific steps include: When the feature map scales of the input and output of the trunk branch are the same, the shortcut connection branch implements identity mapping, and combines the output pulses of the trunk branch and the shortcut branch through an OR operation to maintain the binary attribute of the pulse and reduce pulse redundancy; When the scales of the feature maps input and output by the trunk branch are inconsistent, the shortcut branch implements dimensional transformation, and the sum of the currents of the trunk branch and the shortcut branch is used as the input current of the pulse neuron to avoid information loss; The shortcut connection for constructing the residual block is expressed by the formula: ； in, express Moment The input pulse of the layer, represents the output of the trunk branch, represents a spike neuron, which converts the input into a [0,1] binary pulse output. represents the output current of the shortcut branch of the non-identity mapping, im represents the identity mapping, nim represents the non-identity mapping, Represents or operation; S3. Construct an XOR meta-residual building block, combining the trunk branch and the shortcut branch based on steps S1 and S2 to form a meta-residual building block; S4. Constructing a deep spiking neural network XOResNet: Based on the XOR residual building blocks constructed in step S3, a variable depth spiking neural network framework XOResNet is constructed; The spiking neural network framework XOResNet consists of three main parts: input layer, intermediate encoding layer and output layer; The intermediate coding layer is formed by sequentially stacking a plurality of XOR element residual building blocks, each building block is responsible for further extracting and processing input features; The input layer performs preliminary processing on the input data through convolution operation, and then passes it to the encoding layer composed of XOR residual building blocks, and gradually compresses and transforms the information through layer-by-layer residual learning and feature extraction, and finally performs full connection operation in the output layer to complete the classification or recognition task of the input data; The method also includes performing different types of task training based on the XOR meta-residual building block, including but not limited to image classification, object detection and speech recognition.

2. The method for constructing a deep pulse neural network based on meta-residual according to claim 1, characterized in that: In the shortcut connection step of constructing the residual block, in the identity mapping stage, when the input and output scales of the trunk branch are consistent, the output features of the trunk branch and the shortcut branch are both pulse sequences, and the pulse sequences of the two branches are merged through an OR operation to maintain the binary attributes of the pulses and avoid pulse redundancy; in the non-identity mapping stage, when the input and output scales of the trunk branch are inconsistent, the shortcut branch is The convolution performs a scale transformation on the input feature map, the outputs of the trunk branch and the shortcut branch are both currents, the two branches are merged by current summation, and the current sum is converted into output pulses of the residual block through pulse neurons.

3. The method for constructing a deep pulse neural network based on meta-residual according to claim 1, characterized in that: In the step of extracting and selecting residual features of the main branch, two parallel branches of different scales extract input features, which specifically includes the following steps: The convolution kernel size is used as , convolution with a step size of 1 on the input pulse feature Perform feature extraction, The number of channels is Half the number of channels; The convolution kernel size is used as , the convolution with a step size of 1 is applied to the input pulse feature Perform feature extraction, The number of channels is Half of the number of channels. This step mainly transforms the dimension of the input features, maintains the similarity of the input and output features, and unifies the scale of the output features. Combined by XOR operation and The output features of are used to obtain the meta-residuals that the trunk branch needs to learn.

4. The method for constructing a deep pulse neural network based on meta-residual according to claim 1, characterized in that: In the feature extraction stage of the meta-residual feature extraction and selection step, the scale is , convolution with a step size of 1 or 2 is used to extract features from the XOR element residuals, and the number of convolution channels is consistent with the number of element residual feature channels.

5. The method for constructing a deep pulse neural network based on meta-residual according to claim 4, characterized in that: After feature extraction of the element residual, the scale is , the convolution with a step size of 1 and an output channel number 4 times the number of input channels transforms the dimension of the extracted features, preparing for more advanced feature extraction in the next stage.

6. The method for constructing a deep pulse neural network based on meta-residual according to claim 1, characterized in that: In the step of constructing the deep pulse neural network XOResNet, the number of channels of each residual block is or The number of identity mappings can be set according to requirements, usually according to or The ratio setting, Can be set according to network depth.

7. The method for constructing a deep pulse neural network based on element residuals according to claim 6, characterized in that: The first layer of the pulse neural network XOResNet is a convolutional layer, the last layer is a fully connected layer, and the intermediate coding layer is constructed by meta-residual building blocks.

8. The method for constructing a deep pulse neural network based on meta-residual according to claim 1, characterized in that: The task training includes training using different datasets, wherein the datasets include Fashion-MNIST, CIFAR-10, CIFAR-100 and miniImageNet.

9. The method for constructing a deep pulse neural network based on meta-residuals according to any one of claims 1 to 8, characterized in that: The convolution layer includes a convolution operation and a batch normalization operation. In the identity mapping, the convolution layer also includes a pulse conversion operation.

Images