A method and system for improving the accuracy of deep residual spiking neural network to optimize image classification

By analyzing, standardizing and converting the deep residual neural network, the problem of no size limit for pulse emissivity and activation value is solved, the accuracy of the deep residual pulse neural network is improved, and efficient image recognition and sparse calculation are achieved.

CN111860790BActive Publication Date: 2024-11-22NANJING UNIV
View PDF 12 Cites 0 Cited by

Patent Information

Application Number
CN202010772308.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-08-04
Publication Date
2024-11-22
Estimated Expiration
2040-08-04

AI Technical Summary

Technical Problem

During the conversion process of deep residual neural network to deep residual pulse neural network, there is a linear mismatch problem caused by no size limit on pulse emissivity and activation value, and the prior art lacks effective standardization processing, resulting in a decrease in the accuracy after conversion.

Method used

Optimize the deep residual neural network through three steps: parsing, standardizing and converting, including standardizing parameters with weight and bias, absorbing batch standardization layer parameters, and adding hyperparameter accuracy compensation to the short-circuit connection structure to solve the linear mismatch problem.

Benefits of technology

The accuracy of the deep residual pulse neural network is improved, and efficient recognition is achieved in image classification tasks. The converted network has high sparsity and no multiplication characteristics, which is suitable for high-energy-efficient hardware computing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111860790B_ABST
    Figure CN111860790B_ABST
Patent Text Reader

Abstract

The present invention relates to a method and a system for improving the accuracy of a deep residual pulse neural network to optimize image classification. The method disclosed in the present invention takes the transformation process of a deep residual neural network to a deep residual pulse neural network as an optimization object, parses, standardizes, converts, and optimizes the network layer in the transformation process, and proves the effectiveness of the optimized deep residual pulse neural network in complex image recognition problems; the optimization method proposes an iterative average standardization strategy, which solves the standardization problem faced when transforming short-circuit connections in a residual neural network structure; the algorithm-optimized deep residual pulse neural network has high sparsity and multiplication-free algorithmic properties, which determines the high efficiency and hardware friendliness of the calculation; when tested on the MNIST data set, the deep residual pulse neural network generated after the transformation has no precision loss, and the result of the test on the CIFAR10 data set has only 1.3% precision loss, which is better than the existing technical level.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of deep residual spike neural networks, and in particular to a method and a system for improving the accuracy of a deep residual spike neural network to optimize image classification. Background Art

[0002] With the continuous breakthroughs in deep learning technology, artificial intelligence technology with artificial neural networks as the core is developing rapidly. The third-generation neural network, the spiking neural network, aims to fill the theoretical gap between current neuroscience and machine learning. It uses the model that most closely fits the biological neuron mechanism for calculation. The spiking neural network uses pulses as the basic action unit, which is a discrete event that occurs at a point in time. Pulses enhance the ability of neural networks to process spatiotemporal data. At the spatial level, neurons are only connected to nearby neurons so that they can process input blocks separately, which is somewhat similar to the filter of a convolutional neural network; at the temporal level, pulses occur over time, which allows us to retrieve information that may be lost in the encoding process from the information of the pulse. It turns out that spiking neurons are more powerful computing units than traditional artificial neurons.

[0003] After analysis, it is found that deep neural networks are often over-parameterized, with a large amount of redundant information stored internally, and not all parameters and structures contribute to the high discriminability of deep neural networks. Training of spiking neural networks is still difficult, and research on converting spiking neural networks from neural networks is still limited to shallow neural networks.

[0004] Due to the lack of effective standardization processing in the prior art, the linear mismatch is easily caused because the pulse emission rate of the deep residual spike neural network is limited in size while the activation value of the deep residual neural network is not limited in size. Summary of the invention

[0005] Purpose of the invention: To optimize the conversion process of deep residual neural network to deep residual spiking neural network, and propose a method to improve the accuracy of deep residual spiking neural network to optimize image classification. A further purpose is to provide a system to execute the above method.

[0006] Technical solution: A method for improving the accuracy of deep residual spike neural network to optimize image classification, which is achieved by optimizing the transformation process of deep residual neural network to deep residual spike neural network. The transformation process includes three steps: parsing, standardization, and conversion, that is, parsing, standardizing, and converting the short-circuit connection structure and other network structures unique to deep residual neural network:

[0007] Analysis: Construct a deep residual neural network and analyze various network structures in deep neural networks;

[0008] Standardization: Standardize the parameters in the network model;

[0009] Conversion: Convert the network model with normalized parameters to a deep residual spiking neural network.

[0010] In a further embodiment, the parsing process includes: establishing a network layer of a deep residual pulse neural network, and if a network layer of the corresponding deep residual neural network model has parameters, loading the parameters of the layer into the established network layer of the deep residual pulse neural network.

[0011] In a further embodiment, when establishing the network layer of the deep residual spiking neural network, if a batch normalization layer is encountered, all the parameters of the batch normalization layer are absorbed into the parameters of the convolutional layer of the previous spiking neural network to solve the limitation problem that the spiking neural network does not include a batch normalization layer.

[0012] In a further embodiment, when establishing a network layer of a deep residual spike neural network, if an Add layer unique to the deep residual neural network is encountered, two network layers with the same dimension are added point by point.

[0013] In a further embodiment, parameter normalization is performed on network layers with parameters of the deep residual spike neural network, such as convolutional layers and fully connected layers, including weight normalization and bias normalization, so as to solve the linear mismatch problem caused by the fact that the pulse firing rate of the deep residual spike neural network has a size limit while the activation value of the deep residual neural network has no size limit.

[0014] In a further embodiment, the weight normalization process is performed by scaling the layer and the previous layer by In the normalization process, the parameters are first multiplied by the previous scaling factor. to restore the size, then divide by the layer scale factor Zoom out again:

[0015]

[0016] In the formula, represents the parameters of the current layer before weight normalization, Represents the parameters after weight standardization; Indicates the scaling factor of the current layer, Indicates the scaling factor of the previous layer.

[0017] In a further embodiment, the scaling factor is obtained by activating the layer The 99.9% size of the distribution is determined by:

[0018]

[0019] If we encounter the Add layer of a deep residual neural network, two network layers will enter the Add layer for calculation at the same time, because there will be two scaling factors at the same time in the previous layer. and , this method will take and The middle value of , such as the geometric mean, is used as the uniform scaling factor of the upper layer:

[0020]

[0021] At the same time and The bias normalization process is determined by analyzing the scaling factor of the layer. During the normalization process, the parameter is divided by the scaling factor of the layer. :

[0022]

[0023] In the formula, represents the current layer parameters before bias normalization, represents the parameters after bias normalization.

[0024] The weight normalization formula at this time is:

[0025]

[0026] In order to make the obtained deep residual spike neural network more accurate, two hyperparameters are added to the scaling factor when encountering a short-circuit connection structure: To compensate for the accuracy, Used to increase the scaling factor of the convolutional layer in the short-circuit structure identity mapping, Used to reduce the scaling factor of convolutional layers in short-circuit non-identity mapping.

[0027] Based on the above method for improving the accuracy of deep residual spike neural network to optimize image classification, the present invention further proposes a system for improving the accuracy of deep residual spike neural network to optimize image classification, which includes a parsing module, a standardization processing module, and a conversion processing module.

[0028] In a further embodiment, the parsing module is used to establish a basic network layer of a deep residual spiking neural network. If a certain network layer of the corresponding deep residual neural network model has parameters, the parameters of the network layer are loaded into the network layer of the established deep residual spiking neural network. When establishing the network layer of the deep residual spiking neural network, if a batch normalization layer in the network model is encountered, all the parameters of the batch normalization layer are absorbed into the convolutional layer parameters of the previous layer of the spiking neural network. If an Add layer unique to the deep residual neural network is encountered, two network layers with the same dimension are added point by point.

[0029] In a further embodiment, the standardization processing module is used to perform weight standardization and bias standardization on the convolutional layer and the fully connected layer in the deep residual spike neural network; the weight standardization is determined by the scaling factor of the layer and the previous layer. In the weight standardization process, the weight parameter of the layer is first multiplied by the scaling factor of the previous layer. to restore the size, then divide by the layer scale factor Zoom out again:

[0030]

[0031] In the formula, represents the parameters of the current layer before weight normalization, Represents the parameters after weight standardization; Indicates the scaling factor of the current layer, Indicates the scaling factor of the previous layer;

[0032] The scaling factor is passed through the layer activation value The 99.9% size of the distribution is determined by:

[0033]

[0034] When encountering the Add layer of the deep residual neural network, two network layers enter the Add layer for operation at the same time. At this time, the previous layer of this layer also has and In this case, we take and The geometric mean of is used as the uniform scaling factor for the upper layer:

[0035]

[0036] Then and Update the value of

[0037] The weight normalization formula at this time is:

[0038]

[0039] In the formula, each symbol has the same meaning as above;

[0040] The bias normalization process is determined by analyzing the scaling factor of the layer. During the normalization process, the parameter is divided by the scaling factor of the layer. :

[0041]

[0042] In the formula, represents the current layer parameters before bias normalization, represents the parameters after bias normalization;

[0043] When encountering a short-circuit connection structure, add two hyperparameters to the scaling factor To perform precision compensation; Used to increase the scaling factor of the convolutional layer in the short-circuit structure identity mapping, Used to reduce the scaling factor of convolutional layers in short-circuit non-identity mapping.

[0044] Beneficial effects: The present invention relates to a method and system for improving the accuracy of a deep residual spike neural network to optimize image classification. By weight normalization and bias normalization of the network layer with parameters of the deep residual spike neural network, the linear mismatch problem caused by the size limit of the pulse emission rate of the deep residual spike neural network and the unlimited size limit of the activation value of the deep residual neural network is solved. By absorbing all the parameters of the batch normalization layer into the parameters of the convolutional layer of the previous spike neural network, the limitation problem of not including the batch normalization layer in the spike neural network is solved. When encountering a short-circuited structure, two hyperparameters are added to the scaling factor To compensate for the accuracy, Used to increase the scaling factor of the convolutional layer in the short-circuit structure identity mapping, It is used to reduce the scaling factor of the convolutional layer in the short-circuit structure non-identity mapping, so that the obtained deep residual spike neural network can be more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a schematic diagram of the process of converting deep neural network to pulse neural network.

[0046] Figure 2 Schematic diagram of a deep residual neural network with three residual modules.

[0047] Figure 3 It is a schematic diagram of the iterative average normalization algorithm for deep residual spike neural network parameters.

[0048] Figure 4It is a schematic diagram comparing the correlation between the activation value of the deep residual neural network and the pulse firing rate of the deep residual spike neural network before and after the use of the normalization algorithm.

[0049] Figure 5 This is a schematic diagram of the activation value distribution of the fully connected layer of the deep residual neural network ResNet20 on the CIFAR10 dataset.

[0050] Figure 6 This is a schematic diagram of the pulse firing rate distribution of the fully connected layer of the deep residual spike neural network ResNet20 on the CIFAR10 dataset.

[0051] Figure 7 This is a schematic diagram of the activation value of the ResNet20 fully connected layer on the CIFAR10 dataset and the difference in the converted pulse firing rate. DETAILED DESCRIPTION

[0052] In the following description, a large number of specific details are provided to provide a more thorough understanding of the present invention. However, it is apparent to those skilled in the art that the present invention can be implemented without one or more of these details. In other examples, in order to avoid confusion with the present invention, some technical features well known in the art are not described.

[0053] The present invention relates to a method for improving the accuracy of a deep residual spike neural network to optimize image classification, which is achieved by optimizing the conversion process of a deep residual neural network to a deep residual spike neural network, such as Figure 1 As shown in the figure, the conversion process includes three steps: parsing, standardization, and conversion. Specifically, a deep residual neural network is first established in the neural network computing framework Keras, and trained to obtain a network model that can perform effective image recognition. Then the network is parsed, that is, the various network structures in the deep neural network are parsed. Next, the parameters in the network model are standardized. Finally, the network model with standardized parameters is converted into a deep residual pulse neural network.

[0054] The analysis process includes: first, determining the corresponding network layer in the deep residual spike neural network according to the obtained deep residual neural network model. Here, the deep residual neural network ResNet containing three residual structures (short-circuit connection structures) is taken as an example. Figure 2As shown in the figure, the network model includes input layer, convolution layer, batch normalization layer, activation layer, Add layer (point-by-point addition of two network layers with the same dimension), mean pooling layer, Flatten layer (one-dimensionalization of multi-dimensional input) and fully connected layer. Next, establish the network layer of the deep residual spiking neural network. If the corresponding layer of the deep residual neural network has parameters, load the parameters of this layer into the established network layer of the deep residual spiking neural network.

[0055] Specifically, the input layer, Flatten layer, Add layer, and mean pooling layer keep their functions unchanged and are directly mapped to the corresponding spike network layer; the convolution layer and the fully connected layer migrate their parameters to the spike convolution layer and the spike fully connected layer corresponding to the spike neural network, and calculate them in the forward reasoning mode of the spike neural network; the activation layer is skipped; the batch normalization layer calculation formula is

[0056] …… ...

[0057] When encountering a batch normalization layer, combine all the parameters of the batch normalization layer , They are absorbed into the weights and biases of the convolutional layer of the previous spiking neural network by multiplication and addition respectively.

[0058] The parameters of the network layers with parameters of the deep residual spike neural network, such as convolutional layers and fully connected layers, are normalized, including weight normalization and bias normalization. The weight normalization process is to analyze the scaling factors of this layer and the previous layer. In the normalization process, the parameters are first multiplied by the previous scaling factor. to restore the size, then divide by the layer scale factor Then reduce it. The scaling factor is determined by the activation value of the layer The size of the 99.9% distribution is determined, that is,

[0059] …………………………………………(2)

[0060] The formula for weight normalization is:

[0061] …… ...

[0062] If you encounter the Add layer of a deep residual neural network, Figure 2 As shown, there will be two network layers entering the Add layer for calculation at the same time, because there will be two scaling factors at the same time in the previous layer. and , this method takes and Some intermediate value of , such as the geometric mean, is used as a uniform upper layer scaling factor, that is,

[0063] ……………………………………(4)

[0064] The formula for weight normalization in this case is

[0065] ………………………………………………(5)

[0066] The bias normalization process is determined by analyzing the scaling factor of the layer. During the normalization process, the parameters are divided by the scaling factor of the layer. , the formula for bias normalization is

[0067] …………………………………………………………………………(6)

[0068] In order to make the obtained deep residual spike neural network more accurate, two hyperparameters are added to the scaling factor when encountering a short-circuit connection structure. To compensate for the accuracy, Used to increase the scaling factor of the convolutional layer in the short-circuit structure identity mapping, It is used to reduce the scaling factor of the convolutional layer in the short-circuit structure non-identity mapping. That is, in the identity mapping,

[0069] ………………………………(7)

[0070] In a non-identity mapping,

[0071] …………………………(8)

[0072] like Figure 3 The Deep Residual Spiking Neural Network Weight Normalization Algorithm shown in the figure details the steps of the iterative average normalization algorithm. First, each layer of the network with parameters is traversed to initialize the scaling factor; then, the two scaling factors entering the Add layer are updated for the residual structure unique to the deep residual neural network.

[0073] Using the above method, experiments were conducted on ResNet8, ResNet14, and ResNet20 networks for the MNIST dataset. The experimental results showed that there was no loss of accuracy in the conversion and no hyperparameters were needed. Using the above method, experiments were conducted on the ResNet20 network with a relatively complex structure for the CIFAR10 dataset to determine the hyperparameters. The range is between 1 and 1.1. The range is between 1 and 1.2. The experimental results show that the accuracy of the converted deep residual pulse neural network ResNet20 is 90.28%, which is only 1.3% less than the original deep residual neural network accuracy of 91.58%, which is at the world's leading level. Under the same experimental conditions (CIFAR10 dataset, network model is ResNet20), the accuracy loss shown in "Going Deeper in Spiking Neural Networks: VGG and Residual Architectures" published in the journal "Frontiers in Neuroscience" in 2019 was 1.64%.

[0074] Figure 4 This is a comparison of the correlation between activation value and pulse firing rate in the third convolutional layer of the ResNet20 network model when the iterative average normalization algorithm is used or not. It can be seen that when the iterative average normalization algorithm is not used, a large number of pulse firing rates after conversion have serious overflow, which will lead to a decrease in accuracy. Figure 5 , Figure 6 , Figure 7 As shown in the figure, the activation value distribution at the output of the last fully connected layer of ResNet20 is basically the same as the distribution of the converted pulse firing rate, which proves the effectiveness of the converted deep residual spike neural network for image classification tasks.

[0075] This method effectively improves the image recognition accuracy of the converted deep residual spiking neural network, and achieves the goal of completing image classification tasks with a sparser network. At the same time, the significance of this conversion method is that, under the premise of ensuring recognition accuracy, the converted spiking neural network can use its large sparsity and no multiplication characteristics to achieve energy-efficient hardware-friendly computing.

[0076] In summary, the method disclosed in the present invention takes the transformation process of deep residual neural network to deep residual pulse neural network as the optimization object, parses, standardizes, converts the network layer during the transformation process, and performs targeted optimization, which proves the effectiveness of the optimized deep residual pulse neural network in complex image recognition problems; the optimization method proposes an iterative average normalization strategy, which solves the standardization problem faced when transforming short-circuit connections in the residual neural network structure; the algorithm-optimized deep residual pulse neural network has high sparsity and multiplication-free algorithmic properties, which determines the high efficiency and hardware friendliness of the calculation; when tested on the MNIST data set, the deep residual pulse neural network generated after the transformation has no precision loss, and the result of testing on the CIFAR10 data set has only 1.3% precision loss, which is better than the existing technology level.

[0077] As described above, although the present invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the present invention itself. Various changes may be made to it in form and detail without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A method for improving the accuracy of deep residual spiking neural networks to optimize image classification, characterized in that The steps include: Step 1: Analysis: Construct a deep residual neural network and analyze various network structures in the deep neural network, including: Establishing a basic network layer of a deep residual spiking neural network, if a network layer of the corresponding deep residual neural network model has parameters, then loading the parameters of the network layer into the established network layer of the deep residual spiking neural network; When establishing the network layer of the deep residual spiking neural network, if a batch normalization layer is encountered in the network model, all the parameters of the batch normalization layer are absorbed into the convolutional layer parameters of the previous spiking neural network. When establishing the network layer of the deep residual spike neural network, if the Add layer unique to the deep residual neural network is encountered, two network layers with the same dimension are added point by point; Step 2, standardization: standardizing the parameters in the network model, specifically including: standardizing the parameters of the network layer with parameters in the deep residual spike neural network; The network layer with parameters includes at least a convolutional layer and a fully connected layer; the parameter standardization includes at least weight standardization and bias standardization; The weight normalization is determined by the scaling factors of this layer and the previous layer. In the weight normalization process, the weight parameter of this layer is first multiplied by the scaling factor of the previous layer. to restore the size, then divide by the layer scale factor Zoom out again: ; In the formula, represents the parameters of the current layer before weight normalization, Represents the parameters after weight standardization; Indicates the scaling factor of the current layer, Indicates the scaling factor of the previous layer; The scaling factor is the value of the layer activation. The 99.9% size of the distribution is determined by: ; When encountering the Add layer of the deep residual neural network, two network layers enter the Add layer for operation at the same time. At this time, the previous layer of this layer also has and In this case, we take and The geometric mean of is used as the uniform scaling factor for the upper layer: ; Then and Update the value of The weight normalization formula at this time is: ; In the formula, the symbols have the same meaning as above; The bias normalization process is determined by analyzing the scaling factor of the layer. During the normalization process, the parameter is divided by the scaling factor of the layer. : ; In the formula, represents the current layer parameters before bias normalization, represents the parameters after bias normalization; When encountering a short-circuit connection structure, add two hyperparameters to the scaling factor To perform precision compensation; Used to increase the scaling factor of the convolutional layer in the short-circuit structure identity mapping, Used to reduce the scaling factor of the convolutional layer in the short-circuit structure non-identity mapping; Step 3: Conversion: Convert the network model with standardized parameters into a deep residual spiking neural network that can perform effective image recognition; input the image into the deep residual spiking neural network to achieve the goal of the image classification task.

2. A system for improving the accuracy of deep residual spiking neural networks to optimize image classification, characterized in that Includes the following modules: A parsing module for parsing various network structures in deep neural networks; A standardization processing module for standardizing parameters in a network model; A conversion processing module for converting a network model with standardized parameters into a deep residual spiking neural network; The parsing module is further used to establish a basic network layer of the deep residual spiking neural network. If a certain network layer of the corresponding deep residual neural network model has parameters, the parameters of the network layer are loaded into the network layer of the established deep residual spiking neural network. When establishing the network layer of the deep residual spiking neural network, if a batch normalization layer is encountered in the network model, all the parameters of the batch normalization layer are absorbed into the convolutional layer parameters of the previous spiking neural network; if an Add layer unique to the deep residual neural network is encountered, two network layers with the same dimension are added point by point; The standardization processing module is further used to perform weight standardization and bias standardization on the convolutional layer and the fully connected layer in the deep residual spike neural network; the weight standardization is determined by the scaling factor of the layer and the previous layer. In the weight standardization process, the weight parameter of the layer is first multiplied by the scaling factor of the previous layer. to restore the size, then divide by the layer scale factor Zoom out again: ; In the formula, represents the parameters of the current layer before weight normalization, Represents the parameters after weight standardization; Indicates the scaling factor of the current layer, Indicates the scaling factor of the previous layer; The scaling factor is passed through the layer activation value The 99.9% size of the distribution is determined by: ; When encountering the Add layer of the deep residual neural network, two network layers enter the Add layer for operation at the same time. At this time, the previous layer of this layer also has and In this case, we take and The geometric mean of is used as the uniform scaling factor for the upper layer: ; Then and Update the value of The weight normalization formula at this time is: ; In the formula, the symbols have the same meaning as above; The bias normalization process is determined by analyzing the scaling factor of the layer. During the normalization process, the parameter is divided by the scaling factor of the layer. : ; In the formula, represents the current layer parameters before bias normalization, represents the parameters after bias normalization; When encountering a short-circuit connection structure, add two hyperparameters to the scaling factor To perform precision compensation; Used to increase the scaling factor of the convolutional layer in the short-circuit structure identity mapping, Used to reduce the scaling factor of convolutional layers in short-circuit non-identity mapping.

Citation Information

Patent Citations

  • Image recognition method based on FPGA customized pulse neural network

    CN108470190A

  • An image recognition method based on hierarchical feature extraction and multi-layer impulse neural network

    CN109102000A

  • An image feature description method based on impulse neural network

    CN109214395A

  • Capsule residual error neural network and image classification method of capsule residual error neural network

    CN110009097A

  • Multi-granularity-based deep neural network structured sparse system and method

    CN110276450A