Method for converting ANN model into SNN model for network layer

Abstract

The invention discloses a method for converting an ANN model into an SNN model for a network layer, belongs to the technical field of intelligent information processing, and is used for classifying and identifying a target image. The conversion method comprises the following steps: firstly, establishing a target image data set and an ANN model used for classifying and identifying a target in an image, training the ANN model based on the data set, and then converting the ANN model into an SNN model. Secondly, encoding image data in the data set into a pulse sequence format to form a data set for the SNN model, and calibrating parameters of each layer of the SNN model to obtain the SNN model meeting the precision requirement; and finally, encoding to-be-detected image data into a pulse sequence format, and inputting the to-be-detected image data into the SNN model to obtain a target classification and recognition result. According to the method, high-precision identification of the target is realized in an environment with high interference intensity.

Description

Technical Field

[0001] The present invention relates to a method for converting an ANN model to an SNN model for the network layer, belonging to the technical field of intelligent information processing. Background Art

[0002] The second-generation artificial neural network (ANN) model and the deep learning processor for ANN have been vigorously developed. The deep learning technology based on ANN has been widely applied in fields such as image recognition. However, most research and applications are based on the training and testing under noise-free conditions. Relevant research also shows that ANN still has non-negligible defects - that is, it is vulnerable to various attacks during the training and inference stages and cannot output correct results when affected by external noise and other interferences, which will seriously affect the accuracy of system applications. A typical interference situation is the adversarial sample attack. Adversarial samples refer to the subtle perturbations that attackers add to the original samples, causing the target model to output wrong results with extremely high confidence. Szegedy et al. first proposed adversarial samples (L-BFGS) in 2013, that is, adding some imperceptible subtle perturbations to the benign samples, causing the model to output wrong results with extremely high confidence. For example, a picture originally recognized as a vase may be misclassified as a cat by the ANN model after adversarial modification. This shows the vulnerability and susceptibility of deep learning, making people pay more attention to the stability and robustness of deep neural networks.

[0003] Compared with the second-generation neural network, the spiking neural network of the third-generation neural network is a type closer to the biological neural network, so it has more biological interpretability and lower power consumption. The SNN is a network model based on spiking neurons, which transmits feature information through the temporal and spatial information in the spike train, simulating the working mechanism of synaptic establishment, enhancement and inhibition in the biological nervous system. The SNN combines the biological working principle and the mathematical model to construct a neural network model with both biological interpretability and computational efficiency.

[0004] The SNN has powerful capabilities such as spatio-temporal information representation, asynchronous event information processing, and network self-organizing learning. However, due to the non-differentiable nature of pulses, traditional backpropagation algorithms cannot be directly applied to spiking neural networks, which are difficult to train. In recent years, a large number of studies on SNNs have been conducted at home and abroad, mainly divided into three categories: direct training algorithms based on gradient descent rules, direct training algorithms based on synaptic plasticity rules, and construction methods based on ANN training and conversion. Algorithms based on gradient descent rules approximate the neuron response function by using a differentiable function to solve the problem that the latter cannot be differentiated, but this method has problems such as high power consumption during the training process and difficulty in implementing floating-point operations on neuromorphic hardware. Summary of the Invention

[0005] The technical problem solved by the present invention is: overcoming the deficiencies of the prior art, a method for converting an ANN model to an SNN model for the network layer is proposed, an SNN model with the same network structure as the trained ANN model is built, and the parameters of each layer are calibrated layer by layer to reduce the conversion error, so as to achieve the conversion of the model with less accuracy loss and realize the accurate recognition of the target.

[0006] The technical solution of the present invention is:

[0007] A method for converting an ANN model to an SNN model for the network layer, including:

[0008] S1: For the required target classification and recognition task, establish a target image dataset, and the image data in the dataset is multi-class target image data including the targets to be classified and recognized.

[0009] S2: Establish an ANN model for realizing target classification and recognition, and then constrain the network structure of the ANN model.

[0010] S3: Based on the target image dataset, train the constrained ANN model, and after reaching the set accuracy, convert it into an SNN model.

[0011] S4: Encode the image data in the target image dataset into a spike train format to form a dataset for the SNN model; based on the dataset, calibrate the parameters of each layer of the SNN model to obtain an SNN model that meets the accuracy requirements.

[0012] S5: Encode the image data to be measured into a spike train format and input it into the SNN model to obtain the result of target image classification and recognition.

[0013] Further, in step S3, the method for converting the ANN model that meets the accuracy requirements into an SNN model is:

[0014] S3.1: Extract the parameters of each layer of the ANN model;

[0015] S3.2: Corresponding to the operators in the ANN, establish operators that conform to the neuron calculation mode of the SNN model;

[0016] S3.3: Analyze the network structure of the ANN model and create the network structure of the SNN model;

[0017] Further, in step S3.3, when analyzing the network structure of the ANN model, the analysis method is: split the network structure into data in different stages and different network layers, that is, convert it into a directed acyclic graph composed of data and network layers, where the data is used as nodes and the network layers are used as connecting edges to complete the description of the network structure.

[0018] Further, in step S3.3, analyze the network structure of the ANN model, obtain the network structure of the SNN model, and then extract the ANN model parameters and write them into the corresponding network layers of the SNN model to complete the construction of the SNN model.

[0019] Further, in step S3.2, establish operators that conform to the neuron calculation mode of the SNN model in the ANN model. The operators include SpikeConv2d, SpikeLinear, SpikeMaxPool2d, SpikeReLU, and SpikeConvTranspose2d.

[0020] Further, in step S4, calibrate the parameters of each layer of the SNN model so that each layer of the SNN and the ANN produce the same output for the same input. The calibration method is:

[0021] Based on the dataset for the SNN model, estimate the maximum threshold of each layer of the ANN model to determine the adjustment interval of the dynamic threshold;

[0022] Scale the maximum threshold as the initial threshold of the SNN network layer;

[0023] After determining the initial threshold, set the time step and perform adaptive adjustment to obtain the optimal threshold for each layer: input the data in the dataset into the SNN model for inference, calculate the maximum value of the weighted pulse input of each layer of the network at the current moment. If it is greater than the current threshold, update the current threshold according to the maximum value and ensure that the threshold meets the previously determined adjustment interval after updating.

[0024] Further, in step S4, adjust the time step so that the calibrated SNN model meets the accuracy requirements.

[0025] Further, in step S4, use the frequency encoding method to convert the image data in the target image dataset into a pulse sequence format to meet the input data type requirements of the SNN model.

[0026] Further, in step S2, the network structure of the ANN model is constrained, and the network structure of the ANN model is adjusted. The constraints include:

[0027] Constraining the activation function, using the ReLU activation function as the non - linear activation function, and setting the bias term to zero to reduce the generation of negative values in the network;

[0028] Constraining the pooling function, using average pooling operation.

[0029] Further, adversarial sample data is added to the training dataset, and the anti - interference ability of the SNN model obtained in step S8 that meets the accuracy requirements is tested to verify the anti - interference ability of the SNN model.

[0030] The advantages of the present invention compared with the prior art are as follows:

[0031] (1) The construction method based on ANN training and conversion adopted by the present invention uses the ANN model for training and directly applies the obtained weights to the SNN model, overcoming the problem that the SNN network is difficult to directly train.

[0032] (2) The present invention adopts a conversion idea oriented to the network layer. Based on the neuron characteristics of the SNN model, relevant constraints are imposed on the network structure of the ANN model to adapt to the characteristics of the SNN model, providing a basic condition for the implementation of the model conversion method.

[0033] (3) The present invention uses methods such as parameter optimization for the SNN model to calibrate the parameters of each layer of the SNN layer by layer, which can further reduce the conversion error, thereby realizing the conversion of the model with less accuracy loss and laying a foundation for the practical application of the SNN. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] By reading the detailed description of the preferred embodiments below, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present invention. Moreover, throughout the drawings, the same reference numerals are used to represent the same components. In the drawings:

[0035] Figure 1 is the flowchart of the method for converting the ANN model to the SNN model oriented to the network layer according to the embodiment of the present invention;

[0036] Figure 2 is the schematic diagram of converting the ANN to the constrained ANN according to the embodiment of the present invention;

[0037] Figure 3 is the schematic diagram of the description method for standardizing the network structure according to the embodiment of the present invention;

[0038] Figure 4 Schematic diagram of the method for determining the initial threshold of the SNN network layer in the embodiments of the present invention;

[0039] Figure 5 Schematic diagram of the method for dynamically adjusting the threshold of the SNN network layer in the embodiments of the present invention. Detailed implementation manners

[0040] Hereinafter, exemplary embodiments of the present disclosure will be described in more detail with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be completely conveyed to those skilled in the art.

[0041] In a ship target classification and recognition task, an ANN model based on Resnet was designed. After training based on the HRSC dataset, its anti-interference ability was tested. As adversarial samples were added, the training accuracy of the ANN model dropped sharply and it could no longer output correct results, which would seriously affect the accuracy of system applications. To improve this situation, an SNN model with strong anti-interference ability was adopted to improve the recognition accuracy and meet the project requirements. However, since it is difficult to directly construct and train the SNN model. Based on the above problems, the present invention proposes a method for converting an ANN model to an SNN model for the network layer to achieve model conversion with low accuracy loss and at the same time achieve good anti-interference ability.

[0042] The process of this method is as Figure 1 shown and includes:

[0043] (1) Establish an ANN model

[0044] Based on the ship target classification and recognition task, a small-sample target image dataset, namely the HRSC dataset, is established, which contains three types of samples, namely aircraft carrier, warcraft, and merchant ship, and the ratio of the three types of samples in the dataset is 1:1:1. The entire dataset contains a total of 600 training set data, 75 validation set data, and 75 test set data.

[0045] For the HRSC small-sample dataset, first, an ANN model based on ResNet is established for ship recognition, and the specific structure is listed in Table 1. After training, the recognition accuracy of the ship recognition model based on the ResNet network reaches 92.0%.

[0046] Then, the interference ability test of the model is carried out. The adversarial FGSM attack experiment is conducted on the test set of the HRSC dataset. The adversarial attack results of the ANN on the HRSC dataset are listed in Table 2. With the addition of adversarial samples, the training accuracy of the ANN model drops sharply and it can no longer output correct results, which will seriously affect the accuracy of system applications.

[0047] Table 1 Ship recognition model structure

[0048]

[0049] Table 2 Recognition accuracy of adversarial attacks of the ANN model

[0050]

[0051] To improve this situation, the study adopts the SNN model with strong anti-interference ability to improve the recognition accuracy. However, since it is difficult to directly construct and train the SNN model, the ANN model-to-SNN model conversion for the network layer needs to be carried out below to achieve model conversion with low accuracy loss and good anti-interference ability at the same time.

[0052] (2) Add constraints to the ANN model and adjust the network structure to adapt to the characteristics of the SNN model.

[0053] Based on the neuron characteristics of the SNN model, the network structure of the ANN model is constrained to adapt to the characteristics of the SNN model. The constraints include:

[0054] a) Constrain the activation function. Use the ReLU activation function as the non-linear activation function and set the bias term to zero to reduce the generation of negative values in the network.

[0055] b) Constrain the pooling function. Use average pooling operation instead of max pooling.

[0056] After the above adjustments, the ANN can be converted into a constrained ANN.

[0057] Note that the above network structure constraint method is not the only one. There are slight differences for different network structures. Even if no constraint conditions are taken, the ANN can still be converted into an SNN, but there is a large approximation error before and after the network conversion, making the weights obtained by ANN training difficult to apply to the SNN, resulting in a greater accuracy loss for the SNN.

[0058] Next, an example is used to illustrate how to convert the ANN into a constrained ANN to adapt to the characteristics of the SNN model.

[0059] Figure 2(a) is a typical ANN structure, and its network structure mainly includes a convolutional layer, a pooling layer, a fully connected layer, and a non-linear activation function. The first layer of this structure is the convolutional layer, which realizes the local feature response to the input data through filtering operations to finally extract image features. The second layer is the non-linear activation function layer, usually located after the convolutional layer, which is used to perform point-by-point non-linear transformation on the feature map, enabling the neural network to have stronger non-linear expression ability. There are various types of activation functions, and the more commonly used ones are Softplus and ReLU. The third layer is the spatial max pooling layer, which obtains the maximum output value within the pooling window by downsampling the feature map. In this ANN network structure, a max pooling layer is not adopted after the last convolutional layer, which is a coincidence in the selection of the network structure. The last layer constructs a linear classifier through a fully connected method.

[0060] According to the above method of adding constraints, the ANN in Figure 2 (a) can be converted into a constrained ANN, and the result is as shown in Figure 2 (b).

[0061] In this embodiment, the selected ANN model is the ResNet network model. The average pooling and Relu activation methods adopted therein already meet the constraint conditions. Therefore, for this ANN model, this step can be omitted.

[0062] (3) Optimize and train the constrained ANN model obtained in step (2) to reach the required model accuracy, and extract the parameters of each layer for subsequent steps.

[0063] Since the construction method of the SNN based on the training and conversion of the ANN needs to first train the ANN and then apply the obtained weights to the SNN with the same structure to achieve the conversion. Therefore, it is necessary to optimize and train the constrained ANN model. Since the algorithms used here for optimizing and training the ANN model are all inherent methods of the ANN, they will not be separately described here. If the required accuracy cannot be achieved, it is necessary to further optimize the optimization and training algorithms of the ANN model.

[0064] Here, the constrained ResNet network model obtained in step (1) is trained. The dataset used for training is the HRSC dataset. After training, the accuracy of the constrained ResNet network is 92%.

[0065] (4) Establish the calculation mode of the SNN neurons corresponding to the ANN.

[0066] To implement the computational pattern of SNN neurons, various operators (Spike Layer) that conform to the SNN computational pattern are established at the network layer (Layer) level for the constrained ANN model. The specific corresponding relationships of common operators are shown in Table 3. For example, for the Conv2d operator in the ANN, the corresponding SNN operator, namely the SpikeConv2d operator, needs to be established.

[0067] Table 3 Various Operators of the SNN Computational Pattern

[0068]

[0069] Since for average pooling AvgPool, its output is a floating-point number. If its input is a 0, 1 pulse sequence, then the output of a 2*2 AvgPool will be 0, 0.25, 0.5, 0.75, 1, which destroys the 0, 1 values of the pulses. Therefore, Conv2d is used to replace AvgPool. By setting appropriate parameters such as weights and groups, Conv2d can implement the logic of AvgPool, and then it can be converted into SpikeConv2d. This is a general method and will not be elaborated here.

[0070] (5) For the network layer, parse the constrained ANN network structure and create the SNN network structure accordingly.

[0071] To parse the constrained ANN network structure, first, the description method of the network structure needs to be standardized. The network structure description method adopted here is to split the network into data at different stages and different network layers, and regard the entire network as a directed acyclic graph (DAG) composed of data and network layers. Treat the data as nodes and the network layers as edges to complete the description of the network structure. For example, as shown below Figure 3 The network structure on the left in the figure can be described as the figure on the right.

[0072] According to the given constrained ANN network, after building a DAG through operations such as adding nodes and adding edges, the SNN network can be built.

[0073] Then, based on the training results in step (4), when parsing the constrained ANN network structure, the parameters of the constrained ANN can be extracted simultaneously and written into the corresponding network layers of the SNN. Thus, the construction of the SNN network structure corresponding to the constrained ANN is completed.

[0074] This step parses the constrained ResNet network model obtained in step (2) and creates the corresponding SNN model.

[0075] (6) Perform data format conversion. Here, the input data of the ANN model needs to be converted into the spike train format required by the SNN through an encoding method.

[0076] Since both the input and output of the SNN are spike trains, a new data format also needs to be defined here to represent the spike train data. In the SNN network layer, the input and output are both specific data types, namely SpikeTensor. Compared with the commonly used Tensor in the ANN network layer, this data type has one more dimension to describe the time step timestep.

[0077] In the construction of an SNN based on ANN training and conversion, frequency encoding is generally used to convert pixel data into spike trains, and the implementation of frequency encoding has diversity, and its encoding effects are also different. Here, Poisson encoding is used for implementation. Poisson encoding is a specific implementation of frequency encoding. Since the spike distribution after its encoding is similar to the spike distribution of the biological brain, it has been widely used in spiking neural networks. Since its method is relatively general, it will not be described separately here.

[0078] In this step, Poisson encoding is used to encode the image data in the HRSC dataset used in the embodiment into the spike train format for subsequent processes.

[0079] (7) Extract the parameters of the ANN into the SNN and calibrate the SNN parameters to obtain a converted SNN model that meets the accuracy requirements.

[0080] Based on the SNN model constructed in step 5 and the input data in the spike train format converted in step 6, to improve the model accuracy, it is necessary to further calibrate the parameters of the SNN model. By calibrating the parameters of each layer of the SNN to match the converted activation values, so that each layer of the SNN and the ANN produce the same or approximate output for the same input, thereby transplanting the activation values of the ANN to the SNN.

[0081] Since the selection of the SNN threshold V_thr and the time step timestep greatly affects the conversion performance. The smaller the threshold, after all calculations are completed, the less information that can be "retained" by the membrane potential, and the smaller the error. However, the smaller the threshold, the more spikes are generated by the same input, and a larger time step must be used to release this information, which introduces more loop calculations. After obtaining the DAG describing the ANN network, the threshold of each layer can be determined by searching, and a suitable threshold is set for each SNN layer, and the time step can be specified artificially.

[0082] On the one hand, parameter optimization for adaptive threshold adjustment is carried out based on the network layer. First, a suitable threshold is set for each SNN layer, such as Figure 4As shown below. The specific method is to first estimate the maximum threshold of each layer of the network through a certain number of training samples to determine the adjustment interval of the dynamic threshold, and then scale this maximum threshold as the initial threshold of the SNN network layer. Then, after determining the initial threshold, the adaptive adjustment process is carried out during the SNN inference process, as Figure 5 shown. The steps are as follows: when the encoded test sample is input into the network for inference, calculate the maximum value ∑ω ij X ij (t) of the weighted pulse input of each layer of the network at the current moment. If it is greater than the current threshold V c , then update the current threshold V c = ∑ω ij X ij (t) according to the maximum value, and ensure that the threshold meets the previously determined interval after update. Through this threshold adjustment method, the optimal threshold of each layer of neurons under the current sample can be dynamically set.

[0083] On the other hand, set the time step. By setting different time steps T, simultaneously with the threshold optimization, the optimization and calibration of the SNN model are completed to meet the accuracy requirements. If the required accuracy cannot be achieved, the threshold and step size parameters need to be further adjusted. Thus, the conversion process from the ANN model to the SNN model is completed, and this SNN model is used as the final output.

[0084] (8) Test and verify the SNN model obtained in step (7) on the data set obtained in step (6). The accuracy of the converted ResNet network model is shown in Table 4. Comparing with the accuracy of the constrained ResNet network obtained in step (3) which is 92%, with a relatively short time step T, a small conversion accuracy loss is achieved. After the time step increases, the conversion accuracy loss further decreases. When the time step T is 128, the accuracy comparable to the original ANN model can be achieved.

[0085] Table 4 Comparison of the conversion accuracy of the Resnet model

[0086]

[0087] Further, conduct an adversarial FGSM attack experiment on the test set. By simulating the FGSM adversarial attack on the data set, the adversarial attack results of the ANN and SNN on the HRSC data set are shown in Table 5. With the addition of adversarial samples, the training accuracy of the ANN model drops sharply, while the recognition accuracy loss of the SNN model is relatively small, indicating that the SNN model has better anti-interference ability.

[0088] Table 5 Recognition accuracy of the adversarial attack of the model

[0089]

[0090] The above-described embodiments are merely relatively preferred specific embodiments of the present invention, and the ordinary variations and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included in the protection scope of the present invention.

Claims

1. A network layer-oriented ANN model to SNN model conversion method, characterized in that: include: S1: Establish a target image data set, where the image data in the data set contains multiple categories of target image data that need to be classified and identified; S2: establishing an ANN model for realizing target classification and recognition, and then constraining the network structure of the ANN model; S3: Based on the target image dataset, the constrained ANN model is trained and converted into an SNN model after reaching the set accuracy; S4: Encode the image data in the target image data set into a pulse sequence format to form a data set for the SNN model; based on the data set, calibrate the parameters of each layer of the SNN model to obtain an SNN model that meets the accuracy requirements; S5: Encode the image data to be tested into a pulse sequence format, input it into the SNN model, and obtain the result of target image classification and recognition.

2. The network layer-oriented ANN model to SNN model conversion method according to claim 1, characterized in that: In step S3, the method of converting the ANN model that meets the accuracy requirements into the SNN model is: S3.1: Extract the parameters of each layer of the ANN model; S3.2: Corresponding to the operators in the ANN model, establish operators that conform to the neuron computing mode of the SNN model; S3.3: Analyze the network structure of the ANN model and create the network structure of the SNN model.

3. The method for converting a network-layer-oriented ANN model to a SNN model according to claim 2, characterized in that: In step S3.3, the network structure of the ANN model is parsed. The parsing method is: split the network structure into data at different stages and different network layers, that is, convert it into a directed acyclic graph composed of data and network layers, in which data is used as nodes and network layers are used as edges to complete the description of the network structure.

4. The method for converting a network-layer-oriented ANN model to a SNN model according to claim 3, characterized in that: In step S3.3, the network structure of the ANN model is parsed to obtain the network structure of the SNN model, and then the ANN model parameters are extracted and written into the corresponding SNN model network layer to complete the SNN model construction work.

5. The method for converting a network-layer-oriented ANN model to a SNN model according to claim 2, characterized in that: In step S3.2, corresponding to the operators in the ANN model, operators that conform to the neuron computing mode of the SNN model are established, and the operators include SpikeConv2d, SpikeLinear, SpikeMaxPool2d, SpikeReLU and SpikeConvTranspose2d.

6. The method for converting a network-layer-oriented ANN model to a SNN model according to claim 1, characterized in that: In step S4, the parameters of each layer of the SNN model are calibrated so that the SNN and ANN of each layer produce the same output for the same input. The calibration method is: Based on the data set for the SNN model, the maximum threshold of each layer of the ANN model is estimated to determine the adjustment range of the dynamic threshold; Scaling the maximum threshold as the initial threshold of the SNN network layer; After determining the initial threshold, set the time step and make adaptive adjustments to obtain the optimal threshold for each layer: input the data in the dataset into the SNN model for inference, calculate the maximum value of the weighted pulse input of each layer of the network at the current moment, and if it is greater than the current threshold, update the current threshold based on the maximum value, and ensure that the threshold is updated to meet the previously determined adjustment range.

7. The method for converting a network-layer-oriented ANN model to a SNN model according to claim 6, characterized in that: In step S4, the time step is adjusted so that the calibrated SNN model meets the accuracy requirements.

8. The method for converting a network-layer-oriented ANN model to a SNN model according to claim 1, characterized in that: In step S4, the image data in the target image data set is converted into a pulse sequence format using a frequency encoding method to meet the input data type requirements of the SNN model.

9. The method for converting a network-layer-oriented ANN model to a SNN model according to claim 1, characterized in that: In step S2, the network structure of the ANN model is constrained to adjust the network structure of the ANN model. The constraints include: Constrained activation function, using ReLU activation function as the nonlinear activation function, and setting the bias term to zero to reduce the generation of negative values ​​in the network; Constrained pooling function, using average pooling operation.

10. The method for converting a network-layer-oriented ANN model to a SNN model according to claim 1, characterized in that: Add adversarial sample data to the training data set, and perform an anti-interference ability test on the SNN model that meets the accuracy requirements obtained in step S4 to verify the anti-interference ability of the SNN model.

Images