A digital recognition method based on adaptive activation neuron LSTM network

Abstract

The application discloses a digital recognition method based on an adaptive activation neuron LSTM network, constructs the adaptive activation neuron LSTM network, selects a cross-entropy loss function and an Adam optimization algorithm, divides a preprocessed data set into a training set and a test set, trains the adaptive activation neuron LSTM network built by using the training set, verifies the adaptive activation neuron LSTM network trained by using the test set, and realizes digital recognition. The adaptive activation neuron LSTM network is provided with two activation points on an LSTM network structure, one of the activation points is arranged on a long-term memory gate, and the other activation point is arranged on a short-term memory gate; and the adaptive activation neuron LSTM network determines whether to activate the long-term memory gate and the short-term memory gate according to the size of variance. The adaptive activation neuron LSTM network realizes twice discarding operation, improves operation efficiency, and improves the prediction accuracy.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of image recognition and artificial intelligence, and particularly relates to a digital recognition method based on an adaptive activation neuron LSTM network. BACKGROUND

[0002] With the rapid development of deep learning, more and more neural network technologies are applied to the field of image recognition, and the common one is a recurrent neural network (RNN). The RNN has a self-feedback connection unit and theoretically has the ability to process time sequences of any length. However, in actual operation, it is difficult to establish the dependence relationship between states with a long time interval. Due to the large gradient caused by the large interval, the system will eventually become unstable, and the problem of gradient explosion will be formed. Similarly, the gradient becomes small due to the large interval in the opposite direction, which also causes the problem of gradient disappearance. In view of the long-range dependence problem of the original recurrent neural network, subsequent researchers proposed a gating mechanism. This mechanism can forget and update the continuously accumulated information to ensure that the network gradient is dynamically balanced in size. This kind of network is called a gated recurrent neural network, and the most representative network is called an LSTM network.

[0003] However, the LSTM is still limited to the short-term storage capacity of the original RNN for the historical records of the current sequence. So far, some scholars have increased the network capacity by increasing external memory units, but the prevalence of the Transformer structure makes most of the research based on the recurrent sequence network become powerless. Some research directly replaces the original recurrent network structure with a fast weight network, but completely discards the original sophistication of the LSTM. SUMMARY

[0004] In view of the deficiencies in the prior art, the application provides a digital recognition method based on an adaptive activation neuron LSTM network, which solves the problem of low running efficiency and accuracy of the standard LSTM due to the fully connected recurrent neural network structure of the internal unit.

[0005] The application achieves the above technical purpose through the following technical means.

[0006] A digital recognition method based on an adaptive activation neuron LSTM network comprises the following steps:

[0007] Download a data set and perform preprocessing;

[0008] The adaptive activation neuron LSTM network is constructed by setting two activation points on the LSTM network structure, one of which is set on the long-term memory gate, and the other is set on the short-term memory gate; whether the long-term memory gate and the short-term memory gate are activated is determined according to the size of the variance of the adaptive activation neuron LSTM network;

[0009] The adaptive activation formula of the long-term memory gate is:

[0010] Cell state-long-term memory:

[0011] The adaptive activation formula of the short-term memory gate is:

[0012] Hidden state-short-term memory:

[0013] Wherein: C t is the internal state of the memory unit at t, f t is the forgetting gate, C t-1 is the internal state of the memory unit at t-1, is the input gate, is the candidate state, s() is the variance function, a and b are variance determination parameters, o t is the output gate, sigma() is the Sigmoid function, and tanh() is the hyperbolic tangent function.

[0014] The cross-entropy loss function and the Adam optimization algorithm are selected, the preprocessed data set is divided into a training set and a test set, the adaptive activation neuron LSTM network is trained by using the training set, and the adaptive activation neuron LSTM network after training is verified by using the test set, so as to realize digital recognition.

[0015] Further technical solutions, the adaptive activation neuron LSTM network further comprises:

[0016] Input gate: i t = sigma(W i · [h t-1 , x t ] + b i )

[0017] Forgetting gate: f t = sigma(W f · [h t-1 , x t ] + b f )

[0018] Output gate: o t = sigma(W o · [h t-1 , x t ] + b o )

[0019] candidate state:

[0020] wherein: W * is the weight of each term, b * is the error of each term, h t-1 is the hidden layer state at time t-1, x t is the input vector at time t.

[0021] Further technical solutions, the expression of the cross-entropy loss function in the case of binary classification is:

[0022]

[0023] wherein, y i represents the label of the sample, p i represents the probability of the corresponding sample being predicted as positive, and N is the total number of samples;

[0024] The expression in the case of multi-classification is:

[0025]

[0026] wherein, M represents the number of categories, y ic is an indicator variable, p ic represents the predicted probability that the observed sample belongs to category c.

[0027] Further technical solutions, the Adam optimization algorithm uses the following formula to estimate the first moment of the gradient and the second moment of the gradient:

[0028] M t = beta1M t-1 +(1-beta1)g t

[0029] G t = beta2G t-1 +(1-beta2)g t *g t

[0030] wherein, beta1 and beta2 are the decay rates of the two moving averages, g t is the gradient, M t is the first moment estimate of the gradient, and G t is the second moment estimate of the gradient.

[0031] The Adam optimization algorithm uses the following formula to correct the first moment of the gradient and the second moment of the gradient:

[0032]

[0033]

[0034] wherein, is the correction of the first moment estimation of the gradient, is the correction of the second moment estimation of the gradient, respectively, is the t-th power of

[0035] The parameter update difference of the Adam optimization algorithm is:

[0036]

[0037] wherein, alpha is an initial learning rate, epsilon is a very small parameter, and 10 -8 .

[0038] Further technical solutions, the data set adopts the MNIST data set.

[0039] Further technical solutions, the preprocessing is normalization processing and one-hot encoding representation.

[0040] Further technical solutions, the numerical values of the variance determination parameters a and b are determined by the data set, and s(i t ) and s(sigma(C t )) are replaced respectively.

[0041] Further technical solutions, the adaptive activation neuron LSTM network includes four network morphologies: using a long-term memory door, using a short-term memory door, simultaneously using a long-term memory door and a short-term memory door, and not using a long-term memory door and a short-term memory door.

[0042] The beneficial effects of the present application are:

[0043] (1) The present application sets two activation points on the benchmark LSTM network structure, one activation point is set on the long-term memory door, and the other activation point is set on the short-term memory door, the network structure inside the standard LSTM unit is optimized by changing the activation mode of the neural network neuron node, and the purpose of adaptive activation network neuron is achieved, so that the accuracy of the network is improved, and the gradient vanishing and gradient explosion problems are inhibited to a certain extent.

[0044] (2) The learning structure of the ANNLSTM network of the present application is not fixed, and has four network morphologies through dynamic activation, including using a long-term memory door, using a short-term memory door, simultaneously using a long-term memory door and a short-term memory door, and not using a long-term memory door and a short-term memory door, so that the memory inclusivity and repeated information sparsity of the whole network are high. At the same time, the data variance judgment is introduced for the first time in the learning process, which ensures the balance of network complexity and improves the efficiency of activation condition judgment.

[0045] (3) The ANNLSTM network of the present application, in the long-term memory gate self-adaptive activation formula, in the case of small variance, the input gate does not operate with the candidate state point by point multiplication, which is equivalent to discarding redundant learning values; and in the short-term memory gate self-adaptive activation formula, when the variance is small, the internal state of the memory unit is operated by the Logistic function, which is equivalent to a second discard operation to remove redundancy; thereby improving the running efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 The flow chart of the digital recognition method based on the adaptive activation neuron LSTM network according to the present application;

[0047] Figure 2 The schematic diagram of the reference LSTM network structure in the present application;

[0048] Figure 3 The schematic diagram of the ANNLSTM network structure constructed according to the present application;

[0049] Fig. 4(a) is a loss function convergence chart during training according to the present application;

[0050] Fig. 4(b) is a loss function convergence chart during testing according to the present application;

[0051] Fig. 4(c) is a training accuracy chart according to the present application;

[0052] Fig. 4(d) is a testing accuracy chart according to the present application. DETAILED DESCRIPTION

[0053] The present application will be further described below in conjunction with the drawings and specific embodiments, but the scope of protection of the present application is not limited thereto.

[0054] As shown in the figure, the present application provides an operation flow chart of a digital recognition method of an adaptive activation neuron LSTM (Adaptive activation neuron LSTM, ANNLSTM) network, and the steps specifically include: Figure 1 Step (1), download the data set and perform preprocessing

[0055] The MNIST data set is downloaded from the network, and each pixel point in the original data grayscale image is -256~256. The input data X is saved as a 28x28 matrix for normalization processing to speed up the calculation efficiency, and the label data Y is converted into one-hot encoding representation.

[0056] Step (2), construct the ANNLSTM network

[0057]

[0058] ​Step (2.1), establish the baseline LSTM network

[0059] The LSTM network structure and the operations of each part of LSTM are as follows: Figure 2 As shown, it specifically includes:

[0060] Input gate (threshold): i t =σ(W i ·[h t-1 ,x t ]+b i )

[0061] Forgetting gate (threshold): f t =σ(W f ·[h t-1 ,x t ]+b f )

[0062] Output gate (threshold): o t =σ(w o ·[h t-1 ,x t ]+b o )

[0063] Cellular state (long-term memory):

[0064] Hidden states (short-term memory): h t =o t *tanh(C t )

[0065] Candidate states (new knowledge derived from induction):

[0066] Among them, i t For the input gate, f t For the Gate of Oblivion, o t For output gate, C t For the internal state of the memory unit at time t, C t-1 The internal state of the memory unit at time t-1, As a candidate state, h t-1 Let h be the hidden state at time t-1. t Let x be the hidden state at time t. t W is the input vector at time t; * b represents the weight of each item. * σ represents the various errors; σ is the Sigmoid function, whose output value is "1" or "0", which determines which information can pass (0 means no information can pass, 1 means all information can pass); the tanh function normalizes the values ​​of the operation to [-1, 1].

[0067] Step (2.2) involves setting two activation points on the baseline LSTM network structure to construct an ANNLSTM network.

[0068] After setting two activation points on the baseline LSTM network structure, the ANNLSTM network is constructed as follows: Figure 3 As shown, one activation point is set on the long-term memory gate, and the other on the short-term memory gate. It retains all the gated layers of the original LSTM network, but when processing information, it uses only the appropriate network layer based on the specific situation, rather than the entire LSTM unit. Each layer of the ANNLSTM network has a constraint condition that allows it to activate only under specific conditions; this is called adaptive activation. Each LSTM cell unit contains many nodes, much like neurons in an artificial neural network; the nodes that activate adaptively are called adaptive activation neurons.

[0069] In the adaptive activation formula for long-term memory gates, i t It did not work when the variance was small. Performing pointwise multiplication is equivalent to discarding redundant learned values. In the short-term memory gate adaptive activation formula, C... t When the variance is small, the Sigmoid function is applied, which is equivalent to performing another discard operation to remove redundancy. The entire network not only avoids the problems inherent in the gradient itself, but also reduces the impact of redundant information on network accuracy.

[0070] LSTM networks perform gating operations to learn different features. The data in the initial vector are basically the same. If the variance data does not change much, it means that the samples have not been learned.

[0071] The specific variance formula is as follows:

[0072]

[0073] Where, x i Let i be the value of the i-th data in the vector. This is the average value of all data in the vector.

[0074] The activation of the corresponding network layer is determined based on the magnitude of the variance. The adaptive activation formula for the long-term memory gate is as follows:

[0075]

[0076] Similarly, the adaptive activation formula for short-term memory gates is as follows:

[0077]

[0078] The values ​​of the variance determination parameters a and b are determined by the dataset, and can generally be expressed as s(i t) and s (σ(C t )) are replaced respectively.

[0079] The ANN-LSTM network operation algorithm is as follows:

[0080]

[0081]

[0082] Step (3), selecting a model evaluation index

[0083] The evaluation index adopts cross entropy, and the cross entropy loss function mainly acts on the classification condition.

[0084] In the binary classification condition, the expression of the cross entropy loss function is as follows:

[0085]

[0086] Wherein, y i represents the label of the sample, one class is positive, and the other class is negative, and 1 and 0 are used to represent respectively; p i represents the probability that the corresponding sample is predicted to be positive, and N is the total number of samples.

[0087] In the multi-classification condition, the expression of the cross entropy loss function is as follows:

[0088]

[0089] Wherein, M represents the number of categories, y ic is an indication variable, if the category is the same as the category of the sample, it is 1, otherwise it is 0; p ic represents the prediction probability that the observed sample belongs to the category c.

[0090] Step (4), selecting Adam optimization algorithm

[0091] M t = β1M t-1 + (1-β1)g t (1)

[0092] G t = β2G t-1 + (1-β2)g t *g t (2)

[0093] Wherein, β1 and β2 are the attenuation rates of two moving averages, and the application takes β1 = 0.9 and β2 = 0.999; g t is a gradient, M t is a first moment estimate of the gradient, and G t is a second moment estimate of the gradient.

[0094] Equations (1), (2) are first moment estimation of gradient and second moment estimation of gradient respectively, which can be regarded as unbiased estimation of expectation E|g t |, E|g t *g t | of expectation.

[0095]

[0096]

[0097] wherein, is the correction of first moment estimation of gradient, is the correction of second moment estimation of gradient, is the t power of β1 and β2 respectively;

[0098] Equations (3), (4) are the correction of first moment estimation of gradient and second moment estimation of gradient respectively, which can be approximated to unbiased estimation of expectation.

[0099] The parameter update difference of Adam optimization algorithm is:

[0100]

[0101] wherein, α is initial learning rate, ∈ is very small parameter, and the present application is set to 10 -8 .

[0102] Step (5), after selecting cross-entropy loss function and Adam optimization algorithm, the data set preprocessed in step (1) is divided into training set and test set, and the built ANNLSTM network is trained by using the training set.

[0103] Step (6), experimental verification

[0104] The initial learning rate α of the experiment is set to 0.001, the epoch is set to 20, the batch size batch_size is set to 30, and the hidden layer neuron number hidden_size is set to 196.

[0105] The present application respectively adopts a convolutional neural network (CNN), a standard LSTM network, a JANET network and an AANLSTM network four models to do experiments, inputs a test set, and then compares the experimental results of these models with the experimental results of the AANLSTM. As shown in Table 1, the evaluation indexes of the AANLSTM model are all higher than the CNN, the LSTM and the JANET. It can be seen that the general network prediction accuracy is about 98.5%, which is a network accuracy bottleneck value, and has a strong distinguishing significance; the lowest accuracy of the JANET is below 99%, about 98.88%. But the lowest accuracy of the AANLSTM network of the present application is greater than or equal to 99%, which can be said to have been greatly improved.

[0106] Table 1 Classification test effect of AANLSTM, CNN, LSTM and JANET under Adam optimization algorithm

[0107]

[0108] The ANNLSTM+Adam training test result of the present application is shown in Figure 4, Figure 4(a) is the loss function convergence during training, Figure 4(b) is the loss function convergence during testing, Figure 4(c) is the training accuracy, and Figure 4(d) is the testing accuracy.

[0109] The embodiments are preferred embodiments of the present application, but the present application is not limited to the above embodiments, and any obvious improvements, replacements or modifications made by those skilled in the art without departing from the essential content of the present application shall fall within the protection scope of the present application.

Claims

1. A digital recognition method based on an adaptive activation neuron LSTM network, characterized in that: Download the dataset and preprocess it; Construct an adaptive activation neuron LSTM network: Set two activation points on the LSTM network structure, one activation point is set on the long-term memory gate, and the other activation point is set on the short-term memory gate; the adaptive activation neuron LSTM network determines whether to activate the long-term memory gate and the short-term memory gate based on the size of the variance; The adaptive activation formula for long-term memory gates is: Cellular state - long-term memory: The adaptive activation formula for short-term memory gates is: Hidden layers - short-term memory: Where: C t f is the internal state of the memory unit at time t. t For the Gate of Oblivion, C t-1 For the internal state of the memory unit at time t-1, i t For input gate, Let s() be the candidate state, s() be the variance function, and a and b be the variance decision parameters. t For the output gate, σ() is the Sigmoid function, and tanh() is the hyperbolic tangent function; By selecting the cross-entropy loss function and the Adam optimization algorithm, the preprocessed dataset is divided into a training set and a test set. The adaptive activation neuron LSTM network is trained using the training set, and the trained adaptive activation neuron LSTM network is verified using the test set to achieve digit recognition.

2. The digital recognition method based on an adaptive activation neuron LSTM network according to claim 1, characterized in that, The adaptive activation neuron LSTM network also includes: Input gate: i t =σ(W i ·[h t-1 ,x t ]+b i ) Forgotten Gate: f t =σ(W f ·[h t-1 ,x t ]+b f ) Output gate: o t =σ(W o ·[h t-1 ,x t ]+b o ) Candidate state: Among them: W * b represents the weight of each item. * For each error, h t-1 Let x be the hidden state at time t-1. t Let t be the input vector at time t.

3. The digital recognition method based on an adaptive activation neuron LSTM network according to claim 1, characterized in that, The expression for the cross-entropy loss function in the binary classification case is: Among them, y i The label p represents the sample. i This represents the probability that the corresponding sample is predicted to be positive, where N is the total number of samples; The expression for the multi-class classification case is: Where M represents the number of categories, y ic p is an indicator variable ic This represents the predicted probability that the observed sample belongs to category c.

4. The digital recognition method based on an adaptive activation neuron LSTM network according to claim 1, characterized in that, The Adam optimization algorithm uses the following formulas to estimate the first moment and the second moment of the gradient: M t =β1M t-1 +(1-β1)g t G t =β2G t-1 +(1-β2)g t *g t Where β1 and β2 are the decay rates of two moving averages, respectively, and g t M is the gradient. t For the first moment estimate of the gradient, G t This is the second moment estimate of the gradient; The Adam optimization algorithm uses the following formula to correct the first-moment estimate and the second-moment estimate of the gradient: in, Correction for the first moment estimate of the gradient. Correction for the second-order moment estimate of the gradient. β1 and β2 raised to the power of t, respectively; The parameter update difference of the Adam optimization algorithm is: Where α is the initial learning rate, and ∈ is a very small parameter, which is 10. -8 .

5. The digital recognition method based on an adaptive activation neuron LSTM network according to claim 1, characterized in that, The dataset used is the MNIST dataset.

6. The digital recognition method based on an adaptive activation neuron LSTM network according to claim 5, characterized in that, The preprocessing consists of normalization and one-hot encoding.

7. The digital recognition method based on an adaptive activation neuron LSTM network according to claim 1, characterized in that, The values ​​of the variance determination parameters a and b are determined by the dataset, denoted by s(i t ) and s(σ(C t )) respectively replace.

8. The digital recognition method based on an adaptive activation neuron LSTM network according to claim 1, characterized in that, The adaptive activation neuron LSTM network includes four network morphologies: using long-term memory gates, using short-term memory gates, using both long-term and short-term memory gates, and not using either long-term or short-term memory gates.

Images