A Handwritten Digit Recognition Method Based on Dynamic Feedforward Neural Network Structure and Growth Rate Function

Abstract

The present invention relates to a handwritten digit recognition method based on a dynamic feedforward neural network structure and a growth rate function, including: initializing and training a small-scale neural network, pausing the training to conduct periodic tests on the performance of the neural network; calculating the growth rate based on performance metrics, and calculating the network scale that needs to be increased when resuming training through the growth rate; screening the neurons that need to be split and deleted based on the network performance test results; for the split neurons, keeping the network output unchanged by adjusting the weight values; for the redundant neurons to be deleted, compensating the outputs of adjacent neurons; determining whether the network growth is mature, stopping the growth when the network grows to an appropriate scale, and outputting the current network, so as to obtain a more appropriate network structure and parameters when using a feedforward neural network to classify handwritten digits: avoiding underfitting caused by too small a network scale and overfitting caused by too large a network scale, and reducing the operation time and calculation cost.

Description

Technical Field

[0001] The present invention relates to the application field of handwritten digit recognition, and proposes a handwritten digit recognition method based on a dynamic feedforward neural network structure and a growth rate function. Background Art

[0002] The problem of handwritten digit recognition lies in how to enable a computer to automatically identify handwritten Arabic digits. However, when using a traditional feedforward artificial neural network for classification, due to its fixed network structure, the performance of the network is largely restricted. Artificial neural networks aim to simulate the organizational structure and operating mechanism of the human brain and have become a hot topic for scholars since their inception. With the development of deep learning technology, neural networks have been applied to solve various problems. However, since the performance of neural networks is greatly affected by their network structure and hyperparameters, finding an optimal network structure has also become a key research point for scholars.

[0003] Feedforward neural networks are widely used in fields such as control and identification due to their good learning ability. Many classic neural network computing models, such as radial basis neural networks and Hopfield neural networks, have a common characteristic of a fixed structure, and the structure does not change during training. Users need to have rich design experience to find a suitable network structure. When the scale of the neural network is too large, the expressive ability is too strong, which easily leads to overfitting; when the scale is too small, it easily leads to underfitting, both of which will significantly reduce the generalization ability of the network. Therefore, it is very necessary to select a suitable network scale in applications. When solving the problem of handwritten digit recognition, through a dynamic network structure, continuously find a better structure, train the most effective parameters, and improve the recognition rate of the neural network for handwritten digits. Summary of the Invention

[0004] The purpose of the present invention is to address the above problems and provide a handwritten digit recognition method based on a dynamic feedforward neural network structure and a growth rate function, which optimizes the feedforward neural network to find a suitable network structure for solving the problem of handwritten digit recognition.

[0005] A handwritten digit recognition method based on a dynamic feedforward neural network structure and a growth rate function includes the following steps:

[0006] Train and initialize a small-scale neural network, and pause training to conduct a phased test on the performance of the neural network;

[0007] Calculate the growth rate based on the performance index, and calculate the network scale that needs to be increased when resuming training through the growth rate;

[0008] Based on the network performance test results, screen the neurons that need to be split and deleted;

[0009] For the split neurons, the network output is kept unchanged by adjusting the weight values;

[0010] For the pruned redundant neurons, the outputs of adjacent neurons are compensated;

[0011] Neural network stopping growth condition: Determine whether the network growth is mature. When the network grows to an appropriate scale, stop the growth and output the current network.

[0012] The neural network specifically includes a neural network with RReLU as the activation function and a fully connected feedforward neural network structure with a single hidden layer.

[0013] The method includes: By testing the performance of the current neural network, the neurons in the hidden layer of the neural network with RReLU as the activation function are sorted according to their activity.

[0014] The sorting of neurons includes sorting in the training set and sorting in the test set after pausing training, and screening the neurons to be deleted.

[0015] The method includes: According to the performance of the neural network of the current scale, the number of neurons that the neural network needs to split during continued training is obtained based on the growth rate function.

[0016] The working process of the entire algorithm includes:

[0017] Periodically test various performance indicators of the current neural network during training;

[0018] Calculate the growth rate of the neural network;

[0019] Screen the neurons to be deleted and split;

[0020] Judge whether the growth of the neural network structure is mature.

[0021] The growth of the neural network structure includes four parts:

[0022] I. Growth rate function During the network training process, periodically test the performance of the network and use it as the growth factor of the neural network. Through this growth factor, determine the growth rate of the neural network, so as to obtain the structure that needs to be expanded during continued training. The growth rate function formula and the number of neurons in the hidden layer of the neural network during continued training are as follows:

[0023]

[0024] h r+1 =h r +<gr·h r >

[0025] where gr is the growth rate, a and b are constants, and λ l θl is the growth factor of the neural network, θ l is the calculation formula of the l-th index that can reflect the performance of the neural network, λ l is the weight of the l-th calculation formula containing the performance index. h r is the remaining number after deleting neurons with low hidden layer activation value p before the (r + 1)-th neural network growth, h r+1 is the number of neurons in the hidden layer when entering the next stage of training after growth, <x>Represents rounding, r = 1, 2, ..., l = 1, 2.

[0026]

[0027] θ1 = 1 - Acc r ,

[0028] where a = 12, b = 0.7, λ1 = 1, λ2 = 10, Acc r is the recognition rate of the network after completing the stage training before the (r + 1)-th neural network growth, and loss r is the loss error of the network after completing the stage training before the (r + 1)-th neural network growth. In the method of the present invention, the calculation formulas containing the above two items are selected as the two growth factors for network growth. The former recognition rate item represents the distance between the current network recognition rate and 1, and the latter loss error change rate item represents the degree of severity of the change before and after the previous growth.

[0029] II. Splitting and Deleting Neurons

[0030] In the neural network structure growth stage, first sort the hidden layer neurons according to the activity p, delete the neurons that are ranked behind in both the training set and the test set, and split the neurons with the top-ranked activity p in the training set. The activity of neurons during stage training and testing can be compared by comparing the cumulative output change rate of hidden layer neurons at each moment during stage training and the cumulative output change rate of hidden layer neurons at each moment when the test set is input after completing the stage training, starting from the last growth of the neural network:

[0031]

[0032] where p represents the neuron activity, and u j (t) represents the output of the j-th hidden layer neuron at the t-th moment, n = 1, 2,....

[0033] When splitting and deleting neurons, the convergence of the neural network is not damaged by keeping the sum of the outputs of the split neurons unchanged and compensating the outputs of the neighboring neurons for the deleted neurons:

[0034] (1) When neurons are split:

[0035]

[0036]

[0037] where v ij (t) is the connection weight between the j-th neuron in the hidden layer and the i-th neuron in the input layer at the t-th moment, is the connection weight between the neuron split from the j-th neuron in the hidden layer and the i-th neuron in the input layer; w jk (t) is the connection weight between the j-th neuron in the hidden layer and the k-th neuron in the output layer at time t, is the connection weight between the neuron split from the j-th neuron in the hidden layer and the k-th neuron in the output layer; μ is the mutation factor, which is a random value, 0 < μ < 1; f(x) is the RReLU activation function.

[0038] The output during neuron splitting is:

[0039]

[0040] Among them, y i (t), y′ i (t) are the outputs of the i-th neuron in the output layer at time t before and after the neuron splitting in the hidden layer respectively, w q (t), v iq are the weights at time t between the q-th neuron in the hidden layer and the i-th output neuron and the i-th input neuron, x i (t) is the output of the i-th input neuron at time t. From the above analysis, it can be seen that if the neural network grows at time t, the output of the split neuron remains unchanged and the convergence of the network is not changed.

[0041] (2) When neurons are deleted:

[0042]

[0043] If the neural network grows at time t and there are neurons to be deleted, these neurons are removed and sorted at the end, then the above formula is used to compensate the output of the last neuron among the remaining neurons.

[0044] The output when neurons are deleted is:

[0045]

[0046] Among them, y i (t), y′ i (t) are the outputs of the i-th neuron in the output layer at time t before and after the neuron deletion in the hidden layer respectively, w q (t), v iq are the weights at time t between the q-th neuron in the hidden layer and the i-th output neuron and the i-th input neuron, x i (t) is the output of the i-th input neuron at time t. From the above analysis, it can be seen that when the neural network grows at time t, deleting neurons does not change the network output and does not change the convergence of the network.

[0047] III. Stage Training

[0048] The maximum number of epochs for training before each growth of the neural network is n. After that, the training is paused, and the performance is tested on the test set. After determining the new network structure, the training continues and enters the next stage of training. In addition, to save training time, an early stopping condition for stage training is set:

[0049] |A k+2 -A k |<θ

[0050] where A k is the recognition rate of the neural network in the k-th epoch of stage training, and k = 1, 2,..., n.

[0051] When the change in the neural network recognition rate is not obvious and less than the threshold θ (θ = 0.001), the current training is sufficient and the training is paused in advance.

[0052] Among them, when the neural network is initialized before the first training, the weights are randomly generated in [0, 1].

[0053] IV. Judging Whether the Network Growth is Mature

[0054] When the neural network gradually grows to a scale suitable for solving the current problem, at this time <gr·h r > = 0, and the performance of the network approaches the upper limit, and the growth rate function gradually decreases to the lower limit. At this time, if the neural network continues to grow, the performance of the network hardly changes. Then the condition for judging network growth is: when the number of neurons to be grown is zero <gr·h r > = 0, and there are no neurons to be deleted, stop growing.

[0055] Compared with the prior art, the advantages of the present invention are as follows: Aiming at solving the handwritten digit recognition problem with a feedforward neural network having a fixed structure in the past, a function for guiding the growth of the network structure is proposed in this method. This method has a simple logic and is easy to implement, and can effectively grow to a suitable network scale for solving the handwritten digit recognition problem; Secondly, in this algorithm, the neural network always takes the performance on the test set as one of the important influencing factors for structure growth during training, so that the final generalization ability is strong. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is the flow chart of the method of the present invention;

[0057] Figure 2 is the structure diagram of the feedforward neural network;

[0058] Figure 3 is the schematic diagram of the principle for screening neurons to be deleted;

[0059] Figure 4 It is a curve graph of the neural network training process;

[0060] Figure 5 It is a curve graph of the number of nodes in the hidden layer of the neural network. Specific implementation manners

[0061] Next, the technical solutions in the present invention will be clearly and completely described in conjunction with the accompanying drawings in the present invention. Based on the operation process and proposed methods in the present invention, all other tools obtained by those skilled in the art without creative efforts shall fall within the scope of protection of the present invention.

[0062] As Figure 1 shown, it is a flowchart of the method of the present invention. First, preprocess the data set. The feedforward neural network includes an input layer, a hidden layer, and an output layer. In this experiment, a single-hidden-layer feedforward neural network is used as the research object, and its structure diagram is as Figure 2 shown.

[0063] The number of neurons in the input layer is 784, the number of neurons in the hidden layer is initialized to 1, the number of neurons in the output layer is 10, and the growth rate function is The learning rate is set to lr = 0.02, the maximum number of training rounds before each increase in the number of neurons in the hidden layer is 80, and the growth rate function after the first increase is set to The loss function is set to the cross-entropy loss function based on the pytorch framework. Before each stage of training of the neural network, the neurons are numbered first. When the neural network meets the conditions for early suspension of training or reaches the maximum number of training rounds, the training is suspended, the test set is input, and according to the neuron activity The neurons are sorted from large to small according to their activities in the test set during the stage training and when the training is suspended. Neurons that need to be deleted are screened out and sorted from large to small according to the activity p in both the test set and the training set: as Figure 3 shown, the position of the neuron with the smallest activity p in the index during the stage training (test) in the test set (training set) is sorted, and two neurons and the neurons that are jointly ranked after it are deleted. If the neuron with the smallest activity in the training set and the test set is the same neuron (m = n) during the stage training or when the number of neurons m in the hidden layer ≤ 2, then no neurons are deleted in this stage; the number of hidden layer neurons added each time is <gr·h r >, and the neurons with stronger activities during the stage training process are split in order according to the number of neurons that need to be added; the condition for the neural network to stop growing is that when the stage training ends, the number of neurons that need to grow is zero and there are no neurons that need to be deleted or trimmed, which means that the neural network has grown and matured and stops growing.

[0064] The algorithm steps are as follows:

[0065] S1: Generate an initial small-scale network and initialize the parameters;

[0066] S2: Train the neural network: When the early suspension condition of neural network training is reached or the maximum number of training times is reached, suspend learning, test the network performance, and go to step 3: Judge the stopping growth condition of the neural network; If the stopping condition is not met, the network continues to grow: update the growth rate function, screen the neurons that need to be split and deleted, regenerate the new network structure, and loop step 2;

[0067] S3: Judge the stopping growth condition of the neural network;

[0068] S4: The training ends and the network is output.

[0069] During the neural network training process, the recognition rate and loss error curves are as Figure 4 shown. During the training process, the change in the number of neurons in the hidden layer is as Figure 5 shown. During the neural network training process, whenever the stage training ends and the performance is tested, if there are neurons that need to be pruned, the curve may have a slight jitter when the training is resumed again, which does not affect the convergence of the network.< / x>

Claims

1. A handwritten digit recognition method based on a dynamic feedforward neural network structure and a growth rate function, characterized in that, Including: (1) Obtain a dataset of handwritten digit images and perform data preprocessing on the images: grayscale processing, conversion to tensors, and normalization. (2) Use the dataset to train and initialize a small-scale neural network, and pause the training to conduct periodic tests on the performance of the neural network. (3) Calculate the growth rate based on the performance metrics, and calculate the network scale that needs to be increased when resuming training based on the growth rate. (4) Based on the network performance test results, screen the neurons that need to be deleted and split. (5) For the split neurons, keep the network output unchanged by adjusting the weight values. (6) For the redundant neurons to be deleted, compensate the outputs of adjacent neurons. (7) Neural network stopping growth condition: Determine whether the network growth is mature, and stop growing when the network grows to an appropriate scale, and output the current network. Growth rate function design in step (3): a and b are constants, λ l θ l is the growth factor of the neural network, θ l is the calculation formula of the l-th index that can reflect the performance of the neural network, λ l is the weight of the calculation formula of the l-th performance index. Among them, a = 12, b = 0.7, θ1 = 1 - Acc r , λ1 = 1, λ2 = 10, r = 1, 2,..., l = 1, 2; Acc r is the recognition rate of the network after completing the stage training before the (r + 1)-th neural network growth, loss r is the loss error of the network after completing the stage training before the (r + 1)-th neural network growth.

2. The handwritten digit recognition method based on the dynamic feedforward neural network structure and the growth rate function according to claim 1, characterized in that: The network structure used is a three-layer feedforward neural network, including: an input layer, a hidden layer, and an output layer; at the beginning of training, the number of nodes in each layer of the initialized small-scale neural network is respectively: 784, 1, and 10; the activation function used for the hidden layer is RReLU.

3. The handwritten digit recognition method based on the dynamic feedforward neural network structure and the growth rate function according to claim 1, characterized in that, The method for screening the neurons that need to be deleted and added in step (4) includes: Sort the neurons according to their activities p in the test set during phased training and during suspended training, respectively, from largest to smallest, and select the neurons that need to be deleted with the lowest activity rankings in the test set and the training set. Among them u j (t) represents the output of the j-th hidden layer neuron at the t-th moment. n＝1,2,...； The number of hidden layer neurons increased each time is: <gr·h r >, and the neurons to be added are split in order among the neurons with stronger activity during the stage training; the number of hidden layer neurons after the increase is: h r+1 = h r + <gr·h r > Among them, h r is the remaining number after deleting neurons with poor hidden layer activity before the (r + 1)-th neural network growth, and h r+1 is the number of neurons in the hidden layer when entering the next stage of training after growth, <x>(where) represents rounding to the nearest integer. < / x> 4. The handwritten digit recognition method based on the dynamic feedforward neural network structure and the growth rate function according to claim 1, wherein The neural network stopping growth condition in step (7): Determine whether the network growth is mature. After the stage training is completed, the number of neurons that need to grow <gr·h r >= 0, and when there are no neurons that need to be deleted, it means that the growth of the neural network is mature and the growth stops.

Images