GCNN network optimization method based on Dropout and decorrelation strategy

Abstract

The invention discloses a GCNN network optimization method based on Dropout and a decorrelation strategy, and relates to the technical field of computer vision, and the method specifically comprises two steps: (1) comparing the influence of Dropout on the generalization ability of a network through using Dropout for different combinations of a convolution layer, a sampling layer and a full connection layer, and obtaining an optimal training strategy; and (2) the correlation between GCNN hidden layer activation units is reduced, the GCNN network is optimized by using covariance matrix norm penalty, overfitting is reduced, and the network generalization ability is improved. According to the method, by combining the Dropout technology and the decorrelation strategy, the generalization ability of the network is remarkably improved, the over-fitting possibility is reduced, the method has obvious advantages in the aspects of improving the generalization ability of the GCNN network and reducing the over-fitting, and particularly, when a large-scale and complex data set is processed, the method has the advantage that the method has a wide application prospect. The optimization strategies can greatly improve the network performance and the application value.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a GCNN network optimization method based on Dropout and decorrelation strategies. Background Art

[0002] In the field of computer vision, as the difficulty of tasks increases, building large-scale convolutional neural networks (CNNs) has become an important means to improve visual recognition results. Since the Large Scale Visual Recognition Challenge (ILSVRC), studies have shown that the improvement of network classification accuracy depends largely on the expansion of network size. This discovery has prompted in-depth research on deep learning models, especially innovations in network structure design and training strategies.

[0003] Early researchers, such as LeCun et al., introduced convolution operations with local receptive fields and downsampling to incorporate local spatial topological structure information of the image into the network, effectively alleviating the overfitting problem. However, this method did not solve the problem of too many network parameters. Hinton et al. mined useful information from training images through pre-learning and restricted use of Boltzmann machines, and obtained a network structure with optimized initial weights. Although this method improves network performance to a certain extent, the time required increases with the increase in the number of network layers.

[0004] In order to further improve the generalization ability of the network, Hinton et al. proposed the Dropout strategy, which suppresses the interaction between feature detectors by randomly ignoring the weights of certain hidden nodes in the network during training. This strategy significantly improves the stability of the network model. Goodfellow et al. further improved the network performance by defining a maxout layer between two hidden layers and combining it with the Dropout strategy. The dropconnect strategy proposed by Wan et al. improves the model average ability of the network by randomly setting the connection weights in the nodes to zero. Lian et al. proposed an improved strategy for dropconnect, which sparsely retains the most valuable connection weights based on the output of the previous layer, reducing the complexity of the network.

[0005] GCNN will have overfitting problems without sufficient training data, especially when the network size is too large. How to avoid overfitting in training still has the following problems:

[0006] (1) Limited training samples: When the number of training samples is limited, large-scale CNNs are prone to overfitting, that is, although the network model can fit the training data well, it cannot effectively generalize to unseen test samples.

[0007] (2) Too many network parameters: As the network size increases, the number of parameters also increases, which not only increases the computational burden but also aggravates the overfitting problem.

[0008] (3) Limitations of traditional regularization methods: Without using any regularization techniques, traditional overfitting solutions (such as increasing training samples, early termination, and weight norm penalty) can only achieve good test results when training large networks.

[0009] The present invention proposes a solution to the above problem. By using Dropout on different combinations of convolutional layers, sampling layers and fully connected layers, the effect on the generalization ability of the network is compared to obtain the best training strategy. The correlation between the activation units of the hidden layer of GCNN is reduced, and the covariance matrix norm penalty is used to optimize the GCNN network, reduce overfitting, and improve the generalization ability of the network. Summary of the invention

[0010] The present invention provides a GCNN network optimization method based on Dropout and decorrelation strategy. By applying Dropout in different network layers, its impact on generalization ability is evaluated, and the covariance matrix norm penalty is used to reduce the correlation between hidden layer activation units, thereby reducing overfitting and solving the problems raised in the above background technology.

[0011] The present invention provides the following technical solution: a GCNN network optimization method based on Dropout and decorrelation strategy, comprising the following steps:

[0012] Step 1: Use Dropout on different combinations of convolutional layers, sampling layers, and fully connected layers, compare their effects on the generalization ability of the network, and obtain the best training strategy;

[0013] The specific operation is as follows: design single-layer Dropout experiments and double-layer Dropout experiments. In each group of experiments, three GCNN models are trained. By comparing the application of Dropout at different levels, the impact on the generalization ability of the network is explored. Finally, the classification results are output through the fully connected layer. Based on the experimental results and comparative analysis, the best Dropout combination is obtained: Convolution layer Dropout: 0.3; Sampling layer Dropout: 0.2; Fully connected layer Dropout: 0.5;

[0014] Step 2: Reduce the correlation between the activation units of the hidden layer of GCNN, use the covariance matrix norm penalty to optimize the GCNN network, reduce overfitting, and improve the generalization ability of the network;

[0015] The specific operation is as follows: Step 1, assuming h n ∈R d, represents the activation value of a hidden layer of the nth sample, n∈{1,....,N}, where N represents the number of samples, and the covariance matrix between activations i and j is defined as follows:

[0016]

[0017] in: represents the mean of all sample activation i;

[0018] The second step is to penalize the norm of the covariance matrix and obtain the final loss function:

[0019]

[0020] Where: ||.|| F represents the Frobenius norm;

[0021] The third step is to apply the loss function to the convolutional layer, sampling layer and fully connected layer of GCNN to improve the generalization ability of the network.

[0022] Preferably, the single-layer Dropout experiment uses Dropout at different levels of each network, including GCNN-1: using Dropout at the convolutional layer, GCNN-2: using Dropout at the sampling layer, and GCNN-3: using Dropout at the fully connected layer.

[0023] Preferably, the double-layer Dropout experiment uses Dropout in two layers of each network; including GCNN-4: using Dropout in both the convolutional layer and the sampling layer, GCNN-5: using Dropout in both the sampling layer and the fully connected layer, and GCNN-6: using Dropout in both the convolutional layer and the fully connected layer.

[0024] Preferably, GCNN-1 to GCNN-6 models are trained respectively, using the same training parameters, and the results are obtained after analyzing the evaluation indicators as follows: GCNN-1: Dropout is used in the convolutional layer, which can effectively reduce feature overfitting, but too high Dropout will reduce the feature extraction ability; GCNN-2: Dropout is used in the sampling layer, and sample diversity is increased through max-pooling. Moderate Dropout helps to improve generalization ability; GCNN-3: Dropout is used in the fully connected layer to prevent overfitting, but too high Dropout will cause excessive information loss; GCNN-4: Dropout is used in both the convolutional layer and the sampling layer, and a moderate combination can enhance feature extraction and sample diversity; GCNN-5: Dropout is used in both the sampling layer and the fully connected layer, and a moderate combination can effectively increase sample diversity and prevent overfitting of the fully connected layer; GCNN-6: Dropout is used in both the convolutional layer and the fully connected layer, and a moderate combination can enhance feature extraction and prevent overfitting of the fully connected layer.

[0025] Preferably, the same training parameters are used including Dropout probability, learning rate, batch size, cross entropy loss function, and Adam optimizer.

[0026] Preferably, the loss function L is obtained by minimizing the covariance between activation units. Dcoυ , the loss function L Dcoυ The advantage of is that it is unsupervised and can be applied between any activations.

[0027] Preferably, the GCNN network structure includes an input layer, two convolutional layers, each of which is followed by a sampling layer and a Dropout layer.

[0028] Preferably, the CIFAR-100 dataset is used in both step 1 and step 2.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] 1. This GCNN network optimization method based on Dropout and decorrelation strategy significantly improves the generalization ability of the network and reduces the possibility of overfitting by combining Dropout technology and decorrelation strategy; by implementing Dropout in different network layers (convolutional layer, sampling layer and fully connected layer), it not only improves the generalization ability of the model, but also determines the optimal Dropout ratio through experiments, providing effective guidance for network training.

[0031] 2. The GCNN network optimization method based on Dropout and decorrelation strategy uses the covariance matrix norm penalty to reduce the correlation between hidden layer activation units, which is rarely involved in traditional methods. This strategy further alleviates the overfitting problem and enhances the model's adaptability to new data.

[0032] 3. This GCNN network optimization method based on Dropout and decorrelation strategy provides a comprehensive evaluation framework by comparing the effects of single-layer and double-layer Dropout, ensuring the scientificity and practicality of the optimization strategy.

[0033] 4. The GCNN network optimization method based on Dropout and decorrelation strategy has obvious advantages in improving the generalization ability of the GCNN network and reducing overfitting. Especially when dealing with large-scale and complex data sets, these optimization strategies will greatly enhance network performance and application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a GCNN network optimization method based on Dropout and decorrelation strategy proposed in the present invention. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] The present invention provides a GCNN network optimization method based on Dropout and decorrelation strategy, comprising the following steps:

[0037] Step 1: Use Dropout on different combinations of convolutional layers, sampling layers, and fully connected layers, compare their effects on the generalization ability of the network, and obtain the best training strategy;

[0038] The specific operation is as follows: Use the CIFAR-100 dataset, which contains multiple categories of images and is suitable for testing the generalization ability of image recognition models. Design two sets of experiments: single-layer Dropout experiment and double-layer Dropout experiment. In each set of experiments, three GCNNs are trained to compare the application of Dropout at different levels to explore its impact on the generalization ability of the network. The GCNN (Growing Convolution Neural Network with Progressive) network structure includes an input layer, two convolutional layers, each layer is followed by a sampling and Dropout layer, and finally the classification results are output through a fully connected layer;

[0039] Among them: 1) Single-layer Dropout experiment: Dropout is used at different layers of each network, including GCNN-1: Dropout is used in the convolutional layer, GCNN-2: Dropout is used in the sampling layer (using the max-pooling method), and GCNN-3: Dropout is used in the fully connected layer.

[0040] 2) Double-layer Dropout experiment: Dropout is used in two layers of each network, including GCNN-4: Dropout is used in both convolutional and sampling layers, GCNN-5: Dropout is used in both sampling and fully connected layers, and GCNN-6: Dropout is used in both convolutional and fully connected layers.

[0041] 3) Train GCNN-1 to GCNN-6 models respectively. Use the same training parameters (Dropout probability, learning rate, batch size; use cross entropy loss function; optimizer: Adam), analyze the evaluation indicators and get the following results: GCNN-1: Using Dropout in the convolutional layer can effectively reduce feature overfitting, but too high Dropout will reduce feature extraction ability. GCNN-2: Using Dropout in the sampling layer, increasing sample diversity through max-pooling, and moderate Dropout helps improve generalization ability. GCNN-3: Using Dropout in the fully connected layer, the effect of preventing overfitting is significant, but too high Dropout will cause too much information loss. GCNN-4: Using Dropout in both the convolutional layer and the sampling layer, a moderate combination can enhance feature extraction and sample diversity. GCNN-5: Using Dropout in both the sampling layer and the fully connected layer, a moderate combination can effectively increase sample diversity and prevent overfitting of the fully connected layer. GCNN-6: Using Dropout in both the convolutional layer and the fully connected layer, a moderate combination can enhance feature extraction and prevent overfitting of the fully connected layer. Based on experimental results and comparative analysis, the optimal Dropout combination is obtained: Convolutional layer Dropout: 0.3; Sampling layer Dropout: 0.2; Fully connected layer Dropout: 0.5.

[0042] Step 2: Further solve the overfitting problem of GCNN by reducing the correlation between hidden layer activation units, and use the covariance matrix norm penalty to optimize the GCNN network, reduce overfitting, and improve the generalization ability of the network;

[0043] The specific operation is as follows: Step 1: The correlation between the hidden layer activation units of the convolutional neural network is directly proportional to the overfitting. Therefore, the present invention further solves the overfitting problem of GCNN by reducing the correlation between the hidden layer activation units. Assuming h n ∈R d , represents the activation value of a hidden layer of the nth sample, n∈{1,....,N}, where N represents the number of samples, and the covariance matrix between activations i and j is defined as follows:

[0044]

[0045] in: represents the mean of all sample activation i;

[0046] The second step is to minimize the covariance between activation units, which is equivalent to penalizing the norm of the covariance matrix to obtain the final loss function L Dcoυ for:

[0047]

[0048] Where: ||.|| F represents the Frobenius norm; the advantage of the loss function is that it is unsupervised and can be applied between any activations;

[0049] The third step is to transform the loss function L Dcoυ It acts on the convolutional layer, sampling layer, and fully connected layer of GCNN to improve the generalization ability of the network. In this process, the CIFAR-100 dataset is also used.

[0050] The diagonal elements of the covariance matrix contain the variance of each hidden layer activation. It is necessary to minimize the covariance between different activations, but it is not necessary to require the dynamic range of activation (i.e., the variation of activation output for different samples) to be as small as possible.

[0051] In summary: This GCNN network optimization method based on Dropout and decorrelation strategy evaluates its impact on generalization ability by applying Dropout in different network layers, and uses the covariance matrix norm penalty to reduce the correlation between hidden layer activation units, thereby reducing overfitting. Experiments show that using different combinations of Dropout in convolutional layers, sampling layers, and fully connected layers can effectively improve network performance. In addition, by minimizing the covariance between activation units, the generalization ability of GCNN is further improved, and the adaptability of convolutional neural networks to new data is enhanced.

[0052] The contents not described in detail in this specification belong to the prior art known to professional and technical personnel in the field. Although the embodiments of the present invention have been shown and described, it is understood by ordinary technicians in the field that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principle and spirit of the present invention. The scope of the present invention is defined by the attached claims and their equivalents.

Claims

1. A GCNN network optimization method based on Dropout and decorrelation strategy, characterized in that: The following steps are involved: Step 1: Use Dropout on different combinations of convolutional layers, sampling layers, and fully connected layers, compare their effects on the generalization ability of the network, and obtain the best training strategy; The specific operation is as follows: design single-layer Dropout experiments and double-layer Dropout experiments. In each group of experiments, three GCNN models are trained. By comparing the application of Dropout at different levels, the impact on the generalization ability of the network is explored. Finally, the classification results are output through the fully connected layer. Based on the experimental results and comparative analysis, the best Dropout combination is obtained: Convolution layer Dropout: 0.3; Sampling layer Dropout: 0.2; Fully connected layer Dropout: 0.5; Step 2: Reduce the correlation between the activation units of the hidden layer of GCNN, use the covariance matrix norm penalty to optimize the GCNN network, reduce overfitting, and improve the generalization ability of the network; The specific operation is as follows: Step 1, assuming h n ∈R d , represents the activation value of a hidden layer of the nth sample, n∈{1,....,N}, where N represents the number of samples, and the covariance matrix between activations i and j is defined as follows: in: represents the mean of all sample activation i; The second step is to penalize the norm of the covariance matrix and obtain the final loss function: Where: ||.|| F represents the Frobenius norm; The third step is to apply the loss function to the convolutional layer, sampling layer and fully connected layer of GCNN to improve the generalization ability of the network.

2. The GCNN network optimization method based on Dropout and decorrelation strategy according to claim 1, characterized in that: The single-layer Dropout experiment uses Dropout at different levels of each network, including GCNN-1: using Dropout at the convolutional layer, GCNN-2: using Dropout at the sampling layer, and GCNN-3: using Dropout at the fully connected layer.

3. The GCNN network optimization method based on Dropout and decorrelation strategy according to claim 2, characterized in that: The double-layer Dropout experiment uses Dropout in two layers of each network, including GCNN-4: using Dropout in both the convolutional layer and the sampling layer, GCNN-5: using Dropout in both the sampling layer and the fully connected layer, and GCNN-6: using Dropout in both the convolutional layer and the fully connected layer.

4. The GCNN network optimization method based on Dropout and decorrelation strategy according to claim 3, characterized in that: GCNN-1 to GCNN-6 models were trained separately with the same training parameters. After analyzing the evaluation indicators, the results are as follows: GCNN-1: Dropout is used in the convolutional layer, which can effectively reduce feature overfitting, but too high Dropout will reduce the feature extraction ability; GCNN-2: Dropout is used in the sampling layer, and max-pooling is used to increase sample diversity. Moderate Dropout helps to improve generalization ability; GCNN-3: Dropout is used in the fully connected layer to prevent overfitting, but too high Dropout will cause too much information loss; GCNN-4: Dropout is used in both the convolutional layer and the sampling layer. A moderate combination can enhance feature extraction and sample diversity; GCNN-5: Dropout is used in both the sampling layer and the fully connected layer. A moderate combination can effectively increase sample diversity and prevent overfitting of the fully connected layer; GCNN-6: Dropout is used in both the convolutional layer and the fully connected layer. A moderate combination can enhance feature extraction and prevent overfitting of the fully connected layer.

5. The GCNN network optimization method based on Dropout and decorrelation strategy according to claim 4, characterized in that: The same training parameters are used including Dropout probability, learning rate, batch size, cross entropy loss function, and Adam optimizer.

6. The GCNN network optimization method based on Dropout and decorrelation strategy according to claim 1, characterized in that: The loss function L is obtained by minimizing the covariance between activation units. Dcoυ , the loss function L Dcoυ The advantage of is that it is unsupervised and can be applied between any activations.

7. The GCNN network optimization method based on Dropout and decorrelation strategy according to claim 1, characterized in that: The GCNN network structure includes an input layer, two convolutional layers, and each layer is followed by a sampling layer and a Dropout layer.

8. The GCNN network optimization method based on Dropout and decorrelation strategy according to claim 1, characterized in that: The CIFAR-100 dataset is used in both steps 1 and 2.

Images