Deep learning network generalization error control method based on game method

Abstract

The invention relates to a deep learning network generalization error control method based on a game method, and belongs to the field of deep learning. A general technical method is provided for an existing method for controlling generalization errors through a deep learning model, such as Dropout and DropConnect, a data augmentation technology and the like. However, the above method specifically and difficultly approximates maximum likelihood estimation for different tasks, and controls an error mean value of an actual data set. According to the deep learning network generalization error control method based on the game method, the generative adversarial network is used, the two networks generate some abnormal samples through the game method, the samples are returned to serve as training data, overfitting is avoided, and the generalization ability of the deep learning network is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of deep learning, and more specifically, an embodiment of the present invention relates to a method for controlling generalization error of a deep learning network based on game theory. Background Art

[0002] Deep neural networks have demonstrated impressive performance in computer vision problems such as image classification, object detection, semantic segmentation, and video analysis. However, controlling the generalization error of deep learning models remains a major challenge in machine learning. A model's generalization ability refers to whether its error on a training dataset is close to the mean error on all possible test data. Generalization error can be roughly understood as the difference between the error on the test dataset and the error on the training dataset. In practice, the previously-held concerns about the poor generalization ability of deep neural networks have not materialized significantly in real-world applications. This is partly due to the massive number of samples in the big data era, and partly due to the recent emergence of new, practically effective methods for controlling generalization error (regularization), such as Dropout and DropConnect, as well as highly effective data augmentation techniques. However, these techniques are relatively general and struggle to approximate maximum likelihood estimation or control the mean error on real-world datasets. Therefore, developing more effective methods for further controlling generalization error is crucial. Summary of the Invention

[0003] A brief overview of the present invention is provided below to provide a basic understanding of certain aspects of the present invention. It should be understood that this overview is not an exhaustive overview of the present invention. It is not intended to identify key or important aspects of the present invention, nor is it intended to limit the scope of the present invention. Its purpose is simply to present certain concepts in a simplified form as a prelude to the more detailed description discussed later.

[0004] In view of this, in order to overcome the above technical problems, the present invention provides a deep learning network generalization error control method based on game theory, which generates some abnormal samples through game theory. These samples are returned as training data to avoid overfitting and effectively improve the generalization ability of the deep learning network.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] Solution 1: The present invention provides a method for controlling generalization error of a deep learning network based on game theory. Specifically, it includes the following steps:

[0007] Step 1: Get the sample image information in the dataset. Let x∈Rw×H×C and y represent the training image and its label respectively.

[0008] Step 2: Construct a generative adversarial network, where both the generator G and the discriminator D are multi-layer perceptrons.

[0009] Step 3: Understand the distribution P of the generator over the data x g , defines the input noise variable P z (Z), and then the mapping to the data space is expressed as G(z; θ g ), where G is parameterized by θ g The second multilayer perceptron D(x;θ) outputs a single scalar d ). D(X) means x comes from data instead of P g probability.

[0010] Step 4: Simultaneously train two models: a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that a sample came from the training data rather than G. G is trained to maximize the probability that D makes an error, meaning both networks are "correct." This framework corresponds to a two-person minimax game.

[0011] More specifically: D is trained to maximize the probability of assigning the correct label to the training samples and samples from G. At the same time, G is trained to minimize the following formula:

[0012] log(1-D(G(z)))

[0013] D and G play a two-player minimax game using the function V(G, D):

[0014] min G max D V(D,G)=E x～pdata(x) [logD(x)]+E z～pz(z) [log(1-D(G(z)))]

[0015] Step 5: In the space of arbitrary functions G and D, there exists a unique solution where G recovers the distribution of the training data and D is equal to 0.5 everywhere. In the case where G and D are defined by a multilayer perceptron, the entire system can be trained using backpropagation.

[0016] Beneficial effects:

[0017] Existing methods for controlling generalization error in deep learning models, such as Dropout and DropConnect, as well as data augmentation techniques, provide relatively general technical methods. However, these methods are very difficult to approximate maximum likelihood estimation and control the mean error of actual data sets specifically for different tasks. The technology of the present invention provides a method for controlling generalization error in deep learning networks based on game theory. Using a generative adversarial network, two networks generate some abnormal samples through game theory. These samples are returned as training data to avoid overfitting and effectively improve the generalization ability of deep learning networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a workflow diagram of a deep learning network generalization error control method based on game theory in the present invention. DETAILED DESCRIPTION

[0019] Exemplary embodiments of the present invention are described below with reference to the accompanying drawings. For the sake of clarity and conciseness, not all features of an actual implementation are described in this specification. However, it should be understood that in the process of developing any such actual implementation, many implementation-specific decisions must be made in order to achieve the developer's specific goals, such as meeting those constraints related to the system and business, and these constraints may vary from implementation to implementation. In addition, it should be understood that although the development work may be very complex and time-consuming, it is a routine task for those skilled in the art who benefit from the disclosure of the present invention.

[0020] It is also necessary to explain here that, in order to avoid obscuring the present invention due to unnecessary details, the accompanying drawings only show the device structure and / or processing steps closely related to the solution according to the present invention, while other details that are not closely related to the present invention are omitted.

[0021] step:

[0022] Step 1: Get the sample image information in the dataset. Let x∈Rw×H×C and y represent the training image and its label respectively.

[0023] Step 2: Construct a generative adversarial network, where both the generator G and the discriminator D are multi-layer perceptrons.

[0024] Step 3: Understand the distribution P of the generator over the data x g , defines the input noise variable P z (Z), and then the mapping to the data space is expressed as G(z; θ g ), where G is parameterized by θ g The second multilayer perceptron D(x;θ) outputs a single scalard ). D(X) means x comes from data instead of P g probability.

[0025] Step 4: Simultaneously train two models: a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that a sample came from the training data rather than G. G is trained to maximize the probability that D makes an error, meaning both networks are "correct." This framework corresponds to a two-person minimax game.

[0026] More specifically: D is trained to maximize the probability of assigning the correct label to the training samples and samples from G. At the same time, G is trained to minimize the following formula:

[0027] log(1-D(G(z)))

[0028] D and G play a two-player minimax game using the function V(G, D):

[0029] min G max D V(D,G)=E x～pdata(x) [logD(x)]+E z～pz(z) [log(1-D(G(z)))]

[0030] Step 5: In the space of arbitrary functions G and D, there exists a unique solution where G recovers the distribution of the training data and D is equal to 0.5 everywhere. In the case where G and D are defined by a multilayer perceptron, the entire system can be trained using backpropagation.

Claims

1. A method for controlling generalization error of deep learning networks based on game theory, characterized in that Using a generative adversarial network, the two networks generate some abnormal samples through a game method. These samples are returned as training data to avoid overfitting and effectively improve the generalization ability of the deep learning network. The following steps are included: Step 1: Get the sample image information in the dataset. Let x∈Rw×H×C and y represent the training image and its label respectively; Step 2: Construct a generative adversarial network, where both the generator G and the discriminator D are multi-layer perceptrons; Step 3: Understand the distribution P of the generator over the data x g , defines the input noise variable P z (Z), and then the mapping to the data space is expressed as G(z; θ g ), where G is parameterized by θ g The multilayer perceptron represents a differentiable function, and the second multilayer perceptron outputs a single scalar D(x; θ d ), D(X) means x comes from the data instead of P g probability; Step 4: Simultaneously train two models: a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that a sample came from the training data rather than G. G is trained to maximize the probability that D is wrong, meaning that both networks are "correct." This framework corresponds to a two-person minimax game. More specifically: D is trained to maximize the probability of assigning the correct label to the training samples and samples from G. At the same time, G is trained to minimize the following formula: log(1-D(G(z))) D and G play a two-player minimax game using the function V(G, D): Step 5: In the space of arbitrary functions G and D, there exists a unique solution where G recovers the distribution of the training data and D is equal to 0.5 everywhere. In the case where G and D are defined by a multilayer perceptron, the entire system can be trained using backpropagation.

Images