Data Generation Method Based on Model Feedback for Deep Learning Image Classification

Abstract

The present invention provides a data generation method based on model feedback applicable to deep learning image classification, which relates to the technical field of image classification. The method includes the following steps: (1) acquisition of image data; (2) generation of new image data; (3) establishment of a model; (4) training of the model. The present invention introduces the idea of feedback and combines data synthesis technology to propose a brand-new data generation technology, which can generate new samples adapted to the current model parameters by using the feedback of the model. The present invention can increase the diversity of the generated new samples, enabling the deep learning model to obtain more data support during training and improving its comprehensive performance, and has a certain degree of universality and generality.

Description

Technical Field

[0001] The present invention relates to the technical field of image classification, and in particular, to a data generation method based on model feedback for deep learning image classification. Background Art

[0002] In recent years' deep learning research, in order to obtain an image classification model with good performance, a large amount of labeled data is often required to train the model end-to-end. Experiments have shown that the increase in the amount of data usually has a huge impact on the overall improvement of the model performance. However, the collection and annotation of high-quality data often require a large amount of manpower, material resources and financial resources. Therefore, in order to reduce this part of the expenditure, some image data augmentation algorithms have been proposed to generate virtual data, hoping to use these new image data to improve the model performance.

[0003] There are mainly two categories of data augmentation algorithms. One is to directly generate data using a network model, including using an adversarial generative network (GAN) and a variational auto-encoder (VAE), etc. The other is to generate new data using existing data, including some data augmentation algorithms: flipping, rotating, mirroring, color transformation, etc. of data. These data can be considered as new data within a certain class range. In addition, data augmentation can also use a weight parameter to combine two or more classes to obtain new data, such as the convex combination of data. The new data obtained by these combinations is considered to be between two or more classes.

[0004] Currently, although there are the above-mentioned methods to obtain new virtual data for training the model, there are still some problems that have not been solved in the process of generating new data:

[0005] Effectiveness problem: The GAN network is trained according to a human preset goal, and its result is that it can produce data within the scope of human understanding. VAE assumes that the original data follows a Gaussian distribution, so it can create new data using the distribution. Although these methods can generate a large amount of new data that humans can understand, training the model with such a huge amount of data often causes a huge problem of computing power consumption. The current generation technology can obtain new data by weighted combination of two or more data. However, since most of these new data are simple combinations based on the data itself and do not consider the influence of the model on the generation process in deep learning, these new data also have certain limitations in improving the model performance.

[0006] Diversity problem: The main parameters of the current generation technology, namely the weight parameters mentioned above, are generally randomly sampled from a certain distribution, thus restricting the diversity of new data. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide a model feedback-based data generation method applicable to deep learning image classification in view of the deficiencies of the above-mentioned prior art, which can increase the diversity of generated new data, enable the deep learning model to obtain more data support during training, improve its comprehensive performance, and has a certain degree of universality and generality.

[0008] To solve the above technical problems, the technical solution adopted by the present invention is:

[0009] A model feedback-based data generation method applicable to deep learning image classification, comprising the following steps:

[0010] Step 1: Acquisition and integration of data;

[0011] Obtain the already-made image dataset through the network, and divide it into a training dataset and a validation dataset according to a ratio of 8:2.

[0012] Step 2: Design of the weight generator model;

[0013] Design a weight generator model based on a convolutional neural network, whose input is the image data to be combined, and whose output is a weight matrix with the same length and width as the input data, and each value in the weight matrix is between 0 and 1.

[0014] Step 3: Generation of new image data;

[0015] Use the weight matrix in Step 2 to combine the original image data in the image dataset to obtain new image data; then, average the weight matrix to obtain a constant, and use this constant to combine the labels of the original image data to obtain the labels of the new image data.

[0016] Step 4: Optimization of model parameters;

[0017] Use the loss maximization algorithm to optimize the parameters of the weight generator model, and use the loss minimization algorithm to optimize the parameters of the deep learning classification model; the weight generator model continuously generates new weight matrices according to the optimization process, and uses the new image data obtained from the new weight matrix as training data to increase the diversity of the dataset data and help the deep learning classification model better identify image categories.

[0018] Further, the specific process of Step 1 is:

[0019] Step 1.1: Download the image dataset required for the classification network training from the Internet;

[0020] Step 1.2: Divide the image data in Step 1.1 into a training dataset and a validation dataset according to a ratio of 8:2;

[0021] Step 1.3: Perform data preprocessing on the training dataset, including padding, random cropping, horizontal flipping, and normalization.

[0022] Furthermore, the specific method of Step 2 is: Build a weight generator G based on a convolutional neural network, including three layers of convolutional neural networks. Its structure from top to bottom is: a convolutional layer with a convolution kernel of 3 and a stride of 1, a batch normalization layer, a GELU activation layer, a convolutional layer with a convolution kernel of 3 and a stride of 1, a batch normalization layer, a GELU activation layer, a feature fusion layer, a convolutional layer with a convolution kernel of 1 and a stride of 1, a normalization layer, and a sigmoid activation layer.

[0023] Furthermore, the specific method of Step 3 is:

[0024] Step 3.1: Input two image data x1 and x2 collected from the image dataset into the weight generator model G, and obtain a weight matrix m after the convolutional calculation inside the generator, that is: m = G(x1, x2);

[0025] Step 3.2: Use the generated weight matrix m to combine the original images to obtain new image data, and weight the average value of the weights with the labels y1 and y2 of the two image data to generate the label of the new image data:

[0026]

[0027] r = mean(m)

[0028]

[0029] where, are the new image data and its label respectively, and r is the average value of the weight m.

[0030] Furthermore, the parameter optimization step of the weight generator model is: Input the new image data into the deep learning classification model, calculate the loss function according to the output of the model, and obtain the gradient information of the weight generator model parameters by backpropagation of this loss. Then use the gradient information as feedback to optimize the parameters of the weight generator model.

[0031] Further, the parameter optimization steps of the deep learning classification model are as follows: input the new image data into the deep learning classification model, calculate the loss function based on the output of the model, and then backpropagate from this loss to obtain the gradient information of the classification model parameters. Next, use the gradient information as feedback to optimize the parameters of the classification model.

[0032] Further, the specific method for step 4 is as follows:

[0033] Step 4.1: Construct a deep learning classification network model F with convolution as the basic unit, and train the model using the generated new images and new labels;

[0034]

[0035] where o represents the output value of model F;

[0036] Step 4.2: Calculate the loss function using the predicted labels and the true labels of the output of the deep learning classification network model, as shown in the following formula:

[0037]

[0038] where the subscript i represents the index of label y, n represents the number of all image data in the image dataset, and l represents the calculated loss value;

[0039] Step 4.3: Update the parameters of the weight generator model through the loss maximization algorithm

[0040]

[0041] where TV(m i ) represents calculating the global variational loss for weight m i and using it as a regularization term to restrict the generated weights; Step 4.4: Update the parameters θ of the classification neural network model through the loss minimization algorithm:

[0042] min θ l

[0043] Obtain a deep learning model F that performs excellently in the image classification task.

[0044] The beneficial effects of adopting the above technical solution are as follows: The data generation method based on model feedback applicable to deep learning image classification provided by the present invention utilizes the idea of model feedback to generate new data targeted, and can achieve a greater improvement in model performance with less data. The present invention proposes to use the method of model feedback to generate weights, greatly enhancing the diversity of the weights and providing more help for model training. In addition, since the training process of the deep learning model is constantly changing, in order to ensure that the generated new data can always provide positive guidance for the model training process, the present invention uses the output of the deep learning model as feedback to optimize the parameters of the generator, improving the relevance between the generated new data and the training model. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a framework diagram of the data generation process provided by an embodiment of the present invention;

[0046] Figure 2 It is a complete gradient propagation diagram provided by an embodiment of the present invention;

[0047] Figure 3 It is a complete model framework diagram provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0048] The following combines the drawings and embodiments to further describe in detail the specific embodiments of the present invention. The following embodiments are used to illustrate the present invention, but are not used to limit the scope of the present invention.

[0049] This embodiment takes the image classification task of a certain publicly available dataset in deep learning as an example for detailed description.

[0050] Step 1: Acquisition and integration of data.

[0051] An image dataset that has been made is obtained through the network and divided into a training dataset and a validation dataset in a ratio of 8:2. The specific process is as follows:

[0052] Step 1.1: Download the image dataset required for training the classification network from the Internet, such as CIFAR, ImageNet, etc.;

[0053] Step 1.2: Divide the data in Step 1.1 into a training dataset and a validation dataset according to a ratio of 8:2;

[0054] Step 1.3: Perform data preprocessing on the training dataset, including padding, random cropping, horizontal flipping, and normalization processing.

[0055] Step 2: Design of the weight generator model.

[0056] Design a weight generator model based on a convolutional neural network. Its input is the image data to be combined, and its output is a weight matrix with the same length and width as the input data. Each value in the weight matrix is between 0 and 1. Based on the convolutional neural network as the basic unit, the weight generator model includes three layers of convolutional neural networks. Its structure from top to bottom is: a convolutional layer with a convolution kernel of 3 and a stride of 1, a batch normalization layer, a GELU activation layer, a convolutional layer with a convolution kernel of 3 and a stride of 1, a batch normalization layer, a GELU activation layer, a feature fusion layer, a convolutional layer with a convolution kernel of 1 and a stride of 1, a normalization layer, and a sigmoid activation layer.

[0057] Step 3: Generation of new image data.

[0058] Use the weight matrix in Step 2 to combine two image data randomly sampled from the image dataset to obtain new image data; then, average the weight matrix to obtain a constant, and use this constant to combine the labels of the image data to obtain the label of the new image data. The specific method is as follows:

[0059] Step 3.1: Input two image data x1 and x2 sampled from the image dataset into the generator G. After convolutional calculations inside the generator, obtain a set of weights m, that is: m = G(x1, x2). The specific process of this step is as Figure 1 shown.

[0060] Step 3.2: Use the generated weights m to combine the original images to obtain new images, and weight the average value of the weights with the labels y1 and y2 of the two images to generate the label of the new image:

[0061]

[0062] r = mean(m)

[0063]

[0064] where is the new image and its label, and r is the average value of the weights m.

[0065] Step 4: Optimization of model parameters.

[0066] The parameter optimization steps of the weight generator model: Input the new image data into the deep learning classification model, calculate the loss function based on the output of the model, and then backpropagate this loss to obtain the gradient information of the parameters of the weight generator model. Then, use the gradient information as feedback to optimize the parameters of the weight generator model; it should be noted that the loss maximization algorithm should be used in this process, and its purpose is to generate data that is difficult to easily analyze for the current model.

[0067] Parameter optimization steps for the deep learning classification model: Input the new image data into the deep learning classification model, calculate the loss function based on the output of the model, and then backpropagate from this loss to obtain the gradient information of the classification model parameters. Then, use the gradient information as feedback to optimize the parameters of the classification model. It should be noted that the loss minimization algorithm should be used in this process, and its purpose is to enable the classification model to successfully identify the categories of new data.

[0068] Thus, a closed-loop model feedback-based image data generation method is completed. Specifically, the weight generator model continuously generates new weight matrices according to the feedback information of the training model, and uses the new image data obtained from this matrix as training data to increase the diversity of the image dataset and help the deep learning classification model better identify image categories.

[0069] The process of parameter update is as Figure 2 shown, and the specific method is as follows:

[0070] Step 4.1: Construct a classification neural network model F with convolution as the basic unit, and use the generated new images and new labels to train this model;

[0071]

[0072] Among them, o represents the output value of model F;

[0073] Step 4.2: Calculate the loss function using the predicted label and the true label of the output of the classification neural network model, as shown in the following formula:

[0074]

[0075] Among them, the subscript i represents the index of label y, n represents the number of all data in the dataset, and l represents the calculated loss value.

[0076] Step 4.3: Update the parameters of the weight generator model through the loss maximization algorithm

[0077]

[0078] Among them, TV(m i ) represents calculating the global variational loss for weight m i as a regularization term to restrict the generated weights and ensure that they are smooth enough and do not produce huge mutation values.

[0079] Step 4.4: Update the parameters θ of the classification neural network model through the loss minimization algorithm:

[0080] min θ l

[0081] Thus, a closed-loop data generation method based on model feedback for deep learning image classification models is completed, and the complete process is as Figure 3 shown. It can be seen that the main contributions of this algorithm are as follows: the weight generator model continuously generates new weight matrices according to the feedback information of the training model, and combines this matrix with two randomly sampled data to obtain new image data. This new image data can be used as training data to increase the diversity of the image dataset and make great contributions to improving the performance of deep learning models.

[0082] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope defined by the claims of the present invention.

Claims

1. A data generation method based on model feedback applicable to deep learning image classification, characterized in that: The method includes the following steps: Step 1: Acquisition and integration of data; Acquire the already-made image dataset through the network, and divide it into a training dataset and a validation dataset according to a ratio of 8:2; Step 2: Design of the weight generator model; Design the weight generator model based on a convolutional neural network. Its input is the image data to be combined, and its output is a weight matrix with the same length and width as the input data. Each value in this weight matrix is between 0 and 1; Step 3: Generation of new image data; Use the weight matrix in Step 2 to combine the original image data in the image dataset to obtain new image data; then, average the weight matrix to obtain a constant, and use this constant to combine the labels of the original image data to obtain the labels of the new image data; Step 4: Optimization of model parameters; Use the loss maximization algorithm to optimize the parameters of the weight generator model, and use the loss minimization algorithm to optimize the parameters of the deep learning classification model; the weight generator model continuously generates new weight matrices according to the optimization process, and uses the new image data obtained from this new weight matrix as training data to increase the diversity of the dataset data and help the deep learning classification model better identify image categories.

2. The data generation method based on model feedback applicable to deep learning image classification according to claim 1, characterized in that: The specific process of Step 1 is as follows: Step 1.1: Download the image dataset required for training the classification network from the Internet; Step 1.2: Divide the image data in Step 1.1 into a training dataset and a validation dataset according to a ratio of 8:2; Step 1.3: Perform data preprocessing on the training dataset, including padding, random cropping, horizontal flipping, and normalization processing.

3. The data generation method based on model feedback applicable to deep learning image classification according to claim 2, characterized in that: The specific method of Step 2 is: Build the weight generator G based on the convolutional neural network as the basic unit, including three layers of convolutional neural networks. Its structure from top to bottom is: a convolutional layer with a convolution kernel of 3 and a stride of 1, a batch normalization layer, a GELU activation layer, a convolutional layer with a convolution kernel of 3 and a stride of 1, a batch normalization layer, a GELU activation layer, a feature fusion layer, a convolutional layer with a convolution kernel of 1 and a stride of 1, a normalization layer, and a sigmoid activation layer.

4. The data generation method based on model feedback applicable to deep learning image classification according to claim 3, characterized in that: The specific method of Step 3 is: Step 3.1: Input two image data x1, x2 collected from the image dataset into the weight generator model G, and obtain a weight matrix m after the convolutional calculation inside the generator, that is: m = G(x1, x2); Step 3.2: Use the generated weight matrix m to combine the original images to obtain new image data, and generate the labels of the new image data by weighting the average value of the weights with the labels y1, y2 of the two image data: r = mean(m) Among them, are the new image data and its label respectively, and r is the average value of the weight m.

5. The data generation method based on model feedback applicable to deep learning image classification according to claim 4, wherein: The parameter optimization step of the weight generator model is: Input the new image data into the deep learning classification model, calculate the loss function according to the output of this model, and obtain the gradient information of the parameters of the weight generator model from the backpropagation of this loss. Then use the gradient information as feedback to optimize the parameters of the weight generator model.

6. The data generation method based on model feedback applicable to deep learning image classification according to claim 5, wherein: The parameter optimization steps of the deep learning classification model are as follows: Input the new image data into the deep learning classification model, calculate the loss function based on the output of the model, and then backpropagate from this loss to obtain the gradient information of the classification model parameters. Then, use the gradient information as feedback to optimize the parameters of the classification model.

7. The data generation method based on model feedback applicable to deep learning image classification according to claim 6, characterized in that: The specific method of step 4 is as follows: Step 4.1: Construct a deep learning classification network model F with convolution as the basic unit, and use the generated new images and new labels to train this model; Among them, o represents the output value of model F; Step 4.2: Calculate the loss function using the predicted label and the true label of the output of the deep learning classification network model, as shown in the following formula: Among them, the subscript i represents the index of label y, n represents the number of all image data in the image dataset, and l represents the calculated loss value; Step 4.3: Update the parameters of the weight generator model through the loss maximization algorithm Among them, TV(m i ) represents calculating the global variational loss for the weight m i to restrict the generated weights as a regularization term; Step 4.4: Update the parameters θ of the classification neural network model through the loss minimization algorithm: min θ l Obtain a deep learning model F that performs excellently in image classification tasks.

Images