Method, device, electronic device and storage medium for constructing image classification model

Abstract

The present invention relates to a method, device, electronic device and storage medium for constructing an image classification model. The method comprises: augmenting the original training samples to obtain the augmented training samples of the current round; obtaining the covariance matrix of the augmented training samples of the current round; determining the upper bound function of the first loss function of the image classification model to be optimized after the previous round of secondary optimization on the augmented training samples of the current round according to the covariance matrix of the augmented training samples of the current round; optimizing with the upper bound function as the optimization target to obtain the image classification model to be optimized for the first time of the current round; optimizing the covariance matrix in the upper bound function using a validation set; and performing secondary optimization on the image classification model to be optimized for the first time of the current round according to the optimized covariance matrix. The present invention optimizes the covariance matrix in the upper bound function using a validation set, thereby improving the performance of the model.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device, electronic device and storage medium for constructing an image classification model. Background Art

[0002] This solution aims to solve the problem of neural network training under long-tail training distribution. In real life, long-tail problems are very common, such as the Pareto Principle in economics and Chipf's Law in natural language processing. In the field of computer vision, the long-tail problem encountered is often that the distribution of training samples is long-tail distribution, that is, the category distribution of training samples is uneven, with the head category occupying most of the data and the tail category occupying a small amount of data. The image classification model trained with the training samples of the long-tail distribution has a low accuracy rate for the classification of images in the tail category.

[0003] Existing technologies usually use resampling technology to downsample the training samples of the majority class and upsample the training samples of the minority class to obtain a more even training distribution. However, resampling may sacrifice the classification performance of some majority classes and there is a certain risk of overfitting on the minority class. This is because the minority class naturally has a small amount of data, and overemphasizing its role is prone to overfitting.

[0004] In view of the above-mentioned defects of the prior art, there is an urgent need for a technical solution to improve the generalization performance on the tail categories. Summary of the invention

[0005] The present invention provides a method, device, electronic device and storage medium for constructing an image classification model, so as to solve the defect that the image classification model in the prior art has poor generalization performance on tail categories.

[0006] The present invention provides a method for constructing an image classification model, comprising: iteratively executing the following steps:

[0007] Augment the original training samples to obtain the current round of augmented training samples;

[0008] Get the covariance matrix of the training samples after the current round of augmentation;

[0009] Determine, according to the covariance matrix of the current round of augmented training samples, an upper bound function of a first loss function of the image classification model to be optimized after the previous round of secondary optimization on the current round of augmented training samples;

[0010] Optimize the image classification model to be optimized by taking the upper bound function as the optimization target to obtain the image classification model to be optimized for the first optimization in the current round;

[0011] Optimizing the covariance matrix in the upper bound function using a validation set;

[0012] The image classification model to be optimized that is optimized for the first time in the current round is optimized for the second time according to the optimized covariance matrix.

[0013] According to a method for constructing an image classification model provided by the present invention, before the first round of iteration, the method further includes:

[0014] The original image classification model is trained using the original training samples to obtain the image classification model to be optimized.

[0015] According to a method for constructing an image classification model provided by the present invention, the original training samples are augmented to obtain the training samples after the current round of augmentation, including:

[0016] Extracting feature data of the original training samples according to the image classification model to be optimized after the previous round of secondary optimization, and re-obtaining the covariance matrix of the original training samples according to the feature data of the original training samples;

[0017] According to the characteristic data of the original training sample and the covariance matrix of the original training sample, an augmented sample of the original training sample is obtained.

[0018] According to a method for constructing an image classification model provided by the present invention, the method of determining the upper bound function of the first loss function of the image classification model to be optimized after the previous round of secondary optimization on the training samples after the current round of augmentation according to the covariance matrix of the training samples after the current round of augmentation includes:

[0019] Determine, according to the current round of augmented training samples and the covariance matrix of the current round of augmented training samples, a first loss function of the image classification model to be optimized after the previous round of secondary optimization on the current round of augmented training samples;

[0020] The upper bound function of the first loss function is determined by assuming that the training samples after the current round of augmentation are augmented countless times.

[0021] According to a method for constructing an image classification model provided by the present invention, the optimization is performed with the upper bound function as the optimization target to obtain the image classification model to be optimized for the first optimization in the current round, including:

[0022] Taking minimizing the upper bound function as the optimization goal, obtaining the model parameters of the first optimization in the current round;

[0023] The image classification model to be optimized is updated according to the model parameters optimized for the first time in the current round, so as to obtain the image classification model to be optimized optimized for the first time in the current round.

[0024] According to a method for constructing an image classification model provided by the present invention, the method of optimizing the covariance matrix in the upper bound function by using a validation set includes:

[0025] Obtaining a second loss function of the image classification model to be optimized that is first optimized in the current round on the verification set;

[0026] The covariance matrix in the second loss function is regarded as a hyperparameter, and minimizing the second loss function is taken as the optimization goal to obtain the optimized covariance matrix corresponding to the second loss function.

[0027] According to a method for constructing an image classification model provided by the present invention, the second optimization of the image classification model to be optimized that is optimized for the first time in the current round is performed according to the optimized covariance matrix, including:

[0028] The optimized covariance matrix replaces the covariance matrix of the current round of augmented training samples in the upper bound function;

[0029] Taking the upper bound function after minimizing the replaced covariance matrix as the optimization objective, the model parameters of the current round of quadratic optimization are obtained;

[0030] The image classification model to be optimized is updated according to the model parameters of the current round of secondary optimization.

[0031] According to a method for constructing an image classification model provided by the present invention, after performing a secondary optimization on the image classification model to be optimized that is optimized for the first time in the current round according to the optimized covariance matrix, the method further includes:

[0032] When the current iteration round reaches the preset number of times, or when the difference between the upper bound function value of the current iteration round and the upper bound function value of the previous iteration round is less than a preset threshold, the iteration is stopped;

[0033] The image classification model to be optimized after the secondary optimization in the stopping iteration round is determined as the target image classification model.

[0034] The present invention also provides a device for constructing an image classification model, comprising:

[0035] A sample augmentation module is used to augment the training samples augmented in the previous round to obtain the training samples augmented in the current round; wherein, when it is the first round of iteration, the original training samples are augmented;

[0036] The matrix acquisition module is used to obtain the covariance matrix of the training samples after the current round of augmentation;

[0037] An upper bound function acquisition module, used to determine an upper bound function of a first loss function of the image classification model to be optimized after the previous round of secondary optimization on the training samples after the current round of augmentation according to the covariance matrix of the training samples after the current round of augmentation;

[0038] A first optimization module is used to optimize the image classification model to be optimized by taking the upper bound function as the optimization target, and obtain the image classification model to be optimized in the first optimization of the current round;

[0039] A matrix optimization module, used for optimizing the covariance matrix in the upper bound function using a validation set;

[0040] The secondary optimization module is used to perform secondary optimization on the image classification model to be optimized that is first optimized in the current round according to the optimized covariance matrix.

[0041] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, all or part of the steps of the method for constructing an image classification model as described above are implemented.

[0042] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements all or part of the steps of the method for constructing an image classification model as described in any one of the above.

[0043] The present invention provides a method, device, electronic device and storage medium for constructing an image classification model, which augments training samples, increases the proportion of training samples in minority categories, and improves the model prediction performance of the image classification model to be optimized on minority categories; uses an upper bound function as an optimization objective function to optimize the target model parameters of the image classification model to be optimized, greatly reducing the time complexity of the model training and optimization process; uses a category-balanced validation set to optimize the covariance matrix in the upper bound function, and then performs secondary optimization on the model, further improving the performance of the image classification model to be optimized. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 It is a flowchart of a method for constructing an image classification model provided by the present invention;

[0046] Figure 2It is a structural schematic diagram of a device for constructing an image classification model provided by the present invention;

[0047] Figure 3 It is a structural schematic diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] Combine the following Figure 1-Figure 3 The present invention describes a method, device, electronic device and storage medium for constructing an image classification model.

[0050] Figure 1 It is a flowchart of a method for constructing an image classification model provided by the present invention, such as Figure 1 As shown, the method includes:

[0051] S11, augmenting the original training samples to obtain the training samples after the current round of iterative augmentation;

[0052] Specifically, the method for constructing an image classification model of the present invention augments the original training samples based on the preliminarily trained image classification model to be optimized, and accordingly performs iterative optimization on the image classification model to be optimized, thereby obtaining an optimized image splitting model. The iterative augmentation basis of the training samples is the original training samples, and different data augmentation directions are sought on the basis of the original training samples to perform iterative optimization on the image classification model to be optimized. The basis for each round of iterative optimization of the image classification model to be optimized is the image classification model to be optimized after the previous round of optimization.

[0053] In each round of iteration, data augmentation is performed on the basis of the original training samples to obtain the training samples after the current round of augmentation. The training samples include sample images and reference classification labels corresponding to the sample images. The reference classification labels can be represented by the numbers of the classifications corresponding to the sample images. Data augmentation methods include, for example, flipping, rotating, translating, and other operations on the image in pixel space, and sampling and converting in the deep feature space of the training samples to generate new features to augment the data. It is also possible to combine category weighting to assign different weights to training samples of different categories, so that the image classification model pays more attention to the gradients generated by samples of minority categories. The present invention does not limit the specific method of data augmentation.

[0054] S12, obtaining the covariance matrix of the training samples after the current round of augmentation;

[0055] Specifically, the training samples after the current round of augmentation include the original training samples and some training samples generated by the current round of augmentation. The covariance matrix describes the correlation between samples in each category of training samples. The covariance matrix can be obtained by extracting features from the training samples after the current round of augmentation based on the image classification model to be optimized after the previous round of iterative optimization to obtain feature data, and then statistically analyzing the feature data to obtain the covariance matrix of the training samples after the current round of augmentation.

[0056] S13, determining an upper bound function of a first loss function of the image classification model to be optimized after the previous round of secondary optimization on the training samples after the current round of augmentation according to the covariance matrix of the training samples after the current round of augmentation;

[0057] Specifically, the loss function characterizes the difference between the predicted output of the model (i.e., the image classification result) and the actual value. The larger the loss function, the worse the prediction performance of the model, and the smaller the loss function, the better the prediction performance of the model. According to the sample images in the training samples after the current round of augmentation and the reference classification labels corresponding to the sample images, combined with the covariance matrix, the first loss function of the image classification model to be optimized on the training samples after the current round of augmentation can be determined. Furthermore, based on different samples, the image classification model has different loss functions. Frequent replacement / addition of training samples results in a long process of model training and optimization. The upper bound function of the loss function of the image classification model to be optimized is determined by deduction, and the loss function is always smaller than its upper bound function. Therefore, the prediction performance of the image classification model can be indirectly understood through the upper bound function.

[0058] It should be noted that the above training sample augmentation process can be directly updated in the upper bound function in the form of data update, without explicitly generating actual augmented samples, nor executing the process of training the model to be optimized based on the augmented samples. The upper bound function is directly used as the optimization objective function, and the target model parameters of the image classification model to be optimized can be determined by iteratively updating the upper bound function, which greatly reduces the time complexity of the model training and optimization process.

[0059] S14, optimizing with the upper bound function as the optimization target to obtain the image classification model to be optimized for the first optimization in the current round;

[0060] Specifically, the upper bound function represents the maximum value of the loss function of the image classification model to be optimized on the corresponding training samples. The upper bound function is used as the optimization objective function, and the training samples are iteratively augmented and the upper bound function is iteratively updated accordingly. When the iterative update reaches the preset condition, the iterative update is stopped, and the model parameters corresponding to the upper bound function when the iterative update is stopped are determined as the model parameters of the image classification model to be optimized for the first optimization in the current round (that is, the model parameters of the image classification model to be optimized are updated according to the model parameters corresponding to the upper bound function when the iterative update is stopped).

[0061] S15, optimizing the covariance matrix in the upper bound function using the validation set;

[0062] Specifically, since the original training samples themselves have the defect of imbalanced categories, the augmented samples also have certain defects, so the covariance matrix estimated based on the augmented samples is not precise enough. After thinking, it is found that if a suitable covariance matrix is ​​used for data enhancement, the loss function on the balanced validation set should be minimized. Based on this feature, the present invention considers optimizing the covariance matrix based on the covariance matrix estimated by the augmented samples by using the validation set. The validation set has a small amount of data relative to the original training samples, but the sample category distribution in the validation set is relatively balanced, closer to the actual test sample distribution, and a more accurate covariance matrix can be estimated. Specifically, the covariance matrix can be regarded as a hyperparameter, and the covariance matrix can be optimized by minimizing the loss function of the image classification model to be optimized on the validation set that is optimized for the first time in the current round.

[0063] The present invention utilizes the idea of ​​meta-learning to learn a covariance matrix that is more informative and more suitable for the image classification model to be optimized through the knowledge of a small (i.e., small data volume) balanced validation set, and uses this as the basis for iterative sample augmentation and model optimization.

[0064] S16. Perform secondary optimization on the image classification model to be optimized that is optimized for the first time in the current round according to the optimized covariance matrix.

[0065] Specifically, after the covariance matrix is ​​optimized according to the validation set, the upper bound function can be updated using the optimized covariance matrix, and steps S13 and S14 can be re-referenced to perform secondary optimization on the image classification model to be optimized.

[0066] It should be noted that the first update of the model to be optimized in the current round in step S14 is a pseudo-update implemented on a copy of the image classification model to be optimized (i.e., equivalent to an intermediate auxiliary process implemented on the secondary model). Step S16 is a formal update implemented on the image classification model to be optimized (i.e., the main model) after the previous round of secondary optimization.

[0067] In this embodiment, the training samples are augmented to increase the proportion of minority category training samples, thereby improving the model prediction performance of the image classification model to be optimized on the minority category; the upper bound function is used as the optimization objective function to optimize the target model parameters of the image classification model to be optimized, which greatly reduces the time complexity of the model training and optimization process; the covariance matrix in the upper bound function is optimized using a category-balanced validation set, and then the model is optimized secondary, further improving the performance of the image classification model to be optimized.

[0068] Based on any of the above embodiments, in one embodiment, before the first round of iteration, the method further includes:

[0069] The original image classification model is trained using the original training samples to obtain the image classification model to be optimized.

[0070] Specifically, the image classification model to be optimized is an image classification model that has been preliminarily trained. Before using the image classification model to be optimized to extract the feature data of the original training sample, it is necessary to first use the original training sample to train the original image classification model, adjust the model parameters, and obtain the image classification model to be optimized. The type of image classification model can be VGG network, GoogLenet network, Residual network, ResNet network, etc., and the specific type is not limited. The original image classification model includes a feature extraction network and a classifier network. The feature extraction network can be used to extract features from the training data (i.e., the above-mentioned original training samples) to obtain feature data. The classifier network can be used to process the feature data and output the prediction results. The classification results are presented in the form of probability, and the category corresponding to the value with the largest proportion in the probability is the final predicted label.

[0071] In this embodiment, the original image classification model is trained using the original training samples to obtain the image classification model to be optimized, which facilitates subsequent iterative optimization.

[0072] Based on any of the above embodiments, in one embodiment, the augmenting the original training samples to obtain the current round of augmented training samples includes:

[0073] Extracting feature data of the original training samples according to the image classification model to be optimized after the previous round of secondary optimization, and re-obtaining the covariance matrix of the original training samples according to the feature data of the original training samples;

[0074] According to the feature data of the original training sample and the covariance matrix of the original training sample, an augmented sample of the original training sample is obtained.

[0075] Specifically, the image classification model to be optimized after the previous round of secondary optimization can be used to extract features from the original training samples. The extracted feature data is a deep feature with a high degree of abstraction. The feature data of the original training samples are statistically analyzed to obtain the corresponding covariance matrix. The covariance matrix is ​​composed of covariance matrices of multiple categories. The covariance matrix of each category can be obtained by statistical analysis based on the deep features of the training samples of that category. The covariance in the covariance matrix represents the correlation between the corresponding features. On this basis, the covariance matrix can be sampled and transformed, and the original training samples can be augmented in combination with the extracted feature data to obtain the augmented samples of the current round. The augmented training samples can effectively avoid overfitting on a few categories.

[0076] For example, suppose the original training sample set is: where x i is the image in the i-th training sample, and y i is its corresponding label, and the entire training data set has N samples. The image classification model to be optimized after the previous round of secondary optimization is used to classify the training sample x i Perform feature extraction, and the obtained feature data can be recorded as a i , the covariance matrix of the original training sample can be recorded as: Σ={Σ 1 ,Σ 2 ,…,Σ M},Σ 1 ,Σ 2 ,…,Σ M is the covariance matrix of each category, and M is the total number of categories.

[0077] The feature distribution in the covariance matrix is ​​statistically obtained to obtain the distribution, and the corresponding data augmentation direction can be understood based on the distribution. The data augmentation direction corresponds to the actual sample, such as the different colors of cars in "car" pictures, the different viewing directions of cars, etc. The covariance matrix is ​​sampled according to the distribution, and a transformation vector is generated. The transformation vector can be used to enhance the feature data and generate new feature data (that is, the corresponding training samples are augmented). The feature distribution preferably uses the Gaussian distribution model, which can be recorded as Among them, y i is the classification label corresponding to the i-th image, one of multiple categories, is the covariance matrix of the classification, and λ is a hyperparameter used to adjust the data augmentation strength. The covariance matrix is ​​sampled according to the Gaussian distribution, and a conversion vector is generated to enhance the feature data a corresponding to the i-th image. i .

[0078] In this embodiment, based on the feature data of the training samples after the previous round of augmentation, a covariance matrix representing the correlation between the feature data in the training samples is obtained; the covariance matrix is ​​sampled according to the feature distribution in the covariance matrix, and a transformation vector is generated to augment the training samples after the previous round of augmentation, thereby avoiding overfitting of the image recognition model to be optimized on a minority of categories.

[0079] Based on any of the above embodiments, in one embodiment, determining the upper bound function of the first loss function of the image classification model to be optimized after the previous round of secondary optimization on the training samples after the current round of augmentation according to the covariance matrix of the training samples after the current round of augmentation includes:

[0080] Determine a first loss function of the image classification model to be optimized after the previous round of secondary optimization on the training samples after the current round of augmentation according to the current generation of augmented training samples and the covariance matrix of the training samples after the current round of augmentation;

[0081] The upper bound function of the first loss function is determined by assuming that the training samples after the current round of augmentation are augmented countless times.

[0082] Specifically, the loss function characterizes the difference between the predicted output of the model (i.e., the image classification result) and the actual value. The larger the loss function, the worse the prediction performance of the model, and the smaller the loss function, the better the prediction performance of the model. Each time the training samples are augmented, the loss function of the image classification model to be optimized on the augmented training samples can be obtained. In order to explore all possible data augmentation directions, it is necessary to sample the covariance matrix of the training samples multiple times according to the feature distribution in the covariance matrix, perform augmentation accordingly, and update the loss function of the image classification model to be optimized on the corresponding augmented samples. If we go a step further, assuming that the sampling is infinite, and the training samples are augmented accordingly, the upper bound function of the loss function of the image classification model to be optimized can be determined.

[0083] In this embodiment, by assuming that the training samples after the current round of augmentation are augmented countless times, the upper bound function of the loss function is determined, which facilitates indirect understanding of the prediction performance of the image classification model.

[0084] Based on any of the above embodiments, in one embodiment, the optimization is performed with the upper bound function as the optimization target to obtain the image classification model to be optimized for the first optimization in the current round, including:

[0085] Taking minimizing the upper bound function as the optimization goal, obtaining the model parameters of the first optimization in the current round;

[0086] The image classification model to be optimized is updated according to the model parameters optimized for the first time in the current round, so as to obtain the image classification model to be optimized optimized for the first time in the current round.

[0087] Specifically, the optimization process of the image classification model to be optimized directly with minimizing the loss function as the optimization goal is relatively complicated and requires a large amount of calculation. By minimizing the upper bound function of the first loss function of the image classification model to be optimized on the training samples after the current round of augmentation, the model parameters optimized for the first time in the current round are obtained, which can be used to conveniently update the model parameters.

[0088] In this embodiment, the image classification model to be optimized is conveniently optimized for the first time through the upper bound function of the first loss function on the training samples after the current round of augmentation.

[0089] Based on any of the foregoing embodiments, in one embodiment, the step of optimizing the covariance matrix in the upper bound function using a validation set includes:

[0090] Obtaining a second loss function of the image classification model to be optimized that is first optimized in the current round on the verification set;

[0091] The covariance matrix in the second loss function is regarded as a hyperparameter, and minimizing the second loss function is taken as the optimization goal to obtain the optimized covariance matrix corresponding to the second loss function.

[0092] Specifically, the second loss function of the image classification model to be optimized which is optimized for the first time in the current round on the verification set is obtained; the covariance matrix in the second loss function is regarded as a hyperparameter (i.e., as a variable), and is derived and updated, and the covariance matrix in the second loss function is optimized and adjusted by using a gradient descent algorithm with minimization of the second loss function as the optimization goal, so as to obtain the optimized covariance matrix corresponding to the second loss function.

[0093] In this embodiment, a validation set with a relatively more balanced distribution of sample categories is used to optimize the covariance matrix, so as to obtain more accurate model parameters and improve the model parameters of the image classification model to be optimized.

[0094] Based on any of the above embodiments, in one embodiment, performing a secondary optimization on the image classification model to be optimized that is optimized for the first time in the current round according to the optimized covariance matrix includes:

[0095] The optimized covariance matrix replaces the covariance matrix of the current round of augmented training samples in the upper bound function;

[0096] Taking the upper bound function after minimizing the replaced covariance matrix as the optimization objective, the model parameters of the current round of quadratic optimization are obtained;

[0097] The image classification model to be optimized is updated according to the model parameters of the current round of secondary optimization.

[0098] Specifically, after obtaining the optimized covariance matrix, the optimized covariance matrix replaces the covariance matrix of the training samples after the current round of augmentation in the upper bound function of step S13, and step S14 is executed again, with minimizing the upper bound function after replacing the covariance matrix as the optimization goal, and the model parameters of the current round of quadratic optimization are obtained. The image classification model to be optimized is updated according to the model parameters of the current round of quadratic optimization to achieve the current round of quadratic optimization.

[0099] In this embodiment, the optimized covariance matrix replaces the covariance matrix of the training samples after the current round of augmentation in the upper bound function. Through a single iteration loop, the next step of optimization is performed based on the current optimal model parameters / covariance matrix each time, which speeds up the iterative optimization process, reduces the calculation time, and improves the model performance of the image classification model to be optimized.

[0100] Based on any of the above embodiments, in one embodiment, after performing a secondary optimization on the image classification model to be optimized that is optimized for the first time in the current round according to the optimized covariance matrix, the method further includes:

[0101] When the current iteration round reaches the preset number of times, or when the difference between the upper bound function value of the current iteration round and the upper bound function value of the previous iteration round is less than a preset threshold, the iteration is stopped;

[0102] The image classification model to be optimized after the secondary optimization in the stopping iteration round is determined as the target image classification model.

[0103] Specifically, it can be set to stop iteration when the number of iteration rounds reaches a preset number. The preset number can be set according to the performance requirements of the image classification model to be optimized in combination with historical data, for example, 800 times; it can also be set to stop iteration when the difference between the upper bound function value of the current iteration round and the upper bound function value of the previous iteration round is less than a preset threshold. The difference between the upper bound function value of the current iteration round and the upper bound function value of the previous iteration round is less than the preset threshold, indicating that the space for further optimization is small and the time-consuming further optimization is long. At this time, the iteration can be stopped. The image classification model to be optimized after secondary optimization in the stop iteration round is determined as the target image classification model.

[0104] In this embodiment, when the number of iteration rounds reaches a preset number, or when the difference between the upper bound function value of the current round of iteration and the upper bound function value of the previous round of iteration is less than a preset threshold, the iteration is stopped. The conditions for stopping the iteration are set flexibly and accurately to meet the user's differentiated model building needs.

[0105] The method for constructing the image classification model provided by the present invention is described below in conjunction with a preferred embodiment.

[0106] Assume that the original training sample set is defined as: where x i is the image in the i-th training sample, and y i is its corresponding label, and the entire original training data set has N samples.

[0107] The training sample data of the tail category is extremely lacking, which leads to extremely unbalanced performance of the trained image classification model and poor performance on the tail category. In this embodiment, semantic data augmentation technology (ISDA) is used to augment the training samples of the tail category to improve the performance of the classifier. In order to achieve semantic data augmentation, ISDA dynamically estimates the covariance matrix of each category Σ={Σ 1 ,Σ 2 ,…,Σ C Then, ISDA uses these covariance matrices to construct a Gaussian distribution and samples from the Gaussian distribution to augment the data. Through infinite sampling, ISDA derives the optimal upper bound of the loss function of these augmented training samples:

[0108]

[0109] Among them, x i is the image in the i-th training sample, and y i is its corresponding label, Θ is the model parameter of the image classification model to be optimized, Σ is the covariance matrix of the feature data of the training sample, is the sample x i The cth element in the prediction output of (i.e., the probability that the corresponding prediction belongs to the cth category), is the sample x i The yth prediction output i elements (i.e., the corresponding prediction belongs to the yth i The probability of each class), is the covariance matrix of the classification corresponding to the i-th training sample, w c is the cth column in the weight matrix of the last layer of the image classification model to be optimized is the cth column and the yth column in the weight matrix of the last layer of the image classification model to be optimized i The difference between the columns, yes The transpose of .

[0110] By optimizing this upper bound function, ISDA can effectively implement an equivalent semantic augmentation process. In general, by using formula (1), an image classification model can be built and the data augmentation process can be completed at the same time. However, in the context of long-tail problems, the head category dominates the training set. And through formula (1), it can be observed that the effect of data augmentation depends on the training data. If formula (1) is used directly, it actually mainly augments the majority category, which runs counter to our goal.

[0111] Therefore, in order to solve this problem, this application proposes to combine the category weight and formula (1) to give different gains to the semantically augmented samples according to the number of samples contained in the category, so as to adjust the proportion of minority category training samples. The category weight is defined as:

[0112]

[0113] Among them, ∈ c is the preset category weight of the cth category, n c is the number of training samples in the cth category, β is a hyperparameter, and the value range of β is (0, 1).

[0114] The optimal network parameters can be obtained using the loss function weighted by the category weights:

[0115]

[0116] Among them, Θ * Represents the optimized model parameters corresponding to the optimized loss function.

[0117] To further improve the model, considering that the performance of ISDA depends on the estimation of the covariance matrix, under the long-tail problem, we found that the performance of ISDA is limited by the small number of tail categories, and it is difficult to obtain a reasonable covariance estimate. In order to further solve the problem of category imbalance and improve the performance of the model on a few categories, this embodiment attempts to learn a suitable covariance matrix for data enhancement, leading to better performance. The key idea is: if a suitable covariance matrix is ​​used for data enhancement, then the loss function on a balanced validation set should be minimized. In this embodiment, we use meta-learning to achieve this goal. If the covariance matrix is ​​regarded as a hyperparameter, the validation set can actually be used to search for its optimal value. In detail, consider a small validation set Where N v is the number of samples in the validation set, and the number of samples in the validation set is much smaller than that in the training set. The optimal covariance matrix can be obtained by minimizing the following loss function:

[0118]

[0119] Among them, Σ* Represents the optimized covariance matrix corresponding to the minimized loss function, L ce (·) is the cross entropy loss equation. N v is the number of samples in the validation set, Sample images in the i-th sample number in the validation set, y i is the reference classification label corresponding to the sample image in the i-th training sample in the validation set. Because the validation set is balanced, this embodiment uses the common CE loss function.

[0120] Considering that in formula (3), in order to obtain the optimal values ​​of model parameters and covariance matrix, two nested loops are required, which consumes too much computing resources. Therefore, this embodiment uses an online strategy to update the two quantities of model parameters and covariance matrix through a single iteration loop, and each time the current optimal solution of one quantity is used as the basis to seek the current optimal solution of the other quantity. Given the current iteration round t, we use a statistical method to obtain the current covariance. Then, the following equation is used to update the model parameters:

[0121]

[0122] Where α is the learning rate of the model parameters. t is the covariance matrix of the training samples after the current round of augmentation, Θ t It is the model parameter of the image classification model to be optimized after the previous round of secondary optimization. is the model parameter of the image classification model to be optimized after the first optimization in the current round. After executing this step of back propagation, the model parameters optimized for the first time are obtained. Then, the covariance matrix is ​​updated using the following formula:

[0123]

[0124] Where γ is the learning rate of the covariance matrix, Σ t+1 It is the covariance matrix of the image classification model to be optimized after the first optimization in the current round, which is optimized according to the validation set. Then, the optimized covariance matrix can be used to further improve the model parameters of the image classification model:

[0125]

[0126] Among them, Θ t+1 are the model parameters of the image classification model to be optimized after the current round of secondary optimization. Because the updated covariance matrix is ​​the knowledge learned from the validation set with balanced category distribution, it can be expected that the optimized covariance matrix can help learn better model parameters, thereby achieving further improvement of the model and improving the model performance of the constructed image classification model on a small number of categories.

[0127] For the data augmentation problem of minority classes, this embodiment mainly uses the semantic augmentation technology in deep space, which relies on the estimation of the covariance matrix of the features of each category. If the estimation of the covariance matrix is ​​inaccurate or the amount of information is small, it may lead to poor data augmentation effect. This embodiment uses meta-learning to improve this problem. A loss function of a validation data set with a balanced category distribution is used to help learn a better covariance matrix. The learned covariance matrix is ​​then substituted into the process of semantic data augmentation, with the intention of obtaining better data augmentation effects. The entire data augmentation training process can be replaced by a loss function without the need to explicitly generate many augmented samples. The advantages of this operation are:

[0128] 1. No need to explicitly generate augmented samples, so the training overhead is low.

[0129] 2. It can automatically learn meaningful data augmentation directions without the need for manual selection of augmentation directions.

[0130] 3. No additional data is needed, only a small amount of validation set data is needed to obtain a better covariance matrix.

[0131] 4. Filling the minority categories through data augmentation can reduce the risk of overfitting the model on the minority classes.

[0132] A device for constructing an image classification model provided by the present invention is described below. The device for constructing an image classification model described below and the method for constructing an image classification model described above can refer to each other.

[0133] Figure 2 is a structural schematic diagram of a device for constructing an image classification model provided by the present invention, such as Figure 2 As shown, the device includes: a sample augmentation module 21, a matrix acquisition module 22, an upper bound function acquisition module 23, a first optimization module 24, a matrix optimization module 25, and a second optimization module 26.

[0134] The sample augmentation module 21 is used to augment the original training samples to obtain the current round of augmented training samples;

[0135] A matrix acquisition module 22 is used to obtain the covariance matrix of the training samples after the current round of augmentation;

[0136] An upper bound function acquisition module 23 is used to determine an upper bound function of a first loss function of the image classification model to be optimized after the previous round of secondary optimization on the training samples after the current round of augmentation according to the covariance matrix of the training samples after the current round of augmentation;

[0137] A first optimization module 24 is used to optimize the upper bound function as the optimization target to obtain the image classification model to be optimized in the first optimization of the current round;

[0138] A matrix optimization module 25, used for optimizing the covariance matrix in the upper bound function using a validation set;

[0139] The secondary optimization module 26 is used to perform secondary optimization on the image classification model to be optimized that is first optimized in the current round according to the optimized covariance matrix.

[0140] In this embodiment, the training samples are augmented to increase the proportion of minority category training samples, thereby improving the model prediction performance of the image classification model to be optimized on the minority category; the upper bound function is used as the optimization objective function to optimize the target model parameters of the image classification model to be optimized, which greatly reduces the time complexity of the model training and optimization process; the covariance matrix in the upper bound function is optimized using a category-balanced validation set, and then the model is optimized secondary, further improving the performance of the image classification model to be optimized.

[0141] Based on any of the above embodiments, in one embodiment, the device further includes:

[0142] The model training module is used to train the original image classification model using the original training samples to obtain the image classification model to be optimized.

[0143] In this embodiment, the original image classification model is trained using the original training samples to obtain the image classification model to be optimized, which facilitates subsequent iterative optimization.

[0144] Based on any of the above embodiments, in one embodiment, the sample augmentation module 21 includes:

[0145] A first sample augmentation unit, used for extracting feature data of the original training sample according to the image classification model to be optimized after the previous round of secondary optimization, and re-obtaining the covariance matrix of the original training sample according to the feature data of the original training sample;

[0146] The second sample augmentation unit is used to obtain augmented samples of the original training samples according to the feature data of the original training samples and the covariance matrix of the original training samples.

[0147] In this embodiment, based on the feature data of the training samples after the previous round of augmentation, a covariance matrix representing the correlation between the feature data in the training samples is obtained; the covariance matrix is ​​sampled according to the feature distribution in the covariance matrix, and a transformation vector is generated to augment the training samples after the previous round of augmentation, thereby avoiding overfitting of the image recognition model to be optimized on a minority of categories.

[0148] Based on any of the above embodiments, in one embodiment, the upper bound function acquisition module 23 includes:

[0149] A first acquisition unit is used to determine a first loss function of the image classification model to be optimized after the previous round of secondary optimization on the training samples after the current round of augmentation according to the training samples after the current round of augmentation and the covariance matrix of the training samples after the current round of augmentation;

[0150] The second acquisition unit is used to determine the upper bound function of the first loss function by assuming that the training samples after the current round of augmentation are augmented countless times.

[0151] In this embodiment, by assuming that the training samples after the current round of augmentation are augmented countless times, the upper bound function of the loss function is determined, which facilitates indirect understanding of the prediction performance of the image classification model.

[0152] Based on any of the above embodiments, in one embodiment, the first optimization module 24 includes:

[0153] A first model optimization unit, configured to obtain the model parameters of the first optimization in the current round by taking minimization of the upper bound function as the optimization goal;

[0154] The second model optimization unit is used to update the image classification model to be optimized according to the model parameters optimized for the first time in the current round, so as to obtain the image classification model to be optimized for the first time in the current round.

[0155] In this embodiment, the image classification model to be optimized is conveniently optimized for the first time through the upper bound function of the first loss function on the training samples after the current round of augmentation.

[0156] Based on any of the above embodiments, in one embodiment, the matrix optimization module 25 includes:

[0157] A first matrix optimization unit, used for obtaining a second loss function of the image classification model to be optimized which is first optimized in the current round on the verification set;

[0158] The second matrix optimization unit is used to regard the covariance matrix in the second loss function as a hyperparameter, and to minimize the second loss function as an optimization goal, so as to obtain the optimized covariance matrix corresponding to the second loss function.

[0159] In this embodiment, a validation set with a relatively more balanced distribution of sample categories is used to optimize the covariance matrix, so as to obtain more accurate model parameters and improve the model parameters of the image classification model to be optimized.

[0160] Based on any of the above embodiments, in one embodiment, the secondary optimization module 26 includes:

[0161] A third model optimization unit, used for replacing the covariance matrix of the current round of augmented training samples in the upper bound function with the optimized covariance matrix;

[0162] The fourth model optimization unit is used to obtain the model parameters of the current round of quadratic optimization by minimizing the upper bound function after replacing the covariance matrix as the optimization target;

[0163] The fifth model optimization unit is used to update the image classification model to be optimized according to the model parameters of the current round of secondary optimization.

[0164] In this embodiment, the optimized covariance matrix replaces the covariance matrix of the training samples after the current round of augmentation in the upper bound function. Through a single iteration loop, the next step of optimization is performed based on the current optimal model parameters / covariance matrix each time, which speeds up the iterative optimization process, reduces the calculation time, and improves the model performance of the image classification model to be optimized.

[0165] Based on any of the above embodiments, in one embodiment, the device further includes:

[0166] An iteration detection module is used to stop the iteration when the current iteration round reaches a preset number of times, or when the difference between the upper bound function value of the current iteration round and the upper bound function value of the previous iteration round is less than a preset threshold;

[0167] The model determination module is used to determine the image classification model to be optimized after the secondary optimization in the stop iteration round as the target image classification model.

[0168] In this embodiment, when the number of iteration rounds reaches a preset number, or when the difference between the upper bound function value of the current round of iteration and the upper bound function value of the previous round of iteration is less than a preset threshold, the iteration is stopped. The conditions for stopping the iteration are set flexibly and accurately to meet the user's differentiated model building needs.

[0169] Figure 3 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 3As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communication interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute all or part of the steps of the above-mentioned methods for constructing the image classification model provided, the method comprising: augmenting the original training samples to obtain the augmented training samples of the current round; obtaining the covariance matrix of the augmented training samples of the current round; determining the upper bound function of the first loss function of the image classification model to be optimized after the previous round of secondary optimization on the augmented training samples of the current round according to the covariance matrix of the augmented training samples of the current round; optimizing with the upper bound function as the optimization target to obtain the image classification model to be optimized for the first optimization of the current round; optimizing the covariance matrix in the upper bound function using the validation set; and performing secondary optimization on the image classification model to be optimized for the first optimization of the current round according to the optimized covariance matrix.

[0170] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0171] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute all or part of the steps of the above-mentioned methods for constructing image classification models, the method including: augmenting the original training samples to obtain the training samples after the current round of augmentation; obtaining the covariance matrix of the training samples after the current round of augmentation; determining the upper bound function of the first loss function of the image classification model to be optimized after the previous round of secondary optimization on the training samples after the current round of augmentation according to the covariance matrix of the training samples after the current round of augmentation; optimizing with the upper bound function as the optimization target to obtain the image classification model to be optimized which is optimized for the first time in the current round; optimizing the covariance matrix in the upper bound function using the verification set; and performing secondary optimization on the image classification model to be optimized which is optimized for the first time in the current round according to the optimized covariance matrix.

[0172] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute all or part of the steps of the above-mentioned method for constructing an image classification model, the method comprising: augmenting the original training samples to obtain the training samples after the current round of augmentation; obtaining the covariance matrix of the training samples after the current round of augmentation; determining, based on the covariance matrix of the training samples after the current round of augmentation, an upper bound function of the first loss function of the image classification model to be optimized after the previous round of secondary optimization on the training samples after the current round of augmentation; optimizing with the upper bound function as the optimization target to obtain the image classification model to be optimized which is optimized for the first time in the current round; optimizing the covariance matrix in the upper bound function using a validation set; and performing secondary optimization on the image classification model to be optimized which is optimized for the first time in the current round according to the optimized covariance matrix.

[0173] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0174] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0175] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing an image classification model, characterized in that: include: Iterate and execute the following steps: Augment the original training samples to obtain the current round of augmented training samples; Get the covariance matrix of the training samples after the current round of augmentation; Determine, according to the covariance matrix of the current round of augmented training samples, an upper bound function of a first loss function of the image classification model to be optimized after the previous round of secondary optimization on the current round of augmented training samples; Optimize the image classification model to be optimized by taking the upper bound function as the optimization target to obtain the image classification model to be optimized for the first optimization in the current round; Optimizing the covariance matrix in the upper bound function using a validation set; Performing secondary optimization on the image classification model to be optimized that was optimized for the first time in the current round according to the optimized covariance matrix; The method of augmenting the original training samples to obtain the current round of augmented training samples includes: Extracting feature data of the original training samples according to the image classification model to be optimized after the previous round of secondary optimization, and re-obtaining the covariance matrix of the original training samples according to the feature data of the original training samples; Performing statistics on the characteristic distribution in the covariance matrix to obtain distribution conditions; Sampling the covariance matrix according to the distribution to generate a conversion vector; The feature data is enhanced according to the conversion vector to generate new feature data to augment the original training sample.

2. The method for constructing an image classification model according to claim 1, characterized in that: Before the first iteration, the method also includes: The original image classification model is trained using the original training samples to obtain the image classification model to be optimized.

3. The method for constructing an image classification model according to claim 1, characterized in that: The step of determining the upper bound function of the first loss function of the image classification model to be optimized after the previous round of secondary optimization on the training samples after the current round of augmentation according to the covariance matrix of the training samples after the current round of augmentation includes: Determine, according to the current round of augmented training samples and the covariance matrix of the current round of augmented training samples, a first loss function of the image classification model to be optimized after the previous round of secondary optimization on the current round of augmented training samples; The upper bound function of the first loss function is determined by assuming that the training samples after the current round of augmentation are augmented countless times.

4. The method for constructing an image classification model according to claim 1, characterized in that: The optimization is performed with the upper bound function as the optimization target to obtain the image classification model to be optimized for the first optimization in the current round, including: Taking minimizing the upper bound function as the optimization goal, obtaining the model parameters of the first optimization in the current round; The image classification model to be optimized is updated according to the model parameters optimized for the first time in the current round, so as to obtain the image classification model to be optimized optimized for the first time in the current round.

5. The method for constructing an image classification model according to claim 1, characterized in that: The method of optimizing the covariance matrix in the upper bound function by using the validation set includes: Obtaining a second loss function of the image classification model to be optimized that is first optimized in the current round on the verification set; The covariance matrix in the second loss function is regarded as a hyperparameter, and minimizing the second loss function is taken as the optimization goal to obtain the optimized covariance matrix corresponding to the second loss function.

6. The method for constructing an image classification model according to claim 1, characterized in that: The second optimization of the image classification model to be optimized that is optimized for the first time in the current round according to the optimized covariance matrix includes: The optimized covariance matrix replaces the covariance matrix of the current round of augmented training samples in the upper bound function; Taking the upper bound function after minimizing the replaced covariance matrix as the optimization objective, the model parameters of the current round of quadratic optimization are obtained; The image classification model to be optimized is updated according to the model parameters of the current round of secondary optimization.

7. The method for constructing an image classification model according to any one of claims 1 to 6, characterized in that: After performing secondary optimization on the image classification model to be optimized that was optimized for the first time in the current round according to the optimized covariance matrix, the method further includes: When the current iteration round reaches the preset number of times, or when the difference between the upper bound function value of the current iteration round and the upper bound function value of the previous iteration round is less than a preset threshold, the iteration is stopped; The image classification model to be optimized after the secondary optimization in the stopping iteration round is determined as the target image classification model.

8. A device for constructing an image classification model, characterized in that: include: The sample augmentation module is used to augment the original training samples to obtain the augmented training samples for the current round; The matrix acquisition module is used to obtain the covariance matrix of the training samples after the current round of augmentation; An upper bound function acquisition module, used to determine an upper bound function of a first loss function of the image classification model to be optimized after the previous round of secondary optimization on the training samples after the current round of augmentation according to the covariance matrix of the training samples after the current round of augmentation; A first optimization module is used to optimize the image classification model to be optimized by taking the upper bound function as the optimization target, and obtain the image classification model to be optimized in the first optimization of the current round; A matrix optimization module, used for optimizing the covariance matrix in the upper bound function using a validation set; A secondary optimization module, used for performing secondary optimization on the image classification model to be optimized that is first optimized in the current round according to the optimized covariance matrix; The sample augmentation module is specifically used for: Extracting feature data of the original training samples according to the image classification model to be optimized after the previous round of secondary optimization, and re-obtaining the covariance matrix of the original training samples according to the feature data of the original training samples; Performing statistics on the characteristic distribution in the covariance matrix to obtain distribution conditions; Sampling the covariance matrix according to the distribution to generate a conversion vector; The feature data is enhanced according to the conversion vector to generate new feature data to augment the original training sample.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it implements all or part of the steps of the method for constructing an image classification model as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, all or part of the steps of the method for constructing an image classification model as described in any one of claims 1 to 7 are implemented.

Images