Model training method, device, electronic device and medium based on network data

Abstract

The present invention is applicable to the field of computer vision technology, and provides a model training method, device, electronic device and storage medium based on network data. The method comprises: obtaining the loss value and prediction value of each sample in the network data selected for training at the time, classifying the network data based on the loss value to obtain clean samples and noise samples, classifying the noise samples based on the prediction value to obtain open set noise samples and closed set noise samples, training a pre-trained model using the clean samples and the closed set noise samples, repeating the above steps until a trained model is obtained, and the trained model is used for fine-grained image classification tasks, thereby effectively suppressing the influence of noise labels, improving the utilization rate of network data, and further improving the performance of the model.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a model training method, device, electronic device and storage medium based on network data. Background Art

[0002] Fine-grained image analysis (FGIA) is a long-standing fundamental problem in various research fields of computer vision (CV). Fine-grained image classification has a wide range of research needs and application scenarios both in industry and academia. Fine-grained image classification aims to divide large categories into subcategories, such as distinguishing different brands of cars, different species of birds, and so on. Since these subcategories have large intra-class differences and small inter-class differences, this recognition task is more challenging. With the help of deep learning, fine-grained classification has achieved some success. But its success depends on a large amount of carefully labeled available data, which is expensive and time-consuming to obtain, and data labels can also be very complicated for inexperienced people.

[0003] Since learning with network data does not require manual annotation, this method has attracted widespread attention. Network data refers to a large amount of data obtained from the Internet, such as pictures obtained from search engines such as Baidu and Google. However, there is a major problem in using network data for fine-grained classification, namely, noise samples will affect classification performance. Among them, noise samples refer to data with incorrect labels or completely irrelevant samples. Due to errors in search engine indexing, the returned images may be incorrect. For example, if we use the keyword "bird" to search for pictures on the Internet, the search results will include some pictures that are not birds. If network data is directly used for fine-grained classification, performance will be degraded.

[0004] A lot of research has been done in the prior art to solve the problem of noise samples. However, the impact of noise samples is usually avoided by simply removing them. However, when removing noise samples, a large number of useful samples are also deleted, such as samples with incorrect labels. These useful samples that are mistakenly discarded are conducive to improving the robustness of the model. If used properly, they will improve the performance of the model. Summary of the invention

[0005] The purpose of the present invention is to provide a model training method, device, electronic device and storage medium based on network data, aiming to solve the problem of reduced model performance caused by directly discarding noise samples in network data in the prior art.

[0006] In one aspect, the present invention provides a model training method based on network data, the method comprising the following steps:

[0007] Obtain the loss value and prediction value of each sample in the network data selected for the current training, wherein the network data is obtained based on a fine-grained image classification task;

[0008] Classifying the network data based on the loss value to obtain clean samples and noise samples;

[0009] Classifying the noise samples based on the predicted values ​​to obtain open set noise samples and closed set noise samples;

[0010] Training the pre-trained model using the clean samples and the closed-set noise samples;

[0011] The above steps are repeated until a trained model is obtained, and the trained model is used for the fine-grained image classification task.

[0012] Preferably, the step of classifying the noise sample based on the predicted value comprises:

[0013] The noise samples are classified based on a standard deviation of the predicted values.

[0014] Preferably, the step of classifying the noise samples based on the standard deviation of the predicted values ​​comprises:

[0015] Determine whether the standard deviation of the prediction value of the current sample is greater than or equal to a preset standard deviation threshold, wherein the standard deviation threshold is the mean of the standard deviations of the prediction values ​​of all the clean samples;

[0016] If so, the current sample is determined to be a closed set noise sample; otherwise, the current sample is determined to be an open set noise sample.

[0017] Preferably, the step of training the pre-trained model using the clean samples and the closed set noise samples comprises:

[0018] The clean samples are used as labeled data sets, and the closed set noise samples are used as unlabeled data sets, and the model is trained using a semi-supervised learning method.

[0019] Preferably, the step of training the model using a semi-supervised learning method comprises:

[0020] The labeled data set and the unlabeled data set are mixed using the MixMatch algorithm to obtain a mixed labeled data set and an unlabeled data set;

[0021] The pre-training model is trained based on the mixed labeled data set and unlabeled data set.

[0022] Preferably, the step of training the pre-trained model based on the mixed labeled data set and unlabeled data set includes:

[0023] Performing label smoothing on the mixed labeled data set to obtain a label smoothed labeled data set;

[0024] The pre-training model is trained based on the label-smoothed labeled data set and the unlabeled data set.

[0025] Preferably, the loss function of the model is:

[0026] L=L x +λ u L u +L c

[0027] Among them, L represents the total loss, L x represents the loss of the labeled dataset with label smoothing, L u represents the loss of the mixed unlabeled dataset, L c represents the regularization term, λ u Represents a hyperparameter.

[0028] Preferably, the loss of the label-smoothed labeled data set is a cross entropy loss, which is calculated in the following way:

[0029]

[0030] Among them, L x represents the cross entropy loss of the labeled smoothed labeled data set, x′ represents the mixed labeled data set, x, p represents the labeled data and labels in x′ respectively, L CE represents the cross entropy loss, and f(x;θ) represents the model prediction result output by the network function with θ as a parameter.

[0031] Preferably, the loss of the mixed unlabeled data set is L2 loss, which is calculated in the following way:

[0032]

[0033] Among them, L u represents the L2 loss of the mixed unlabeled dataset, u′ represents the mixed unlabeled dataset, and u,q represent the unlabeled data and predicted labels in u′ respectively.

[0034] Preferably, the regularization term is:

[0035]

[0036] Among them, C represents the number of categories, pj represents the probability distribution of a certain category in the training samples, n represents the number of training samples in the training set, x i represents the i-th sample of the network data.

[0037] On the other hand, the present invention provides a model training device based on network data, the device comprising:

[0038] A data acquisition unit, used to acquire the loss value and the predicted value of each sample in the network data selected for the current training, wherein the network data is obtained based on a fine-grained image classification task;

[0039] A first classification unit, used for classifying the network data based on the loss value to obtain clean samples and noise samples;

[0040] A second classification unit is used to classify the noise samples based on the predicted values ​​to obtain open set noise samples and closed set noise samples; and

[0041] A model training unit is used to train a pre-trained model using the clean samples and the closed-set noise samples.

[0042] On the other hand, the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0043] On the other hand, the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0044] The present invention obtains the loss value and the prediction value of each sample in the network data selected for training at the time, classifies the network data based on the loss value to obtain clean samples and noise samples, classifies the noise samples based on the prediction value to obtain open-set noise samples and closed-set noise samples, trains a pre-trained model using the clean samples and the closed-set noise samples, repeats the above steps until a trained model is obtained, and the trained model is used for fine-grained image classification tasks, thereby effectively suppressing the influence of noise labels, improving the utilization rate of network data, and further improving the performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1A is a flow chart for implementing a model training method based on network data provided in Embodiment 1 of the present invention;

[0046] Figure 1B is an example diagram of the model training process provided in the first embodiment of the present invention;

[0047] Figure 2 This is a visualization effect diagram of the network data classification result provided by the second embodiment of the present invention;

[0048] Figure 3 is a schematic diagram of the structure of a model training device based on network data provided in Embodiment 3 of the present invention; and

[0049] Figure 4 It is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0051] The specific implementation of the present invention is described in detail below in conjunction with specific embodiments:

[0052] Embodiment 1:

[0053] Figure 1A The implementation process of the model training method based on network data provided in the first embodiment of the present invention is shown. For the convenience of explanation, only the part related to the embodiment of the present invention is shown, which is described in detail as follows:

[0054] In step S101, the loss value and predicted value of each sample in the network data selected for the current training are obtained.

[0055] In an embodiment of the present invention, network data can be obtained based on a fine-grained image classification task. The network data usually includes clean samples and noise samples. The noise samples may include closed-set noise samples and open-set noise samples. Among them, clean samples refer to correctly labeled samples, that is, valid data with correct labels; closed-set noise samples refer to incorrectly labeled samples, that is, valid data with incorrect labels, and their true category labels are included in the training set; open-set noise samples refer to completely irrelevant samples, and their true category labels are not included in the training label set. The above-mentioned fine-grained image classification tasks may include but are not limited to vehicle model classification tasks, animal or plant species or varieties classification tasks, or aircraft model classification tasks. Accordingly, the network data may be vehicle image data of various models, animal or plant image data of various types or varieties, aircraft images of various models, etc.

[0056] The model training goal is to use network data to train the classifier model f to achieve the correct classification of any input image. Specifically, the cross entropy loss L can be optimized. CE To update the parameters θ:

[0057]

[0058] Among them, a represents the learning rate, D represents the network data selected for one training, and x i represents the i-th sample in the network data D, y i Represents the sample x in D i The corresponding label, f(x i ; θ) represents the neural network function with θ as parameter.

[0059] Before obtaining the loss value and predicted value of each sample in the network data selected for the current training, the network data can be used to pre-train the model for several rounds to obtain a pre-trained model. Among them, the model is usually a deep neural network model. Considering that the pre-training process will lead to overfitting of noise labels, so that noise samples will also have smaller loss values, which will affect the selection of subsequent samples. Therefore, in the pre-training stage, a regularization term H can be added on the basis of the original cross entropy loss (Pereyra G, Tucker G, Chorowski J, et al. Regularizing neural networks by penalizing confident output distributions [J]. arXiv preprint arXiv: 1701.06548, 2017.). The regularization term H is the predicted entropy of the input x, that is:

[0060]

[0061] Where C represents the number of categories.

[0062] In step S102, the network data is classified based on the loss value of each sample to obtain clean samples and noise samples.

[0063] In an embodiment of the present invention, after obtaining the pre-trained model, sample selection and classification are started. The learning speed of clean samples based on deep neural networks is often faster than the learning speed of noisy samples, so that the loss of clean samples is smaller. Therefore, the loss-based method proposed by Eran Malach et al. can be used to select samples (Malach E, Shalev-Shwartz S. Decoupling "when to update" from "how to update" [J]. arXiv preprint arXiv: 1706.02613, 2017). Specifically, samples with small cross entropy loss can be selected as clean samples, and samples with large cross entropy loss can be selected as noise samples. Clean sample D c It can be calculated as follows:

[0064]

[0065]

[0066] Among them, D c represents a clean sample in D, x i represents the i-th sample, y i Represents x i The corresponding label, r represents the sample discard rate, f(x i ; θ) represents the prediction result output by the network function with θ as parameter, L CE represents the cross entropy loss of the model, p i Represents the probability prediction value of the i-th sample, y i represents the label of the i-th sample, and n represents the number of training samples.

[0067] Then, using formula D n =DD c The noise sample can be obtained, where D n represents a noise sample.

[0068] In step S103, the noise samples are classified based on the predicted values ​​of the noise samples to obtain open set noise samples and closed set noise samples.

[0069] In an embodiment of the present invention, open set noise and closed set noise are distinguished according to the predicted value of the sample. Since the true category label of the closed set noise is included in the training label set, and the true category label of the open set noise is not included in the training label set, the probability prediction value of the closed set noise will have a maximum value, while the probability prediction value of the open set noise is closer. If the probability of a certain label is close to other labels, it means that the sample is an irrelevant open set noise sample. Preferably, the noise samples are classified based on the standard deviation of the predicted value, so as to classify the noise samples based on the standard deviation calculation result. Further, it is determined whether the standard deviation of the predicted value of the current sample is greater than or equal to a preset standard deviation threshold, wherein the standard deviation threshold is the mean of the standard deviation of the predicted values ​​of all clean samples. If so, it is determined that the current sample belongs to a closed set noise sample, otherwise, it is determined that the current sample belongs to an open set noise sample, thereby achieving the classification of noise samples.

[0070] Specifically, the predicted value of the open set noise is distributed smoothly, the predicted value is close to the mean, and the standard deviation is small. The predicted value of the closed set noise will have a peak to represent the true label of the image, and its standard deviation will be large. If the standard deviation of the predicted value of the current noise sample is When , the current noise sample is determined to be a closed set noise sample. Among them, D Crepresents the clean samples in the network data selected for training, C represents the number of categories, μ represents the average value of the probability prediction value of all samples, P j Represents the probability distribution of a certain category in the training samples.

[0071] After the open-set noise is obtained, it can be directly discarded.

[0072] In step S104, the pre-training model is trained using clean samples and closed-set noise samples.

[0073] In an embodiment of the present invention, clean samples and closed-set noise samples can be used as training sets to train the pre-trained model. Preferably, the clean samples are used as labeled data sets, and the closed-set noise samples are used as unlabeled data sets, and the pre-trained model is trained using a semi-supervised learning method to improve the model training effect. Specifically, for labeled data sets, the labels of its samples are kept unchanged; for unlabeled data sets, the model is used to predict its labels. At the same time, a sharpening function is applied to reduce the entropy of the label distribution (Berthelot D, Carlini N, Goodfellow I, et al. MixMatch: Aholistic approach to semi-supervised learning [J]. arXiv preprint arXiv: 1905.02249, 2019.), and the predicted label of the unlabeled data after the sharpening function is Among them, q b represents the probability prediction value of the sample, C represents the number of categories, and T is a hyperparameter used to adjust the classification entropy.

[0074] When the pre-trained model is trained using a semi-supervised learning method, preferably, the MixMatch algorithm is used to mix the labeled data set and the unlabeled data set to obtain a mixed labeled data set and an unlabeled data set, and the pre-trained model is trained based on the mixed labeled data set and the unlabeled data set to further improve the network training effect.

[0075] Specifically, for a pair of samples (x1, x2) and their corresponding labels (p1, p2), the calculation method of the mixed (x′, y′) is:

[0076] λ~Beta(α,α)

[0077] λ′=max(λ,1-λ)

[0078] x′=λ′x1+(1-λ′)x2

[0079] p′=λ′p1+(1-λ′)p2

[0080] Where α is the Beta distribution hyperparameter and λ is the mixing coefficient calculated from the Beta distribution with parameter α.

[0081] When training a pre-trained model based on the mixed labeled dataset and unlabeled dataset, preferably, label smoothing is performed on the mixed labeled dataset to obtain a label-smoothed labeled dataset, and the pre-trained model is trained based on the label-smoothed labeled dataset and unlabeled dataset to improve the generalization performance of the model.

[0082] The true probability distribution after adding label smoothing is as follows:

[0083]

[0084] Where ε is a hyperparameter used to reduce the confidence of the model, C is the number of categories, i represents a category among multiple categories, and y represents a fixed category.

[0085] During the model training process, the loss function of the model can be calculated based on the loss of the mixed labeled data set and the loss of the unlabeled data set. Preferably, a regularization term is introduced in the loss function of the model to avoid assigning all labels to a single class.

[0086] Specifically, the loss function of the model is:

[0087] L=L x +λ u L u +L c

[0088] Among them, L represents the total loss, L x represents the loss of the labeled dataset with label smoothing, L u represents the loss of the mixed unlabeled dataset, L c represents the regularization term, λ u Represents a hyperparameter.

[0089] Preferably, the loss of the labeled dataset with label smoothing is the cross entropy loss, which is calculated using the following formula:

[0090]

[0091] Among them, L x represents the cross entropy loss of the labeled data set with smoothed labels, x′ represents the mixed labeled data set, x,p represents the labeled data and labels in x′ respectively, and L CE represents the cross entropy loss, and f(x;θ) represents the model prediction result output by the network function with θ as a parameter.

[0092] L CE It can be specifically expressed as follows:

[0093]

[0094] Preferably, the loss of the mixed unlabeled dataset is L2 loss, which is calculated as follows:

[0095]

[0096] Among them, L u represents the L2 loss of the mixed unlabeled dataset, u′ represents the mixed unlabeled dataset, and u,q represent the unlabeled data and predicted labels in u′ respectively. The predicted label is the one mentioned above. The prediction method is obtained.

[0097] Preferably, the regularization term is:

[0098]

[0099] Among them, C represents the number of categories, p j represents the probability distribution of a certain category in the training samples, n represents the number of training samples, x i Represents the i-th sample of network data.

[0100] Figure 1B This is an example diagram of the model training process. In the model training process, steps S101-104 are repeated until a trained model is obtained. For example, when the number of model training times reaches a preset value, or the loss value of the model converges, it is determined that a trained model is obtained. The trained model is used for fine-grained image classification tasks. Specifically, based on the trained model, the input image to be classified is classified and the classification result is output.

[0101] In an embodiment of the present invention, the loss value and the predicted value of each sample in the network data selected for the current training are obtained, the network data is classified based on the loss value to obtain clean samples and noise samples, the noise samples are classified based on the predicted value to obtain open-set noise samples and closed-set noise samples, the clean samples and the closed-set noise samples are used to train the pre-trained model, and the above steps are repeated until a trained model is obtained. The trained model is used for fine-grained image classification tasks, thereby effectively suppressing the influence of noise labels, improving the utilization rate of network data, and further improving the performance of the model.

[0102] Embodiment 2:

[0103] This embodiment further illustrates the method described in the first embodiment in conjunction with an experimental example.

[0104] (1) Dataset used in this experiment

[0105] This experimental example was conducted on three commonly used fine-grained datasets, CUB200 (Wah C, Branson S, Welinder P, et al. The caltech-ucsd birds-200-2011 dataset [J]. 2011.), FGVC-Aircraf (Maji S, Rahtu E, Kannala J, et al. Fine-grained visual classification of aircraft [J]. arXiv preprint arXiv: 1306.5151, 2013.) and Cars-196 (Krause J, Stark M, Deng J, et al. 3d object representations for fine-grained categorization [C] / / Proceedings of the IEEE international conference on computer vision workshops. 2013: 554-561.).

[0106] CUB200 (Wah C, Branson S, Welinder P, et al. The caltech-ucsd birds-200-2011 dataset [J]. 2011.) contains 200 bird subcategories, with 5994 images in the training dataset and 5794 images in the test set, and each image provides image class labeling information. FGVC-Aircraft (Maji S, Rahtu E, Kannala J, et al. Fine-grained visual classification of aircraft [J]. arXiv preprint arXiv: 1306.5151, 2013.) contains 10,000 images of 100 aircraft model variants, with 6667 images for training and 3333 images for testing. Cars-196 (Krause J, Stark M, Deng J, et al. 3D object representations for fine-grained categorization [C] / / Proceedings of the IEEE international conference on computer vision workshops. 2013: 554-561.) contains 196 subcategories of cars, a total of 16,185 images, of which 8,144 are used for training and 8,041 are used for testing.

[0107] The network dataset used for training in this experimental example adopts the dataset published in the paper by Chuanyi Zhang et al. (Zhang C, Yao Y, Liu H, et al. Web-supervised network with softly update-drop training for fine-grained visual classification [C] / / Proceedings of the AAAIConference on Artificial Intelligence. 2020, 34 (07): 12781-12788.). This dataset uses the label categories in the benchmark dataset to retrieve images from the image search engine. For the network bird set, a total of 18,388 images were collected for training, and 5,794 test images from the original benchmark dataset CUB200 were used as the test set. For the network aircraft dataset, a total of 13,503 images were collected for training, and 3,333 images from the test set in the original benchmark dataset FGVC-aircraft were used for testing. For the network car set, a total of 21,448 images were collected for training, and a total of 8,041 images from the test set in the original benchmark dataset Cars-196 were used for testing.

[0108] (2) Experimental description

[0109] This experimental example uses the pre-trained model VGG-16 (Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition [J]. arXiv preprint arXiv: 1409.1556, 2014.) as the backbone network and adopts a two-stage training strategy. Specifically, the convolution layer parameters are first frozen, and only the last fully connected layer is optimized in the first step. Then in the second step, all layers in the previously learned model are optimized. The drop rate r is selected from {0.20, 0.25, 0.30, 0.35}. According to the experimental results, this experimental example selects r values ​​of 0.30, 0.20 and 0.20 for the CUB200, FGVC-Aircraft and Cars-196 datasets respectively, and sets the hyperparameters T = 0.5, α = 0.75, λ u= 100, and set the hyperparameter ∈ = 0.5 for label smoothing. In the experiment, the Adam optimizer (Kingma DP, Ba J.Adam:A method for stochastic optimization[J].arXiv preprint arXiv:1412.6980,2014.) is used, and the momentum is set to 0.9. In the first step, the learning rate, batch size, and number of training rounds are set to 0.001, 64, and 80, respectively, and in the second step, they are set to 0.0001, 32, and 100, respectively. This experimental example first uses all the data for pre-training for 5 rounds, and then decays the learning rate by cosine annealing.

[0110] (3) Experimental results

[0111] In order to evaluate the effectiveness of the method described in Example 1 (hereinafter referred to as the method, which includes two-stage sample selection and semi-supervised training), this experimental example conducts a fine-grained classification test based on network data in the above-mentioned dataset, and conducts comparative experiments with the current mainstream methods for solving noisy labels, including Decoupling (Malach E, Shalev-Shwartz S. Decoupling "when to update" from "how to update" [J]. arXiv preprint arXiv: 1706.02613, 2017.), Co-teaching (Han B, Yao Q, Yu X, et al. Co-teaching: Robust training of deep neural networks with extremely noisy labels [J]. arXiv preprint arXiv: 1804.06872, 2018.), FGVC (Zhang C, Yao Y, Liu H, et al. Web-supervised network with softly update-drop training for fine-grained visual classification [C] / / Proceedings of the AAAI Conference on ArtificialIntelligence.2020,34(07):12781-12788.). The experimental results are shown in Table 1, which shows that this method has achieved better performance than other methods on different data sets. For the network supervision method, the same backbone network and the same training data are used. By observing Table 1, it can be noted that this method performs better than other network supervision methods on the three benchmark data sets. The experimental results show that this method can effectively alleviate the impact of noisy labels during model training. Compared with the method using manually annotated data, the method using only network data still has a lot of room for improvement.

[0112]

[0113] Table 1

[0114] In addition, in order to verify the effectiveness of this method, this experimental example uses this method to conduct experiments on different backbone networks to study the influence of different backbone networks. The experimental results are shown in Table 2. This experimental example is experimented on ResNet-18 (He K, Zhang X, Ren S, et al. Deep residual learning for image recognition [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 770-778.) and ResNet-50 (He K, Zhang X, Ren S, et al. Deep residual learning for image recognition [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 770-778.). For ResNet-18 and ResNet-50, this experimental example uses one-step training. The experimental results are shown in Table 2. As can be seen from Table 2, this method has a significant improvement over the standard network on the three benchmark datasets, which illustrates the applicability of this method. It can also be observed in this experimental example that using a deeper backbone network can achieve better performance, but it requires more computing power accordingly. For the FGVC-Aircraft dataset, the accuracy of this method is very small, which may be due to the fact that the FGVC-Aircraft dataset contains fewer noisy labels.

[0115]

[0116] Table 2

[0117] In order to prove the effectiveness of each step of this method, this experimental example conducted ablation experiments on different steps. In this experiment, ResNet-18 was selected as the backbone network and the experiment was conducted on the CUB200 dataset. First, the experiment was conducted only on the standard network, and then the method steps described in Example 1 were gradually added. The first step is to add the first-stage sample selection, which is used to divide the data into clean samples and noise samples; then add the second-stage sample selection, which is used to distinguish closed-set noise from open-set noise; finally, add the semi-supervised method as the last step, and train the clean data of the first step as labeled data and the closed-set noise of the second step as unlabeled data. The experimental results are shown in Table 3. As can be seen from Table 3, the experimental accuracy rate increased by 4.65% after adding the first-stage sample selection, and the experimental accuracy rate increased by 2.03% after adding the second-stage sample selection, which shows that two-stage sample selection can effectively utilize data. By adding the semi-supervised method, the accuracy rate increased by 2.18%, indicating that adding a semi-supervised method to train the network can effectively improve the performance of the model.

[0118]

[0119] Table 3

[0120] In order to visually observe the results of data separation, this experiment visualizes the sample selection results on the online bird set. Figure 2 As shown. Figure 2 It can be seen that this method separates three types of data. Clean samples are simple data used directly to train the network. Open set noise is also valid data, but its label is wrong. This experiment will use these samples correctly to improve the accuracy of the model. Among them, open set noise includes buildings, text, etc. These are completely irrelevant data and will affect the classification performance of the model. This experiment will directly discard these open set noise.

[0121] From the above experiments, it can be seen that after two stages of sample screening, the method described in Example 1 can effectively suppress the influence of noise labels and improve the utilization of network data. A large number of comparative experiments and ablation experiments were conducted on three benchmark fine-grained datasets, and the experimental results show the superiority of the method described in Example 1.

[0122] Embodiment three:

[0123] Figure 3 The structure of the model training device based on network data provided by the third embodiment of the present invention is shown. For the convenience of description, only the part related to the embodiment of the present invention is shown, including:

[0124] A data acquisition unit 31 is used to acquire the loss value and the prediction value of each sample in the network data selected for the current training, where the network data is obtained based on a fine-grained image classification task;

[0125] A first classification unit 32, used to classify the network data based on the loss value to obtain clean samples and noise samples;

[0126] A second classification unit 33 is used to classify the noise samples based on the predicted values ​​to obtain open set noise samples and closed set noise samples; and

[0127] The model training unit 34 is used to train the pre-trained model using clean samples and closed-set noise samples.

[0128] Preferably, the second classification unit includes:

[0129] The second classification subunit is used to classify the noise samples based on the standard deviation of the predicted values.

[0130] Preferably, the second classification subunit includes:

[0131] The standard deviation judgment unit is used to judge whether the standard deviation of the predicted value of the current sample is greater than or equal to a preset standard deviation threshold. If so, the current sample is judged to be a closed set noise sample; otherwise, the current sample is judged to be an open set noise sample, wherein the standard deviation threshold is the mean of the standard deviations of the predicted values ​​of all clean samples.

[0132] Preferably, the model training unit includes:

[0133] The first training subunit is used to use the clean samples as labeled data sets and the closed set noise samples as unlabeled data sets, and train the pre-trained model using a semi-supervised learning method.

[0134] Preferably, the first training subunit comprises:

[0135] A data mixing unit, used to mix the labeled data set and the unlabeled data set using the MixMatch algorithm to obtain a mixed labeled data set and an unlabeled data set; and

[0136] The second training subunit is used to train the pre-training model based on the mixed labeled data set and the unlabeled data set.

[0137] Preferably, the second training subunit comprises:

[0138] a label smoothing unit, used to perform label smoothing on the mixed labeled data set to obtain a label smoothed labeled data set; and

[0139] The third training subunit is used to train the pre-training model based on the label-smoothed labeled data set and the unlabeled data set.

[0140] In the embodiment of the present invention, each unit of the network data-based model training device can be implemented by a corresponding hardware or software unit, and each unit can be an independent software or hardware unit, or can be integrated into a software or hardware unit, which is not intended to limit the present invention. The specific implementation of each unit of the network data-based model training device can refer to the description of the aforementioned method embodiment, which will not be repeated here.

[0141] Embodiment 4:

[0142] Figure 4 The structure of an electronic device provided by a fourth embodiment of the present invention is shown. For the convenience of description, only the parts related to the embodiment of the present invention are shown.

[0143] The electronic device 4 of the embodiment of the present invention includes a processor 40, a memory 41, and a computer program 42 stored in the memory 41 and executable on the processor 40. When the processor 40 executes the computer program 42, the steps in the above-mentioned method embodiments are implemented, for example: Figure 1A Alternatively, when the processor 40 executes the computer program 42, the functions of each unit in the above-mentioned device embodiments are realized, for example Figure 3 The functions of the units 31 to 34 are shown.

[0144] In an embodiment of the present invention, the loss value and the predicted value of each sample in the network data selected for the current training are obtained, the network data is classified based on the loss value to obtain clean samples and noise samples, the noise samples are classified based on the predicted value to obtain open-set noise samples and closed-set noise samples, the clean samples and the closed-set noise samples are used to train the pre-trained model, and the above steps are repeated until a trained model is obtained. The trained model is used for fine-grained image classification tasks, thereby effectively suppressing the influence of noise labels, improving the utilization rate of network data, and further improving the performance of the model.

[0145] Embodiment five:

[0146] In an embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above method embodiment are implemented, for example, Figure 1A Alternatively, when the computer program is executed by a processor, the functions of each unit in the above-mentioned device embodiments are realized, for example Figure 3 The functions of the units 31 to 34 are shown.

[0147] In an embodiment of the present invention, the loss value and the predicted value of each sample in the network data selected for the current training are obtained, the network data is classified based on the loss value to obtain clean samples and noise samples, the noise samples are classified based on the predicted value to obtain open-set noise samples and closed-set noise samples, the clean samples and the closed-set noise samples are used to train the pre-trained model, and the above steps are repeated until a trained model is obtained. The trained model is used for fine-grained image classification tasks, thereby effectively suppressing the influence of noise labels, improving the utilization rate of network data, and further improving the performance of the model.

[0148] The computer-readable storage medium of the embodiment of the present invention may include any entity or device or recording medium capable of carrying computer program code, for example, ROM / RAM, magnetic disk, optical disk, flash memory and other memories.

[0149] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A model training method based on network data, characterized in that: The method comprises the following steps: Obtain the loss value and prediction value of each sample in the network data selected for the current training, wherein the network data is obtained based on a fine-grained image classification task; Classifying the network data based on the loss value to obtain clean samples and noise samples; Classifying the noise samples based on the predicted values ​​to obtain open set noise samples and closed set noise samples, wherein classifying the noise samples based on the predicted values ​​includes: determining whether the standard deviation of the predicted value of the current sample is greater than or equal to a preset standard deviation threshold, wherein the standard deviation threshold is the mean of the standard deviations of the predicted values ​​of all the clean samples, and if so, determining that the current sample belongs to a closed set noise sample, otherwise, determining that the current sample belongs to an open set noise sample; The pre-trained model is trained using the clean samples and the closed-set noise samples, wherein the step of training the pre-trained model using the clean samples and the closed-set noise samples comprises: using the clean samples as labeled data sets and the closed-set noise samples as unlabeled data sets, and training the model using a semi-supervised learning method; The above steps are repeated until a trained model is obtained, and the trained model is used for the fine-grained image classification task.

2. The method according to claim 1, characterized in that The step of training the model using a semi-supervised learning method comprises: The labeled data set and the unlabeled data set are mixed using the MixMatch algorithm to obtain a mixed labeled data set and an unlabeled data set; The pre-training model is trained based on the mixed labeled data set and unlabeled data set.

3. The method according to claim 2, characterized in that The step of training the pre-trained model based on the mixed labeled data set and the unlabeled data set includes: Performing label smoothing on the mixed labeled data set to obtain a label smoothed labeled data set; The pre-training model is trained based on the label-smoothed labeled data set and the unlabeled data set.

4. The method according to claim 3, characterized in that The loss function of the model is: L=L x +λ u L u +L c Among them, L represents the total loss, L x represents the loss of the labeled dataset with label smoothing, L u represents the loss of the mixed unlabeled dataset, L c represents the regularization term, λ u Represents a hyperparameter.

5. The method according to claim 4, characterized in that The loss of the labeled smoothed labeled dataset is the cross entropy loss, which is calculated as follows: Among them, L x represents the cross entropy loss of the labeled smoothed labeled data set, x′ represents the mixed labeled data set, x, p represents the labeled data and labels in x′ respectively, L CE represents the cross entropy loss, f(x;θ) represents the model prediction result output by the network function with θ as parameter; The loss of the mixed unlabeled dataset is L2 loss, which is calculated as follows: Among them, L u represents the L2 loss of the mixed unlabeled data set, u′ represents the mixed unlabeled data set, and u, q represent the unlabeled data and predicted labels in u′ respectively; The regularization term is: Among them, C represents the number of categories, p j represents the probability distribution of a certain category in the training samples, n represents the number of training samples in the training set, x i represents the i-th sample of the network data.

6. A model training device based on network data, characterized in that: The device comprises: A data acquisition unit, used to acquire the loss value and the predicted value of each sample in the network data selected for the current training, wherein the network data is obtained based on a fine-grained image classification task; A first classification unit, used for classifying the network data based on the loss value to obtain clean samples and noise samples; A second classification unit is used to classify the noise samples based on the predicted values ​​to obtain open set noise samples and closed set noise samples. When the second classification unit classifies the noise samples based on the predicted values, it includes: judging whether the standard deviation of the predicted value of the current sample is greater than or equal to a preset standard deviation threshold, wherein the standard deviation threshold is the mean of the standard deviations of the predicted values ​​of all the clean samples. If so, it is judged that the current sample belongs to the closed set noise sample; otherwise, it is judged that the current sample belongs to the open set noise sample; and A model training unit is used to train a pre-trained model using the clean samples and the closed-set noise samples. When the pre-trained model is trained using the clean samples and the closed-set noise samples, it includes: taking the clean samples as labeled data sets, taking the closed-set noise samples as unlabeled data sets, and training the model using a semi-supervised learning method.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images