An image classification-oriented internet dataset label noise proportion evaluation method
By training an M-class classifier to measure the accuracy difference on clean and noisy test sets, the noise ratio of internet datasets is automatically evaluated, solving the problem of time-consuming and labor-intensive manual cleaning in existing technologies and achieving efficient label noise evaluation.
Patent Information
- Application Number
- CN202310305967.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-27
- Publication Date
- 2026-02-06
- Estimated Expiration
- 2043-03-27
AI Technical Summary
In existing technologies, label noise assessment for large-scale image datasets relies on manual cleaning, which is time-consuming, labor-intensive, and difficult to automate.
By training an M-class classifier, the noise ratio of an internet dataset is automatically estimated by utilizing the difference in classification accuracy of the classifier on clean and noisy test sets.
It enables automatic evaluation of label noise ratio in the construction of large-scale Internet datasets, eliminating the reliance on manual cleaning and improving efficiency and accuracy.
Smart Images
Figure CN116342946B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of image dataset construction, and in particular to an internet dataset label noise proportion evaluation method for image classification. BACKGROUND
[0002] Image dataset construction is a crucial link in computer vision tasks, and obtaining large-scale clean image datasets is necessary for subsequent research work. Obtaining a large number of picture data through the Internet is an important way to construct large-scale image datasets, but the obtained image data will contain certain noise data. Generally, the cleaning work of the internet dataset is completed by manual methods. In the context of large-scale dataset construction, such cleaning methods are time-consuming and labor-intensive, which is an important reason for the lack of large-scale image datasets. At present, some automatic noise data detection methods have emerged, but these methods still need manual cleaning of a part of the noise data to obtain the noise proportion.
[0003] The automatic internet dataset label noise proportion evaluation method liberates manpower and breaks through the limitation of the existing label noise proportion evaluation method which relies on manual data cleaning. The label noise evaluation method based on manual data cleaning brings the following challenges to large-scale image dataset construction: long cleaning period; large amount of manpower; and dependence on human judgment. SUMMARY
[0004] In view of the deficiencies in the prior art, a first object of the present application is to provide an internet dataset label noise proportion evaluation method for image classification. The present application breaks through the limitations of the existing manual noise evaluation method in the context of large-scale internet dataset noise proportion evaluation, trains a model on a noisy training set, and uses the model to make class predictions on a clean test set and a noisy test set, respectively. The accuracy difference value caused by the noise is used to realize automatic noise proportion estimation.
[0005] To solve the above technical problems, the present application realizes the following technical scheme:
[0006] An internet dataset label noise proportion evaluation method for image classification, characterized in that it comprises the following steps:
[0007] S1: Obtain an internet picture dataset Web-M, and divide it into a noisy training set Train n and a noisy test set Test n ;
[0008] S2: Obtain a clean test set Test c containing the same categories as the dataset Web-M;
[0009] S3: Train an M-class classifier with the noisy training set Train n Train an M-class classifier with the noisy training set Train
[0010] S4: Test the M-class classifier on Test c Test the M-class classifier on Test c
[0011] S5: Test the M-class classifier on Test n Test the M-class classifier on Test n
[0012] S6: Compare the accuracy of the M-class classifier on Test c and Test n , and obtain the estimate of the noise ratio Ratio n of the Web-M dataset
[0013] Further, in step S1,
[0014] The dataset Web-M used can be any large-scale noisy Internet dataset, where M is the number of classes; the classes of the dataset Web-M are divided into a noisy training set Train n and a noisy test set Test n , where M is the number of classes, the noisy labels are not cleaned, and the division of data of each class is strictly according to the label;
[0015] Further, in step S2,
[0016] Obtain a clean test set Test c with the same classes as the dataset Web-M; Test c The clean test set can come from an existing small-scale clean dataset, or be constructed by manually cleaning a small amount of network data; Test c The labels of each image data in the test set can be regarded as its true class, and the noise ratio is close to zero, so the test set can be regarded as a 100% clean dataset.
[0017] Further, in step S3,
[0018] Train an M-class classifier for M classes according to the noisy training set Train n obtained in step S1; since the training set is noisy, the model training method on a general noisy dataset needs to be used in the training process to avoid overfitting of the model to the noisy data.
[0019] Furthermore: In step S4,
[0020] Based on the M-class classifier trained in step S3, on the clean test set Test c The model inference is performed; the given label is compared with the predicted category given by the M-class classifier, and finally the M-class classifier's performance on the clean test set Test is obtained. c Classification accuracy Acc c ; this accuracy rate Acc c This is considered as the classification accuracy of the M-class classifier on a typical clean test set.
[0021] Furthermore: In step S5,
[0022] Based on the M-class classifier trained in step S3, on the noisy test set Test n The model inference is performed on the test set Test; the predicted categories of the given noise labels are compared with those of the M-class classifier, and finally the performance of the M-class classifier on the noisy test set Test is obtained. n Classification accuracy Acc n The noise test set Test n Corresponding to a clean test set The M-class classifier in Test n Corresponding clean test set The classification accuracy on the Test set is approximately the same as that on the clean test set. c Classification accuracy Acc c .
[0023] Furthermore: In step S6,
[0024] Based on the M-class classifier obtained in steps S4 and S5, on the clean test set Test c Classification accuracy Acc c and in the noisy test set Test n Classification accuracy Acc n The noise ratio of the Web-M network dataset is obtained. n The estimated value
[0025] M-class classifier on clean test set Test c With noisy test set Test n Classification accuracy Acc c Acc n The difference between them is Δ acc , where Δ acc =Acc c -Acc n ;
[0026] Based on Acc c and Acc n , the noise ratio Ratio n of the Web-M network data set can be further obtained wherein
[0027] A second object of the present application is to provide an electronic device, characterized by comprising:
[0028] one or more processors;
[0029] a storage device for storing one or more programs,
[0030] when the one or more programs are executed by the one or more processors, so that the one or more processors implement the image classification-oriented internet data set label noise ratio evaluation method as described in any one of the above.
[0031] A third object of the present application is to provide a computer readable medium having stored thereon a computer program, characterized in that the program, when executed by a processor, implements the image classification-oriented internet data set label noise ratio evaluation method as described in any one of the above.
[0032] Compared with the prior art, the present application has the following advantages and beneficial effects:
[0033] The present application combines noise model training, test model classification accuracy, and comparison of noise test set and clean test set classification accuracy, wherein the comparison of noise test set and clean test set classification accuracy makes the noise ratio evaluation tend to be automated and independent of manual cleaning methods. The present application provides a solution to the challenges of long cleaning cycle, large amount of manpower, and dependence on human judgment of manual noise evaluation method, and realizes an automatic internet data set label noise ratio evaluation method. Specifically, comparing the classification accuracy of the model on the noise test set and the clean test set fully utilizes the differences of the model on the two test sets, and utilizes the connection between the differences and the noise ratio to realize noise ratio evaluation. The automatic noise ratio evaluation method replaces manual discrimination with model classification prediction. In addition, the accuracy of the model on the existing clean test set is used to approximate the accuracy that can be obtained by the cleaned noise test set. Finally, the connection between the classification accuracy difference of the model on the noise test set before and after cleaning and the noise ratio is utilized to realize automatic label noise ratio evaluation. The present application is independent of manual cleaning methods and can realize automatic label noise ratio evaluation in large-scale internet data set construction scenarios. BRIEF DESCRIPTION OF DRAWINGS
[0034] Figure 1 is a general framework diagram of the method of the present application;
[0035] Figure 2 Flowchart of the method of the present application. DETAILED DESCRIPTION
[0036] In order for those skilled in the art to better understand the technical solutions of the present application, the preferred embodiments of the present application will be described below in combination with specific examples, but it should be understood that the drawings are only used for illustrative description and cannot be understood as a limitation on the present application; in order to better illustrate the present embodiment, some components in the drawings will be omitted, enlarged or reduced, and do not represent the actual size of the product; it is understandable for those skilled in the art that some well-known structures and their descriptions in the drawings can be omitted. The positional relationship described in the drawings is only used for illustrative description and cannot be understood as a limitation on the present application.
[0037] The present application will be further described below in combination with the drawings and examples, but it should not be used as a basis for limiting the present application.
[0038] As shown in Figure 1 , Figure 2 , the present application provides an image classification-oriented internet dataset label noise proportion evaluation method, which is realized based on comparison of classification accuracy on clean test set and noise test set, and the method comprises model training, model inference on clean test set, model inference on noise test set, classification accuracy comparison, and noise proportion estimation; for details, please refer to Figure 1 , which is an automatic label noise proportion evaluation module.
[0039] The method comprises the following steps:
[0040] S1: Obtain an internet picture dataset Web-M, and divide it into a noisy training set Train n and a noisy test set Test n ;
[0041] In step S1,
[0042] The dataset Web-M used can be any large-scale noisy internet dataset, wherein M is the number of categories. The dataset Web-M is divided into a noisy training set Train n and a noisy test set Test n according to a certain proportion, wherein M is the number of categories, the noisy labels are not cleaned, and the division of data of each category is strictly performed according to the labels. The implementation of the existing artificial noise proportion estimation method is to select a part of noisy data, clean the part of data artificially, obtain the noise proportion, and take the noise proportion of the part of data as the noise proportion of the original dataset. As can be seen, the estimation of the noise proportion of the dataset Web-M can be realized by estimating the noise proportion of the noisy test set Test n .
[0043] S2: Obtain a clean test set Test containing the same categories as the dataset Web-M c ;
[0044] In step S2,
[0045] Obtain a clean test set Test containing the same categories as the dataset Web-M c . Test c The clean test set can be constructed by selecting data from an existing clean dataset, or by manually cleaning a small amount of noisy web data. Since there is no scale requirement for the clean test set Test c , it will not consume a large amount of manual work even if it is obtained by manual cleaning. Of course, if the clean test set Test c can be obtained directly from an existing clean dataset, the size of the dataset should be as large as possible. Test c The labels of each image data in the test set can be regarded as its true category, and the noise ratio is almost zero, so the test set is regarded as a 100% clean dataset.
[0046] S3: Train an M-class classifier with the noisy training set Train n as training data;
[0047] In step S3,
[0048] According to the noisy training set Train n obtained in step S1, train an M-class classifier for M categories. Since the training set is noisy, ordinary training methods may cause the classifier to overfit to the noise label, and a model training method on a noisy dataset is needed to avoid model overfitting. The model is not overfit to the noise data, and the classification accuracy on the clean test set will be more reasonable.
[0049] S4: The M-class classifier trained in step S3 is used for model inference on the test set Test c , and the classification accuracy Acc c is measured;
[0050] In step S4,
[0051] According to the M-class classifier trained in step S3, the model is inferred on the clean test set Test c . By comparing the given label of each image data in the test set with the predicted category given by the M-class classifier, the classification accuracy Acc c of the M-class classifier on the clean test set Test c is finally obtained. The accuracy Acc cIt can be used as the classification accuracy of the M-class classifier on a typical clean test set.
[0052] S5: The trained M-class classifier performs well in Test. n The model inference was performed on the test set, and the classification accuracy Acc was measured. n ;
[0053] In step S5,
[0054] Using the M-class classifier trained in step S3, on the noisy test set Test... n Model inference is performed on the test set. The given label of each image in the test set is compared with the predicted category given by the M-class classifier. Finally, the performance of the M-class classifier on the noisy test set Test is determined. n Classification accuracy Acc n The noise test set Test n Corresponding to a clean test set To obtain the M-class classifier in Test n Corresponding clean test set The classification accuracy is obtained without manually cleaning the data. This M-class classifier can be used on a clean test set Test c Classification accuracy Acc c To approximate the substitution.
[0055] S6: Compare the accuracy (Acc) of the M-class classifier on the two test sets. c With Acc n The noise ratio of the Web-M network dataset is obtained. n The estimated value
[0056] In step S6,
[0057] Based on the M-class classifier obtained in steps S4 and S5, on the clean test set Test c Classification accuracy Acc c and in the noisy test set Test n Classification accuracy Acc n The noise ratio of the Web-M network dataset is obtained. n The estimated value
[0058] M-class classifier on clean test set Test c With noisy test set Test n Classification accuracy Acc c Acc n The difference between them is Δacc wherein:
[0059] Δ acc = Acc c - Acc n .
[0060] Based on Acc c and Acc n , the estimation of the noise ratio Ratio n of the Web-M network dataset can be further obtained wherein:
[0061]
[0062] The theoretical analysis and demonstration of the formula are as follows:
[0063] The noise ratio evaluation method is based on two reasonable premises: first, the noise ratio of the Web-M network dataset can be approximately replaced by the noise ratio of the noisy test set Test n ; second, the classification accuracy of the M-class classifier on the corresponding clean test set of the noisy test set Test n can be approximately replaced by the classification accuracy Acc c of the M-class classifier on the clean test set Test c .
[0064] Based on the above two premises, it is only necessary to demonstrate the relationship among the noise ratio of the noisy test set Test n the classification accuracy Acc c of the M-class classifier on the corresponding clean test set of the noisy test set Test n , and the classification accuracy Acc n of the M-class classifier on the noisy test set Test n . That is, it is necessary to demonstrate the relationship among the data amount PT n in the noisy test set Test n that is judged as correct by the M-class classifier, the data amount in the clean test set that is judged as correct by the M-class classifier, and the data amount Noise of the noise in the noisy test set Test n . Let the total data amount of the noisy test set Test n be N, it is necessary to prove that that is, to prove that:
[0065]
[0066] Only proof is required:
[0067]
[0068] make
[0069] Proof:
[0070]
[0071] Proof: On the clean test set The amount of noisy data in the portion of the data that can be correctly classified by the M-class classifier For a clean test set The total amount of data that can be correctly classified by the M-class classifier With noisy test set Test n The total amount of data PT that can be correctly classified by the M-class classifier n difference.
[0072] This interpretation is based on a reasonable premise: on a clean test set In this context, whether a single image data point can be correctly classified by an M-class classifier, and whether that data is noisy, are two independent events; that is, on a clean test set... The proportion of noise in the data that can be correctly classified by the M-class classifier, compared to the clean test set. The overall noise level is the same.
[0073] For the formula The specific analysis and argumentation are as follows:
[0074] For the noisy test set Test n The single image data in the Test corresponds to the Test. n The given noisy label n , Given the true label GT and the prediction Pred given by the M-class classifier. If Label n =GT, then the image data is non-noise data; otherwise, if Label n If the noise level is not equal to GT, then the image data is noisy data. By capturing all the noisy data, the noise ratio can be estimated.
[0075] Table 1 shows the analysis of partial data, i.e., the clean test set. Analysis of the cases where the data is correctly classified by the M-class classifier, specifically based on the clean test set. The portion of data that can be correctly classified by the M-class classifier is categorized by its corresponding label. nThe data is categorized into three values: GT value, Pred value, and GT value; it is analyzed whether the data in each case is noisy; and the impact of each case on... Value, PT n Value, and The influence of these three factors, namely Analysis of whether it can pass The difference Δ with PTn Capture noise and the magnitude of the error value δE.
[0076] As shown in the data analysis in Table 1, the clean test set... The data correctly classified by the M-class classifier is given by Pred = GT; according to the corresponding noisy test set Test... n Those classes that can be correctly classified by the M-class classifier can be analyzed separately. The analysis is as follows:
[0077] Test with noise n In the above, the data that can be correctly classified includes Pred = Label. n Thus, Label came into being. n =GT, then the data is clean data; Right now This data label is a non-noise label, according to... The value is used for noise capture, with an error δE of 0;
[0078] Test with noise n In this context, data that cannot be correctly classified includes Pred ≠ Label. n Thus, Label came into being. n If the value is not equal to GT, then the data is noise data; ΔPT n =0, that is This data label is a noise label, according to... The value is used for noise capture, with an error δE of 0;
[0079] Table 1: Data Situation Analysis Table
[0080]
[0081] In summary, for a clean test set In all cases, the data correctly classified by the M-class classifier is categorized as follows: The value is used for noise capture, with an error of 0. Proof obtained.
[0082] The application strictly demonstrates the relationship between the difference of classification accuracy of the model on the two test sets before and after cleaning and the label noise ratio, and carries out experimental verification in a way of artificially manufacturing label noise. The superiority of the automatic noise ratio estimation method is embodied, which can well estimate the label noise ratio in the large-scale Internet data set construction process without relying on the artificial cleaning method. In the scene of artificially constructing 20% label noise, the error of the automatically estimated noise ratio value can be stabilized within 1%, which proves the effectiveness of the automatic noise ratio estimation method.
[0083] From the above description of the embodiments, those skilled in the art can clearly understand that the facility of the present application can be realized by means of software and necessary general hardware platform. The embodiments of the present application can be realized by using existing processors, or by special-purpose processors used for this purpose or other purposes for appropriate systems, or by hardwired systems. The embodiments of the present application also include non-transitory computer-readable storage media, which include machine-readable media for carrying or having stored thereon machine-executable instructions or data structures; such machine-readable media can be any available media that can be accessed by general or special-purpose computers or other machines with processors. For example, such machine-readable media can include RAM, ROM, EPROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program codes in the form of machine-executable instructions or data structures, and can be accessed by general or special-purpose computers or other machines with processors. When information is transmitted or provided to a machine through a network or other communication connection (hardwired, wireless, or a combination of hardwired or wireless), the connection is also considered a machine-readable medium.
[0084] According to the description and drawings of the present application, those skilled in the art can easily manufacture or use the biological sequence processing and model training method of the present application, and can produce the positive effects described in the present application.
[0085] The above is only a preferred embodiment of the present application, and does not limit the present application in any form. Any simple modification or equivalent change to the above embodiment according to the technical essence of the present application falls within the protection scope of the present application.
Claims
1. A method for evaluating the noise ratio of labels in internet datasets for image classification, characterized in that: Includes the following steps: S1: Obtain the Web-M internet image dataset, and divide the Web-M dataset into noisy training sets according to a certain proportion for each category. and noisy test set ,in To determine the number of categories, retain the noise label and divide the data into categories based on that label; S2: Obtain a clean test set containing the same categories as the Web-M dataset. ; S3: Using the aforementioned noisy training set Train a using training data Class classifier; S4: Based on the training obtained in step S3 Class classifier on clean test set The model performs inference, comparing the given label with... The classifier provides the predicted category, which is the final result. Class classifier on clean test set Classification accuracy ; S5: Based on the training in step S3 Class classifier on noisy test set The model performs inference, comparing the given noise label with... The classifier provides the predicted category, which is the final result. Class classifier on noisy test set Classification accuracy ; S6: Compare the results obtained based on steps S4 and S5. Class classifier on clean test set Classification accuracy and in the noisy test set Classification accuracy The noise ratio of the Web-M network dataset was obtained. The estimated value ; Class classifier on clean test set With noisy test set Classification accuracy , The difference between them is ,in ; based on and This allows us to further obtain the noise ratio of the Web-M network dataset. The estimated value ,in .
2. The method for evaluating the noise ratio of labels in internet datasets for image classification according to claim 1, characterized in that, In step S2, a clean test set with the same category as the Web-M dataset is obtained. ; The labels for each image data in the test set are their true categories, and the noise ratio is almost zero and can be ignored.
3. The method for evaluating the label noise ratio of internet datasets for image classification according to claim 1, characterized in that, In step S3, based on the noisy training set obtained in step S1 ,for Training for each category Class classifier.
4. An electronic device, characterized in that: include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method for evaluating the label noise ratio of Internet datasets for image classification as described in any one of claims 1-3.
5. A computer-readable medium having a computer program stored thereon, characterized in that: When executed by the processor, the program implements the method for evaluating the noise ratio of labels in Internet datasets for image classification as described in any one of claims 1-3.