A classification method for TCM tongue image rottenness and greasyness based on noisy labeled data

Abstract

A method for classifying TCM tongue images of greasy and rottenness with noisy labeled data belongs to the field of computer vision. The method uses multiple machine learning classification models to evaluate and update the label confidence of noisy labeled tongue image datasets, puts the label dataset label quality update and the classification network model parameter update in one iterative process, introduces the active learning idea, and provides a novel method for interactively iteratively improving the label quality of TCM tongue image samples of greasy and rottenness and the classification model. The present invention improves the confidence of labeled samples, thereby screening high-quality labeled samples, and can be applied to other noisy labeled tongue image sample problems, which can effectively solve the problem of strong subjectivity of TCM tongue image labeling and noisy labels, and has higher generalizability.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a method for classifying greasy and rotten tongue images in traditional Chinese medicine (TCM) with noisy labeled data. Background Art

[0002] Tongue diagnosis is a method of understanding the physiological functions and pathological conditions of the human body by observing the changes in tongue images. It is one of the most widely used and popular diagnostic methods in the four diagnostic methods of traditional Chinese medicine. The analysis of tongue image characteristics in traditional Chinese medicine is the core content of realizing the objectivity of tongue diagnosis. The characteristics of tongue images reflect the pathological changes of cold, heat, deficiency and excess in the internal organs and blood of the human body. In tongue image analysis, different aspects of tongue characteristics are usually analyzed according to needs, such as tongue coating color, thickness and moisture of tongue coating, greasy tongue texture, prickling, cracks, tongue shape (tooth marks, crooked and fat and thin) and other characteristics. Greasy characteristics reflect the texture of tongue coating, mainly referring to the changes in the size and density of tongue coating particles, which are usually divided into three categories: non-greasy, greasy, and greasy. The extracted greasy characteristics are mainly used by doctors to diagnose symptoms, which is one of the key links in the diagnosis and treatment of traditional Chinese medicine.

[0003] Tongue image and tongue diagnosis are usually visually observed by doctors and diagnosed based on experience. Compared with Western medicine, it lacks quantitative and objective measurement and analysis. Affected by subjective factors such as the doctor's knowledge level and experience, tongue diagnosis results vary from person to person, are highly subjective, and have poor repeatability, which directly affects the standardization of syndrome differentiation and treatment. Extracting and classifying tongue image features through computer algorithms is of great significance to the objectivity of tongue diagnosis.

[0004] The data annotation cost of the sample data of the TCM tongue image features of greasy and rotten is high, and there are few labeled samples of greasy and rotten. It is necessary to effectively use unlabeled samples to distinguish the features between different classes. There is a certain similarity between different classes of greasy and rotten features, and doctors' annotations are subjective, so there are inconsistent noise labeled samples. How to extract high-quality labeled samples is the prerequisite for efficient training of classification models.

[0005] Deep learning methods effectively extract deep features of images and learn the inherent laws and deep feature expressions of sample data, thereby achieving higher classification accuracy and stronger model generalization ability. Deep learning models usually require a large amount of labeled data for supervised training, and the limited number of noisy labeled tongue samples becomes an important factor affecting the accuracy and generalization ability of the model.

[0006] To this end, the present invention aims at the characteristics of the greasy and rotten features of the tongue, combines active learning and semi-supervised learning technology, and proposes a method for automatically classifying the greasy and rotten features of TCM tongue images with noisy labeled data. Multiple classification networks are used as the basic network to extract deep features of the tongue image, predict the noisy labeled data, statistically predict the consistency, select samples with large amounts of information for manual re-labeling, achieve the purpose of data cleaning, update high-quality labeled samples, introduce unlabeled samples, and mine the implicit information of unlabeled samples. Semi-supervised learning is used to effectively assist the training of the final classifier and improve the classification performance. Summary of the invention

[0007] The present invention provides a method for classifying greasy and rotten tongue images in traditional Chinese medicine based on noisy labeled data, the method comprising:

[0008] 1) Constructing a dataset of greasy tongue texture

[0009] The data set used in the present invention is collected by a traditional Chinese medicine tongue imager, and the greasy and rotten category is calibrated by a professional doctor. The greasy and rotten category of a single tongue image is mainly determined by the tongue center block image block. To construct the training and test data sets, three steps are required: tongue area extraction, tongue center block extraction, and tongue image block slider processing:

[0010] Step 1.1: Extract the tongue area from the original image and use it as the basic tongue image;

[0011] Step 1.2: Extract the center block of the tongue as the representative block of its greasy characteristics;

[0012] Step 1.3: Slider processing is performed on the extracted tongue image blocks to obtain tongue image blocks for constructing a large batch of tongue image block data sets;

[0013] 2) Sample category prediction and consistency statistics based on multi-classification models

[0014] The prediction results of noisy labeled samples through multiple models are unified. Samples with large amounts of information that are difficult to distinguish are screened for manual re-labeling, and the labeled data sets are updated to improve the confidence of data annotation.

[0015] Step 2.1: Use the noisy annotated tongue image data to train multiple classification models;

[0016] Step 2.2: Use multiple classification models to predict the category of labeled data, and estimate the confidence of the data label based on the multi-model prediction results. Classify the samples into the following three categories:

[0017] Category 1: For samples with consistent prediction results from multiple models and the same labeled category, they are classified as labeled consistent samples with high label reliability and are directly used for optimization training of subsequent classification models;

[0018] Category 2: Samples whose prediction results are consistent but inconsistent with the annotation results are classified as suspected incorrectly labeled samples and submitted for manual verification of the annotation results;

[0019] Category 3: For samples with inconsistent prediction results from multiple models, the label confidence is calculated based on the specific prediction results and their relationship with the labeled category. Based on the confidence, the samples are weighted or manually calibrated;

[0020] Step 2.3: Update the labeled samples

[0021] The three types of samples are processed accordingly and the labeled data sets are updated. Samples of category 1 are directly used for training and testing; samples of category 2 are added to the training and testing sets after manual labeling and verification; samples of category 3 are excluded and manually calibrated according to the sample ratio and training method, and weighted applications are assigned;

[0022] 3) Classification model training

[0023] Step 3.1: Use the updated labeled samples to divide the training set and test set; perform network training on the multi-classification model. You can use retraining the model or optimize the training on the existing network model;

[0024] Step 3.2: Train multiple classification models until the models converge, that is, the test classification accuracy no longer changes significantly, and record the test accuracy;

[0025] Step 3.3: Repeat steps 2) and 3) to perform a new round of label prediction and consistency statistics, data label update, and network model training on the labeled samples;

[0026] Step 3.4: When the proportion of consistent label samples in the annotated data increases significantly and the overall classification performance of the network no longer improves, that is, the recorded accuracy no longer changes, the iteration is stopped. According to the system accuracy and complexity requirements, the best performance classifier is selected to implement the tongue image classification network;

[0027] 4) Optionally, the multi-classification model label prediction scheme described in the present invention can also be used to predict labels for unlabeled data; for samples with consistent prediction results, directly assign prediction pseudo labels and add them to the training set; for samples with poor consistency in prediction results, select them for manual labeling for active learning; for samples with intermediate confidence, flexibly apply them. The technology of the present invention can conveniently clean noisy labeled data, predict pseudo labels for semi-supervised learning, select hard-to-distinguish samples for active learning, etc.

[0028] A method for classifying greasy and rotten tongue images in traditional Chinese medicine based on noisy labeled data, the method comprising:

[0029] 1) Constructing a dataset of greasy tongue texture

[0030] Step 1: Extract the tongue area from the original image and use it as the basic tongue image; use the existing Chinese medicine tongue image segmentation method to segment the tongue, and unify the size of the extracted tongue image;

[0031] Step 2: Extract the central block of the tongue as the representative block of its greasy and rotten features; the central part of each segmented tongue retains the greasy and rotten features of the tongue, so for each tongue image, extract its central tongue image block, adopt the small area filtering method, set the minimum area threshold, and experimentally set the threshold to 100×100 pixels. By obtaining all connected areas in the image, determine whether the area of ​​each area is less than the threshold, and filter out areas with an area less than the threshold, retaining areas equal to and greater than the threshold, obtaining the maximum and minimum pixel values ​​of the target area, determining the target area, and then locating the center point, setting the cropping area, and sliding to obtain the central block of the tongue;

[0032] Step 3: Construct a tongue image block training data set; perform slider processing on the extracted tongue image blocks, set the target tongue image block size, and set a uniform step size to slide and obtain the tongue image blocks;

[0033] 2) Multiple model integration prediction statistics

[0034] Step 1: Use the first batch of noisy and annotated tongue data to train multiple classification models; use the convolutional network InceptionV3_v1, MobileNet_v1, ResNet50, and Efficient Net b4 network structure training, and use the SGD gradient descent algorithm to train the network model parameters for the four classification model trainings. Set the learning rate range to 0.00001 to 1, the momentum range to 0.5 to 0.99, and the decay coefficient range to 1e-9 to 1e-2;

[0035] Step 2: Select InceptionV3, MobileNet_v1, and ResNet50 network models as the basic network. According to the integrated prediction method of multiple models, a unified prediction is made for all the first batch of labeled samples, as shown in formula (1). a represents the sample confidence, the number of prediction consistent models is set to k, the total number of models is K, z is the predicted label, and Z is the original label. If the three models predict the same and the samples are the same as the originally labeled categories, they are classified into the labeled set, the confidence is set to 1, the label reliability is high, and they are directly used for subsequent cyclic training; if the prediction results are consistent but the samples are inconsistent with the originally labeled categories, the confidence is set to 1 / 2, and they are classified as suspected incorrectly labeled samples. The idea of ​​active learning is introduced and the labeling results are submitted for manual verification; if the three models predict inconsistently, According to the specific prediction results and the relationship with the labeled category: set the threshold to 1 / 2, when the k / K ratio is greater than the threshold, that is, most models in the model predict the same, and only a few are inconsistent, then determine whether this sample is consistent with the original label; if consistent, the confidence is set to 1, and it is classified into the labeled set; if inconsistent, the confidence is set to 1 / 2, and manual label verification is performed; when the k / K ratio is equal to and less than the threshold, that is, a few models predict the same, which means that most models predict inconsistently, the confidence is set to 0, and it is classified as an unlabeled data set; when the k / K ratio is 0, that is, all models predict inconsistently, it means a high probability noise label sample, the confidence is set to 0, and it is classified as an unlabeled data set; after all labeling is completed, the labeled sample set is updated to obtain the manually involved denoised labeled sample set X;

[0036]

[0037] Step 3: Divide the denoised labeled samples into training and test sets, perform integrated prediction based on the new model after training, keep the parameters unchanged, and record the classification accuracy of the optimal model; update the labeled sample set again according to the criteria in step 2, and iterate until the classification accuracy of the sample test set remains unchanged;

[0038] 3) Semi-supervised model training

[0039] Step 1: Based on the data screening in 2), the denoised annotation set is used as the labeled sample set, and the remaining unlabeled tongue image data is used as the unlabeled sample set. Data enhancement is applied to both the labeled and unlabeled samples. batch is the enhanced data batch. One data enhancement is performed on a batch of labeled data, and M data enhancements are performed on a batch of unlabeled data. The experimental setting M is 2. The standard enhancement methods for data enhancement are random cropping and horizontal flipping, random cropping to different sizes and aspect ratios, and then scaling the cropped images to the target size to obtain the enhanced labeled data set X and unlabeled data U;

[0040] Step 2: After comparing the results of K (K=3) classification models in 2), the trained Efficient Net b4 classification model is used as a semi-supervised learning classifier. The classifier predicts and classifies the M enhancement results of a single data frame. The classification results are finally averaged to determine the final pseudo label. The specific calculation method is shown in formula (2);

[0041]

[0042] Among them, u b represents unlabeled data, y is the predicted label, m is the number of enhancements, the total number of enhancements is M, θ is the model parameter, and P is the probability of being predicted as category y by the classifier. is the final average predicted probability value; in the process of generating pseudo labels, the model introduces the minimization entropy operation, and the specific operation is to introduce the "sharpen" function, as shown in formula (3);

[0043]

[0044] Among them, Q is the average predicted probability value of the enhanced data, corresponding to formula (1) T is the temperature hyperparameter. The experiment sets T value to 0.5, which can adjust the classification entropy. L is the total number of categories. i and j refer to a single category. When T value approaches 0, the output will be close to one-hot distribution. Reducing T value will encourage the model to make low entropy predictions for the enhanced unlabeled data, thus improving the prediction accuracy of the model. Through this step, the pseudo labels of the enhanced unlabeled data are obtained.

[0045] Step 3: In order to relatively enhance the generalization ability of the model, the mix up method is used to mix the data. The labeled dataset is X and the unlabeled dataset is U. The specific mixing operation is to mix the labeled dataset X and the unlabeled dataset U to form a mixed dataset W. Then, the labeled data X and the first X data in the W dataset are mixed up to form a new labeled dataset denoted as X′. The unlabeled data and the last U data in the W dataset are mixed up to form a new unlabeled dataset denoted as U′. A mixed labeled and unlabeled tongue texture dataset is constructed.

[0046] Step 4: For the labeled dataset X′, calculate the cross entropy loss between the label and the model prediction. The calculation method is shown in formula (4):

[0047]

[0048] Among them, X′ represents all labeled samples, |X′| is the number of all labeled samples, x is a single labeled sample, p is the true sample label distribution, P is the probability that sample x is predicted as category y by the classifier, and the H function is the cross entropy between distribution p and P.

[0049] For the unlabeled dataset U′, the mean square loss function between the model prediction and the pseudo label is calculated. The error is obtained by subtracting the estimated value from the target value and then squaring it. Specifically, as shown in formula (5),

[0050]

[0051] Where U′ represents all unlabeled samples, |U′| is the number of all unlabeled samples, u is a single unlabeled sample, q is the true sample label distribution, P is the probability that sample u is predicted as category y by the classifier, θ is the model parameter, and L is the total number of categories;

[0052] The overall loss term is shown in formula (6). Loss represents the total loss, which is the sum of the two losses.

[0053] Loss = L x +λL u #(6)

[0054] λ is a hyperparameter and is set to 100. On the newly constructed data set, the above loss constraint network is introduced, and the network is continuously iterated and trained. The learning rate is set to 0.00001 to 1, the momentum is set to 0.5 to 0.99, and the decay coefficient is set to 1e-9 to 1e-2. The training is carried out until the classification model loss converges and the decay classification index does not change. The model with the highest verification result index is selected as the optimal model for classification network training.

[0055] The technical solution described in this application provides a method for classifying TCM tongue images with greasy and rotten features based on noisy annotated data, which is used for computer-assisted classification of tongue images with greasy and rotten features. This method uses multiple machine learning classification models to evaluate and update the label confidence of the noisy annotated tongue image dataset, and puts the label dataset annotation quality update and the classification network model parameter update in one iterative process. The introduction of active learning ideas provides a novel method for interactively iteratively improving the annotation quality of TCM tongue image samples with greasy and rotten features and the classification model.

[0056] Compared with the single model decision method in deep learning, the method for classifying the greasy and rotten TCM tongue images of noisy labeled data provided by the present invention can improve the confidence of labeled samples, thereby screening high-quality labeled samples, and can be applied to other tongue image sample problems containing noisy labels. It can effectively solve the problems of strong subjectivity and noisy labels in TCM tongue image annotation, and has higher generalizability. The methods for data annotation quality assessment and update, and classification model update training provided by the present invention can also be extended to other similar TCM tongue image feature classification tasks, and can also be widely used in other classification tasks of noisy labeled data.

[0057] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above technical solution of the present invention and its objectives, features and advantages more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0059] Figure 1 The process of making a dataset of tongue greasy blocks for classification provided by the present invention is shown;

[0060] Figure 2 A schematic diagram of data cleaning provided by the present invention is shown;

[0061] Figure 3 A schematic diagram of the framework of the semi-supervised classification model for greasy tongue quality provided by the present invention is shown; DETAILED DESCRIPTION

[0062] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0063] This method predicts all labeled data based on multiple models, uses active learning ideas to screen out inconsistent data and re-label them manually, obtains high-quality denoised labeled data sets, and optimizes network training methods. It makes full use of the feature information of unlabeled data so that the model can learn more distinguishable features of rottenness and greasyness, so that the network can make a clearer distinction between the three types of data: non-rottenness and greasyness, rottenness, and greasyness. The present invention proposes a method for automatically classifying rottenness and greasyness in TCM tongue images with noisy labeled data, as shown in the attached figure. Figure 2 and 3 As shown in the figure, in this method, multiple trained models are used to predict noisy labeled data, and the labeled data set and the unlabeled data set are distinguished by the prediction consistency. The data is updated after manual labeling, and then data enhancement measures are taken for other unlabeled data. The average prediction of the enhanced data is introduced to introduce the minimization entropy operation. Finally, the labeled data and the unlabeled data are fused, the data is updated, the network is iteratively trained, and the network robustness is enhanced. Finally, a semi-supervised training model is obtained for category testing of greasy tongue samples.

[0064] The following is a description of the implementation examples of the present invention in conjunction with the accompanying drawings:

[0065] The data set used in the present invention is collected by a traditional Chinese medicine tongue imager, and its category is calibrated by a traditional Chinese medicine physician. The tongue image samples used are 1423 full tongue images collected from the Department of Traditional Chinese Medicine of a certain traditional Chinese medicine hospital. The types of greasy tongue are divided into three categories: non-greasy, greasy, and greasy. The training set is divided into labeled samples and unlabeled samples, of which the labeled samples are 618 tongue images. According to the tongue image data set production process in 1), the tongue images are preprocessed into 2472 tongue image blocks, including 1512 non-greasy blocks; 432 greasy blocks; 528 greasy blocks, and a total of 10,000 unlabeled samples. The test set has a total of 405 tongue images, corresponding to 405 tongue center blocks, of which 249 are non-greasy, 9 are greasy, and 147 are greasy.

[0066] 1) Constructing a dataset of greasy tongue texture

[0067] Step 1: Extract the tongue area from the original image and use it as the basic tongue image. Since the analysis of specific tongue texture information needs to focus on the tongue coating area, the tongue image collected by the Chinese medicine tongue imager usually contains interfering information such as human face and lips. Therefore, it is necessary to extract the tongue body from the original image and segment the tongue body using the existing Chinese medicine tongue image segmentation method. After extraction, the tongue image has a unified standard size of 3456×3456 pixels.

[0068] Step 2: Extract the center block of the tongue as the representative block of its greasy and rotten features. The center part of each segmented tongue retains the characteristics of greasy tongue, so for each tongue image, extract its center tongue image block. First, adopt the small area filtering method and set the minimum area threshold. The experimental threshold is set to 100×100 pixels. By obtaining all the connected areas in the image, determine whether the area of ​​each area is less than the threshold. The area with an area less than the threshold is filtered out, and the area equal to and greater than the threshold is retained. This method is used to distinguish the interference area in the image background and prevent deviation in the positioning of the target area. Obtain the maximum and minimum pixel values ​​of the target area, determine the target area, and then locate the center point. Set the cropping area size to 512×512 pixels to obtain the center block of the tongue.

[0069] Step 3: Construct a large batch of tongue image block training data sets. The extracted tongue image blocks are processed by sliders, and the target tongue image block size is set to 224×224 pixels, the step size is set to 65, and 2472 tongue image blocks are obtained by sliding, which meets the large demand for training sets in deep learning networks.

[0070] 3) Multiple model integration prediction statistics

[0071] Step 1: Use the first batch of noisy annotated tongue data to train multiple classification models. At present, the commonly used convolutional networks in the classification field are InceptionV3, MobileNet, and ResNet series models. The Efficient Net network structure is also used because it can maintain high classification accuracy while taking a short time. The convolutional network InceptionV3_v1, MobileNet_v1, ResNet50, and Efficient Net b4 network structures are used for training. The SGD gradient descent algorithm is used to train the network model parameters for the four classification model training. In the experiment, the initial learning rate is set to 0.001, the momentum is set to 0.9, and the decay coefficient is set to 1e-6.

[0072] Step 2: Select InceptionV3, MobileNet_v1, and ResNet50 network models as the basic network. According to the integrated prediction method of multiple models, a unified prediction is made for all the first batch of labeled samples, as shown in formula (1). a represents the sample confidence, the number of models with consistent predictions is set to k, the total number of models is set to K, z is the predicted label, and Z is the original label. If the three models predict the same sample and the sample is the same as the original labeled category, it is classified into the labeled set, the confidence is set to 1, the label reliability is high, and it is directly used for subsequent cyclic training; if the prediction result is consistent but the sample is inconsistent with the original labeled category, the confidence is set to 1 / 2, and it is classified as a suspected incorrectly labeled sample. The idea of ​​active learning is introduced and submitted for manual labeling. Note the result verification; if the predictions of the three models are inconsistent, according to the specific prediction results and the relationship with the labeled category: the threshold is set to 1 / 2, when the k / K ratio is greater than the threshold, that is, when most models in the model predict the same, only a few are inconsistent, then determine whether this sample is consistent with the original label; if consistent, the confidence is set to 1, and it is classified as a labeled set; if inconsistent, the confidence is set to 1 / 2, and the label verification is performed manually; when the k / K ratio is equal to and less than the threshold, that is, a few models predict the same, and most models predict inconsistently, the confidence is set to 0, and it is classified as an unlabeled data set; when the k / K ratio is 0, that is, all models predict inconsistently, it indicates a high probability noise label sample, the confidence is set to 0, and it is classified as an unlabeled data set. After all the annotations are completed, the labeled sample set is updated to obtain the manually involved denoised labeled sample set X.

[0073]

[0074] Step 3: Divide the denoised labeled samples into training and test sets, perform integrated prediction based on the new model after training, keep the parameters unchanged, and record the classification accuracy of the optimal model; update the labeled sample set again according to the criteria in the second step, and iterate until the classification accuracy of the sample test set remains basically unchanged.

[0075] 3) Semi-supervised model training

[0076] Step 1: Based on the data screening in 2), the denoised annotation set is used as the labeled sample set, and the remaining unlabeled tongue image data is used as the unlabeled sample set. Data enhancement is applied to both labeled and unlabeled samples. A specific batch size is set. The batch size of this experiment is set to 4. Data enhancement is performed once for a batch of labeled data, and M times for a batch of unlabeled data. The experiment sets M to 2. The standard enhancement methods for data enhancement are random cropping and horizontal flipping, random cropping to different sizes and aspect ratios, and then scaling the cropped images to 380×380 pixels to obtain the enhanced labeled data set X and unlabeled data U.

[0077] Step 2: After comparing the results of the three classification models in 2), the Efficient Net b4 network model has a good effect on the classification of grease. This experiment uses the trained Efficient Net b4 classification model as a semi-supervised learning classifier. The classifier predicts and classifies the M enhancement results of a single data frame. The classification results are finally determined by the average method to determine the final pseudo label. The specific calculation method is shown in formula (2).

[0078]

[0079] Among them, u b represents unlabeled data, y is the predicted label, m is the number of enhancements, the total number of enhancements is M, θ is the model parameter, and P is the probability of being predicted as category y by the classifier. is the final average prediction probability value. In the process of generating pseudo labels, in order to make the model's prediction confidence for unlabeled samples as high as possible, the model introduces the minimization entropy operation, which is to introduce the "sharpen" function, as shown in formula (3).

[0080]

[0081] Among them, Q is the average predicted probability value of the enhanced data, corresponding to formula (1) T is the temperature hyperparameter. The experiment sets T to 0.5, which can adjust the classification entropy. L is the total number of categories. i and j refer to a single category. When T approaches 0, the output will be close to the one-hot distribution. Reducing T will encourage the model to make low-entropy predictions for the enhanced unlabeled data, improving the prediction accuracy of the model. Through this step, the pseudo-labels of the enhanced unlabeled data are obtained.

[0082] Step 3: In order to relatively enhance the generalization ability of the model, the mix up method is used to mix the data. The labeled dataset is X and the unlabeled dataset is U. The specific operation of mixing is to mix the labeled dataset X and the unlabeled dataset U to form a mixed dataset W. Then, the labeled data X and the first X data in the W dataset are mixed up to form a new labeled dataset denoted as X′. The unlabeled data and the last U data in the W dataset are mixed up to form a new unlabeled dataset denoted as U′. A mixed labeled and unlabeled tongue texture dataset is constructed.

[0083] Step 4: For the labeled dataset X', calculate the cross entropy loss between the label and the model prediction. The calculation method is shown in formula (4):

[0084]

[0085] Among them, X′ represents all labeled samples, |X′| is the number of all labeled samples, x is a single labeled sample, p is the true sample label distribution, P is the probability that sample x is predicted as category y by the classifier, and the H function is the cross entropy between distribution p and P.

[0086] For the unlabeled dataset U′, the mean square loss function between the model prediction and the pseudo label is calculated. The error is obtained by subtracting the estimated value from the target value and then squaring it. Specifically, as shown in formula (5),

[0087]

[0088] Among them, U′ represents all unlabeled samples, |U′| is the number of all unlabeled samples, u is a single unlabeled sample, q is the true sample label distribution, P is the probability that sample u is predicted as category y by the classifier, θ is the model parameter, and L is the total number of categories.

[0089] The overall loss term is shown in formula (6). Loss represents the total loss, which is the sum of the two losses.

[0090] Loss = L x +λL u #(6)

[0091] λ is a hyperparameter, which is set to 100 in the experiment. On the constructed new data set, the above loss constraint network is introduced, and the network is continuously iterated and trained. The initial learning rate in the training parameters is set to 0.001, the decay coefficient is set to 0.999, and epochs is the number of iterations, which is set to 100. The training is carried out until the classification model loss converges and the decay classification index is close to unchanged. The model with the highest verification result index is selected as the optimal model for classification network training.

[0092] The present invention selects 618 labeled tongue greasy images from the database as the initial labeled training samples, which are preprocessed into 2472 tongue image blocks, 10000 unlabeled training samples, and 405 tongue images as the test samples. The same training strategy is adopted, and fully supervised training is firstly performed on 4 models including InceptionV3, Mobile Net, Res Net, and Efficient Net b0, and the average classification accuracy is 82.47%, 89.62%, 90.86%, and 91.11% respectively. The Efficient Net model performs well, and then the training is performed on the Efficient Net b0-b4 network architecture, and the optimal classification result of 93.09% is obtained on the b4 model. 10000 blocks of unlabeled data are introduced for semi-supervised learning, and the classification result is 94.50%, which improves the accuracy rate by 1.41%. Preliminary verification shows that the features after the introduction of unlabeled data increase the diversity of samples, and the classifier can extract more effective information, and at the same time, the unlabeled data is utilized, so that the model has higher generalization ability.

[0093] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0094] Similarly, it should be understood that in order to streamline the present invention and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the following intention: that the claimed invention requires more features than the features explicitly recited in each claim. More specifically, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present invention.

[0095] It should be noted that the above embodiments illustrate the present invention rather than limit the present invention, and those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation to the claims.

Claims

1. A method for classifying greasy and rotten tongue images in traditional Chinese medicine based on noisy labeled data, characterized in that: The method includes: 1) Constructing a dataset of greasy tongue texture The application data set is collected by a traditional Chinese medicine tongue imager and calibrated for the category of rotten and greasy. To construct the training and test data sets, three steps are required: tongue area extraction, tongue center block extraction, and tongue image block slider processing: Step 1.1: Extract the tongue area from the original image and use it as the basic tongue image; Step 1.2: Extract the center block of the tongue as the representative block of its greasy characteristics; Step 1.3: Perform slider processing on the extracted tongue image blocks to obtain tongue image blocks for constructing a tongue image block dataset; 2) Sample category prediction and consistency statistics based on multi-classification models The noisy labeled samples are predicted by multiple models to have uniform results, which updates the labeled data set and improves the confidence of data annotation; Step 2.1: Use the noisy annotated tongue image data to train multiple classification models; Step 2.2: Use multiple classification models to predict the categories of labeled data. According to the prediction results of multiple models, estimate the confidence of data labels; classify samples into the following three categories: Category 1: For samples with consistent prediction results from multiple models and the same labeled category, they are classified as labeled consistent samples with high label reliability and are directly used for optimization training of subsequent classification models; Category 2: Samples whose prediction results are consistent but inconsistent with the annotation results are classified as suspected incorrectly labeled samples and submitted for manual verification of the annotation results; Category 3: For samples with inconsistent prediction results from multiple models, the label confidence is calculated based on the specific prediction results and their relationship with the labeled category; based on the confidence, the samples are weighted or manually calibrated; Step 2.3: Update the labeled samples The three types of samples are processed accordingly and the labeled data sets are updated; samples of category 1 are directly used for training and testing; samples of category 2 are added to the training and testing sets after manual labeling and verification; for category 3, samples are excluded and manually calibrated according to the sample ratio and training method, and weighted applications are given; 3) Classification model training Step 3.1: Use the updated labeled samples to divide the training set and the test set; perform network training on the multi-classification model; use the retrained model or perform optimization training on the existing network model; Step 3.2: Train multiple classification models until the models converge and record the test accuracy; Step 3.3: Repeat steps 2) and 3) to perform a new round of label prediction and consistency statistics, data label update, and network model training on the labeled samples; Step 3.4: When the proportion of consistent label samples in the annotated data increases and the overall classification performance of the network no longer improves, that is, the recorded accuracy no longer changes, the iteration is stopped; the classifier with the best performance is selected to realize the classification of greasy tongue images.

2. The method according to claim 1, characterized in that: 1) Constructing a dataset of greasy tongue texture Step 1: Extract the tongue area from the original image and use it as the basic tongue image; use the existing Chinese medicine tongue image segmentation method to segment the tongue, and unify the size of the extracted tongue image; Step 2: Extract the central block of the tongue as the representative block of its greasy and rotten features; the central part of each segmented tongue retains the greasy and rotten features of the tongue, so for each tongue image, extract its central tongue image block, adopt the small area filtering method, set the minimum area threshold, and experimentally set the threshold to 100×100 pixels. By obtaining all connected areas in the image, determine whether the area of ​​each area is less than the threshold, and filter out areas with an area less than the threshold, retaining areas equal to and greater than the threshold, obtaining the maximum and minimum pixel values ​​of the target area, determining the target area, and then locating the center point, setting the cropping area, and sliding to obtain the central block of the tongue; Step 3: Construct a tongue image block training data set; perform slider processing on the extracted tongue image blocks, set the target tongue image block size, and set a uniform step size to slide and obtain the tongue image blocks; 2) Multiple model integration prediction statistics Step 1: Use the first batch of noisy and annotated tongue data to train multiple classification models; use the convolutional network InceptionV3_v1, MobileNet_v1, ResNet50, and Efficient Net b4 network structure training, and use the SGD gradient descent algorithm to train the network model parameters for the four classification model trainings. Set the learning rate range to 0.00001 to 1, the momentum range to 0.5 to 0.99, and the decay coefficient range to 1e-9 to 1e-2; Step 2: Select InceptionV3, MobileNet_v1, and ResNet50 network models as the basic network. According to the integrated prediction method of multiple models, a unified prediction is made for all the first batch of labeled samples, as shown in formula (1). a represents the sample confidence, the number of prediction consistent models is set to k, the total number of models is K, z is the predicted label, and Z is the original label. If the three models predict the same and the samples are the same as the originally labeled categories, they are classified into the labeled set, the confidence is set to 1, the label reliability is high, and they are directly used for subsequent cyclic training; if the prediction results are consistent but the samples are inconsistent with the originally labeled categories, the confidence is set to 1 / 2, and they are classified as suspected incorrectly labeled samples. The idea of ​​active learning is introduced and the labeling results are submitted for manual verification; if the three models predict inconsistently, According to the specific prediction results and the relationship with the labeled category: set the threshold to 1 / 2, when the k / K ratio is greater than the threshold, that is, most models in the model predict the same, and only a few are inconsistent, then determine whether this sample is consistent with the original label; if consistent, the confidence is set to 1, and it is classified into the labeled set; if inconsistent, the confidence is set to 1 / 2, and manual label verification is performed; when the k / K ratio is equal to and less than the threshold, that is, a few models predict the same, which means that most models predict inconsistently, the confidence is set to 0, and it is classified as an unlabeled data set; when the k / K ratio is 0, that is, all models predict inconsistently, it means a high probability noise label sample, the confidence is set to 0, and it is classified as an unlabeled data set; after all labeling is completed, the labeled sample set is updated to obtain the manually involved denoised labeled sample set X; Step 3: Divide the denoised labeled samples into training and test sets, perform integrated prediction based on the new model after training, keep the parameters unchanged, and record the classification accuracy of the optimal model; update the labeled sample set again according to the criteria in step 2, and iterate until the classification accuracy of the sample test set remains unchanged; 3) Semi-supervised model training Step 1: Based on the data screening in 2), the denoised annotation set is used as the labeled sample set, and the remaining unlabeled tongue image data is used as the unlabeled sample set. Data enhancement is applied to both the labeled and unlabeled samples. batch is the enhanced data batch. One data enhancement is performed on a batch of labeled data, and M data enhancements are performed on a batch of unlabeled data. The experimental setting M is 2. The standard enhancement methods for data enhancement are random cropping and horizontal flipping. Random cropping is performed to different sizes and aspect ratios, and then the cropped image is scaled to the target size to obtain the enhanced labeled data set X and unlabeled data U; Step 2: After comparing the results of K classification models in 2), where K = 3, the trained Efficient Net b4 classification model is used as a semi-supervised learning classifier. The classifier predicts and classifies the M enhancement results of a single data frame. The classification results are finally averaged to determine the final pseudo label. The specific calculation method is shown in formula (2); Among them, u b represents unlabeled data, y is the predicted label, m is the number of enhancements, the total number of enhancements is M, θ is the model parameter, and P is the probability of being predicted as category y by the classifier. is the final average predicted probability value; in the process of generating pseudo labels, the model introduces the minimization entropy operation, and the specific operation is to introduce the "sharpen" function, as shown in formula (3); Among them, Q is the average predicted probability value of the enhanced data, corresponding to formula (1) T is the temperature hyperparameter. The experiment sets T value to 0.5, which can adjust the classification entropy. L is the total number of categories. i and j refer to a single category. When T value approaches 0, the output will be close to one-hot distribution. Reducing T value will encourage the model to make low entropy predictions for the enhanced unlabeled data, thus improving the prediction accuracy of the model. Through this step, the pseudo labels of the enhanced unlabeled data are obtained. Step 3: In order to relatively enhance the generalization ability of the model, the mix up method is used to mix the data. The labeled dataset is X and the unlabeled dataset is U. The specific mixing operation is to mix the labeled dataset X and the unlabeled dataset U to form a mixed dataset W. Then, the labeled data X and the first X data in the W dataset are mixed up to form a new labeled dataset denoted as X′. The unlabeled data and the last U data in the W dataset are mixed up to form a new unlabeled dataset denoted as U′. A mixed labeled and unlabeled tongue texture dataset is constructed. Step 4: For the labeled dataset X′, calculate the cross entropy loss between the label and the model prediction. The calculation method is shown in formula (4): Among them, X′ represents all labeled samples, |X′| is the number of all labeled samples, x is a single labeled sample, p is the true sample label distribution, P is the probability that sample x is predicted as category y by the classifier, and the H function is the cross entropy between distribution p and P. For the unlabeled dataset U′, the mean square loss function between the model prediction and the pseudo label is calculated. The error is obtained by subtracting the estimated value from the target value and then squaring it. Specifically, as shown in formula (5), Where U′ represents all unlabeled samples, |U′| is the number of all unlabeled samples, u is a single unlabeled sample, q is the true sample label distribution, P is the probability that sample u is predicted as category y by the classifier, θ is the model parameter, and L is the total number of categories; The overall loss term is shown in formula (6). Loss represents the total loss, which is the sum of the two losses. Loss=L x +λL u (6) λ is a hyperparameter and is set to 100. On the newly constructed data set, the above loss constraint network is introduced, and the network is continuously iterated and trained. The learning rate is set to 0.00001 to 1, the momentum is set to 0.5 to 0.99, and the decay coefficient is set to 1e-9 to 1e-2. The training is carried out until the classification model loss converges and the decay classification index does not change. The model with the highest verification result index is selected as the optimal model for classification network training.

Images