A tongue image constitution identification method based on deep learning

Abstract

The method for distinguishing tongue constitution based on deep learning described in the present invention is deployed on a mobile terminal, which is convenient for users to identify tongue images anytime and anywhere. The method of the present invention adopts a convolutional neural network algorithm, and performs data enhancement on the tongue image data set by combining the same type enhancement and mixed type enhancement methods. This enables the model to adapt to tongue image data in various environments, and the extracted features can directly output the classification results, which greatly reduces the amount of calculation and has stronger generalization ability.

Description

Technical Field

[0001] The present invention relates to the field of images, and in particular to a tongue image and constitution identification method based on deep learning. Background Art

[0002] At present, the use of artificial intelligence technologies such as machine learning and deep learning for image recognition has become mature. Some scholars have established a tongue image database by collecting a large number of tongue images with constitution labels, and designed algorithms to match tongue images with the established database to achieve the purpose of recognition. Considering the complexity of traditional manual extraction of tongue features and the accuracy of tongue area segmentation, deep learning has been applied to constitution identification due to its advantages in automatic feature extraction. Some scholars have used various models of convolutional neural networks to identify tongue constitution. Although these studies have promoted the development of intelligent constitution identification of tongue diagnosis, there are also some common problems. First, deep learning requires a large number of data sets as support, and the sample data collection of tongue images is difficult. Second, it is simply using the model for classification without improving and optimizing the model, resulting in the low accuracy of constitution identification using the above method. Summary of the invention

[0003] To solve the above problems, this application proposes a tongue image constitution identification method based on deep learning, specifically a tongue image recognition method based on an improved convolutional neural network GoogleNet model. This method is used to identify constitution based on the collected tongue image data set. The main contributions of this application are: constructing a tongue image data set with constitution labels, and proposing a GoogleNet network model (TCR-GoogleNet) suitable for tongue image constitution identification by adding batch normalization operations, improving and optimizing the Inception module, adding an attention mechanism, and using different Inception structures in combination. Ablation experiments were designed to verify the contribution of each improved module to the prediction accuracy, and comparative experiments were designed with 5 mainstream networks for image classification, proving that the model proposed in this application has greatly improved accuracy and stability.

[0004] The embodiment of the present invention provides the following technical solutions:

[0005] A tongue image and constitution identification method based on deep learning, the method is deployed in a mobile terminal, and the method includes:

[0006] Obtain a tongue image to be identified, use the trained improved convolutional neural network GoogleNet model to identify the tongue image to be identified, and obtain a tongue image recognition result;

[0007] Display the recognition result of the tongue image on the mobile terminal;

[0008] Among them, the training process of the improved convolutional neural network GoogleNet model is as follows:

[0009] Obtain a tongue image data set, and perform enhancement processing on the tongue image data set to obtain a tongue image training data set, wherein the enhancement processing includes two methods: same-class enhancement and mixed-class enhancement;

[0010] The tongue image training data set is used to train the improved convolutional neural network GoogleNet model to obtain a trained improved convolutional neural network GoogleNet model;

[0011] Among them, the improved convolutional neural network GoogleNet model improvements include:

[0012] Each intermediate layer of the model is standardized, which effectively avoids gradient vanishing and gradient exploding, speeds up network convergence, and optimizes the network structure.

[0013] The residual is integrated into the Inception structure, and the Inception structure is improved and optimized. The input features are fused with the features output by the Inception structure, and it is named Inception B.

[0014] The 3×3 convolution branch in the Inception module is decomposed into 1×3 and 3×1 convolutions, and the 5×5 convolution is decomposed into 3×3 convolutions and 1×3, 3×1 convolutions. Batch Normalization is added after each branch, which effectively reduces the number of parameters of the model and is named Inception C.

[0015] Based on the original GoogleNet model, the unimproved Inception module was retained and named InceptionA. The InceptionA, InceptionB and InceptionC structures were fused and applied to the improved GoogleNet network model. The InceptionA module was added to the shallow layer, the InceptionB module was added to the middle layer, and the InceptionC module was added to the deep layer. The A and B modules were used twice respectively, and the C module has stronger feature extraction and perception capabilities, so it was used five times.

[0016] Among them, similar enhancement refers to the use of data normalization, horizontal or vertical flipping, proportional enlargement or reduction of images, random rotation, random shearing transformation, brightness change and pixel filling methods to expand the data set for images with the same label. Color is also an important feature in tongue constitution identification, so no color channel transformation is performed on the tongue image data set.

[0017] Among them, the mixed class enhancement method is to solve the problem of imbalance between classes of sample data sets. The mixed class enhancement method mixes sample data of two categories with different labels to construct a virtual training sample to enrich the tongue image data set. Its mathematical expression is shown in (1):

[0018]

[0019] Where (x i ,y i )(x j ,y j ) are two randomly selected samples and their corresponding labels, λ is a random number that follows beta distribution, and λ∈[0,1].

[0020] Compared with the prior art, the above technical solution has the following advantages:

[0021] The tongue image constitution identification method based on deep learning described in this application is deployed on a mobile terminal, which is convenient for users to identify tongue image images anytime and anywhere. The method of the present invention adopts a convolutional neural network algorithm to enhance the tongue image data set by combining the same type enhancement and mixed type enhancement methods. This allows the model to adapt to tongue image data in various environments, and the extracted features can directly output the classification results, which greatly reduces the amount of calculation and has stronger generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 This is a flowchart of the method for distinguishing tongue constitution based on deep learning described in this application;

[0024] Figure 2 To frame the proposed method;

[0025] Figure 3 is a sample image;

[0026] Figure 4 The image after data enhancement;

[0027] Figure 5 Visualize Mixup;

[0028] Figure 6 It is the Inception B structure;

[0029] Figure 7It is the Inception C structure;

[0030] Figure 8 It is the SE module;

[0031] Fig. 9 It is the InceptionA structure;

[0032] Fig.10 This is the structural diagram of the tongue constitution identification model;

[0033] Fig.11 The accuracy curve (1) and loss value curve (2) during training;

[0034] Fig.12 The accuracy training curves of the test sets of the four models. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined in this application may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown in this application, but will conform to the widest scope consistent with the principles and novel features disclosed in this application.

[0037] like Figure 1 As shown, the present application provides a tongue image constitution identification method based on deep learning, which is as follows:

[0038] Obtain a tongue image to be identified, use the trained improved convolutional neural network GoogleNet model to identify the tongue image to be identified, and obtain a tongue image recognition result;

[0039] Display the recognition result of the tongue image on the mobile terminal;

[0040] Among them, the training process of the improved convolutional neural network GoogleNet model is as follows:

[0041] Obtain a tongue image data set, and perform enhancement processing on the tongue image data set to obtain a tongue image training data set, wherein the enhancement processing includes two methods: same-class enhancement and mixed-class enhancement;

[0042] The tongue image training data set is used to train the improved convolutional neural network GoogleNet model to obtain a trained improved convolutional neural network GoogleNet model;

[0043] Among them, the improved convolutional neural network GoogleNet model improvements include:

[0044] Each intermediate layer of the model is standardized, which effectively avoids gradient vanishing and gradient exploding, speeds up network convergence, and optimizes the network structure.

[0045] The residual is integrated into the Inception structure, and the Inception structure is improved and optimized. The input features are fused with the features output by the Inception structure, and it is named Inception B.

[0046] The 3×3 convolution branch in the Inception module is decomposed into 1×3 and 3×1 convolutions, and the 5×5 convolution is decomposed into 3×3 convolutions and 1×3, 3×1 convolutions. Batch Normalization is added after each branch, which effectively reduces the number of parameters of the model and is named Inception C.

[0047] Based on the original GoogleNet model, the unimproved Inception module was retained and named InceptionA. The InceptionA, InceptionB and InceptionC structures were fused and applied to the improved GoogleNet network model. The InceptionA module was added to the shallow layer, the InceptionB module was added to the middle layer, and the InceptionC module was added to the deep layer. The A and B modules were used twice respectively, and the C module has stronger feature extraction and perception capabilities, so it was used five times.

[0048] Among them, similar enhancement refers to the use of data normalization, horizontal or vertical flipping, proportional enlargement or reduction of images, random rotation, random shearing transformation, brightness change and pixel filling methods to expand the data set for images with the same label. Color is also an important feature in tongue constitution identification, so no color channel transformation is performed on the tongue image data set.

[0049] Among them, the mixed class enhancement method is to solve the problem of imbalance between classes of sample data sets. The mixed class enhancement method mixes sample data of two categories with different labels to construct a virtual training sample to enrich the tongue image data set. Its mathematical expression is shown in (1):

[0050]

[0051] Where (x i ,y i )(x j ,y j ) are two randomly selected samples and their corresponding labels, λ is a random number that follows beta distribution, and λ∈[0,1].

[0052] The technical effects achieved by this application are as follows:

[0053] 1. Add various improvement modules to the original GoogleNet model, such as adding a combination of the same category enhancement and mixup mixed category enhancement methods to enhance the tongue image dataset, adding batch normalization operations, and adding an attention mechanism module.

[0054] 2. Improve and optimize the Inception structure, the basic module of GoogleNet. For example, add a residual structure to the original Inception structure for feature fusion, add convolution replacement and decomposition operations to the Inception structure, and combine the two improved Inception structures with the original Inception structure to jointly construct a model.

[0055] 3. The method of the present invention adopts a convolutional neural network algorithm to enhance the tongue image dataset by combining the same type enhancement and mixed type enhancement methods. This enables the model to adapt to tongue image data in various environments, and the extracted features can directly output the classification results, which greatly reduces the amount of calculation and has stronger generalization ability.

[0056] 4 The method of the present invention is improved and optimized based on the current mainstream GoogleNet model. The classifier adopts the softmax classifier, which is a supervised learning method suitable for multi-classification problems such as physical fitness identification, and achieves better results in calculation methods and recognition effects.

[0057] 5 Compared with the traditional Chinese medicine constitution identification method, the method of the present invention reduces the diagnosis time and improves the diagnosis efficiency. Compared with the current method of using machine learning and deep learning, it improves the recognition accuracy.

[0058] 6 The method of the present invention is based on a large amount of tongue image data sets and applies deep learning technology to the field of traditional Chinese medicine constitution identification. It can not only perform constitution identification through the PC side, but also can perform constitution identification by porting to the mobile phone side. It is very convenient, highly accurate and saves time.

[0059] 7 The method of the present invention combines deep learning methods with traditional Chinese medicine constitution identification, and performs identification based on big data, effectively solving the problem of traditional constitution identification. This method has certain market value and promotion value.

[0060] The following introduces the specific method of tongue image and constitution identification based on deep learning.

[0061] 1. Introduction

[0062] Traditional Chinese Medicine (TCM) is a treasure of the Chinese nation, with a history of thousands of years of disease prevention and treatment. It not only plays an irreplaceable role in China's medical care and "preventive treatment", but is also accepted and frequently used by more and more people internationally. Treating the same disease with different treatments and treating different diseases with the same treatment are the basic principles of TCM in understanding and treating diseases. The premise and foundation of this dialectical treatment method depends on the individual's constitution. In TCM theory, constitution is considered to be a relatively stable inherent characteristic of the human body based on innate conditions and acquired acquired conditions, including human morphological structure and psychological temperament. TCM constitution is closely related to human health. Constitutional studies are also the basis and core content of TCM constitution research, affecting the diagnosis and treatment of the entire clinical discipline of TCM. Combining the definition of constitution proposed in "Huangdi Neijing" and the principles of modern clinical application of constitution, Professor Wang Qi proposed a nine-point constitution method, which classifies constitution types into balanced constitution, qi deficiency constitution, yang deficiency constitution, yin deficiency constitution, phlegm-damp constitution, damp-heat constitution, blood stasis constitution, qi stagnation constitution, and special constitution. Among them, the balanced constitution is a healthy constitution, and the other eight constitutions are biased constitutions.

[0063] Tongue diagnosis is the most commonly used method in TCM diagnosis. TCM believes that tongue images are related to the body's meridians and internal organs, and reflect the cold and heat, emptiness and fullness, and the rise and fall of yin and yang in the body's internal environment. Doctors judge the patient's health status and susceptibility to diseases by observing the changes in the color, texture, texture and morphology of the tongue coating, and give reasonable suggestions. Clinical practice has also proved that tongue diagnosis is a simple, effective and side-effect-free diagnostic method, and the tongue image contains information directly related to the human constitution, so it is feasible to use tongue diagnosis for constitution identification. The "Classification and Determination of TCM Constitution" published by China Traditional Chinese Medicine in 2009 elaborates on the tongue image characteristics corresponding to each constitution. For example, the book stipulates that "the tongue shape is plump and tender, the edges have tooth marks, and the tongue color is light or slightly dark blue, and the tongue coating is smooth" is the tongue image characteristic of people with yang deficiency constitution. In addition, other physical characteristics of constitution are introduced, which can be used to design questionnaires to identify constitution. However, questionnaires are very subjective, and it is difficult for individuals to make objective choices. There are many questions in the scale, which takes a lot of time and also affects the accuracy of judgment. With the popularization of artificial intelligence methods such as machine learning and deep learning, if it is possible to capture an individual's tongue image and store it in the form of a digital image, and then combine it with artificial intelligence technology, it will be possible to realize intelligent TCM constitution identification based on tongue diagnosis, thereby obtaining more objective and accurate results.

[0064] At present, the use of artificial intelligence technologies such as machine learning and deep learning for image recognition has become mature. Some scholars have established a tongue image database by collecting a large number of tongue images with constitution labels, and designed algorithms to match tongue images with the established database to achieve the purpose of recognition. Other scholars have designed quantitative methods to extract objective tongue image features such as the hue and saturation of tongue quality and tongue coating, the texture characteristics of tongue coating, smoothness and tooth mark characteristics, and established a TCM constitution identification model based on machine learning methods such as support vector machine (SVM), K-means, and artificial neural network. Considering the complexity of traditional manual extraction of tongue image features and the accuracy of tongue area segmentation, deep learning has been applied to constitution identification due to its advantages in automatic feature extraction. Some scholars have used various models of convolutional neural networks to identify tongue constitution. Although these studies have promoted the development of intelligent constitution identification by tongue diagnosis, there are also some common problems. First, deep learning requires a large number of data sets as support, and the sample data collection of tongue images is difficult. Second, simply using the model for classification without improving and optimizing the model leads to a low accuracy rate of constitution identification using the above methods. In order to address these issues, this application proposes a new method based on an improved convolutional neural network GoogleNet model. This method is used to identify physical constitution based on the collected tongue image dataset. The main contributions of this application are: constructing a tongue image dataset with physical constitution labels, and proposing a GoogleNet network model (TCR-GoogleNet) suitable for tongue image physical constitution identification by adding batch normalization operations, improving and optimizing the Inception module, adding an attention mechanism, and using different Inception structures in combination. Ablation experiments were designed to verify the contribution of each improved module to the prediction accuracy, and comparative experiments were designed with 5 mainstream networks for image classification, proving that the model proposed in this application has greatly improved accuracy and stability.

[0065] 2. Methods

[0066] This chapter introduces the specific process of body constitution identification through tongue images. The first part briefly describes the specific steps and architecture of the method. The second part briefly describes the construction of the tongue image set, including the basic process and requirements of image acquisition and the image preprocessing method. The third part introduces the preprocessing of tongue images and the data enhancement operation of the tongue image dataset. The fourth part introduces the improvement method of the module and the overall construction of the model.

[0067] 2.1 Method Architecture

[0068] In order to better identify tongue constitution, this application proposes a TCR-Googlenet model, which is trained on a large amount of data in the constructed tongue image dataset in order to obtain better constitution identification results. The basic architecture of this method is as follows Figure 2 As shown in the figure, there are five steps in total. The first step is to collect tongue images. The second and third steps are image preprocessing, including segmenting and cropping from the face image to extract the tongue area. The quality of the tongue image dataset can be greatly improved after segmentation and cropping. The fourth step is to input the extracted tongue image into the built deep neural network model for training and physical constitution recognition. Finally, different physical constitution types are classified and identified.

[0069] 2.2 Dataset Construction

[0070] (1) Description of the characteristics of the nine common body types

[0071] In TCM tongue diagnosis, different constitutions have different tongue characteristics, and doctors often judge the patient's constitution and pathology through these different tongue characteristics. Based on the theory proposed by Professor Wang Qi [6], this application takes the tongue characteristics of nine constitutions, namely, peaceful constitution, qi deficiency constitution, yang deficiency constitution, yin deficiency constitution, phlegm-damp constitution, damp-heat constitution, blood stasis constitution, qi stagnation constitution, and special constitution, as the research object. Their characteristics are shown in Table 1. These characteristic descriptions can provide a visual basis for subjective judgment of the accuracy of TCM constitution recognition by deep learning network models.

[0072] Table 1 Description of tongue characteristics of nine body types

[0073]

[0074]

[0075] (2) Construction of tongue image and constitution dataset

[0076] The deep learning network model imitates the human brain to extract image features from low-level to high-level abstraction based on a large-scale image dataset. In order to perform tongue image and constitution recognition based on deep learning, it is necessary to construct a tongue image dataset. By collecting tongue images of patients in the outpatient clinic of a traditional Chinese medicine hospital, a total of 1,752 tongue image data were collected. At the same time, the patients were asked to fill in a traditional Chinese medicine constitution questionnaire to determine their constitution type and label the tongue images. Each image was manually processed to accurately extract the tongue area image, thereby reducing the influence of other parts of the original collected image on the tongue image and constitution recognition. The tongue images were taken under natural conditions with bright light. The sample distribution of the tongue image dataset is shown in Table 2, where the numerical value represents the number of tongue images for each constitution. The samples corresponding to each constitution are shown in Table 2. Figure 3 shown.

[0077] Table 2 Dataset sample distribution

[0078]

[0079] 2.3 Data Augmentation

[0080] Since images taken with mobile phone cameras are affected by external factors such as light and angle, and the number of tongue images collected is relatively small, in order to prevent overfitting and low model robustness, data enhancement of the collected images is required. This application mainly uses a combination of similar enhancement and mixed enhancement to enhance the dataset.

[0081] (1) Same type of enhancement

[0082] Similar enhancement mainly refers to the use of data normalization, horizontal or vertical flipping, proportional enlargement or reduction of images, random rotation, random shearing, brightness change, and pixel filling to expand the data set for images with the same label. Color is also an important feature in tongue constitution identification, so no color channel transformation is performed on the tongue image data set. Figure 4 It is a set of pictures with similar data enhancement.

[0083] (2) Mixed Class Enhancement

[0084] The mixed class enhancement method is to solve the problem of imbalance between classes of sample data sets. For example, special constitution is an uncommon type of constitution, so the sample data is relatively small. This application uses the Mixup mixed class enhancement method to mix the sample data of two different labels to construct a virtual training sample to enrich the tongue image data set. Its mathematical expression is shown in (1):

[0085]

[0086] Where (x i ,y i )(x j ,y j ) are two randomly selected samples and their corresponding labels, λ is a random number that follows beta distribution, and λ∈[0,1]. Figure 5 This is the effect picture randomly generated by Mixup twice.

[0087] 2.4 Construction of tongue image constitution recognition model

[0088] This application improves the GoogleNet model for tongue image recognition. GoogleNet is the winner of the 2014 ILSVRC competition and is a deep convolutional neural network model. Since the development of the VGG model, convolutional neural networks have developed towards a deeper architecture, containing more and more network layers and parameters. However, blindly increasing the network will inevitably burden computing resources. The Inception module proposed in GoogleNet effectively alleviates the gradient vanishing and gradient exploding problems caused by deepening the network, and improves the operational efficiency of the model. This application proposes a deep neural network model based on the improved GoogleNet model, which is more suitable for this data set and has a higher accuracy rate. This section mainly introduces the construction of the tongue image recognition model and the improvement strategy.

[0089] (1) Batch Normalization (BN)

[0090] As the parameters of the neural network are continuously updated, the data distribution of each intermediate layer input is often quite different from that before the parameter update, which causes the network to constantly adapt to the new data distribution, increasing the difficulty of model training. The BN operation proposed in 2015 effectively solves this problem. This application performs standardization processing on each intermediate layer of the model (before the activation function), effectively avoiding problems such as gradient vanishing and gradient exploding, speeding up the network convergence, and optimizing the network structure. The calculation process of BN is as follows:

[0091] Suppose a batch of input variables are x1, x2,..., x k , calculate their mean and variance respectively, and their mathematical expressions are shown in (2)(3):

[0092]

[0093]

[0094] Then the input batch of x values ​​is normalized to a mean of 0 and a variance of 1. The mathematical expression is shown in (4):

[0095]

[0096] Formula (4) i Correcting the edges where the gradient disappears to areas where the gradient is larger increases the training speed, where ε is a very small value to avoid the variance denominator being 0 and not affecting the calculation.

[0097] Finally, the normalized value is reconstructed, and its mathematical expression is shown in (5):

[0098]

[0099] Among them, α and β are two learnable variables, the purpose of which is to compensate for the nonlinear expression ability of the model and restore the distribution that this layer of the network needs to learn.

[0100] (2) Inception structure residual design

[0101] He proposed the residual structure in 2015 and proved the advantages of feature additive merging through theory and practice. In image recognition, the residual structure can effectively avoid the network degradation problem caused by deepening the depth, and it also solves the gradient problem, so that the performance of the network is also improved. This application draws on this idea to integrate the residual into the Inception structure, improves and optimizes the Inception structure, and fuses the input features with the features output by the Inception structure. Figure 6 A residual Inception module is added for this application and named Inception B.

[0102] (3) Inception structure convolution decomposition and replacement

[0103] In Inceptionv3, the decomposition and replacement operations of convolution are proposed: decomposing large convolution kernels into symmetrical small convolution kernels; decomposing into asymmetric convolution kernels. The first method is to replace a 5ⅹ5 convolution kernel with two 3ⅹ3 convolution kernels. The second method is to replace the n*n convolution kernel with 1*n and n*1 convolution kernels and then stack them. Both methods can reduce the number of parameters and increase more nonlinear capabilities while keeping the receptive field unchanged. Therefore, these two methods are used in this application to improve the Inception module, decomposing the 3ⅹ3 convolution branch in the Inception module into 1ⅹ3 and 3ⅹ1 convolutions, decomposing the 5ⅹ5 convolution into 3ⅹ3 convolutions and 1ⅹ3, 3ⅹ1 convolutions, and adding Batch Normalization after each branch, which effectively reduces the number of parameters of the model. Figure 7 The Inception structure improved by this method is named InceptionC.

[0104] (4) SE attention module

[0105] The SE module consists of two operations: Squeeze and Excitation. Squeeze is a low-dimensional embedding of global information, which is a compression process. Excitation is a mapping transformation process. The SE module mainly explores the relationship between channels and assigns weights to network channels to enhance the learning ability of the network, which is also called channel attention. Figure 8 is a schematic diagram of the SE module.

[0106] (5) Inception structure fusion and model construction

[0107] This application retains the original GoogleNet model and retains the Inception module without any improvement. We name it Inception A. Fig. 9 As shown in the figure, we merged Inception A with the Inception B and Inception C structures mentioned above and applied them to the improved GoogleNet network model. Inception A modules were added to the shallow layer, Inception B modules were added to the middle layer, and Inception C modules were added to the deep layer. The A and B modules were used twice respectively, and the C module had a stronger ability to extract and perceive features, so it was used five times.

[0108] Combining the above methods and improvements, the overall structure of the improved tongue image constitution recognition model is proposed as follows: Fig.10 shown.

[0109] 3. Experimental design and result analysis

[0110] This experiment uses the win10 operating system, the CPU is Intel(R) Core(TM) i9-7900X3.30GHz, the GPU is GeForce RTX 3090, the Python version is 3.8.4, the Cuda and Cudnn versions are 11.2 and 8.1.0 respectively, and the tensorflow2.5.0 framework is used as the experimental environment.

[0111] 3.1 Parameter selection and optimization

[0112] In the process of deep learning model training, the selection of parameters often affects the accuracy of the model. In the process of parameter selection and optimization, this application uses multiple ablation experiments for fine-tuning. When using this model for training, a batch training method is adopted. The batch size (Batch size) is 32, and a total of 100 epochs are performed. The optimization function uses the Adam algorithm (Adaptive Moment Estimation), the momentum is 0.9, and the weight decay value is 1×10 -5 , the initial learning rate is 0.0001. In order to further prevent overfitting, the Dropout value is set to 0.3, that is, 30% of the nodes are randomly discarded, and the classifier adopts SoftMax.

[0113] 3.2 Evaluation indicators

[0114] In order to evaluate the quality of the model, the number of rounds (E) required for the model to converge, i.e., the convergence speed, confusion matrix, loss value (L), and Top-1 accuracy (Acc) are selected as evaluation indicators in model training. The number of rounds required for model convergence means the evaluation of the model convergence speed under the same conditions. Physical identification is a multi-classification problem, and the loss function adopts the cross entropy loss function of multi-classification. The loss value expression is:

[0115]

[0116] Where y i is the true label corresponding to the i-th sample; p i is the predicted value of the image for model training; N is the total number of constitution categories; K is the total number of tongue image samples.

[0117] The Top-1 accuracy reflects the proportion of correctly identified images in the recognition results to all recognized images, reflecting the training effect of the model on the data set. Its mathematical expression is:

[0118]

[0119] Among them, TP (True positive) is the positive sample predicted by the model as positive; TN (True negative) is the negative sample predicted by the model as negative; FN (False negative) is the negative sample predicted by the model as positive; FP (False positive) is the positive sample predicted by the model as negative.

[0120] 3.3 Ablation Experiment

[0121] As a common method of model evaluation, ablation experiment refers to deleting a part of the network model so that we can better understand the impact and contribution of the improved network module on the accuracy of the network model, and further illustrate the effectiveness of the improved algorithm. This application uses ablation experiments to verify the effectiveness of each improvement strategy and analyze the impact of each improved network module on the model accuracy.

[0122] This experiment designed experiments by adding batch normalization operations, using and combining Inception A, Inception B, and Inception C structures, and adding attention mechanisms. A total of 7 groups of experiments were designed, numbered 1-7. The experimental design is shown in Table 3.

[0123] Table 3 Ablation experiment design

[0124]

[0125] Table 4 Comparison of ablation experiment results

[0126]

[0127] The experimental results are shown in Table 4. The following conclusions can be drawn from the experimental results of each group:

[0128] (1) Batch Normalization (BN)

[0129] Experiment 1 only used the Inception A module and did not make any changes to the model, which was the original GoogleNet model. In Experiment 2, the BN operation was added. Comparing the results, the Top-1 accuracy of Experiment 2 was improved by 1.6% compared with that of Experiment 1. This shows that the BN operation optimized the network structure and improved the model accuracy to a certain extent.

[0130] (2)Mixup

[0131] In Experiment 3, the Mixup mixed-class enhancement method was used to expand the dataset. Compared with Experiment 2, the accuracy and loss value were optimized to varying degrees. The combination with similar data enhancement made up for the defect of less tongue image data to a certain extent, and made the model better fit the dataset with almost no additional computing overhead.

[0132] (3) Inception structure

[0133] In Experiment 4, all Inception A modules in the model were replaced with Inception B modules with residual structures, but the accuracy of the model dropped by about 2.5%, proving that simply stacking residual structures to widen the network is not a good way to adapt to the dataset of this application.

[0134] In Experiment 5, all Inception C modules were used, and the Top-1 accuracy was significantly improved, indicating that the network structure deepened by convolution decomposition and replacement had a better effect on extracting tongue feature information, reducing network model parameters while improving recognition accuracy.

[0135] Experiment 6 combines the operations of Experiment 2, Experiment 4, and Experiment 5, and uses a fusion method of three Inception structures to improve the model. At the same time, the network model is widened and deepened, and the extraction of tongue image features is more comprehensive. Its accuracy and loss value are better than using the Inception structure alone.

[0136] (4) SE module

[0137] Experiment 7 is the final model after improvement. It can be seen from the table that compared with the previous experiments, the Top-1 accuracy can be improved by 1.3% by adding the SE module. This shows that the introduction of the attention mechanism can improve the Top-1 accuracy of the model without increasing the network depth, width, or significantly increasing the network parameters.

[0138] 3.4 Experimental Results and Analysis

[0139] Based on the construction of the above model, we use 85% of the data set as the training set and 15% as the test set. When dividing the data set, we ensure that the data is random and reproducible. This application analyzes the experimental effects of the model of this application from three aspects: confusion matrix, training set and test set indicators of this model, and model comparison.

[0140] (1) Confusion Matrix

[0141] This model is applied to the test set of the tongue image dataset, and the resulting confusion matrix is ​​shown in Table 5. It includes the number of samples predicted correctly and incorrectly in the 9 physical images in the test set. In the confusion matrix, the values ​​on the main diagonal are the number of samples predicted correctly, and the values ​​in other positions are the number of samples predicted incorrectly.

[0142] Table 5 Confusion matrix of the test set

[0143]

[0144]

[0145] It can be seen from the table that Qi deficiency and Yang deficiency are easily confused in the classification process because these two constitutions may have similar tongue characteristics and small differences between categories, so the model is easy to confuse these two constitutions. The tongue characteristics of blood stasis and qi stagnation are obviously different from other constitutions, and the classification effect is better. Among them, the recognition rate of special constitution is the lowest, because there are very few samples of special constitution, and there are too few people with special constitution in reality. It is difficult to collect a large number of training samples and the tongue characteristics of special constitution are variable. The recognition accuracy of phlegm dampness and damp heat is the highest, thanks to the sufficient training samples, the model can fully extract the tongue characteristics.

[0146] (2) Recognition results of training set and test set

[0147] Fig.11The accuracy curve and loss value curve of the training set and test set during training. The training accuracy curve (1) can reflect the changing trend of the model prediction accuracy as the number of Epoch rounds increases. The training loss value curve (2) shows the deviation and fluctuation between the model prediction value and the true value as the number of Epoch rounds increases. The lower the loss value, the higher the prediction accuracy of the model, which means that the probability of the model prediction error is smaller. The training set accuracy of the model of this application reached 92.2%, the loss value was 0.375, and the test set accuracy reached 87.4%, the loss value was 0.501. Compared with other similar studies at this stage, this model has higher recognition accuracy and stability.

[0148] (3) Model comparison results

[0149] In order to verify the effectiveness of the model in this application and the effect after improvement, this application selected five mainstream convolutional neural network models for image recognition for comparative experiments. Fig.12 This is a line chart comparison of the accuracy of the model of this application and five models, namely VGG-16, resnet-50, mobilenet-v3-large, mobilenet-v3-small, and GoogleNet original model, for the test set training to show the difference from other models. Table 3 intuitively lists the Top-1 accuracy, loss value, and number of epochs to converge (convergence speed) of the test sets of the six models. We use the accuracy of the test set as the standard for model comparison because the test set can better characterize the model's ability to fit the data set and provide a more accurate assessment of the model's training accuracy. In model evaluation, the priority of model prediction accuracy is greater than the convergence speed.

[0150] In this experiment and the comparative experiment, the neural network model was trained from scratch, so the initial accuracy was low, but this model was able to reach the highest Top-1 accuracy of 87.4% at a relatively fast speed. Although the model in this application converges slower than the three models of GoogleNet, mobilenet-v3-large, and mobilenet-v3-small, the Top-1 accuracy of this model is 11.5% higher than that of the GoogleNet model, 17.1% higher than that of the mobilenet-v3-large model, and 19.5% higher than that of the mobilenet-v3-small model. The advantage of the mobilenet-v3 model is that it uses deep separable convolution, which greatly reduces the amount of calculation and parameters, so the mobilenet-v3 series has a great advantage in terms of convergence speed, but after convergence on this data set, the oscillation is large, the model is not stable, and the accuracy is low. The Resnet-50 network model with residuals has good performance in both Top-1 accuracy and loss value, indicating that the fused residual structure has a good representation ability for tongue image features. Therefore, we also added a residual structure during the construction of this model. Combined with other improved modules, the recognition effect was 5.6% higher than the Top-1 accuracy of Resnet-50, the loss value decreased by 0.112, and the convergence speed was also slightly improved. The VGG-16 model improves the model training accuracy by deepening the number of network layers and reducing the size of the convolution kernel, but the effect did not meet expectations in the comparative experiment. The accuracy rate was only 72.8%, which was a big gap from the experimental results of the model in this application, proving that simply deepening the network structure cannot extract tongue image features well. In the network model of this application, we not only improved the model in terms of network depth and network width, but also proved through ablation experiments that the multiple improved modules added on this basis have improved the prediction accuracy of the model to a certain extent.

[0151] Table 6 Comparison of model results

[0152]

[0153] Overall, the improved model in this application has a significant improvement in recognition accuracy compared with other models, and its convergence speed is moderate, but the priority of model prediction accuracy is higher than the convergence speed. Therefore, the improved model proposed in this application has better recognition effect on the tongue image dataset than the other five models.

[0154] This application proposes the TCR-GoogleNet model based on GoogleNet and constructs a mature tongue image dataset. At the same time, we should continue to collect images and expand the tongue image dataset to provide more powerful data support for subsequent research work. With the advantages of network width and depth and various improvements, this model shows strong feature extraction capabilities and good recognition accuracy for tongue image datasets. For ordinary tongue images, nine types of TCM constitutions can be effectively identified, including peaceful constitution, qi deficiency constitution, yang deficiency constitution, yin deficiency constitution, phlegm and dampness constitution, damp-heat constitution, blood stasis constitution, qi stagnation constitution, and special constitution. To a certain extent, it alleviates the difficulty that constitution classification requires rich expert experience. The comparison questionnaire method greatly shortens the time for constitution identification and provides reference and ideas for the objectification process of tongue diagnosis.

[0155] There is still more room for development in the future for constitution identification based on tongue diagnosis. The tongue image dataset can be expanded more abundantly, and the recognition accuracy based on this can be further improved. Since everyone's constitution is diverse, a person may have multiple constitutions, so subsequent research can be carried out around multi-label learning of constitution identification based on tongue images. In addition, tongue diagnosis research can not only be used to identify constitution, but there are now studies that combine tongue diagnosis with Western medicine symptoms to prevent and treat diseases, but research in this area needs further development.

[0156] The various parts in this manual are described in a progressive manner, and each part focuses on the differences from other parts. The same or similar parts between the various parts can be referenced to each other.

[0157] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined in this application may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown in this application, but will conform to the widest range consistent with the principles and novel features disclosed in this application.

Claims

1. A tongue image constitution identification method based on deep learning, characterized in that: The method is deployed in a mobile terminal, and the method includes: Obtain a tongue image to be identified, use the trained improved convolutional neural network GoogleNet model to identify the tongue image to be identified, and obtain a tongue image recognition result; Display the recognition result of the tongue image on the mobile terminal; Among them, the training process of the improved convolutional neural network GoogleNet model is as follows: Obtain a tongue image data set, and perform enhancement processing on the tongue image data set to obtain a tongue image training data set, wherein the enhancement processing includes two methods: same-class enhancement and mixed-class enhancement; The tongue image training data set is used to train the improved convolutional neural network GoogleNet model to obtain a trained improved convolutional neural network GoogleNet model; Among them, the improved convolutional neural network GoogleNet model improvements include: Each intermediate layer of the model is standardized, which effectively avoids gradient vanishing and gradient exploding, speeds up network convergence, and optimizes the network structure. The residual is integrated into the Inception structure, and the Inception structure is improved and optimized. The input features are fused with the features output by the Inception structure, and it is named Inception B. The 3×3 convolution branch in the Inception module is decomposed into 1×3 and 3×1 convolutions, and the 5×5 convolution is decomposed into 3×3 convolutions and 1×3, 3×1 convolutions. Batch Normalization is added after each branch, which effectively reduces the number of parameters of the model and is named Inception C. Based on the original GoogleNet model, the unimproved Inception module was retained and named Inception A. The Inception A, Inception B and Inception C structures were fused and applied to the improved GoogleNet network model. The Inception A module was added to the shallow layer, the Inception B module was added to the middle layer, and the Inception C module was added to the deep layer. The A and B modules were used twice respectively, and the C module has stronger feature extraction and perception capabilities, so it was used five times.

2. The method according to claim 1, characterized in that Similar enhancement refers to the use of data normalization, horizontal or vertical flipping, proportional enlargement or reduction of images, random rotation, random shearing, brightness change, and pixel filling methods to expand the data set for images with the same label. Color is also an important feature in tongue constitution identification, so no color channel transformation is performed on the tongue image data set.

3. The method according to claim 2, characterized in that The mixed class enhancement method is to solve the problem of imbalance between classes of sample data sets. The mixed class enhancement method mixes sample data of two categories with different labels to construct a virtual training sample to enrich the tongue image data set. Its mathematical expression is shown in (1): Where (x i ,y i )(x j ,y j ) are two randomly selected samples and their corresponding labels, λ is a random number that follows beta distribution, and λ∈[0,1].

Images