Traditional Chinese medicine tongue picture multi-examination feature classification method based on multi-label learning

Abstract

The invention provides a traditional Chinese medicine tongue picture multi-examination feature classification method based on multi-label learning, and aims to realize automatic analysis of various features in traditional Chinese medicine tongue diagnosis through computer vision and deep learning technologies. According to the method, DenseNet-121 is adopted as a backbone network, and tongue image features are extracted in combination with an efficient multi-scale attention (EMA) mechanism. A multi-label learning network is constructed by using a Gaussian mixture variational auto-encoder and comparative learning so as to collaboratively classify the characteristics of tongue color, fur color, greasy taste, dryness and the like. According to the network, the feature expression and distinguishing capability is improved through integration of probability modeling and feature mapping. In addition, a symmetric correction cross entropy loss function and a comparative learning loss function are used for model training, so that the classification precision and robustness are improved. The method has the advantages that the classification precision is high, the system implementation complexity is low, and an effective technical scheme is provided for automation of traditional Chinese medicine tongue diagnosis.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and medical image processing, and specifically relates to technologies such as computer image processing, deep learning, and traditional Chinese medicine tongue image analysis. Background Art

[0002] Tongue diagnosis is an important part of the "four diagnoses" of traditional Chinese medicine. Doctors understand the physiological functions and pathological changes of the human body by observing various appearances of the tongue, such as tongue color, coating color, tongue shape, and tongue dynamics. Tongue diagnosis focuses on more than 30 tongue image diagnostic features, such as tongue color, coating color, moistness, greasy, tooth marks, cracks, punctures, fatness, etc. When existing methods automatically analyze tongue images, they generally adopt a separate analysis method, that is, design a classification model for each diagnostic feature. Although ideal classification performance can be obtained, it will also cause the system implementation complexity to be too high and difficult to deploy and implement.

[0003] The present invention regards the analysis of multiple tongue diagnostic features as a multi-label classification problem, and adopts a multi-label learning method to simultaneously realize the automatic analysis of multiple tongue diagnostic features such as tongue color, tongue coating color, greasy and moist, etc. in a unified network framework. Specifically, by using contrastive learning and Gaussian mixture variational autoencoder, by utilizing the correlation between various tongue diagnostic features, not only can the classification performance be improved, but also the complexity of the overall system implementation can be reduced. Summary of the invention

[0004] The present invention proposes a classification method for multiple diagnostic features of tongue images in traditional Chinese medicine, aiming to achieve accurate classification of multiple tongue image diagnostic features such as tongue color, tongue coating color, greasy and moist, and dry, by using a multi-label learning strategy under a unified network framework.

[0005] In order to achieve the above objectives, the present invention proposes the following technical solutions: First, DenseNet-121 is used as the backbone network, and the efficient multi-scale attention (EMA) mechanism is combined as the feature extraction network to extract the features of the tongue image; then, a Gaussian mixture variational autoencoder is used to construct a multi-label learning network. Specifically, a Gaussian mixture variational autoencoder is used to learn a latent space containing multiple Gaussian distributions, each Gaussian distribution corresponding to a different label category. By integrating probabilistic modeling with feature mapping, the model can learn the deep latent feature expression shared by different tongue image diagnostic features and labels. At the same time, contrastive learning is used to maximize the similarity between the features of samples of the same category in different latent spaces, while minimizing the similarity between the features of samples of different categories. By combining the above two methods, the expression and discrimination capabilities of the features are improved, and the collaborative classification of multiple tongue image diagnostic features is achieved.

[0006] The solution includes four steps: construction of tongue image multi-diagnosis feature dataset, design of lightweight convolutional neural network based on efficient multi-scale attention mechanism, design of multi-label learning network based on contrastive learning and Gaussian mixture variational autoencoder, and training of multi-label classification network model. The implementation details of these steps are introduced in detail below.

[0007] Step 1: Construction of tongue image multi-diagnosis feature classification dataset

[0008] The present invention uses a traditional Chinese medicine tongue imager as a data set acquisition device, cooperates with three well-known Chinese medicine hospitals in China to collect tongue images, and labels the tongue images one by one. Each tongue image is labeled with four labels: tongue color, tongue coating color, greasy and moist, and the category of each label is shown in Table 1.

[0009] Diagnostic characteristics category Tongue color Dark red, light red, red, purple 4 categories Moss Yellow, white, gray, gray-black, black 5 categories Rotten 3 categories: rotten, greasy, normal Moisturizing Dry, smooth, normal 3 categories

[0010] Three tongue image multi-diagnostic feature classification datasets were constructed in turn, denoted by SIPL-A, SIPL-B, and SIPL-C respectively.

[0011] Step 2: Design of lightweight convolutional neural network based on efficient multi-scale attention mechanism

[0012] The DenseNet network uses a dense connection mechanism, that is, all layers are connected to each other, and each layer is connected to all previous layers in the channel dimension to achieve feature reuse as the input of the next layer. This not only slows down the gradient disappearance phenomenon, but also enables it to achieve better classification performance with fewer parameters and less computation.

[0013] The present invention uses DenseNet-121 as the backbone network to construct a lightweight convolutional neural network structure. The EMA mechanism is embedded after the Dense Block4 layer to enhance important channels and suppress unimportant channels. The constructed network structure is as follows: Figure 2 As shown, the EMA network structure is as follows Figure 3 As shown, the number of parameters of the network model is 8.43M.

[0014] Step 3: Multi-label learning network design based on contrastive learning and Gaussian mixture variational autoencoder The overall framework of the multi-label learning network designed by the present invention is as follows Figure 4As shown. The network uses Gaussian mixture variational autoencoders to learn multiple latent spaces that conform to Gaussian distribution, and integrates probability modeling with feature mapping to learn the four diagnostic features of tongue color, tongue coating color, greasy and moist, and the potential deep feature space shared by labels. Contrastive learning is used to minimize the distance between the features of the same samples in different latent spaces, while maximizing the feature distance between samples of different categories. By combining the above two methods, the expression and differentiation capabilities of features are improved, and the collaborative classification of multiple diagnostic features of tongue images is achieved.

[0015] Step 4: Tongue image multi-label classification network model training

[0016] The present invention divides the tongue image multi-diagnosis feature classification data set constructed in step 1 into a training set and a test set, with a division ratio of 8: 2. The total loss function combines the symmetric modified cross-entropy loss function (SymmetricModified Cross-Entropy, SMCE) with noise resistance and the loss function based on contrastive learning (Information NoiseContrastive Estimation, InfoNCE) to optimize and supervise the training process of the network model.

[0017] Through training, a tongue image multi-label classification network model was obtained, which can simultaneously analyze and predict the four diagnostic characteristics of tongue color, tongue coating color, greasy and moist.

[0018] Compared with the prior art, the present invention has the following obvious advantages and practical application value:

[0019] 1. High classification accuracy. Compared with the traditional single-label classification model, the multi-label classification network can effectively utilize the correlation between various diagnostic features through collaborative learning of different labels, thereby improving the accuracy of the overall prediction of the model.

[0020] The overall implementation complexity of the system is low. Compared with the need to train a model for each label separately, the multi-label classification network can predict multiple labels under a unified framework, thereby greatly reducing the overall implementation complexity of the system. In addition, the backbone network uses DenseNet, which improves the expressiveness of features by introducing an efficient multi-scale attention mechanism EMA, and can achieve better classification performance with a lightweight network structure. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 These are some tongue image samples from the SIPL-A, SIPL-B, and SIPL-C datasets.

[0022] Figure 2 This is the DenseNet network structure diagram based on the EMA mechanism.

[0023] Figure 3 It is the EMA network structure diagram.

[0024] Figure 4 This is the overall framework diagram of the multi-label learning network. DETAILED DESCRIPTION

[0025] In combination with the above method description and the accompanying drawings, the process of the present invention is further described in detail.

[0026] Step 1: Construct a tongue image multi-diagnosis feature classification dataset

[0027] The present invention uses a traditional Chinese medicine tongue imager as a data set acquisition device, and each tongue image sample is annotated with multi-label data. The SIPL-A data set contains 430 samples, the SIPL-B data set contains 383 samples, and the SIPL-C data set contains 720 samples. Each sample contains four labels: tongue color, coating color, greasy and moist.

[0028] Step 2: Design of lightweight convolutional neural network based on efficient multi-scale attention mechanism

[0029] The model uses DenseNet-121 as the backbone network, adds an efficient multi-scale attention mechanism EMA, and uses Gaussian Error Linear Unit (GeLU) as the activation function to construct a lightweight convolutional neural network.

[0030] DenseNet-121 consists of 4 Dense Blocks, each of which contains multiple convolutional layers. Each layer of the network uses a dense connection mechanism so that the output of each layer is connected to all previous layers in the channel dimension, thereby achieving feature reuse. For example, in Dense Block 1, the output of the first convolutional layer is connected to the input of the second convolutional layer, and the output of subsequent convolutional layers is continuously spliced ​​to the subsequent layers in the channel dimension. This design not only maximizes the reuse of previous features, but also avoids the problem of gradient disappearance. Another advantage of the DenseNet structure is that it retains low-dimensional features and fuses them with high-level features, allowing the network to make full use of the different characteristics of low-level and high-level features.

[0031] In the transition layer after Dense Block4, an efficient multi-scale attention mechanism EMA is introduced for feature extraction. The transition layer first reduces the number of channels through 1×1 convolution and the feature map size through 2×2 average pooling layer. Then the EMA mechanism divides the input features into g sub-features across the channel dimension (g is 16 or 32, within which the network can learn more fine-grained multi-scale features without excessively increasing the computational complexity). These sub-features are processed independently and learn different semantic information at different scales.

[0032] The features obtained through the EMA mechanism are fused with the features of the Dense Block 2 layer after the 1×1 convolution layer and the 2×2 average pooling layer, and sent to the activation function layer for processing. Compared with the traditional DenseNet-121, the network with the EMA mechanism allows the model to be more flexible in processing multi-scale features, can capture finer-grained local information and richer global semantic relationships, and improves the model's adaptability to complex data patterns, especially in multi-label classification tasks.

[0033] In addition, the network uses Gaussian Error Linear Unit (GeLU) as the activation function. Compared with traditional activation functions such as Rectified Linear Unit (ReLU), it avoids the derivative being always equal to 0 and eliminates the gradient vanishing problem. In addition, the nonlinear transformation of GeLU introduces a transformation similar to the logistic function (Sigmoid), which helps to accelerate the convergence speed of the model.

[0034] Step 3: Multi-label learning network design based on contrastive learning and Gaussian mixture variational autoencoder

[0035] The tongue image sample is subjected to feature extraction by the lightweight convolutional neural network designed in step 2 to generate the corresponding feature representation. The Gaussian mixture variational autoencoder structure inherits the general variational autoencoder framework (VariationalAutoencoder), in which the label encoder, feature encoder and decoder are all implemented using the variational autoencoder structure, but the idea of ​​Gaussian mixture model (GMM) is introduced to construct a Gaussian mixture variational autoencoder, which further enables the network to effectively handle multi-label classification tasks.

[0036] In this framework, the label category of each sample is converted into a label embedding vector w i, represents the embedding vector corresponding to the i-th label category, and then these embedding vectors are sent to the label encoder. The label encoder processes the input label embedding vectors in turn. Finally, the Gaussian distribution parameters Ν(μ,σ 2 )(where the mean is μ and the variance is σ 2 ), that is, the potential representation of this label category obeys a specific Gaussian distribution. In this way, the Gaussian mixture variational autoencoder can effectively incorporate label information into the learning process of the latent space, maintaining the excellent properties of the variational autoencoder while using the Gaussian mixture model to capture the complexity of the data.

[0037] When multiple label categories coexist, the Gaussian distributions of different categories can be used to represent the complex structure of the entire latent space through the Gaussian mixture model. i =1, indicating that the label information exists), the label corresponds to a Gaussian distribution, and these Gaussian distributions are combined to form a Gaussian mixture space. This space retains the label information and is easier to decode back to the label. At the same time, the high-dimensional feature vector w obtained by the feature extraction network of the tongue image sample x After the feature encoder, it is mapped to the latent space. This feature vector is aligned with the label embedding vector in the same latent space, and finally the decoder outputs the feature embedding vector w′ x Therefore, different features are mapped to the latent space together, and the model can learn the deep latent feature expression shared by different tongue image diagnosis features and label information to promote correct label prediction.

[0038] At the same time, the present invention adopts the paradigm of contrastive learning, combined with the contrastive loss function InfoNCE, to learn useful features by constructing positive sample pairs and negative sample pairs. Contrastive learning minimizes the feature distance of positive sample pairs in the latent space, making samples of the same type have a higher similarity; at the same time, it maximizes the feature distance of negative sample pairs in the latent space, ensuring that samples of different categories are distinguished from each other in the latent space, thereby improving the model's discriminative ability.

[0039] Step 4: Tongue image multi-label classification network model training

[0040] The objective function of the present invention adopts a combination of SMCE and InfoNCE to train the multi-label learning classification network. Considering that there may be noisy labels in the labeled samples, the present invention adopts a noise-robust loss function, namely the symmetrically modified cross entropy loss function SMCE, to suppress the influence of noise samples on network training performance and improve the robustness of classification. Its expression is as follows:

[0041]

[0042] Where x is the input data, y is the true label, py(y|x) represents the probability distribution when the label prediction is correct, d is the dynamic adjustment parameter, N represents the number of categories, α and β are adjustable parameters, and the loss function with the best model performance can be achieved by adjusting the appropriate α and β. Here, the parameters are set to α=0.4 and β=0.6.

[0043] The loss function InfoNCE is a contrastive learning loss function, which is derived from the NCE Loss (NoiseContrastive Estimation Loss) function. It can maximize the similarity of features between samples of the same category and minimize the similarity of features between samples of different categories. Its expression is:

[0044]

[0045] Where τ is the temperature parameter, which is used to adjust the sensitivity of similarity between different samples. τ is set to 0.4, K is the number of negative samples, and z i represents the feature embedding vector of the query sample (i.e., the target sample we are trying to learn), z j represents the feature embedding vector of positive sample j (positive sample usually represents the target category sample), z k Represents the feature embedding vector of negative sample k (negative samples usually represent samples of non-target categories). sim(z i ,z j ) and sim(z i ,z k ) represent the similarity between the query sample and the positive and negative samples respectively, and the cosine similarity is used for calculation.

[0046] The loss function of each type of tongue image label can be expressed as:

[0047] Loss i =L SMCE +μL InfoNCE (3)

[0048] Where μ is a hyperparameter (ranging between 0 and 1) used to adjust the weights of the two loss functions, Loss iThey represent Loss1 for tongue color label training, Loss2 for moss color label training, Loss3 for greasy label training, and Loss4 for moist label training. For each category, the data set is first divided into a training set and a validation set (in a ratio of 8:2), and the model is trained under different μ values, and evaluated by the loss on the validation set, and the μ value that minimizes the validation set loss is selected as the optimal weight parameter for each type of loss. The present invention also adds a KL divergence term (Kullback-Leibler) to the target loss function to measure the difference between the latent space distribution p(z|x) and the unit Gaussian distribution q(z), thereby helping the model learn a meaningful latent space that conforms to the expected distribution and better classifies labels.

[0049] Therefore, the total loss function Loss can be composed of five parts, and the formula is as follows:

[0050] Loss = L K-L +λ1Loss1+λ2Loss2+λ3Loss3+λ4Loss4(4)

[0051] The expression of KL divergence formula is:

[0052]

[0053] Wherein p(z|x) is the potential space distribution, which indicates the approximate posterior distribution of the potential variable z given the input data x, and q(z) is the prior distribution of the potential variable, which is usually set to the standard normal distribution N(0,1) (i.e., a Gaussian distribution with a mean of 0 and a variance of 1) in the variational autoencoder. λ1, λ2, λ3, and λ4 are the weight coefficients of each loss function respectively. The network structure proposed in the present invention is built and trained on the PyTorch platform, and the training GPU is NVIDIA GeForce GTX1080Ti. During model training, the Adam gradient descent algorithm is adopted, the learning rate is set to 0.001, the batch size (Batch Size) is set to 32, and the number of iterations (Epoch) reaches 500 times. After experimental observation, when Loss1 reaches the minimum, λ1, λ2, λ3, and λ4 can be determined at this time, the model converges, and the total loss function Loss value is the minimum, that is, the model performs well on multiple tasks at the same time, and a lightweight and efficient tongue image multi-diagnosis feature classification network model is obtained.

Claims

1. A method for classifying multiple features of tongue images in traditional Chinese medicine based on multi-label learning, characterized in that: The following steps are involved: Step 1: Construct a tongue image multi-diagnosis feature classification dataset A traditional Chinese medicine tongue imager was used as the data set collection device, and each tongue image sample was annotated with multi-label data; each sample contained four labels: tongue color, coating color, greasy and moist; Step 2: Design of lightweight convolutional neural network based on efficient multi-scale attention mechanism The model uses DenseNet-121 as the backbone network, adds an efficient multi-scale attention mechanism EMA, and uses Gaussian error linear unit GeLU as the activation function to construct a lightweight convolutional neural network; the details are as follows: DenseNet-121 consists of 4 Dense Blocks, each of which contains multiple convolutional layers. Each layer of the network uses a dense connection mechanism, so that the output of each layer is connected to all previous layers in the channel dimension, thereby achieving feature reuse. In the transition layer after Dense Block4, an efficient multi-scale attention mechanism EMA is introduced for feature extraction; the transition layer first reduces the number of channels through 1×1 convolution and the feature map size through 2×2 average pooling layer, and then the EMA mechanism divides the input features into g sub-features across the channel dimension; g is 16 or 32; these sub-features are processed independently and learn different semantic information at different scales; The features obtained through the EMA mechanism are fused with the features of the Dense Block 2 layer after passing through the 1×1 convolution layer and the 2×2 average pooling layer, and sent to the activation function layer for processing; Step 3: Multi-label learning network design based on contrastive learning and Gaussian mixture variational autoencoder In this framework, the label category of each sample is converted into a label embedding vector w i , represents the embedding vector corresponding to the i-th label category, and then these embedding vectors are sent to the label encoder; the label encoder processes the input label embedding vectors in turn; finally, the Gaussian distribution parameters Ν(μ,σ 2 ), where the mean is μ and the variance is σ 2 , that is, the potential representation of this label category obeys a specific Gaussian distribution; When multiple label categories coexist, the Gaussian distributions of different categories can be used to represent the complex structure of the entire latent space through the Gaussian mixture model; when a label i is activated, that is, y i =1, indicating that the label information exists, and the label corresponds to a Gaussian distribution. These Gaussian distributions are combined together to form a Gaussian mixture space; At the same time, the tongue image sample obtains a high-dimensional feature vector w through the feature extraction network x After the feature encoder, it is mapped to the latent space; This feature vector is aligned with the label embedding vector in the same latent space, and finally the decoder outputs the feature embedding vector w′ x ;; It adopts the contrastive learning paradigm and combines the contrastive loss function InfoNCE to learn useful features by constructing positive and negative sample pairs. Step 4: Tongue image multi-label classification network model training The objective function adopts a combination of SMCE and InfoNCE to train the multi-label learning classification network. A noise-robust loss function, namely the symmetrically modified cross entropy loss function SMCE, is used to suppress the impact of noise samples on network training performance and improve the robustness of classification. Its expression is as follows: Where x is the input data, y is the true label, p(y|x) represents the probability distribution when the label prediction is correct, d is the dynamic adjustment parameter, N represents the number of categories, α=0.4 and β=0.6; The contrast loss function InfoNCE is expressed as: Where τ is the temperature parameter, which is used to adjust the sensitivity of similarity between different samples. τ is set to 0.4, K is the number of negative samples, and z i represents the feature embedding vector of the query sample (i.e., the target sample we are trying to learn), z j represents the feature embedding vector of positive sample j, where positive sample represents the target category sample, z k Represents the feature embedding vector of negative sample k, where negative samples are samples of non-target categories; sim(z i ,z j ) and sim(z i ,z k ) represent the similarity between the query sample and the positive and negative samples respectively, and are calculated using cosine similarity; The loss function of each type of tongue image label is expressed as: Loss i =L SMCE +μL InfoNCE (3) Where μ is a hyperparameter ranging from 0 to 1, Loss i They represent Loss1 for tongue color label training, Loss2 for moss color label training, Loss3 for greasy label training, and Loss4 for moist label training. For each category, the dataset is first divided into a training set and a validation set. The models are trained under different μ values ​​and evaluated by the loss on the validation set. The μ value that minimizes the loss of the validation set is selected as the optimal weight parameter for each category of loss. The total loss function Loss consists of five parts, and the formula is as follows: Loss=L K-L +λ1Loss1+λ2Loss2+λ3Loss3+λ4Loss4(4) The expression of KL divergence formula is: Where p(z|x) is the latent space distribution, which means the approximate posterior distribution of the latent variable z given the input data x. q(z) is the prior distribution of the latent variable, which is set to the standard normal distribution N(0,1) in the variational autoencoder. λ1, λ2, λ3, and λ4 are the weight coefficients of each loss function.

Images