A collaborative classification method of tongue color and fur color in traditional Chinese medicine based on convolutional neural network

Abstract

A convolutional neural network-based collaborative classification method for tongue color and fur color in traditional Chinese medicine belongs to the field of computer vision and traditional Chinese medicine diagnosis. Since both tongue color and fur color are identified by color features, both require the extraction of color features from the tongue body area, and the two tasks have similarities. The method first designs a shared deep neural network architecture to extract the common deep features of tongue color and fur color contained in the traditional Chinese medicine tongue image, as well as the unique semantic features of the tongue image; then, the labels of tongue color and fur color are encoded and combined to obtain the combined label vector of tongue color and fur color; finally, a deep neural network is designed to establish a mapping model between the common deep features of tongue color and fur color and the combined label vector through training. By adopting such a mapping method, the recognition of the two diagnostic features of tongue color and fur color can be realized at the same time, which is not only simple to implement, but also makes full use of the intrinsic correlation between the two attributes of tongue color and fur color, and can obtain a higher recognition accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and Chinese medicine diagnosis, and specifically relates to computer image processing, deep learning, Chinese medicine tongue diagnosis and other technologies. Background Art

[0002] Tongue diagnosis is the core content of the four diagnoses in traditional Chinese medicine. Doctors diagnose symptoms by observing the patient's tongue color, tongue coating, greasy, fat and thin tongue characteristics (hereinafter referred to as tongue diagnosis characteristics), which is one of the key links in the diagnosis and treatment of syndromes in traditional Chinese medicine. With the changes of the times, the modernization of tongue diagnosis has become an inevitable path for the development of traditional Chinese medicine.

[0003] Accurate analysis of TCM diagnostic features is a fundamental problem in the objectification of TCM tongue images. Commonly used TCM tongue image diagnostic features include tongue color, tongue coating color, thickness of tongue coating, greasy, moist, cracked and other categories. Among them, tongue color and tongue coating color are the two most important diagnostic features in TCM diagnosis and treatment. TCM tongue color can be divided into five categories: light white, light red, red, dark red, and purple. The coating color can be divided into nine categories: thin white coating, white coating, thick white coating, thin yellow coating, yellow coating, thick yellow coating, brown coating, gray coating, and black coating. When using computers to automatically analyze TCM tongue color and coating color, it is often regarded as a classification problem and implemented using pattern recognition methods.

[0004] Traditional Chinese medicine tongue color and fur color analysis generally adopts the processing framework of "manually designed features + classifier". First, special feature extraction methods are designed according to the characteristics of tongue color and fur color, and then different classifiers are trained using the extracted features to identify tongue color and fur color. This processing method has the following shortcomings:

[0005] (1) Due to the limited expressive power of artificial features, the accuracy of tongue color and coating color analysis is not high, and there is still a long way to go before clinical application;

[0006] (2) For tongue color and coating color, different features are extracted and different classifiers are designed to implement them, resulting in a relatively high complexity in the implementation of the algorithm.

[0007] In recent years, deep learning has achieved great success in many fields such as image classification, object detection, and semantic segmentation. Compared with traditional machine learning methods, deep learning can automatically learn hierarchical feature representations directly from massive data. By building a multi-layer neural network, the machine can automatically learn the relationships implicit in the data. Compared with manually designed features, deep features are more robust and have stronger expressive power. They have outstanding advantages in expressing multi-level and contextual information of images.

[0008] The present invention applies deep learning technology to tongue diagnosis in traditional Chinese medicine, and proposes a tongue color and fur color collaborative analysis method based on a deep convolutional neural network. Since both tongue color and fur color are identified by color features, both need to extract color features from the tongue body area, and the two tasks have similarities. Therefore, the present invention adopts the idea of ​​multi-task learning to achieve collaborative identification of tongue color and fur color. This method does not adopt the common idea of ​​"feature extraction + classifier", but adopts the idea of ​​"feature extraction + mapping", that is: first, a shared deep neural network architecture is designed to extract the common deep features of tongue color and fur color contained in the traditional Chinese medicine tongue image, as well as the unique semantic features of the tongue image; then, the labels of tongue color and fur color are encoded and combined to obtain the combined label vector of tongue color and fur color; finally, a deep neural network is designed, and a mapping model between the common deep features of tongue color and fur color and the combined label vector is established through training. By adopting such a mapping method, the identification of the two diagnostic features of tongue color and fur color can be realized simultaneously under one framework, which is not only simple to implement, but also makes full use of the intrinsic correlation between the two attributes of tongue color and fur color, and can obtain a higher recognition accuracy. Summary of the invention

[0009] Objective: To propose a collaborative classification method for tongue color and coating color in traditional Chinese medicine. By mining and utilizing the potential correlation between tongue color and coating color, only one deep convolutional neural network architecture can be used to simultaneously achieve accurate analysis of the two diagnostic features of tongue color and coating color.

[0010] In order to achieve the above objectives, the present invention proposes the following technical solutions: first, ResNeXt is used as the backbone network to extract the common deep features of tongue color and tongue coating color in tongue images; then, the original labels of tongue color and tongue coating color are binary encoded and combined to form a binary label combination vector; finally, a deep neural network is designed to establish a mapping relationship between the deep features of tongue images and the binary label combination vector, so as to realize the collaborative recognition of tongue color and tongue coating color in traditional Chinese medicine.

[0011] The scheme includes three steps: constructing a data set, extracting the common depth features of tongue color and fur color in the tongue body area, and establishing a tongue color and fur color collaborative recognition model. Each step is described in detail below.

[0012] Step 1: Build the dataset

[0013] Deep learning needs to be supported by big data. The present invention first needs to construct a data set for training a deep neural network model.

[0014] 1.1 Tongue segmentation

[0015] The present invention collects TCM tongue images through a SIPL-type TCM tongue imager, and the obtained image contains not only tongue images but also other areas such as human face and lips. Since the colors of the human face and the tongue body are similar, in order to reduce the interference of the human face on the classification of tongue color and fur color, the tongue body area needs to be segmented. The present invention uses a SegNet segmentation network to segment the tongue body and obtain the tongue body area.

[0016] 1.2 Binary coding combination of tongue color and coating color labels

[0017] The tongue color and fur color of each tongue image are manually calibrated by an experienced Chinese medicine physician. The present invention performs binary coding on the tongue color and fur color labels of the original tongue image and combines them to obtain a binary label combination vector.

[0018] After such processing, each segmented tongue area corresponds to a binary label combination vector. These one-to-one corresponding data sample pairs constitute a data set for training and testing deep neural network models.

[0019] Step 2: Extraction of common depth features of tongue color and coating color in the tongue area

[0020] The present invention uses a deep residual convolutional neural network ResNeXt to extract the common deep features of tongue color and fur color. This method can learn the two related tasks of tongue color and fur color classification together, and by sharing the knowledge implicit in the two related tasks, train the two tasks at the same time, and improve the performance of the two tasks at the same time, thereby improving the efficiency of learning. Compared with single-task learning, due to the use of shared representation information between similar tasks, when training multiple tasks simultaneously, the amount of training data and the scale of the overall model parameters can be reduced, making the prediction more efficient and having a better generalization effect.

[0021] Step 3: Establishment of tongue color and coating color collaborative recognition model

[0022] For the classification task of tongue color and fur color, the present invention designs a deep neural network structure including multiple fully connected layers, the input of which is the common deep features of tongue color and fur color in the tongue body region obtained in step 2, and the output is the binary label combination vector of the tongue body obtained in step 1.2. Through training, a mapping model between shared features and label combination vectors is established.

[0023] During recognition, the deep features extracted from the tongue area are input into the trained mapping model, and the output is a 14-dimensional vector containing the discrimination probability. The category with the highest probability in the first 5 dimensions is selected as the tongue color recognition result; and the category with the highest probability in the last 9 dimensions is selected as the moss color recognition result.

[0024] Compared with the prior art, the present invention has the following obvious advantages and beneficial effects:

[0025] 1. High classification accuracy. Compared with the traditional method of identifying tongue color and fur color separately, the sharing of tongue color and fur color features is realized based on the idea of ​​multi-task learning, so that the classification tasks can complement each other and learn from each other, which increases the generalization ability of the network and improves the classification accuracy. In addition, since the correlation between tongue color and fur color is fully utilized, it also helps to improve the classification accuracy and meet the actual application needs;

[0026] 2. Low implementation complexity. The present invention realizes the mapping between shared features and label combination vectors by using a deep neural network through the combination of tongue color and fur color feature sharing and labels. In this way, the recognition of tongue color and fur color can be realized simultaneously under one architecture, without the need to design different feature extraction methods and classifiers for tongue color and fur color recognition respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is the SegNet network structure diagram for tongue segmentation

[0028] Figure 2 This is the ResNeXt network structure diagram

[0029] Figure 3 It is the ResNeXt network training graph

[0030] Figure 4 It is a deep neural network structure for tongue color and coating color collaborative recognition DETAILED DESCRIPTION

[0031] Step 1: Build the dataset

[0032] Step 1.1 Tongue segmentation

[0033] The present invention uses SegNet network to segment the tongue, and its network structure is as follows: Figure 1 As shown. The SegNet network adopts the encoder + decoder architecture, and the encoding and decoding correspond to 4 layers of downsampling and 4 layers of upsampling respectively. The encoding part uses the first 13 layers of the VGG-16 convolutional network to extract deep features and save the pooled index values; the decoding part is just the opposite of the encoding part, using a combination of deconvolution and upsampling. Deconvolution can reproduce the classified features, and upsampling can restore the size of the original image. Finally, Softmax is used to output the maximum values ​​of different classifications to obtain the segmented tongue area.

[0034] 1.2 Binary coding combination of tongue color and coating color labels

[0035] The present invention performs binary coding and combination on the original labels of tongue color and fur color to obtain a label combination vector. Each tongue image corresponds to two attributes, a total of 14 categories, of which the tongue color attribute has 5 categories (light white, light red, red, dark red, purple), and the fur color attribute has 9 categories (thin white fur, medium thick white fur, thick white fur, thin yellow fur, medium thick yellow fur, thick yellow fur, gray black fur, brown fur, less fur). The 14 attribute categories are arranged in fixed positions, as shown in Table 1.

[0036] Table 1 Tongue color and fur color attribute label combination order

[0037]

[0038]

[0039] According to the original labeling results of the tongue image attributes, if the tongue color and coating color of the tongue image correspond to a certain category, the value at the corresponding position is set to 1, and the values ​​at other positions are set to 0.

[0040] For example, if a tongue image is labeled as: tongue color is red, coating color is thin white coating, then after encoding and combining according to the above rules, a 14-dimensional binary label vector [0,0,1,0,0,1,0,0,0,0,0,0,0,0] will be obtained. The first 5 dimensions are used to represent the category corresponding to the tongue color, and the last 9 dimensions represent the category corresponding to the coating color. Then, in the 14-dimensional binary data, only one value can be 1 in the first 5 dimensions, and the other 4 values ​​should be 0. Similarly, in the last 9 dimensions, only one value can be 1, and the other 8 values ​​should be 0.

[0041] In this way, each segmented tongue area corresponds to a binary label combination vector, and these one-to-one corresponding data sample pairs constitute a data set for subsequent training of the recognition network model.

[0042] Step 2: Extraction of common depth features of tongue color and coating color in the tongue area

[0043] The present invention uses the ResNeXt network as the backbone network to extract the deep features of the tongue area. ResNeXt is a variation of the ResNet network, which incorporates the Inception network concept. The network architecture is as follows: Figure 2 As shown in the figure, the network has a depth of 50 layers and contains 4 layers. Each layer contains bottlenecks of [3, 4, 6, 3]. Each bottleneck is completed by converting each original convolution operation into 32 parallel convolution operations. In other words, the original 1*1->3*3->1*1 convolutions are replaced by 3×32 convolutions.

[0044] Step 2.1: Training of ResNeXt network

[0045] The training of deep neural networks requires a large number of data samples. Since the collection of tongue image data in practical applications of traditional Chinese medicine is difficult, the size of the tongue image data set is relatively limited. When training ResNeXt, the present invention adopts the strategy of "pre-training + fine-tuning", such as Figure 3 That is, first, the ResNeXt network is pre-trained using the large-scale dataset Imagenet to obtain the initial parameters of the network model; then the tongue image dataset is used to fine-tune the parameters of the pre-trained ResNeXt network model. When the fine-tuning is completed until the network converges, the optimized network model is obtained.

[0046] The training process is divided into forward propagation and back propagation: forward propagation is the result obtained after calculating the input data through the neural network, and back propagation is mainly to reduce the gap between the network's calculation results and the actual results through the loss function, so that the results calculated by the network and the actual label results are getting closer and closer, and finally end when they are equal. Back propagation continuously optimizes the network training performance by continuously updating the network parameters.

[0047] Step 2.2 Extraction of common depth features of tongue color and coating color

[0048] First, the segmented tongue image is normalized to a size of 224*224 and input into the trained ResNeXt network model. The output of the last pooling layer of the ResNeXt network is extracted as the deep feature of the tongue image. This feature has a total of 2048 dimensions.

[0049] Step 3: Establishment of tongue color and coating color collaborative recognition model

[0050] The present invention designs a deep neural network, the network input is the 2048-dimensional deep feature obtained by step 2.2, and the output is a binary label combination vector with 14 dimensions. Through training, a mapping model between the shared deep features of tongue color and fur color and the label combination vector is established to achieve collaborative recognition of tongue color and fur color.

[0051] Step 3.1 Mapping network structure design

[0052] The network structure is as follows Figure 4 As shown in the figure, the network is composed of fully connected layers, including 1 input layer, 3 hidden layers and 1 output layer, and the corresponding numbers of neurons are 2048, 1024, 256, 128 and 14 respectively.

[0053] The network uses batch normalization (BN) after each layer of the input layer and hidden layer (the normalization size is 2048, 1024, 256, 128 respectively), ReLU and Dropout methods to improve the classification performance of the network, and realizes the coordinated recognition of tongue color and fur color by designing loss functions respectively. The relevant description is as follows:

[0054] (1) Batch Normalization

[0055] The normalization operation makes the input data distribution of each layer of the network more stable, speeds up the convergence of the network model, and makes the data more stable during the training process. At the same time, it makes the network model less sensitive to parameters, making network learning more stable. Batch normalization also has a regularization effect to a certain extent.

[0056] (2) ReLU

[0057] As the depth of the network increases, the traditional activation function will easily cause the gradient to become smaller and smaller during the back propagation process due to the multiplication relationship between gradients, and the loss converges more and more slowly, which will lead to the so-called gradient vanishing phenomenon. In order to solve this problem, the present invention adds a ReLU activation function after each layer of the deep neural network, which not only solves the problem of gradient vanishing, but also speeds up the convergence of the network.

[0058] (3) Dropout

[0059] Dropout randomly inactivates neurons in the hidden layer with a probability of 0.5, and the inactivated neurons do not participate in the forward and backward propagation of the neural network. The random inactivation of neurons not only greatly reduces the number of parameters in the training network, but also determines that the structure of the neural network trained each time will be different, thereby enhancing the robustness of the model. Therefore, the present invention uses the Dropout strategy in the fully connected layer to improve the classification performance of the neural network.

[0060] (4) Design of loss function

[0061] Since there is mutual exclusivity between attribute labels, the present invention designs loss functions for tongue color and fur color respectively, and the activation function uses the Softmax function. The Softmax function maps the output of the network to between (0, 1), and at the same time, it widens the gap between different categories through the effect of the exponential function, increases the distinguishability, and can better output the category corresponding to the maximum probability. The calculation formula of the Softmax function is shown in formula (1):

[0062]

[0063] p n,irepresents the probability corresponding to the i-th attribute in the n-th sample, e i Represents the mathematical exponent of the i-th attribute of the sample with the natural constant e as the base.

[0064] The present invention uses two Softmax functions to activate the first 5 dimensions and the last 9 dimensions of the network output respectively. The loss function designed for tongue color and fur color is the cross entropy loss, as shown in formulas (2) and (3). s represents the loss of tongue color attributes, C represents the number of all tongue color attributes, p = [p 0 ,p 1 ,p 2 ......p C-1 ] is the probability distribution after Softmax activation, p i represents the probability of the i-th attribute, y = [y 0 ,y 1 ,y 2 ........y D-1 ] is the onehot encoding of the sample label, and it is required that there can only be one label value 1 in the onehot encoding. When the sample belongs to the i-th category, y i =1 all other labels are 0; L t represents the loss of moss color attributes, D represents the number of all moss color attributes, p j represents the probability of the jth attribute of moss color, y j The specific meaning and i similar:

[0065]

[0066] The total loss L during training is the sum of the two losses, as shown in formula (4):

[0067] L=L s +L t (4)

[0068] Step 3.2: Establishment of tongue color and coating color collaborative recognition model

[0069] The deep features of the tongue image obtained in step 2.2 are used as the input of the mapping network, and the 14-dimensional binary label combination vector of the tongue image is used as the output. The mapping network is trained to obtain the optimized network model.

[0070] During recognition, the tongue image is first segmented, and then the deep features are extracted. The extracted deep features are input into the trained mapping network. The output of the network is a 14-dimensional combination vector, in which each dimension represents the discrimination probability. The category corresponding to the maximum probability in the first 5 dimensions is used as the discrimination result of tongue color, and the category corresponding to the maximum probability in the last 9 dimensions is used as the discrimination result of moss color.

[0071] The present invention proposes a tongue color and fur color collaborative classification method based on a deep convolutional neural network. Unlike the traditional method of classifying tongue color or fur color separately, this method can classify both tongue color and fur color in a unified framework. Compared with the traditional classification method, it is not only simple to implement, but also has a significantly improved classification accuracy.

Claims

1. A collaborative classification method of tongue color and fur color in traditional Chinese medicine based on convolutional neural network. Features: Step 1: Build the dataset 1.1 Tongue segmentation The SegNet segmentation network is used to segment the tongue and obtain the tongue area; 1.2 Binary coding combination of tongue color and coating color labels The tongue color and fur color of each tongue image are calibrated; the tongue color and fur color labels of the original tongue image are binary-encoded and combined to obtain a binary label combination vector; Each segmented tongue region corresponds to a binary label combination vector. These one-to-one corresponding data sample pairs constitute a data set for training and testing deep neural network models. Step 2: Extraction of common depth features of tongue color and coating color in the tongue area The ResNeXt network is used as the backbone network to extract the deep features of the tongue area. The network has a depth of 50 layers, including 4 layers, where each layer contains a bottleneck of [3, 4, 6, 3], and each bottleneck is completed by converting each original convolution operation into 32 parallel convolution operations. Step 2.1: Training of ResNeXt network The training of deep neural networks requires a large number of data samples. Since it is difficult to collect tongue image data for practical applications, the size of the tongue image data set is relatively limited. When training ResNeXt, the "pre-training + fine-tuning" strategy is adopted. That is, first, the ResNeXt network is pre-trained using the large-scale dataset Imagenet to obtain the initial parameters of the network model. Then, the tongue image dataset is used to fine-tune the parameters of the pre-trained ResNeXt network model. When the fine-tuning reaches the point where the network converges, the optimized network model is obtained. The training process is divided into forward propagation and back propagation: forward propagation is to calculate the result obtained after the input data passes through the neural network, and back propagation is mainly to reduce the gap between the network's calculation result and the actual result through the loss function, so that the result calculated by the network and the result of the actual label are getting closer and closer, and finally end when they are equal; Back propagation continuously optimizes network training performance by continuously updating network parameters; Step 2.2 Extraction of common depth features of tongue color and coating color First, the segmented tongue image is normalized to a size of 224*224 and input into the trained ResNeXt network model. The output of the last pooling layer of the ResNeXt network is extracted as the deep feature of the tongue image; this feature has a total of 2048 dimensions. Step 3: Establishment of tongue color and coating color collaborative recognition model A deep neural network was designed. The network input was the 2048-dimensional deep features obtained in step 2.2, and the output was a binary label combination vector with 14 dimensions. Through training, a mapping model between the shared deep features of tongue color and coating color and the label combination vector was established to achieve collaborative recognition of tongue color and coating color. Step 3.1 Mapping network structure design The network is composed of fully connected layers, including 1 input layer, 3 hidden layers and 1 output layer, and the corresponding number of neurons is 2048, 1024, 256, 128 and 14 respectively; The network uses normalization after each layer of the input layer and hidden layer, where the normalized sizes are 2048, 1024, 256, and 128 respectively. A ReLU activation function is added after each layer of the deep neural network; Dropout randomly inactivates neurons in the hidden layer with a probability of 0.5; (4) Design of loss function Since there is mutual exclusivity between attribute labels, loss functions are designed for tongue color and fur color respectively, and the activation function uses the Softmax function; the Softmax function maps the output of the network to between (0, 1), and at the same time, it widens the gap between different categories through the effect of the exponential function, increases the discrimination, and better outputs the category corresponding to the maximum probability; the calculation formula of the Softmax function is shown in formula (1): p n,i represents the probability corresponding to the i-th attribute in the n-th sample, e i Represents the mathematical exponent of the i-th attribute of the sample with the natural constant e as the base; Two Softmax functions are used to activate the first 5 dimensions and the last 9 dimensions of the network output respectively. The loss function designed for tongue color and fur color is the cross entropy loss, as shown in formulas (2) and (3); where L s represents the loss of tongue color attributes, C represents the number of all tongue color attributes, p = [p 0 ,p 1 ,p 2 ......p C-1 ] is the probability distribution after Softmax activation, p i represents the probability of the i-th attribute, y = [y 0 ,y 1 ,y 2 ........y D-1 ] is the onehot encoding of the sample label, and it is required that there can only be one label value 1 in the onehot encoding. When the sample belongs to the i-th category, y i =1 all other labels are 0; L t represents the loss of moss color attributes, D represents the number of all moss color attributes, p j represents the probability of the jth attribute of moss color, y j The specific meaning and i similar: The total loss L during training is the sum of the two losses, as shown in formula (4): L=L s +L t (4) Step 3.2: Establishment of tongue color and coating color collaborative recognition model The tongue image depth feature obtained in step 2.2 is used as the input of the mapping network, and the 14-dimensional binary label combination vector of the tongue image is used as the output, and the mapping network is trained to obtain the optimized network model; During recognition, the tongue image is first segmented, and then the deep features are extracted. The extracted deep features are input into the trained mapping network. The output of the network is a 14-dimensional combination vector, in which each dimension represents the discrimination probability. The category corresponding to the maximum probability in the first 5 dimensions is used as the discrimination result of tongue color, and the category corresponding to the maximum probability in the last 9 dimensions is used as the discrimination result of tongue color.

2. According to the method of collaborative classification of tongue color and fur color in traditional Chinese medicine based on convolutional neural network in claim 1, Features: Step 1: Build the dataset The SegNet network is used for tongue segmentation. The SegNet network adopts an encoder + decoder architecture. The encoding and decoding correspond to 4 layers of downsampling and 4 layers of upsampling respectively. The encoding part uses the first 13 layers of the VGG-16 convolutional network to extract deep features and save the pooled index values. The decoding part is just the opposite of the encoding part, using a combination of deconvolution and upsampling. Deconvolution allows the classified features to be reproduced, and upsampling restores the size of the original image. Finally, Softmax is used to output the maximum values ​​of different classifications to obtain the segmented tongue area. 1.2 Binary coding combination of tongue color and coating color labels The original labels of tongue color and fur color are binary-encoded and combined to obtain a label combination vector; each tongue image corresponds to 2 attributes, a total of 14 categories, of which the tongue color attribute has 5 categories and the fur color attribute has 9 categories; the 14 attribute categories are arranged in fixed positions, and according to the original tongue image attribute annotation results, if the tongue color and fur color of the tongue image correspond to a certain category, the value at the corresponding position is set to 1, and the values ​​at other positions are set to 0; In this way, each segmented tongue area corresponds to a binary label combination vector, and these one-to-one corresponding data sample pairs constitute a data set for subsequent training of the recognition network model.

Images