Tongue surface detection and segmentation and constitution classification method based on deep learning

Abstract

The tongue surface detection segmentation and constitution classification method based on deep learning comprises the following steps: (1) the collected tongue surface images are subjected to denoising treatment so as to meet the objective and input requirements of the training network; (2) the denoised image data are subjected to detection and segmentation labeling to obtain a training data set; (3) the training data set is encapsulated into a network model with a modified loss function, an added channel and spatial attention mechanism and a deep separable module, training is started, a real-time detection and segmentation prediction model is obtained and is saved; and (4) under the assistance of professional doctors, a network fusion constitution classification method of migration learning is established according to the relationship between tongue and constitution. The present application not only can establish an objective standard of computer tongue diagnosis, but also can help to reduce the pressure of medical staff at the grassroots level, and has important application value.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a tongue surface detection and segmentation and constitution classification method based on deep learning BACKGROUND

[0002] With the development of modern society, people's pressure is getting bigger and bigger, and with the improvement of people's material living standard, some old diseases have also been gradually becoming younger, and most people are in a sub-healthy state. Some hidden and potential diseases are gradually eroding our body. How to take a painless, non-contact and effective diagnostic method is becoming more and more popular.

[0003] As the quintessence of China, traditional Chinese medicine has accumulated five thousand years of development experience, and in the long river of history and culture, traditional Chinese medicine has been continuing four methods of diagnosing diseases, namely, four diagnoses of looking, smelling, asking and touching. Among them, looking diagnosis is the first step of four diagnoses of traditional Chinese medicine, and is also the most important and basic part, and tongue diagnosis is the most important part of looking diagnosis, and is the most effective and direct method of diagnosing diseases. It can reflect the cold and heat, deficiency and excess, yin and yang of the body environment, and it is feasible to identify the constitution, so as to improve the constitution and reduce the occurrence of diseases.

[0004] Traditional tongue diagnosis relies on the subjective judgment of doctors, and there are differences in the judgment of the same tongue image by different doctors due to the different experience and diagnosis environment of doctors. With the rapid development of artificial intelligence, intelligent tongue diagnosis is gradually emerging. Intelligent tongue diagnosis can process and analyze the tongue image collected in different environments, realize tongue detection, tongue segmentation and human constitution classification to assist doctors in diagnosis and improve the accuracy of diagnosis. At the same time, intelligent tongue diagnosis can realize non-contact intelligent diagnosis and avoid cross infection.

[0005] The prior art usually adopts a convolutional neural network algorithm to detect and segment the tongue surface image, and there are many methods to improve the network structure, but there are still several limitations: first, these methods only segment the tongue surface and external image, do not use the positional relationship between each pixel point in the image, and ignore the tongue surface detail information; second, the problem of mobile terminal deployment with limited computer resources is not considered; third, the constitution classification standard is single. These problems are crucial in computer medicine and constitution classification. SUMMARY

[0006] The purpose of the present application is to improve the constitution, reduce the national medical burden and promote the health of the whole people. The tongue surface detection and segmentation and constitution classification method based on deep learning is proposed, which can denoise the image, modify the loss function of deep learning, add attention mechanism and depth separable convolution block. This method can extract the detail information of the tongue surface, reduce the calculation parameter amount of the model, and also realize the constitution classification.

[0007] The technical scheme of the present application comprises the following steps:

[0008] Step 1: Set the original collected tongue surface image as V1, with a size of m x n, and the denoised image as U, the pixel gray value at the (N i ,N j ) position in U is:

[0009]

[0010] where ω(i,j) represents the similarity rate at the (i,j) position, satisfying d(i,j) = ||V(N i )-V(N j ) || 2 2,α ; i,j are the pixel centers of V1(N i ), V1(N j ) respectively, h is a filter parameter for controlling smoothing, α is the standard deviation of the Gaussian kernel, and Z(i) is a normalized coefficient, which finds similar areas in the image through image blocks, and then averages these areas to obtain the denoised image V2;

[0011] Step 2: Tongue surface labeling: use LabelImg and LabelMe to detect and segment label V2 to obtain tongue surface image dataset V3;

[0012] Step 3: Model training: use an improved deep learning model to train the dataset V3; the trained model can detect and segment the tongue surface in real time, and the detection and segmentation results are saved;

[0013] Step 4: Physical constitution labeling: the saved segmentation results are labeled by a physician for physical constitution, and a physical constitution classification dataset V4 is obtained;

[0014] Step 5: Physical constitution classification: use a transfer network fusion model to train V4, save the optimal training model, and perform physical constitution research and judgment on V2.

[0015] Further, the method in step 3, characterized in that the improved deep learning model specifically comprises:

[0016] Step 1: Design a truncated loss function:

[0017]

[0018] where x ∈ R n is a variable, f0: R n → R∪{+∞}, f i ,h i : R n→ R, i = 1, 2, …, m, min{f(x), h(x)} is called the truncated function; because the truncated loss function is a non-differentiable function, the idea of smoothing function is applied to approximate the convex non-smooth surface, and the smoothing function used is defined as:

[0019]

[0020]

[0021] The truncated loss function is equivalent to:

[0022]

[0023] Step 2: Solve the extreme value: using the augmented Lagrange multiplier method, get

[0024]

[0025] where x, y ∈ R n is a variable, ω ∈ R n is a Lagrange multiplier, and β > 0 is a penalty parameter.

[0026] Step 3: variable initialization: (y 0 , ω 0 ) ∈ R n × R m , p 0 > 0, σ ∈ (0, 1), β > 0, ε1 > 0 and ε2 > 0.

[0027] Step 4: parameter iteration: k = 0, 1, 2, …, m, t = 0, 1, 2, …, m, sequence (x k+1 , y k+1 , ω k+1 ) is generated by formula (7) (8) (9):

[0028] x k+1 = arg x minL β (x, y k , ω k ), (7)

[0029] y k+1 = arg y minL β (x k+1 , y, ω k ), (8)

[0030] ω k+1 = ω k + β (x k+1 - y k+1 ), (9)

[0031] If ||x k+1 -y k+1 || < ε1, ||- β(y k -y k+1 )|| < ε1 and then stop the calculation and output (x k+1 ,y k+1 , ω k+1 );

[0032] Step 5: Channel convolution: First, the feature map F passes through two parallel MaxPool and AvgPool layers in the channel convolution, changing the dimension of the feature map from CxHxW to Cx1x1; second, the MLP (Multi-Layer Perceptron) compresses the channel number to 1 / r of the original; third, the ReLU activation function is used to expand to the original channel number, and the two output results are added element by element; finally, a Sigmoid activation function is used to output the result multiplied by the original graph, changing back to CxHxW, outputting the channel convolution feature map M c (F) as shown in equations (10) and (11):

[0033] M c (F) = σ(MLP(AvgPool(F)) + MLP(MaxPool(F)), (10)

[0034]

[0035] where σ represents the Sigmoid activation function, W0∈R C / r×C , W1∈R C×C / r , and the weights W0 and W1 of MLP are shared for input, and the ReLU activation function is located after W0 and before W1;

[0036] Step 6: Spatial convolution: The output of the channel convolution is passed through MaxPool and AvgPool to obtain two 1xWxH feature maps; the two feature maps are spliced, and then a 7x7 convolution is performed to change the channel number of the feature map to 1, and an activation function is used to obtain the spatial convolution feature map. The output result is multiplied by the original input feature map to change back to CxHxW size, outputting the spatial convolution feature map M s (F) as shown in equations (12) and (13):

[0037] M s (F) = σ(f 7×7 ([AvgPool(F); MaxPool(F)])), (12)

[0038]

[0039] wherein, sigma represents a Sigmoid activation function, f 7×7 represents a convolution process with a convolution kernel of 7x7;

[0040] Step 7: Deep separable convolution: including channel-by-channel convolution and point-by-point convolution, the original C3 module is composed of convolution, pooling and activation module, all the C3 modules in the feature extraction network are replaced by deep separable convolution module (DW_PW_C3, DP_C3), which greatly reduces the network parameters and speeds up the training speed; the calculation amount of traditional convolution:

[0041] T1 = M x N x C x C x D x D; (14)

[0042] The calculation amount of 1 deep convolution is MxCxCxDxD; the calculation amount of 1 point convolution is MxNx DxD;

[0043] The calculation amount of deep separable convolution:

[0044] T2 = M x C x C x D x D + M x N x D x D; (15)

[0045] The calculation amount ratio of deep separable convolution to traditional convolution is:

[0046]

[0047] Wherein, M is the channel number of input feature map, N is the number of convolution kernel, D is the size of feature map, C is the size of convolution kernel, the more the number of convolution kernel N, the less the parameter amount and calculation amount of model, which is convenient for embedding development board and physical classification.

[0048] Further, the physical classification in step 7 is specifically: according to the tongue color, moss color and tongue shape weight and physical corresponding relationship, a network fusion physical classification system of transfer learning is established for the denoising image V2, including VGG16, ResNet50, MobileNetV2, respectively classifying and learning the features of tongue color, moss color and tongue shape, according to the corresponding relationship between tongue image and physical, the physical of human body is divided into 9 types, including qi deficiency, qi depression, flatness, phlegm dampness, damp heat, yang deficiency, yin deficiency, blood stasis and special endowment, and the physical type is trained and judged.

[0049] The beneficial effects of this invention are as follows: (1) It can detect and segment from images containing the tongue surface to obtain the complete tongue region; (2) It modifies the loss function in image segmentation, solving the problems of sample imbalance and blurred edges; (3) It incorporates the attention mechanism and depthwise separable convolutional blocks of deep learning, improving the feature extraction capability and accelerating the network training speed, making it suitable for implementation on limited computers and lightweight development boards; (4) It designs a physical fitness classification system that can judge the physical fitness of the human body and output the results, which meets the requirements of computer medicine and provides a priori basis for improving physical fitness and the Healthy China strategy. Attached Figure Description

[0050] Figure 1 Flowchart of tongue surface detection, segmentation, and constitution classification based on deep learning;

[0051] Figure 2 Schematic diagram of tongue surface detection and segmentation model;

[0052] Figure 3 Attention mechanism module;

[0053] Figure 4 Depthwise separable convolutional modules;

[0054] Figure 5 A physical constitution classification model based on network fusion using transfer learning; Detailed Implementation

[0055] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0056] Figure 1 This is a flowchart of tongue surface detection, segmentation, and body constitution classification based on deep learning;

[0057] Step 1: Let the original acquired tongue surface image be V1, with a size of m×n, and the denoised image be U, where the (N)th denoised image in U is... i N j The pixel grayscale value at position ) is:

[0058]

[0059] Where ω(i,j) represents the similarity rate at position (i,j), satisfying d(i,j)=||V(N i )-V(N j )|| 2 2,α i and j are respectively V1(N i ), V1(N jPixel center, h is the filter parameter for controlling smoothing, alpha is the standard deviation of the Gaussian kernel, Z(i) is the normalized coefficient, find similar areas in the image by image block, and then average these areas to get the denoising image V2;

[0060] Step 2: Obtain the data set: use N tongue surface original pictures as detection and segmentation text, and use LabelImg and LabelMe for manual annotation, LabelImg annotated detection data set is txt type, which meets the detection training input; LabelMe annotation tool generates json file, which contains image file itself and annotated object box information, we convert json file to txt file to meet the segmentation model training, and obtain the detection and segmentation data set.

[0061] Step 3: Network training: use improved YoLov5 6.2, as shown in Figure 2 , then train the data set, set training epoch = 300, lr = 0.001, batch_size = 8, and the model starts training;

[0062] Step 4: Design the truncated loss function:

[0063]

[0064] Where, x∈R n is a variable, f0:R n →R∪{+∞}, f i ,h i :R n →R, i = 1, 2, …, m, min{f(x), h(x)} is called truncated function; Because the truncated loss function is a non-differentiable function, the idea of smooth function is applied to approximate the convex non-smooth surface, and the smooth function used is defined as:

[0065]

[0066]

[0067] The truncated loss function is equivalent to:

[0068]

[0069] Step 5: Solve the extreme value: use augmented Lagrange multiplier method to get

[0070]

[0071] Where, x,y∈R n is a variable, ω∈R nIt is a Lagrange multiplier, with a penalty parameter β > 0;

[0072] Step 6: Variable initialization: (y 0 ,ω 0 )∈R n ×R m p 0 >0, σ∈(0,1), β>0, ε1>0 and ε2>0;

[0073] Step 7: Parameter iteration: k = 0, 1, 2, ..., m, t = 0, 1, 2, ..., m, sequence (x k+1 y k+1 ω k+1 This is generated by formulas (7)(8)(9):

[0074] x k+1 =arg x minL β (x,y k ,ω k (7)

[0075] y k+1 =arg y minL β (x k+1 ,y,ω k (8)

[0076] ω k+1 =ω k +β(x k+1 -y k+1 (9)

[0077] If ||x k+1 -y k+1 ||<ε1,||-β(y k -y k+1 )||<ε1 and Then stop the calculation and output (x) k+1 ,y k+1 ,ω k+1 );

[0078] Step 8: Channel Convolution: First, the feature map F undergoes channel convolution, such as... Figure 3 As shown in (1), two parallel MaxPool and AvgPool layers transform the dimension of the feature map from C×H×W to C×1×1; then, a multilayer perceptron (MLP) is used to compress the number of channels to 1 / r of the original number; next, a ReLU activation function is used to expand the number of channels to the original number, and the two outputs are added element by element; finally, a Sigmoid activation function is used to multiply the output with the original image to transform it back to C×H×W, outputting the channel convolutional feature map M shown in formulas (10) and (11).c (F):

[0079] M c (F) = σ(MLP(AvgPool(F)) + MLP(MaxPool(F)), (10)

[0080]

[0081] where σ represents a Sigmoid activation function, W0∈R C / r×C , W1∈R C×C / r , and the weights W0 and W1 of MLP are shared for input, and a ReLU activation function is located after W0 and before W1;

[0082] Step 9: Spatial convolution: the channel convolution output result is subjected to spatial convolution, as shown in formula (12) and (13). Figure 3 (2), two 1×W×H feature maps are obtained through maximum pooling and average pooling; the two feature maps are spliced, and the channel number of the feature map is changed to 1 through 7×7 convolution, and the spatial convolution feature map is obtained through an activation function, the output result is multiplied by the original input feature map, and the size is changed back to C×H×W, and the spatial convolution feature maps M s (F):

[0083] M s (F) = σ(f 7×7 ([AvgPool(F); MaxPool(F)])), (12)

[0084]

[0085] where σ represents a Sigmoid activation function, f 7×7 represents a convolution process with a 7×7 convolution kernel;

[0086] Step 10: Depthwise separable convolution: including channel-by-channel convolution and point-by-point convolution, as shown in formula (15). Figure 4 The original C3 module is composed of convolution, pooling and activation modules, and the C3 module in the feature extraction network is replaced by a depthwise separable convolution module (DW_PW_C3, DP_C3), which greatly reduces the network parameters and speeds up the training speed; the calculation amount of a traditional convolution:

[0087] T1 = M×N×C×C×D×D; (14) The calculation amount of a depthwise convolution is M×C×C×D×D; the calculation amount of a pointwise convolution is M×N×D×D; and the calculation amount of a depthwise separable convolution:

[0088] T2 = M×C×C×D×D + M×N×D×D; (15) The calculation amount ratio of the depth separable convolution to the traditional convolution is:

[0089]

[0090] Wherein, M is the channel number of the input feature map, N is the number of convolution kernels, D is the size of the feature map, C is the size of the convolution kernel, the more the number of convolution kernels N, the less the parameter quantity and the calculation amount of the model, which is convenient for embedding the development board and classifying the constitution.

[0091] Further, the method of claim 2, wherein the constitution classification in step 5 is specifically: for the denoised image V2, a network fusion constitution classification system of transfer learning is established according to the weight and constitution corresponding relationship of tongue color, moss color and tongue shape, as shown in Figure 5 The constitution of human body is divided into 9 types, including qi deficiency constitution, qi depression constitution, flat constitution, phlegm-damp constitution, damp-heat constitution, yang deficiency constitution, yin deficiency constitution, blood stasis constitution and special constitution, and the types are trained and predicted.

[0092] Step 11: predicting constitution: predicting constitution by the corresponding relationship between tongue image and constitution, and outputting constitution type. The corresponding relationship between tongue image and constitution is shown in Table 1:

[0093] Table 1: Corresponding relationship between tongue image and constitution

[0094]

[0095]

Claims

1. A tongue surface detection segmentation and constitution classification method based on deep learning, characterized in that, The segmentation and classification process comprises the following steps: Step 1: image denoising: the original collected tongue surface image is V1, with a size of m*n, and the denoised image is U, the pixel gray value of the (i, j) position in U is: （1） where ω(i, j) represents the similarity rate of (i, j) position, satisfying , , d(i, j) = ; i, j are the centers of V1(N i ), V1(N j ) pixels respectively, h is a filter parameter for controlling smoothing, a is the standard deviation of the Gaussian kernel, and Z(i) is a normalized coefficient. Similar regions in the image are found through the image block, and the average of these regions is obtained to obtain the denoising image V2. Step 2: tongue surface labeling: using LabelImg and LabelMe to detect and segmentally label V2 to obtain a tongue surface image dataset V3; Step 3: model training: using an improved deep learning model to train the dataset V3; the trained model can detect and segment the tongue surface in real time, and the segmentation result is saved; Step 4: constitution labeling: the saved segmentation result is labeled by a physician to obtain a constitution classification dataset V4; Step 5: constitution classification: using a transfer network fusion model to train V4, saving the optimal training model, and using V2 to determine the constitution type; The improved deep learning model in step 3 comprises the following steps: Step 1: design a truncated loss function: where x e R n is a variable; It is called a truncated function; because the truncated loss function is a non-differentiable function, the idea of a smooth function is used to approximate the convex non-smooth surface, and the smooth function used is defined as: The truncated loss function is equivalent to: Step 2: Solve for the extreme value: Using the method of Lagrange multipliers, we get , wherein is a variable, is a Lagrange multiplier, a penalty parameter β > 0; Step 3: variable initialization: ； Step 4: Parameter iteration: , sequence Resulting from equations (7) (8) (9): If and then stop the calculation and output ; Step 5: Channel convolution: Firstly, the feature map F passes through two parallel MaxPool and AvgPool layers in the channel convolution, which changes the dimension of the feature map from CxHxW to Cx1x1; secondly, the channel number is compressed to 1 / r times of the original through the multi-layer perception MLP; thirdly, the output results of the two are added element by element after passing through the ReLU activation function and expanding to the original channel number; finally, the output result is multiplied by the original graph after passing through a Sigmoid activation function, and the dimension is changed back to CxHxW, outputting the channel convolution feature map M shown in formulas (10) and (11) c (F): where σ represents a Sigmoid activation function, and the weights W0 and W1 of the MLP are shared for the inputs, with a ReLU activation function after W0 and before W1; Step 6: spatial convolution: the channel convolution output result is subjected to maximum pooling and average pooling to obtain two 1xWxH feature maps; the two feature maps are spliced, then subjected to 7x7 convolution to change the channel number of the feature map to 1, and subjected to an activation function to obtain a spatial convolution feature map, the output result is multiplied by the original input feature map to change back to CxHxW size, and the spatial convolution feature maps Ms(F) of formulas (12) and (13) are output: wherein, σ represents a Sigmoid activation function, f 7×7 represents a convolution process with a convolution kernel of 7x7; Step 7: Depth separable convolution: including channel-by-channel convolution and point-by-point convolution, the original C3 module is composed of convolution, pooling and activation module, and all the C3 modules in the feature extraction network are replaced by depth separable convolution modules , greatly reducing the network parameters and accelerating the training speed; the original convolution calculation amount: The calculation amount of 1 depth convolution is MxCxCDxD; the calculation amount of 1 point convolution is MxNx DxD; The calculation amount of the depth separable convolution is: The calculation amount ratio of the depth separable convolution to the traditional convolution is: Wherein, M is the channel number of the input feature map, N is the number of convolution kernels, D is the size of the feature map, C is the size of the convolution kernel, the more the number of convolution kernels N, the less the parameter amount and the calculation amount of the model, which is convenient for embedding the development board and performing constitution classification.

2. The method of claim 1, wherein, The constitution classification in step 5 is specifically: a network fusion constitution classification system of transfer learning is established according to the weight and constitution corresponding relationship of tongue color, moss color and tongue shape, including VGG16, ResNet50, MobileNetV2, respectively, for feature classification learning of tongue color, moss color and tongue shape, according to the corresponding relationship between tongue image and constitution, the constitution of human body is divided into 9 types, i.e. qi deficiency constitution, qi depression constitution, balanced constitution, phlegm-damp constitution, damp-heat constitution, yang deficiency constitution, yin deficiency constitution, blood stasis constitution and special constitution, and the optimal training model is used to determine the constitution type of the denoised image V2.

Images