Intelligent Chinese medicine tongue diagnosis method based on artificial intelligence and computer storage medium

Abstract

A smart Chinese medicine tongue diagnosis method based on artificial intelligence and a computer storage medium belong to the field of smart medical technology. The method is divided into three stages: tongue segmentation based on the TransFuse network; tongue segmentation post-processing based on OPENCV; and intelligent analysis method based on ensemble learning. The computer storage medium stores at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the smart Chinese medicine tongue diagnosis method based on artificial intelligence. Based on Transfuse, the present invention makes full use of CNN networks and Transformer networks, combines the strengths of the two networks, can capture low-level spatial and high-level semantic features, greatly improves the training and reasoning efficiency of the model, and improves the efficiency of model deployment in the cloud or smart terminals. The present invention has significant advantages, and the accuracy, robustness and robustness of the algorithm have been significantly improved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and more specifically, relates to an intelligent Chinese medicine tongue diagnosis method based on artificial intelligence and a computer storage medium. Background Art

[0002] Traditional Chinese medicine uses the four-diagnosis method of "looking, smelling, asking and touching" to explore the cause, nature and location of the disease. Tongue diagnosis, as the key content of visual diagnosis, has always been one of the important branches of traditional Chinese medicine. With the development of information technology and artificial intelligence, more and more researchers are using artificial intelligence methods to help traditional Chinese medicine experts perform auxiliary diagnosis and quantitative analysis. However, due to the diverse morphology of the tongue, different tongue collection locations and equipment, the tongue images collected by mobile terminals contain a lot of data noise and redundant data, resulting in low accuracy and lack of robustness and robustness of the current research algorithm, which cannot perform intelligent analysis on the changing tongue images.

[0003] Therefore, in order to solve the above problems, an intelligent tongue analysis method with strong robustness and robustness is urgently needed. Summary of the invention

[0004] In view of this, the purpose of the present invention is to address the shortcomings and defects of the above-mentioned prior art and propose an intelligent Chinese medicine tongue diagnosis method based on artificial intelligence, which realizes intelligent analysis of tongue images by using deep learning and intelligent image processing methods.

[0005] To achieve the above object, the present invention adopts the following technical solution: an intelligent Chinese medicine tongue diagnosis method based on artificial intelligence, characterized in that the method is divided into three stages: tongue segmentation stage, segmentation preprocessing stage and triple-jiao area detection and intelligent analysis stage, and the three stages are specifically:

[0006] The first stage, tongue segmentation based on the TransFuse network, includes the following steps:

[0007] 1) Obtain tongue images and perform data enhancement preprocessing on each tongue image, wherein the data enhancement preprocessing is to perform random data enhancement operations using the imgaug data enhancement library;

[0008] 2) Two independent channels are used to extract features from the tongue image after data preprocessing. The first channel is extracted through the CNN network, and the second channel is extracted through the Transformer network.

[0009] When using the Transformer network for feature extraction, the tongue image is first cut into multiple image blocks, and then the image blocks are flattened and passed to the linear layer to obtain the feature D of each image block, and the position information and feature information of each image block are input into the encoding layer of the Transformer together;

[0010] 3) Use channel attention, spatial attention, 3×3 convolution and residual connection to fuse the features extracted by the CNN network and the features extracted by the Transformer network. The formula is as follows:

[0011]

[0012] Among them, t i Represents the features obtained by the Transformer network at different scales. Similarly, g i Represents the features obtained by the CNN network at different scales; ChannelAttn represents the channel attention mechanism, and SpatialAttn represents the spatial attention mechanism; and Represent the results after channel attention and spatial attention respectively; W1 i and Respectively represent and The obtained learnable weight matrix; Conv represents 3×3 convolution; ⊙ represents the Hadamard product; Residual represents the residual block; represents the features obtained after 3×3 convolution, f i Represents the features obtained after feature fusion;

[0013] 4) Training the segmentation network: Calculate the loss between the predicted mask and the mask label obtained by forward propagation of the segmentation network, and perform back propagation through the loss function to train the segmentation network;

[0014] 5) Use the trained segmentation network to perform inference calculation on the tongue image to obtain the first-stage tongue segmentation mask image;

[0015] In the second stage, the post-processing operation of the tongue segmentation mask image based on OPENCV is performed to obtain the positioning of the three burners and five internal organs in the tongue area, including the following steps:

[0016] 1) Based on the tongue segmentation mask image obtained in the first stage, hole filling and redundant contour removal are performed to obtain a tongue segmentation mask image with only one contour;

[0017] The specific process is: use the function cv2.findContours() to find all the contours in the tongue segmentation mask image, find the largest contour, and fill the pixels of the largest contour using the cv2.drawContours() function to get a new tongue segmentation mask image. The formula is as follows:

[0018] Contours=cv2.findContours(img)

[0019] maxContours=find(Area(Contours))

[0020] Img new =cv2.drawContours(maxContours)

[0021] Among them, Contours represents the set of all contours found; Area() represents the function of calculating the area of ​​a contour; find() is the function of finding the index of the maximum area in Contours; maxContours represents the set of coordinate values ​​of all points of the largest contour; Img new Represents the new tongue segmentation mask image after pixel filling;

[0022] 2) merging the tongue segmentation mask image with only one contour with the initial image in the first stage to obtain a segmented tongue image;

[0023] 3) The segmented tongue image obtained in step 2) of the second stage is resized and transformed in perspective to standardize the tongue segmentation image; and before the perspective transformation, the cv2.minAreaRect() function is used to obtain the coordinates of the four points of the external matrix of the tongue contour, and the positions of the four coordinates are obtained by intelligent judgment through the coordinate points, and the coordinates are sorted according to the upper left, upper right, lower right, and lower left. The positions of the four coordinate points in the new image after the customized perspective transformation are shown in the following formula:

[0024]

[0025] Among them, width start , height start ,width end , height end Respectively represent the starting and ending positions of the width and height of the external matrix after the perspective transformation; width and height represent the width and height of the original tongue image before the transformation; therefore, the coordinates of the external matrix after the perspective transformation are:

[0026]

[0027] Then, the cv2.getPerspectiveTransform() function is used to perform intelligent transformation of the perspective, transforming the coordinates of the four points of the external matrix of the original tongue contour into the coordinates of the four points of the external matrix after the perspective transformation, and obtaining the new image after the perspective transformation;

[0028] 4) Using the method of traversing the image pixel values, the standardized tongue segmentation image is used to locate and identify the pixels, and the corresponding organ area is determined according to the pixel position. The ImageDraw.Draw.text() function is used to fill in the text to obtain the intelligent positioning of the tongue triple energizer and five internal organs area;

[0029] The third stage: intelligent tongue analysis method based on ensemble learning

[0030] 1) inputting the standardized tongue segmentation image obtained in step 3) of the second stage into multiple CNN networks for feature extraction;

[0031] 2) Calculate the cross entropy loss of the outputs and labels of multiple CNN networks independently, and back-propagate to train a single network;

[0032] 3) Perform image reasoning on multiple trained CNN networks to obtain corresponding multiple probability distributions and perform integrated operations on the probability distributions, and finally obtain intelligent analysis results of the tongue image.

[0033] Furthermore, in step 1) of the first stage, the data enhancement method is Dropout, Sharpen, Affine, ElasticTransformation, GaussianBlur, Fliplr or PiecewiseAffine.

[0034] Furthermore, in step 2) of the second stage, the cv2.add() function is used to add the tongue segmentation mask image having only one contour to the initial image.

[0035] Furthermore, in step 3) of the second stage, the cv2.resize() function is used to perform size transformation.

[0036] As a preferred solution of the present invention, in the third stage, three CNN networks are used for feature extraction. The three CNN networks respectively use the ResNet model, the DenseNet model and the GoogLeNet model. The three trained CNN networks are used for image reasoning to obtain three probability distributions and perform an integrated operation of the probability distributions, as shown in the following formula:

[0037]

[0038] Among them, pred ResNet , pred DenseNet , pred GooLeNet They represent the probability distributions predicted by the ResNet model, the DenseNet model, and the GooLeNet model respectively. By calculating the threshold of the integrated probability distribution, as shown in the following formula, the final prediction value pred can be obtained. overall ;

[0039]

[0040] Among them, if is a conditional statement, which means if, It represents the mean of the probability distribution of multiple network pairs belonging to the i-th category.

[0041] To achieve the purpose of the present invention, a technical solution adopted by an embodiment of the present invention is: providing a computer storage medium, in which at least one executable instruction is stored, and the executable instruction enables the processor to execute the operation corresponding to the artificial intelligence-based intelligent Chinese medicine tongue diagnosis method.

[0042] Through the above design scheme, the present invention can bring the following beneficial effects:

[0043] 1. The present invention proposes a complete intelligent tongue analysis process with a wide range of applications and strong robustness and robustness.

[0044] 2. Based on Transfuse, the present invention makes full use of CNN network and Transformer network, integrating the strengths of the two networks, can capture low-level spatial and high-level semantic features, greatly improves the training and reasoning efficiency of the model, and improves the efficiency of model deployment in the cloud or smart terminals.

[0045] 3. The present invention uses a variety of intelligent image processing algorithms to achieve perspective conversion and data standardization of the segmented tongue image.

[0046] 4. The present invention uses the idea of ​​ensemble learning to integrate the prediction results of multiple networks, solves the overfitting problem of a single model, and improves the generalization performance of the deep learning model. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] A more complete understanding of exemplary embodiments of the present invention may be obtained by referring to the following drawings:

[0048] Figure 1 This is a flow chart of the intelligent TCM tongue diagnosis method based on artificial intelligence;

[0049] Figure 2 This is the second stage flow chart of the intelligent Chinese medicine tongue diagnosis method based on artificial intelligence. DETAILED DESCRIPTION

[0050] The specific implementation steps and working principles of the present invention are further described in detail below with reference to the accompanying drawings.

[0051] like Figure 1 and Figure 2 As shown, the present invention proposes a complete set of operation procedures for intelligent tongue analysis, and divides the entire process into three stages: tongue segmentation stage, segmentation post-processing stage, and triple-focus area detection and intelligent analysis stage.

[0052] The first stage: Segment the tongue using the existing tongue segmentation method to reduce irrelevant noise and redundancy in the tongue image data;

[0053] The second stage: post-process the segmented tongue image to achieve a unified perspective and further confirm the tongue image segmentation effect of the first stage, so as to facilitate the subsequent third stage of intelligent analysis;

[0054] The third stage: Train multiple neural networks and integrate them together through the idea of ​​ensemble learning to improve the accuracy, robustness and robustness of intelligent analysis.

[0055] The specific implementation plan is as follows:

[0056] The first stage is tongue segmentation based on the TransFuse network, which includes the following steps:

[0057] 1) Preprocess the tongue images taken by mobile terminals, perform data enhancement on each tongue image, and use the imgaug data enhancement library to perform random data enhancement operations. The data enhancement methods include Dropout, Sharpen, Affine, ElasticTransformation, GaussianBlur, Fliplr, PiecewiseAffine, etc.

[0058] 2) Two feature extraction methods are performed on the tongue image after data preprocessing. The first is to extract features through the CNN network branch, and the second is to extract features through the Transformer network branch. Specifically, in this embodiment, the CNN network used is ResNet34. The design of the Transformer network branch follows the encoding-decoding structure. The image input will be cut into multiple image blocks (16 image blocks in this embodiment), and then the image blocks are flattened and passed to the linear layer to obtain the features D of each image block. The position information and feature information of each image block are input into the encoding layer of the Transformer. In the Transformer structure, multi-layer multi-head self-attention (MSA) and multi-layer perceptron (MLP) are used.

[0059] Among them, the self-attention mechanism (SA) formula is:

[0060]

[0061] Where [q, k, v] = zW qkv , W qkv is a learnable weight matrix. i ,q i They represent the i-th row of z and q respectively. is the scaling factor. q represents the query matrix, k represents the key matrix, and v represents the value matrix. z represents the attention matrix obtained after the self-attention mechanism. softmax represents the flexible maximum value, k T represents the transpose of the key matrix.

[0062] 3) Fuse the features of the CNN network branch and the Transformer network branch: Use channel attention, spatial attention, 3×3 convolution and residual connection to fuse the features. The formula is as follows:

[0063]

[0064] Among them, t i is the feature obtained by the Transformer network at different scales. Similarly, g i Represents the features obtained by the CNN network at different scales; ChannelAttn represents the channel attention mechanism. The channel attention mechanism used in this embodiment is SE-Block (a method known in the field); SpatialAttn represents the spatial attention mechanism. The spatial attention mechanism used in this embodiment is CBAM (a method known in the field); and Represent the results after channel attention and spatial attention respectively; W1 i , Respectively for and The obtained learnable weight matrix; Conv represents 3×3 convolution; ⊙ represents the Hadamard product; Residual represents the residual block; represents the features obtained after 3×3 convolution; f i Represents the features obtained after feature aggregation.

[0065] 4) Training the segmentation network: Calculate the loss of the predicted mask and mask label obtained by forward propagation of the segmentation network, and perform back propagation through the loss function to train the segmentation network. The mask label is obtained by manual annotation. In the segmentation data set, each image has an original image and its mask image, which belongs to the prior art and will not be described in detail here. In this embodiment, the loss function used is Dice loss, and Adam is used as the optimizer.

[0066] 5) Use the trained segmentation network to perform inference calculations on the tongue image to obtain the first-stage tongue segmentation mask image.

[0067] In the second stage, the post-processing operation of the tongue segmentation mask image is based on OPENCV, and the positioning of the three burners and five internal organs in the tongue area is obtained, which specifically includes the following steps:

[0068] 1) Based on the tongue segmentation mask image obtained in the first stage, perform hole filling and remove redundant contours. The function used in the present invention is cv2.findContours(), which finds all contours in the mask image, finds the largest contour, and fills the pixels of the largest contour using the cv2.drawContours() function to obtain a new mask image. The formula is as follows:

[0069] Contours=cv2.findContours(img)

[0070] maxContours=find(Area(Contours))

[0071] Img new =cv2.drawContours(maxContours)

[0072] Among them, Contours represents the set of all contours. Area() represents the function of calculating the area of ​​a contour. find() is the function of finding the index of the maximum area in Contours. maxContours represents the set of coordinate values ​​of all points of the largest contour. Img new Represents the new mask image after pixel filling.

[0073] 2) After image processing based on step 1) of the second stage, a tongue segmentation mask image with only one contour is obtained, and the mask image is merged with the initial image of the first stage to obtain a segmented tongue image. The cv2.add() function is used in the present invention.

[0074] 3) The segmented tongue image obtained in step 2) of the second stage is resized and transformed in perspective to standardize the tongue segmentation image. The cv2.resize() function is used in the present invention for resizing. Before the perspective transformation, the present invention obtains the coordinates of the four points of the external matrix of the tongue contour through the cv2.minAreaRect() function, obtains the positions of the four coordinates through intelligent judgment of the coordinate points, and sorts the coordinates according to the upper left, upper right, lower right, and lower left. The positions of the four coordinate points in the new image after the customized perspective transformation are shown in formula (3) in this embodiment:

[0075]

[0076] Among them, width start , height start ,width end , height end Respectively represent the starting and ending positions of the width and height of the external matrix after the perspective transformation. Width and height represent the width and height of the original tongue image before the transformation. Therefore, the coordinates of the external matrix after the perspective transformation are:

[0077]

[0078] Then, the cv2.getPerspectiveTransform() function is used to perform intelligent transformation of the perspective, and the coordinates of the four points of the circumscribed matrix of the original tongue are transformed into the coordinates in formula (4) to obtain the new image after the perspective transformation.

[0079] 4) The standardized tongue image is subjected to pixel location recognition to obtain intelligent location of the tongue's three-jiao and five-zang regions. The present invention uses a method of traversing image pixel values ​​to intelligently identify the pixel position of the tongue, and determines the region of the corresponding organ based on the pixel position, and uses the ImageDraw.Draw.text() function to fill in text.

[0080] Phase 3: Intelligent tongue analysis based on ensemble learning:

[0081] 1) The standardized tongue segmentation image obtained in step 3) of the second stage is input into multiple CNN networks for feature extraction. In the present invention, three types of CNN networks are used, namely ResNet, DenseNet and GoogLeNet. It is worth noting that the networks used may not be limited to these three CNN networks.

[0082] 2) The outputs of the three CNN networks are independently cross-entropy-computed with the labels, and a single network is trained by back-propagation.

[0083] 3) Perform image reasoning on the three trained CNN networks to obtain three probability distributions and perform integrated operations on the probability distributions, and finally obtain the intelligent analysis results of the tongue image. The calculation process is shown in formula (5):

[0084]

[0085] Among them, pred ResNet , pred DenseNet , pred GooLeNet Represent the probability distributions predicted by the ResNet model, the DenseNet model, and the GooLeNet model respectively. By performing threshold calculation on the probability distribution after integration, as shown in formula (6), the final prediction value pred overall .

[0086]

[0087] Among them, if is a conditional statement, which means if, The multiple network pairs represented are the means of the probability distribution of the i-th class.

[0088] The present invention may also be implemented as a device or apparatus program (e.g., a computer program and a computer program product) for performing a portion or all of the methods described herein. Such a program implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0089] It should be noted that, in the description of the present invention, “plurality” means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0090] In summary, the above embodiment discloses a complete process of tongue image intelligent analysis method, which is divided into three stages: tongue segmentation based on TransFuse network; tongue segmentation post-processing based on OPENCV; intelligent analysis based on ensemble learning. Compared with previous studies, the present invention has significant advantages, and the accuracy, robustness and robustness of the algorithm have been significantly improved.

Claims

1. A smart Chinese medicine tongue diagnosis method based on artificial intelligence, characterized in that: The method is divided into three stages: tongue segmentation stage, segmentation preprocessing stage, and triple focal area detection and intelligent analysis stage. The three stages are specifically: The first stage, tongue segmentation based on the TransFuse network, includes the following steps: 1) Obtain tongue images and perform data enhancement preprocessing on each tongue image, wherein the data enhancement preprocessing is to perform random data enhancement operations using the imgaug data enhancement library; 2) Two independent channels are used to extract features from the tongue image after data preprocessing. The first channel is extracted through the CNN network, and the second channel is extracted through the Transformer network. When using the Transformer network for feature extraction, the tongue image is first cut into multiple image blocks, and then the image blocks are flattened and passed to the linear layer to obtain the feature D of each image block, and the position information and feature information of each image block are input into the encoding layer of the Transformer together; 3) Use channel attention, spatial attention, 3×3 convolution and residual connection to fuse the features extracted by the CNN network and the features extracted by the Transformer network. The formula is as follows: Among them, t i Represents the features obtained by the Transformer network at different scales. Similarly, g i Represents the features obtained by the CNN network at different scales; ChannelAttn represents the channel attention mechanism, and SpatialAttn represents the spatial attention mechanism; and Represent the results after channel attention and spatial attention respectively; W1 i and Respectively represent and The obtained learnable weight matrix; Conv represents 3×3 convolution; ⊙ represents the Hadamard product; Residual represents the residual block; represents the features obtained after 3×3 convolution, f i Represents the features obtained after feature fusion; 4) Training the segmentation network: Calculate the loss between the predicted mask and the mask label obtained by forward propagation of the segmentation network, and perform back propagation through the loss function to train the segmentation network; 5) Use the trained segmentation network to perform inference calculations on the tongue image to obtain the first-stage tongue segmentation mask image; In the second stage, the post-processing operation of the tongue segmentation mask image based on OPENCV is performed to obtain the positioning of the three burners and five internal organs in the tongue area, including the following steps: 1) Based on the tongue segmentation mask image obtained in the first stage, hole filling and redundant contour removal are performed to obtain a tongue segmentation mask image with only one contour; The specific process is: use the function cv2.findContours() to find all the contours in the tongue segmentation mask image, find the largest contour, and fill the pixels of the largest contour using the cv2.drawContours() function to get a new tongue segmentation mask image. The formula is as follows: Contours=cv2.findContours(img) maxContours=find(Area(Contours)) Img new =cv2.drawContours(maxContours) Among them, Contours represents the set of all contours found; Area() represents the function of calculating the area of ​​a contour; find() is the function of finding the index of the maximum area in Contours; maxContours represents the set of coordinate values ​​of all points of the largest contour; Img new Represents the new tongue segmentation mask image after pixel filling; 2) merging the tongue segmentation mask image with only one contour with the initial image in the first stage to obtain a segmented tongue image; 3) The segmented tongue image obtained in step 2) of the second stage is resized and transformed in perspective to standardize the tongue segmentation image; and before the perspective transformation, the cv2.minAreaRect() function is used to obtain the coordinates of the four points of the external matrix of the tongue contour, and the positions of the four coordinates are obtained by intelligent judgment through the coordinate points, and the coordinates are sorted according to the upper left, upper right, lower right, and lower left. The positions of the four coordinate points in the new image after the customized perspective transformation are shown in the following formula: Among them, width start , height start ,width end , height end Respectively represent the starting and ending positions of the width and height of the external matrix after the perspective transformation; width and height represent the width and height of the original tongue image before the transformation; therefore, the coordinates of the external matrix after the perspective transformation are: Then, the cv2.getPerspectiveTransform() function is used to perform intelligent transformation of the perspective, transforming the coordinates of the four points of the external matrix of the original tongue contour into the coordinates of the four points of the external matrix after the perspective transformation, and obtaining the new image after the perspective transformation; 4) Using the method of traversing the image pixel values, the standardized tongue segmentation image is used to locate and identify the pixels, and the corresponding organ area is determined according to the pixel position. The ImageDraw.Draw.text() function is used to fill in the text to obtain the intelligent positioning of the tongue triple energizer and five internal organs area; The third stage: intelligent tongue analysis method based on ensemble learning 1) inputting the standardized tongue segmentation image obtained in step 3) of the second stage into multiple CNN networks for feature extraction; 2) Calculate the cross entropy loss of the outputs and labels of multiple CNN networks independently, and back-propagate to train a single network; 3) Perform image reasoning on multiple trained CNN networks to obtain corresponding multiple probability distributions and perform integrated operations on the probability distributions, and finally obtain intelligent analysis results of the tongue image.

2. The intelligent Chinese medicine tongue diagnosis method based on artificial intelligence according to claim 1 is characterized by: In step 1) of the first stage, the data enhancement method is Dropout, Sharpen, Affine, ElasticTransformation, GaussianBlur, Fliplr or PiecewiseAffine.

3. The intelligent Chinese medicine tongue diagnosis method based on artificial intelligence according to claim 1 is characterized by: In step 2) of the second stage, the cv2.add() function is used to add the tongue segmentation mask image having only one contour to the initial image.

4. The intelligent Chinese medicine tongue diagnosis method based on artificial intelligence according to claim 1 is characterized by: In step 3) of the second stage, the cv2.resize() function is used to perform size transformation.

5. The intelligent Chinese medicine tongue diagnosis method based on artificial intelligence according to claim 1 is characterized by: In the third stage, three CNN networks are used for feature extraction. The three CNN networks respectively use the ResNet model, the DenseNet model and the GoogLeNet model. The three trained CNN networks are used for image reasoning to obtain three probability distributions and perform integrated operations on the probability distributions, as shown in the following formula: Among them, pred ResNet , pred DenseNet , pred GooLeNet They represent the probability distributions predicted by the ResNet model, the DenseNet model, and the GooLeNet model respectively. By calculating the threshold of the integrated probability distribution, as shown in the following formula, the final prediction value pred can be obtained. overall ; Among them, if is a conditional statement, which means if, It represents the mean of the probability distribution of multiple network pairs belonging to the i-th category.

6. A computer storage medium, wherein at least one executable instruction is stored in the computer storage medium, and the executable instruction enables a processor to perform operations corresponding to the artificial intelligence-based intelligent Chinese medicine tongue diagnosis method as described in any one of claims 1-5.

Images