Traditional Chinese medicine tongue diagnosis auxiliary diagnosis method and system fusing multiple characteristics of tongue body

Abstract

The invention relates to a traditional Chinese medicine tongue diagnosis auxiliary diagnosis method and system fusing multiple characteristics of a tongue body, and belongs to the technical field of medical information and health data processing. Firstly, a BlendMask network is utilized to carry out tongue segmentation model training, an obtained tongue segmentation model is utilized to complete data segmentation, tongue shape analysis is carried out on a segmented tongue data set, whether the shape is normal or not is judged by calculating a tongue skew index, and a data set with the normal shape is utilized to input a feature extraction network to carry out classification training. And classifying the tongue body image by using a tongue body identification and classification model, carrying out further regional tongue color analysis on the tongue body with abnormal tongue color, and outputting the comprehensive health state of the patient by combining the tongue body form skew feature, the tongue body identification and classification and the regional tongue color analysis. According to the invention, comprehensive analysis is carried out from the aspects of tongue body morphology, color and regional tongue color, the accuracy of tongue diagnosis analysis is improved, and efficient diagnosis of organs is realized.

Description

Technical Field

[0001] The present invention relates to a TCM tongue diagnosis auxiliary diagnosis method and system integrating multiple tongue features, belonging to the technical field of medical information and health data processing. Background Art

[0002] Tongue diagnosis is an important part of TCM visual diagnosis. The Medical Mirror says: "All diseases are manifested on the tongue... the tip of the tongue governs the heart, the middle of the tongue governs the spleen and stomach, the sides of the tongue govern the liver and gallbladder, and the root of the tongue governs the kidneys." TCM believes that the tongue is the only external organ that can observe the internal organs. Because the tongue is exposed, the characteristics of tongue changes rapidly and significantly. It is the most sensitive organ to reflect changes in the condition and is an indispensable objective basis for clinical syndrome differentiation and diagnosis.

[0003] In traditional Chinese medicine, the tongue diagnosis process is performed by the doctor observing the morphological characteristics of the tongue with the naked eye. Since there is no unified quantitative tool such as colorimetric cards, there are errors in determining the patient's attributes. In addition, the differences and subjectivity of perception, as well as the influence of environmental factors such as light, severely limit the development of Chinese medicine diagnostic methods and technologies. Therefore, it has become a top priority for the modernization of Chinese medicine diagnosis to achieve objective and quantitative tongue analysis with the help of multidisciplinary technologies such as digital image processing technology and artificial intelligence. Patent CN 106510636 A discloses a system and method for automatic detection of tongue images in traditional Chinese medicine, which is applied to a computer. The method includes the following steps: controlling an image acquisition device to capture a clear tongue image; segmenting the tongue image to obtain a tongue body image; extracting the color and texture features of the tongue coating from the tongue body area, and analyzing the characteristic values ​​of the thickness of the tongue coating, the dryness of the tongue coating, and the degree of greasy tongue coating according to the color and texture features of the tongue coating; detecting the length and width of the tongue pattern and quantifying them into a characteristic value describing the condition of the tongue pattern; detecting the size of the tongue surface and quantifying them into a characteristic value describing the fatness or thinness of the tongue body; combining the characteristic values ​​of the thickness of the tongue coating, the dryness of the tongue coating, the degree of greasy tongue coating, the condition of the tongue pattern, and the fatness or thinness of the tongue body into a multi-dimensional tongue image characteristic vector. This patent mainly focuses on traditional Chinese medicine diagnosis from the perspective of tongue coating. Patent CN 116311386 A discloses a tongue shape recognition method based on image segmentation, including S1 obtaining a tongue image; S2 detecting the tongue in the tongue image and segmenting it to obtain a tongue image; S3 extracting characteristic parameters of the tongue image; S4 identifying and distinguishing tongue types according to the characteristic parameters, identifying enlarged tongue, thin tongue and normal tongue through the tongue contour, and identifying old tongue and tender tongue through color characteristics and texture characteristics; classifying and separating the tongue image, and accurately automatically segmenting the tongue image to obtain the tongue image; extracting the characteristic parameters of the tongue image, and analyzing the tongue shape through the characteristic parameters, so that doctors with less experience can accurately determine the patient's tongue shape to ensure subsequent accurate judgment and treatment. The patent uses MaskR-CNN to classify the tongue shape. In the existing tongue image diagnosis, identification and judgment are mostly carried out from certain aspects such as tongue coating and tongue shape, which cannot comprehensively analyze the impact on the accuracy of tongue image diagnosis. On the other hand, the calculation process in classification and recognition is complicated, which affects the efficiency of diagnosis. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a traditional Chinese medicine tongue diagnosis auxiliary diagnosis method that integrates multiple tongue features, conducts a comprehensive analysis from the aspects of tongue shape, color and regional tongue color, improves the accuracy of tongue diagnosis analysis, and realizes organ diagnosis.

[0005] The technical solution adopted by the present invention is: A traditional Chinese medicine tongue diagnosis auxiliary diagnosis method integrating multiple tongue features comprises the following steps: S1. Collect tongue images and store them in the image sample library; S2. Select some tongue images from the sample library, add labels, and input them into the BlendMask network to train the tongue segmentation model. Use the obtained tongue segmentation model to complete all tongue data segmentation and integrate them into a tongue data set; S3. Performing tongue morphology analysis on the segmented tongue dataset, and determining whether tongue skew exists by calculating the tongue skew index. If tongue skew exists, it can be predicted that the patient has neurological damage, and if tongue skew does not exist, the patient is classified into a normal morphology dataset; S4. Use the normal morphological data set to input the feature extraction network with ResNet-34 as the backbone to extract color, shape and texture features and perform classification training to obtain a tongue recognition classification model. Use the tongue recognition classification model to classify the tongue image and determine whether it belongs to the category of abnormal tongue color, normal tongue, cracked tongue or tooth-marked tongue; S5. Further regional tongue color analysis is performed on the tongue with abnormal tongue color, tongue color parameters are extracted from the tongue image, and the hue of the HSV color space model is obtained using a region-based tongue color recognition method. H , Saturation S , Brightness V Value, according to the tongue color conditions of different areas, the tongue tip tongue color H t , S t Value, middle of tongue H m Value, root of tongue V r Value, both sides of tongue H s value, and the overall tongue color H z , S z , V z The values ​​are compared to determine the regional tongue color abnormality, and the health status of different organs is determined based on the regional tongue color abnormality; S6. Based on the above-mentioned tongue morphology and crookedness characteristics, tongue identification and classification, and regional tongue color analysis, the patient's comprehensive health status is output.

[0006] In the above method, the BlendMask network described in step S2 is composed of a detection network and a mask branch, the mask branch has three parts, a bottom module for predicting a score map, a top module for predicting instance attention, and a hybrid module for fusing the score and attention; the preprocessed tongue image is input into the ResNet101-FPN backbone neural network to extract features, the extracted features are input into the bottom module to obtain a score map, the extracted features are input into the top module to predict the top-level attention, the score map, the attention and the candidate bounding box are input into the hybrid module for calculation to obtain a mask, the mask and the original tongue color image are subjected to a bitwise "AND" operation, wherein the marked area in the mask is retained and the black area is cleared, and a new image containing only the tongue is obtained, thereby completing the segmentation operation of the tongue image.

[0007] The integration described in step S2 is: standardizing the image, cropping it, scaling the cropped image, and then adding labels to the processed tongue image.

[0008] The calculation and judgment process of the tongue deflection index in step S3 is as follows: (1) Determine the central axis AB of the tongue; call the Gdal library to find the target centroid of the tongue, and then find a series of straight lines passing through the target centroid. Each straight line divides the tongue into two parts, the left and right parts. The ratio of the area difference to the average area of ​​these two parts is calculated respectively. Finally, the straight line with the smallest area difference and area ratio is compared. This straight line is the central axis of the target. (2) Determine the tail line of the tongue. The line connecting the two left and right points at the end of the divided tongue is the tail line of the tongue. (3) Determine the median perpendicular line CD of the tongue body and find the midpoint of the tail line of the tongue body to determine the median perpendicular line of the tongue body; (4) Calculate the angle. Based on the coordinates of the central axis and the perpendicular bisector, calculate the angle between the two lines: , , In the formula, A ( x A ,y A )and B(x) B ,y B ） are two points on the central axis; C(x) C ,y C ） and D(x) D ,yD ） are two points on the perpendicular bisector, i is the angle between the central axis and the perpendicular bisector; (5) Calculation of tongue deflection index: , In the formula, , th1 and th2 is the empirical threshold. The central axis and the median perpendicular line intersect with the tongue body and tail line at points A and C respectively. AC represents the distance between the two intersection points. L The tail line of the tongue. i is the angle between the central axis and the perpendicular bisector, abs () indicates absolute value, Askew It is the tongue deflection index; Principles for judging the direction of tongue deflection: Askew =0, normal tongue; Askew When >0, it is judged as tongue deviation.

[0009] The tongue recognition and classification process in step S4 is as follows: (1) Use the ResNet-34 network to extract the deep visual features of the tongue image and generate the corresponding feature binarization map. Multiply the feature binarization map with each feature channel map of the deep visual feature to obtain the region of interest features of the tongue; (2) Select the region of interest features of three different scales, low, medium, and high layers, in the ResNet-34 network respectively, and then use the residual sampling structure to transform the dimensions of the multi-scale features, complete the fusion of multi-scale features, and obtain the normalized multi-scale features of the region of interest; (3) Using the normalized multi-scale features of the region of interest, a multi-level double quadratic pooling structure is constructed, that is, the interaction between the same-layer and cross-layer features is simultaneously constructed to obtain the final representation vector; (4) Use the fully connected layer for classification.

[0010] In step S4, a fissured tongue can predict spleen and stomach problems, and a tooth-marked tongue can predict lung problems.

[0011] The regional tongue color analysis described in step S5 first divides the tongue into regions, and divides the tongue image into regions based on empirical values. The horizontal division rule is: the root of the tongue occupies 1 / 6, the tip of the tongue occupies 1 / 6, and the middle of the tongue occupies 2 / 3. The vertical division rule is: the left and right sides of the tongue each occupy 1 / 10, and the middle of the tongue occupies 4 / 5.

[0012] The calculation formulas for the HSV values ​​of the tongue color of the entire tongue and different areas in step S5 are as follows: , , , , , in H It's the color tone. S is saturation, V is brightness, R , G , B It is the proportion of red, green and blue in the three primary colors.

[0013] In step S5, the health status of different organs is judged according to the abnormal tongue color in the region as follows: like H t or S t Greater than H z , S z , it means that the tongue tip and tongue color are abnormal, predicting that the patient has problems with cardiopulmonary function; like H m Greater than H z , it means the color of the middle part of the tongue is abnormal, predicting that the patient has problems with spleen and stomach function; like V r Greater than V z , it means that the color of the tongue root is abnormal, which predicts that the patient has problems with kidney function; like H s Greater than H z , it means that the color of both sides of the tongue is dark, which predicts that the patient has problems with liver and gallbladder function.

[0014] Another object of the present invention is to provide a TCM tongue diagnosis auxiliary diagnosis system integrating multiple tongue features, comprising An image acquisition module is used to acquire tongue images and store them in a picture sample library; The tongue segmentation module uses the trained tongue segmentation model to complete all tongue data segmentation and integrate them into a tongue data set; The morphological discrimination module performs tongue morphological analysis on the segmented tongue data set to determine whether the tongue is skewed; The tongue recognition and classification module uses the trained tongue recognition and classification model to classify the tongue images with normal morphology and determine whether they belong to abnormal tongue color, normal tongue, cracked tongue or tooth-marked tongue; The regional tongue analysis module extracts tongue color parameters from the tongue image for tongues with abnormal tongue color and obtains the hue of the HSV color space model H , Saturation S , Brightness V Value, according to the tongue color conditions of different areas, the tongue tip tongue color H t , S t Value, middle of tongue H m Value, root of tongue V r Value, both sides of tongue H s value, and the overall tongue color H z , S z , V z The values ​​are compared to determine the abnormal tongue color area; The overall health prediction module combines the tongue morphology and deviation characteristics, tongue recognition and classification, and regional tongue analysis to output the patient's comprehensive health status.

[0015] The beneficial effects of the present invention are: The present invention first uses the BlendMask network to segment the collected images and removes unnecessary interference areas such as lips and teeth; secondly, by judging the skewness of the tongue shape, the diagnosis of whether there is nervous system damage is achieved; then, in order to identify subtle differences in tongue images to achieve tongue classification and organ health diagnosis, the ResNet-34 network is used, which can use deep convolution to extract complex color and shape texture features in the image; finally, through further analysis of abnormal tongue color and comprehensive classification and analysis results, the diagnosis and prediction of organ abnormalities are completed.

[0016] The present invention conducts a comprehensive analysis from the aspects of tongue shape, color and regional tongue color, improves the accuracy of tongue diagnosis and analysis, and realizes efficient diagnosis of organs. In terms of tongue segmentation, the calculation of the center of mass and contour of the tongue is not adopted, but the BlendMask network is used for segmentation, which simplifies the calculation process and also improves the accuracy of tongue segmentation; in terms of tongue type recognition and classification, the tongue coating is not involved, but the analysis is conducted from the two aspects of tongue shape and tongue color, which not only realizes the judgment of tongue crookedness, but also completes the distinction of tongue types; finally, on the basis of classification, the prediction of organ health is further completed, which is closer to the diagnosis result of traditional Chinese medicine tongue diagnosis. The present invention combines computer image analysis technology with traditional Chinese medicine tongue diagnosis, which is a favorable supplement to the weak link of lack of quantification in traditional Chinese medicine diagnosis, provides objective quantitative analysis data for traditional Chinese medicine doctors, and assists traditional Chinese medicine clinical diagnosis, which will greatly promote the development of objectivity and standardization of medical diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic diagram of the process of the present invention; Figure 2 Schematic diagram of the network of the tongue segmentation model in the method of the present invention; Figure 3 A schematic diagram of tongue deflection determination in the method of the present invention; Figure 4 This is a network diagram of the tongue recognition classification model in the method of the present invention. DETAILED DESCRIPTION

[0018] The present invention is further described below with reference to specific embodiments.

[0019] Example 1 A TCM tongue diagnosis auxiliary diagnosis method integrating multiple tongue features comprises the following steps: S1. Collect tongue images and store them in the image sample library: When collecting image data, in order to control variables and ensure uniform light, a standard tongue diagnosis collection instrument is used. When shooting, volunteers are required to sit upright with their chins placed on the chin support. This not only ensures uniform light and the color of the tongue itself, but also ensures that the mouth is placed in the right position, making it easier to obtain a tongue image that is in a consistent position and not tilted. Tongue images are collected and stored in the image sample library.

[0020] S2. Select some tongue images from the sample library, add labels, and input them into the BlendMask network to train the tongue segmentation model. Use the obtained tongue segmentation model to complete all tongue data segmentation and integrate them into the tongue data set: Randomly select some tongue images from the image sample library to train the tongue segmentation model. First, in order to obtain the training set of the segmentation model, use Labelme software to manually segment the separate tongue area, discard irrelevant parts such as the mouth and teeth, and add tongue labels. Then input the labeled tongue image and the original color tongue image into the BlendMask network, and set the relevant parameters (for experimental reference only): To reduce memory overhead, Dataloader.num_workers can be set to 2, the initial learning rate can be set to 0.002, and the number of training rounds epoch can be set to 20. Finally, save the trained tongue segmentation model.

[0021] The BlendMask network consists of a detection network and a mask branch. The mask branch has three parts: a bottom module that predicts the score map, a top module that predicts the instance attention, and a blending module that combines the score and attention. The construction and segmentation process of the tongue segmentation model is as follows: (a) Input the preprocessed tongue image into the backbone neural network (ResNet101-FPN) to obtain the feature map; (ii) The four feature maps C2, C3, C4 and C5 from the ResNet101 network are 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the original tongue map size respectively. These four feature maps are then sent to FPN (Feature Pyramid Network) for feature extraction to obtain P2-P5. The feature maps C2-C5 and P2-P5 are input to the bottom module and the score map B is output. The shape of the score map is N × K × H s × W s ,in N is the batch size, K is the number of score graphs, H × W is the size of the input graph, s is the step size of the score map output; (III) Each detection tower (the detection tower extracts the feature information of the region of interest from the top feature area and the bottom feature area) is followed by a convolutional layer to predict the top-level attention A, where A is a tensor with a size of N ×( K × M × M )× H l × W l ,in M × Mis the resolution of A. Through the three-dimensional structure, A can encode instance-level information and use K × M × M The prediction results are first passed through D bounding boxes P and the corresponding attention module A to facilitate subsequent calculations. in , , R represents a real number, and the letter in the upper right corner represents the dimension; (IV) Input the obtained score maps B, D attentions A and candidate bounding boxes P into the hybrid module for calculation, and finally obtain the mask. The detailed steps are as follows: (1) Candidate bounding boxes in step (3) p d Crop the score map and rescale the area to a fixed size of R × R Feature map r d : , In the formula PoIPool The layer function is to map different sizes to a fixed size R × R On the feature map r d , R refers to the resolution of the underlying region of interest, B represents the score graph of step (ii), p d is a candidate bounding box, the number of which is D; (2) Along K dimensions, and the score map is obtained s d The collection consists of: , Use in the formula softmax The function is to normalize. Interpolate represents bilinear interpolation, M × M is the resolution of attention A, R × R is the feature map size, a d represents an attention vector; (3) For each feature map r d , score graph s d Perform an element-wise product operation, and then KThe results of the dimensions are summed to get the final mask. , K The default value is 4. m d Represents a mask, For Hadamard; (V) The mask and the original tongue color image are subjected to a bitwise “AND” operation, in which the marked area in the mask is retained and the black area is cleared, thereby obtaining a new image containing only the tongue, thus completing the segmentation operation of the tongue image.

[0022] Using the tongue segmentation model, all tongue data segmentation is completed and integrated into a tongue data set. Since the tongue images are not uniform in size, the images are standardized, and the RandomResizeCrop method is used to crop the tongue images, and the cropped images are scaled to size. Finally, Labelme software is used to add labels of healthy, abnormal tongue color, cracked tongue, and tooth-marked tongue to the processed tongue images.

[0023] S3. Perform tongue morphology analysis on the segmented tongue dataset and determine whether there is tongue deflection by calculating the tongue deflection index. If there is tongue deflection, it can be predicted that the patient has neurological damage. If there is no tongue deflection, it is classified into the normal morphology dataset: For the segmented tongue dataset, we first perform tongue skew analysis, which is divided into the following steps: (I) Determine the central axis AB of the tongue. First, call the Gdal library to find the target centroid of the tongue. Then, find a series of straight lines passing through the target centroid. Each straight line divides the tongue into two parts, the left and right parts. The ratio of the area difference to the average area of ​​these two parts is calculated. Finally, compare the straight line with the smallest area difference and area ratio. This straight line is the central axis of the target. (ii) Determine the tail line of the tongue. The line connecting the two left and right points at the end of the divided tongue is the tail line of the tongue, denoted by L. (III) Determine the median perpendicular line CD of the tongue body. Find the midpoint of the tail line of the tongue body, and then the median perpendicular line of the tongue body can be determined. (iv) Calculate the angle. Based on the coordinates of the central axis and the perpendicular bisector, calculate the angle between the two lines: , , In the formula, A ( x A ,y A )and B(x) B ,yB ） are two points on the central axis; C(x) C ,y C ） and D(x) D ,y D ） are two points on the perpendicular bisector, i is the angle between the central axis and the perpendicular bisector; (V) Calculation of tongue deflection index: , In the formula, , th1 and th2 is the empirical threshold, the central axis and the median perpendicular line intersect the tongue body and tail line at point A and point C respectively. AC That is, it represents the distance between the two intersection points. L The tail line of the tongue. i is the angle between the central axis and the perpendicular bisector, abs () indicates absolute value, Askew It is the tongue deflection index.

[0024] Principles for judging the direction of tongue deflection: Askew =0, normal tongue; Askew When the value is >0, it is judged as tongue deviation. If the tongue is deviated, it can be preliminarily predicted that the person has neurological damage.

[0025] S4. Use the normal morphological data set to input the feature extraction network with ResNet-34 as the backbone to extract color, shape and texture features and perform classification training to obtain a tongue recognition classification model. Use the tongue recognition classification model to classify the tongue image and determine whether it belongs to the category of abnormal tongue color, normal tongue, cracked tongue or tooth-marked tongue; ResNet-34 was used as the backbone feature extraction network to establish a tongue recognition classification model. This network uses deep convolution to extract complex color, shape and texture features in images, and has unique advantages in identifying subtle differences in tongue images.

[0026] (I) Use the ResNet-34 network to extract the deep visual features of the tongue image and generate the corresponding feature binary map, which is then multiplied by each feature channel map of the deep visual feature to obtain the region of interest features of the tongue. The implementation process is as follows: Using ResNet-34 network to obtain deep visual features of tongue images ,Will X Add in the channel direction to get the feature map: , in , Representation Dimension h w real number; X is three-dimensional, the first two dimensions are required, and the x channel takes 1~k, so It means that the height and width of the image are not processed, and only the sum of 1 to k is performed on the X channel; k express X The number of channels in the direction of h , w , c They represent the height (the number of pixels from top to bottom), width (the number of pixels from left to right), and number of color channels of the tongue image respectively.

[0027] The feature map A x The mean of is set as the threshold α , A x Inner greater than or equal to α The area is regarded as the target main area, and its value is set to 1. α The area is considered as the background and its value is set to x , in x is a decimal that is infinitely close to 0, and the binary image of the feature map is obtained M X , and its generation process is: , Where i and j are the horizontal and vertical coordinates of the image pixels. A x is the feature map, x is a real number, α is the threshold value, i is the threshold weight, which ranges from [0.5, 1] ​​and can be obtained through cross-validation experiments. h x , w X It is a feature X height and width.

[0028] The figure M X Do a dot product with each feature channel map of the image deep visual feature X.

[0029] The convolutional features of the three layers Conv5_3, Conv4_6 and Conv3_4 in ResNet are selected as the feature expressions of different levels and scales of the image, which are recorded as , , , for the three features X, Y, and Z, use the above formula to generate the corresponding binary images, and then use the following formula to fuse the three binary images , In the formula, M It is a unified binary image, which can effectively improve the accuracy. M x , M y , M z The feature binarization images corresponding to X, Y, and Z respectively. is the maximum pooling downsampling function, M y Downsample to M x Same size, represents the Hadamard product.

[0030] Binarized image of fusion M Perform bilinear interpolation to adapt to the size of the Y and Z features to obtain a normalized binary image , and use the Hadamard product operation to obtain the corresponding final feature map of the region of interest: , in, Represents the features of the region of interest finally extracted from X, Y, and Z. Function represent M By bilinear interpolation, it can be enlarged to Size, top right It means that the height and width of the image are not processed, and only the color channel is operated.

[0031] (II) Select the features of the regions of interest at three different scales, low, medium, and high levels, in the ResNet-34 network, respectively, and then use the residual sampling structure to transform the dimensions of the multi-scale features to complete the fusion of the multi-scale features. The specific steps are: Main line route K × K It consists of a pooling layer and a 1×1 convolution layer. The pooling layer is used to change the size of the feature map, and the convolution layer changes the number of feature channels. The convolution layer is used to construct a residual branch to make up for the feature information lost in the pooling layer in the main line. The convolution layer in this branch has the same number of convolution kernels as the convolution layer in the main line. The main line and the residual branch are added together to obtain the final sampling network, and the features of the region of interest are normalized: , in, Q and Q’ Indicates characteristics, is the original dimension of the feature, is the target dimension; BN is the normalization layer; For use c p The size is K × K Convolution kernel, in the feature Q’ Perform convolution operation on After applying the residual sampling structure, the normalized multi-scale features of the region of interest are obtained. .

[0032] 3. Multi-scale features A multi-level double quadratic pooling structure is constructed. First, the convolution features are operated on by Hadamard products, and the convolution features at the same level are interacted. Then, each convolution feature is matrix-outer-producted with its own transpose, so that the features at different positions and channels are associated: , in, P is the classification parameter matrix, concat represents the concatenation operation, U , V , S For each convolutional feature, the cross-layer mapping matrix represents the aggregation of features, and at the same time, the interaction between the same-layer and cross-layer features is constructed to obtain the final representation vector O , using fully connected layers for classification.

[0033] (IV) Construct an end-to-end network for training according to the above steps to obtain a trained tongue recognition model. Use the tongue recognition model to classify the tongue images in the test set and output their categories: abnormal tongue color, normal tongue, cracked tongue or tooth-marked tongue.

[0034] A fissured tongue can predict spleen and stomach problems, while a tongue with tooth marks can predict lung problems.

[0035] S5. Further regional tongue color analysis is performed on the tongue with abnormal tongue color, tongue color parameters are extracted from the tongue image, and the hue of the HSV color space model is obtained using a region-based tongue color recognition method. H , Saturation S , Brightness V Value, according to the tongue color conditions of different areas, the tongue tip tongue color H t , S t Value, middle of tongue H m Value, root of tongue V r Value, both sides of tongue H s value, and the overall tongue color Hz , S z , V z The values ​​are compared to determine the regional tongue color abnormality, and the health status of different organs is determined based on the regional tongue color abnormality: In order to further analyze the tongue with abnormal tongue color, the tongue is identified by region. The tongue color parameters are extracted from the tongue image photos, and the H, S, and V values ​​of the HSV color space model are obtained by using the region-based tongue color recognition method. The HSV color space model is the abbreviation of hue, saturation, and value. First, the tongue is segmented into regions. The tongue image is segmented into regions based on empirical values. The horizontal division rule is: the root of the tongue accounts for 1 / 6, the tip of the tongue accounts for 1 / 6, and the middle of the tongue accounts for 2 / 3. The vertical division rule is: the left and right sides of the tongue each account for 1 / 10, and the middle of the tongue accounts for 4 / 5.

[0036] Calculate the HSV value of the tongue color of the whole tongue and different areas: , , , , , in H (Hue) is the hue, S (Saturation) is saturation, V (Value) is brightness. RGB is the three primary colors: color = R (percentage of red) + G (Green percentage) + B (Percentage of blue).

[0037] According to the tongue color conditions in different areas, the tongue tip H t , S t Value, middle of tongue H m Value, root of tongue V r Value, both sides of tongue H s value, and the overall tongue color H z , S z , V z Values ​​to compare.

[0038] like Ht or S t Greater than H z , S z , it means that the tongue tip and tongue color are abnormal, predicting that the patient has problems with cardiopulmonary function; like H m Greater than H z , it means the color of the middle part of the tongue is abnormal, predicting that the patient has problems with spleen and stomach function; like V r Greater than V z , it means that the color of the tongue root is abnormal, which predicts that the patient has problems with kidney function; like H s Greater than H z , it means that the color of both sides of the tongue is dark, which predicts that the patient has problems with liver and gallbladder function.

[0039] Through the above analysis of the color of the tongue area, we can find out which internal organs the patient is more likely to have problems with, assist the doctor in diagnosis, and help the patient to regulate his body.

[0040] S6. Based on the above-mentioned tongue morphology and crookedness characteristics, tongue identification and classification, and regional tongue color analysis, the patient's comprehensive health status is output.

[0041] Embodiment 2 A TCM tongue diagnosis auxiliary diagnosis system integrating multiple tongue features to implement the method described in Embodiment 1 comprises: An image acquisition module is used to acquire tongue images and store them in a picture sample library; The tongue segmentation module uses the trained tongue segmentation model to complete all tongue data segmentation and integrate them into a tongue data set; The morphological discrimination module performs tongue morphological analysis on the segmented tongue data set to determine whether the tongue is skewed; The tongue recognition and classification module uses the trained tongue recognition and classification model to classify the tongue images with normal morphology and determine whether they belong to abnormal tongue color, normal tongue, cracked tongue or tooth-marked tongue; The regional tongue analysis module extracts tongue color parameters from the tongue image for tongues with abnormal tongue color and obtains the hue of the HSV color space model H , Saturation S , Brightness V Value, according to the tongue color conditions of different areas, the tongue tip tongue color H t , St Value, middle of tongue H m Value, root of tongue V r Value, both sides of tongue H s value, and the overall tongue color H z , S z , V z The value is compared to determine the abnormal tongue color area; the overall health prediction module combines the tongue morphology and crookedness characteristics, tongue recognition and classification, and regional tongue analysis to output the patient's comprehensive health status.

[0042] The above is a further description of the present invention in combination with the embodiments, and the protection scope of the present invention is not limited thereto.

Claims

1. A TCM tongue diagnosis auxiliary diagnosis method integrating multiple tongue features, characterized in that: The steps include: S1. Collect tongue images and store them in the image sample library; S2. Select some tongue images from the sample library, add labels, and input them into the BlendMask network to train the tongue segmentation model. Use the obtained tongue segmentation model to complete all tongue data segmentation and integrate them into a tongue data set; S3. Performing tongue morphology analysis on the segmented tongue dataset, and determining whether tongue skew exists by calculating the tongue skew index. If tongue skew exists, it can be predicted that the patient has neurological damage, and if tongue skew does not exist, the patient is classified into a normal morphology dataset; S4. Use the normal morphological data set to input the feature extraction network with ResNet-34 as the backbone to extract color, shape and texture features and perform classification training to obtain a tongue recognition classification model. Use the tongue recognition classification model to classify the tongue image and determine whether it belongs to the category of abnormal tongue color, normal tongue, cracked tongue or tooth-marked tongue; S5. Further regional tongue color analysis is performed on the tongue with abnormal tongue color, tongue color parameters are extracted from the tongue image, and the hue of the HSV color space model is obtained using a region-based tongue color recognition method. H , Saturation S , Brightness V Value, according to the tongue color conditions of different areas, the tongue tip tongue color H t , S t Value, middle of tongue H m Value, root of tongue V r Value, both sides of tongue H s value, and the overall tongue color H z , S z , V z The values ​​are compared to determine the regional tongue color abnormality, and the health status of different organs is determined based on the regional tongue color abnormality; S6. Based on the above-mentioned tongue morphology and crookedness characteristics, tongue identification and classification, and regional tongue color analysis, the patient's comprehensive health status is output.

2. According to claim 1, a TCM tongue diagnosis auxiliary diagnosis method integrating multiple tongue features is characterized in that: The BlendMask network described in step S2 is composed of a detection network and a mask branch. The mask branch has three parts: a bottom module for predicting a score map, a top module for predicting instance attention, and a hybrid module for fusing the score and attention. The preprocessed tongue image is input into the ResNet101-FPN backbone neural network to extract features, the extracted features are input into the bottom module to obtain a score map, the extracted features are input into the top module to predict the top-level attention, the score map, the attention and the candidate bounding box are input into the hybrid module for calculation to obtain a mask, the mask and the original tongue color image are subjected to a bitwise "AND" operation, wherein the marked area in the mask is retained and the black area is cleared, and a new image containing only the tongue is obtained, thereby completing the segmentation operation of the tongue image.

3. According to claim 1, a TCM tongue diagnosis auxiliary diagnosis method integrating multiple tongue features is characterized in that: The integration described in step S2 is: standardizing the image, cropping it, scaling the cropped image, and then adding labels to the processed tongue image.

4. The TCM tongue diagnosis auxiliary diagnosis method integrating multiple tongue features according to claim 1 is characterized in that: The calculation and judgment process of the tongue deflection index in step S3 is as follows: (1) Determine the central axis AB of the tongue; call the Gdal library to find the target centroid of the tongue, and then find a series of straight lines passing through the target centroid. Each straight line divides the tongue into two parts, the left and right parts. The ratio of the area difference to the average area of ​​these two parts is calculated respectively. Finally, the straight line with the smallest area difference and area ratio is compared. This straight line is the central axis of the target. (2) Determine the tail line of the tongue. The line connecting the two left and right points at the end of the divided tongue is the tail line of the tongue. (3) Determine the median perpendicular line CD of the tongue body and find the midpoint of the tail line of the tongue body to determine the median perpendicular line of the tongue body; (4) Calculate the angle. Based on the coordinates of the central axis and the perpendicular bisector, calculate the angle between the two lines: ， ， In the formula, A ( x A ,y A )and B(x B ,y B ） are two points on the central axis; C(x C ,y C ） and D(x D ,y D ） are two points on the perpendicular bisector, θ is the angle between the central axis and the perpendicular bisector; (5) Calculation of tongue deflection index: ， In the formula, , th1 and th2 is the empirical threshold, the central axis and the median perpendicular line intersect the tongue body and tail line at point A and point C respectively. AC That is, it represents the distance between the two intersection points. L The tail line of the tongue. θ is the angle between the central axis and the perpendicular bisector, abs () indicates absolute value, Askew It is the tongue deflection index; Principles for judging the direction of tongue deflection: Askew =0, normal tongue; Askew When >0, it is judged as tongue deviation.

5. The TCM tongue diagnosis auxiliary diagnosis method integrating multiple tongue features according to claim 1 is characterized in that: The tongue recognition and classification process in step S4 is as follows: (1) Use the ResNet-34 network to extract the deep visual features of the tongue image and generate the corresponding feature binarization map. Multiply the feature binarization map with each feature channel map of the deep visual feature to obtain the region of interest features of the tongue; (2) Select the region of interest features of three different scales, low, medium, and high layers, in the ResNet-34 network respectively, and then use the residual sampling structure to transform the dimensions of the multi-scale features, complete the fusion of multi-scale features, and obtain the normalized multi-scale features of the region of interest; (3) Using the normalized multi-scale features of the region of interest, a multi-level double quadratic pooling structure is constructed, that is, the interaction between the same-layer and cross-layer features is simultaneously constructed to obtain the final representation vector; (4) Use the fully connected layer for classification.

6. The TCM tongue diagnosis auxiliary diagnosis method integrating multiple tongue features according to claim 1 is characterized in that: In step S4, a fissured tongue can predict spleen and stomach problems, and a tooth-marked tongue can predict lung problems.

7. The TCM tongue diagnosis auxiliary diagnosis method integrating multiple tongue features according to claim 1 is characterized in that: The regional tongue color analysis described in step S5 first divides the tongue into regions, and divides the tongue image into regions based on empirical values. The horizontal division rule is: the root of the tongue occupies 1 / 6, the tip of the tongue occupies 1 / 6, and the middle of the tongue occupies 2 / 3. The vertical division rule is: the left and right sides of the tongue each occupy 1 / 10, and the middle of the tongue occupies 4 / 5.

8. The TCM tongue diagnosis auxiliary diagnosis method integrating multiple tongue features according to claim 1 is characterized in that: The calculation formulas for the HSV values ​​of the tongue color of the entire tongue and different areas in step S5 are as follows: ， ， ， ， ， in H It's the color tone. S is saturation, V is brightness, R , G , B It is the proportion of red, green and blue in the three primary colors.

9. The TCM tongue diagnosis auxiliary diagnosis method integrating multiple tongue features according to claim 1 is characterized in that: In step S5, the health status of different organs is judged according to the abnormal tongue color in the region as follows: like H t or S t Greater than H z , S z , it means that the tongue tip and tongue color are abnormal, predicting that the patient has problems with cardiopulmonary function; like H m Greater than H z , it means the color of the middle part of the tongue is abnormal, predicting that the patient has problems with spleen and stomach function; like V r Greater than V z , it means the tongue root color is abnormal, predicting that the patient has problems with kidney function; like H s Greater than H z , it means that the color of both sides of the tongue is dark, which predicts that the patient has problems with liver and gallbladder function.

10. A TCM tongue diagnosis auxiliary diagnosis system integrating multiple tongue features, characterized in that: include An image acquisition module is used to acquire tongue images and store them in a picture sample library; The tongue segmentation module uses the trained tongue segmentation model to complete the segmentation of all tongue data and integrate them into a tongue data set; The morphological discrimination module performs morphological analysis on the segmented tongue data set to determine whether the tongue is skewed; The tongue recognition and classification module uses the trained tongue recognition and classification model to classify the tongue images with normal morphology and determine whether they belong to abnormal tongue color, normal tongue, cracked tongue or tooth-marked tongue; The regional tongue analysis module extracts tongue color parameters from the tongue image for tongues with abnormal tongue color and obtains the hue of the HSV color space model H , Saturation S , Brightness V Value, according to the tongue color conditions of different areas, the tongue tip tongue color H t , S t Value, middle of tongue H m Value, root of tongue V r Value, both sides of tongue H s value, and the overall tongue color H z , S z , V z The values ​​are compared to determine the abnormal tongue color area; The overall health prediction module combines the tongue morphology and deviation characteristics, tongue recognition and classification, and regional tongue analysis to output the patient's comprehensive health status.

Images