Tongue image feature classification method and device and computer readable storage medium

Abstract

The invention relates to a tongue image feature classification method, a tongue image feature classification device and a computer readable storage medium, belongs to the field of image processing, and solves the problems of relatively large tongue image region division error, low accuracy and high reliability of the existing tongue image feature classification method in the prior art. And the accuracy of classifying the tongue image by the classifier obtained by subsequent training is low. The tongue image feature classification method comprises the following steps: acquiring a tongue image to be classified; performing image segmentation on the tongue image by using a trained segmentation network to obtain a tongue region in the tongue image; dividing a tongue area in the tongue image into a plurality of subareas; extracting texture features of different subarea images of the tongue area; and classifying the tongue image by using a trained classifier according to the texture features of the different subarea images of the tongue area to obtain a tongue image classification result. According to the method, the regional division error of the tongue image is small, and the accuracy is high.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a tongue image feature classification method, device and computer-readable storage medium. Background Art

[0002] The tongue is one of the important organs of the human body. It is composed of crisscrossing striated muscles and covered with a special mucosal structure. In traditional Chinese medicine diagnosis, tongue diagnosis is one of the important links. In traditional Chinese medicine theory, the tongue is connected to the internal organs through meridians. Tongue diagnosis is an effective and non-invasive diagnostic method that determines the occurrence, development and prognosis of diseases by observing the tongue quality, the shape and color of the tongue coating. When a disease occurs, the tongue's quality, coating, tooth marks and other characteristics often show regular changes, which can be captured and analyzed by the naked eye or images.

[0003] In traditional Chinese medicine diagnosis, tongue diagnosis is done through visual inspection. However, traditional diagnostic methods are mainly based on experience or subjectivity, and the quality and results of visual inspection of the tongue vary among different Chinese medicine physicians. If you are an experienced expert, you can accurately judge the characteristics of the tongue image and make a diagnosis, but the same skills cannot be easily transferred to other medical professionals. Moreover, these diagnostic skills can only be limited to qualitative descriptions, and cannot be quantified or expressed mathematically to form widely applicable methods and standards. As a result, the classification of tongue image features in Chinese medicine has always lacked objective and standardized standards, which has brought great limitations to the clinical Chinese medicine syndrome differentiation and judgment of the various image features of the tongue and modern research in Chinese medicine.

[0004] With the development of computer science, mathematics, and biological technology, the existing technology has achieved qualitative and quantitative research on tongue images. However, the existing classification method of tongue image features lacks standardized processing of tongue image segmentation, resulting in poor performance of the model for extracting image features for subsequent modeling. Summary of the invention

[0005] In view of the above analysis, the embodiments of the present invention aim to provide a tongue image feature classification method, device and computer-readable storage medium, so as to solve the problem that the existing tongue image feature classification method has large errors in regional division of tongue images and low accuracy, resulting in low accuracy of classifying tongue images by the classifier obtained by subsequent training.

[0006] The purpose of the present invention is mainly achieved through the following technical solutions: The present invention provides a tongue image feature classification method, comprising the following steps: Step S1: obtaining a tongue image to be classified; Step S2: performing image segmentation on the tongue image using a trained segmentation network to obtain a tongue region in the tongue image; Step S3: dividing the tongue area in the tongue image into a number of partitions; Step S4: extracting texture features of different partition images of the tongue area; Step S5: classifying the tongue image using a trained classifier according to the texture features of the images of different partitions of the tongue area to obtain a tongue image classification result.

[0007] Furthermore, in step S2, the segmentation network is trained by the following steps: Step 21: obtaining a plurality of tongue images, and marking the tongue region in each tongue image; Step 22: Divide the tongue images into a training set and a validation set according to the specified ratio according to the number of tongue images obtained, use the training set as the training basis, and use the segmentation models pytorch segmentation framework to train multiple decoder automatic segmentation networks; If the number of acquired tongue images is ≥500, the tongue images are divided into a training set and a validation set according to the specified ratio of 9:1; If the number of acquired tongue images is less than 500, the tongue images are divided into a training set and a validation set according to a specified ratio of 7:3 or 6:4.

[0008] Furthermore, the tongue region division includes dividing the tongue region into a plurality of fixed-size partitions or a plurality of adaptive-size partitions.

[0009] Further, in step S1, the lower left corner of the circumscribed rectangle of the tongue image is the origin, the horizontal direction is the X-axis, and the vertical direction is the Y-axis; The coordinates of the lower left, upper left, lower right, and upper right of the rectangular image block of the tongue area are (t x , t y ), t x , t y +t h ), t x + t w , t y ), t x + t w , t y + t h ); The plurality of fixed-size partitions include a left tongue partition, a right tongue partition, a middle tongue partition and a tongue tip partition, and the left tongue partition, the right tongue partition, the middle tongue partition and the tongue tip partition are all square image blocks; The coordinates of the upper left corner vertex and the lower right corner vertex of the square image block of the tongue tip partition are respectively: ; The coordinates of the upper left corner vertex and the lower right corner vertex of the square image block of the left tongue partition are respectively: ; The coordinates of the upper left corner vertex and the lower right corner vertex of the square image block of the tongue right partition are respectively: ; The coordinates of the upper left corner vertex and the lower right corner vertex of the square image block of the tongue partition are respectively: ; Blk refers to a fixed-size pixel data block.

[0010] Further, the plurality of adaptive size partitions include a left tongue partition, a right tongue partition, a middle tongue partition and a tongue tip partition, and the left tongue partition, the right tongue partition, the middle tongue partition and the tongue tip partition are all rectangular image blocks; The coordinates of the upper left corner vertex and the lower right corner vertex of the rectangular image block of the tongue tip partition are respectively: ; The coordinates of the upper left corner vertex and the lower right corner vertex of the rectangular image block of the left tongue partition are respectively: ; The coordinates of the upper left corner vertex and the lower right corner vertex of the rectangular image block of the tongue right partition are respectively: ; The coordinates of the upper left corner vertex and the lower right corner vertex of the rectangular image block of the tongue partition are respectively: .

[0011] Further, in step S4, the texture features of the images of different partitions of the tongue region include contrast, correlation, dissimilarity, inverse disparity, angular second-order moment, and energy of different directions of different partitions of the tongue region; The texture features of the tongue region images in different regions are extracted through a gray level co-occurrence matrix.

[0012] Further, in step S5, the classifier is trained by the following steps: S51: Acquire several tongue images after the image texture features are extracted in step S4; S52: dividing the tongue images into a training set and a test set according to a specified ratio according to the number of tongue images after the image texture features are extracted, taking the training set as a training basis, and using a linear kernel function to train and test the classifier; If the number of tongue images obtained is ≥500, the tongue images are divided into a training set and a test set according to a specified ratio of 9:1; if the number of tongue images obtained is <500, the tongue images are divided into a training set and a test set according to a specified ratio of 7:3 or 6:4.

[0013] The present invention also provides a tongue image feature classification device for implementing the above-mentioned tongue image feature classification method, comprising: an acquisition unit, a segmentation unit, a division unit, an extraction unit and a classification unit; The acquisition unit includes a camera, a data transmission module, and a result output module; The segmentation unit includes an image recognition module, an image segmentation module, and a result output module; The division unit includes an image division module and a result output module; The extraction unit includes an image extraction module and a result output module; The classification unit includes an image classification module and a result output module.

[0014] Furthermore, the acquisition unit uses a camera to collect the tongue image to be classified; The segmentation unit performs image segmentation on the tongue image using a trained segmentation network to obtain a tongue region in the tongue image; The division unit divides the tongue region in the tongue image into a plurality of partitions of fixed size or adaptive size by using tongue diagnosis habits in medical practice; The extraction unit extracts texture features of different partition images of the tongue region based on a gray level co-occurrence matrix that reflects the roughness and distribution state of the image texture; The classification unit classifies the tongue image features according to the texture features of the images of different partitions of the tongue region using a trained classifier to obtain a classification result.

[0015] The present invention also provides a computer storage medium, comprising computer executable instructions stored therein, wherein the computer executable instructions are used to execute the above-mentioned tongue image feature classification method.

[0016] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects: 1. The present invention uses a camera to collect tongue images to be classified; uses a trained segmentation network to perform image segmentation on the tongue image to obtain the tongue area in the tongue image; uses the tongue diagnosis habits in medical practice to divide the tongue area in the tongue image into multiple fixed-size or adaptive-size partitions; the error of the area division of the tongue image is small and the accuracy is high.

[0017] 2. The present invention can classify tongue image features according to requirements by extracting image texture features by partitioning the tongue area in the tongue image. The tongue image feature classification method according to the present invention can provide objective, qualitative or quantitative classification information of tongue image features based on image processing technology without being restricted by various subjective factors, so as to better assist doctors in diagnosis and improve the accuracy of diagnosis.

[0018] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. In the entire drawings, the same reference symbols represent the same components; Figure 1 A flowchart of a tongue image feature classification method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of photographing a tongue image using a colorimetric card according to Embodiment 3 of the present invention; Figure 3 illustrative diagram of different directionality in the gray level co-occurrence matrix of embodiment 1 of the present invention; Figure 4 This is an example diagram of the gray-level co-occurrence matrix at 0° direction of Example 1 of the present invention; Figure 5 Schematic diagram of tongue region segmentation in a tongue image according to Embodiment 1 of the present invention; Figure 6 Schematic diagram of fixed-size partitions in the tongue region according to Example 1 of the present invention; Figure 7 Schematic diagram of adaptive size partitioning in the tongue area according to Example 1 of the present invention; Figure 8 This is a schematic diagram of the classifier accuracy corresponding to different partitions of the tongue area in Example 1 of the present invention; Fig. 9 is a block diagram of a tongue image feature classification device according to an embodiment of the present invention; Fig.10 Schematic diagram of the corresponding coordinate system when the tongue area is partitioned with a fixed size in Example 3 of the present invention.

[0020] Reference numerals: (t x , t y ) - the lower left coordinate of the rectangular image block of the tongue area in Example 1; (tx , t y + t h ) - the upper left coordinate of the rectangular image block of the tongue area in Example 1; (t x + t w , t y ) - the lower right coordinate of the rectangular image block of the tongue area in Example 1; (t x + t w , t y + t h )-the upper right coordinate of the rectangular image block of the tongue area in Example 1. DETAILED DESCRIPTION

[0021] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0022] The present invention provides a tongue image feature classification method, comprising the following steps: Step S1: obtaining a tongue image to be classified; Step S2: performing image segmentation on the tongue image using a trained segmentation network to obtain a tongue region in the tongue image; Step S3: dividing the tongue area in the tongue image into a plurality of partitions; Step S4: extracting texture features of different partition images of the tongue area; Step S5: classifying the tongue image using a trained classifier according to the texture features of the images of different partitions of the tongue area to obtain a tongue image classification result.

[0023] Specifically, in step S1, the tongue image to be classified can be collected by a camera; during the collection process of the tongue image, factors such as lighting, background, different tongue coating colors of the subjects, camera type, etc. will affect the collected tongue image characteristics, so a colorimetric card can be used to perform color correction on the tongue image collected by the camera.

[0024] Specifically, in step S2, the segmentation network is mainly trained through the following steps: Step 21: obtaining a plurality of tongue images, and marking the tongue region in each tongue image; Step 22: According to the number of tongue images obtained, the tongue images are divided into a training set and a validation set according to a specified ratio. The training set is used as the training basis, and the segmentation models pytorch segmentation framework is used to train multiple decoder automated segmentation networks.

[0025] It should be noted that the step S21 of marking the tongue area in each tongue image is to manually outline the tongue contour in each tongue image; It should be noted that in step S22, if the number of tongue images obtained is ≥500, the tongue images are divided into a training set and a validation set according to a specified ratio of 9:1; if the number of tongue images obtained is <500, the tongue images are divided into a training set and a validation set according to a specified ratio of 7:3 or 6:4; During the training process, the training set is used as the training basis, and the segmentation models pytorch segmentation framework is used to train multiple decoders to automatically segment the network. Under this segmentation framework, the segmentation network can be trained for multiple rounds through multiple decoders, and then the segmentation network obtained in each round of training is immediately verified through the verification set; during verification, the segmentation intersection-over-union ratio is used as the segmentation network performance evaluation parameter. If the segmentation intersection-over-union ratio meets the requirement for stopping iteration, the iteration is stopped, and the segmentation network obtained in this round of training is used as the first candidate segmentation network; if the segmentation intersection-over-union ratio does not meet the requirement for stopping iteration, the multiple decoders under the segmentation models pytorch segmentation framework automatically iterate and adjust the training parameters of the segmentation network obtained in this round of training, and then the obtained segmentation network is verified again, and the training and verification process is repeated until the segmentation intersection-over-union ratio of the obtained segmentation network meets the requirement for stopping iteration, and the segmentation network obtained at this time is used as the first candidate segmentation network; Raise the stop iteration requirement and repeat the training process. If after multiple iterations, the segmentation intersection-and-union ratio that meets the stop iteration requirement can still be obtained, the obtained segmentation network will be used as the second candidate segmentation network. Then raise the stop iteration requirement again and repeat the training process. After the Nth increase in the stop iteration requirement, after multiple iterations, the segmentation intersection-and-union ratio that meets the stop iteration requirement cannot be obtained, and the segmentation network cannot be successfully trained. The segmentation network that meets the N-1th stop iteration requirement is the final segmentation network. At this time, the decoder that can meet the N-1th stop iteration requirement among the multiple decoders under the segmentation models pytorch segmentation framework is the segmentation training model selected for training.

[0026] It should be noted that the above training method for the segmentation network is only an example. In practical applications, any training method suitable for the segmentation network of tongue images can be adopted, and there is no limitation here.

[0027] Specifically, in step S3, in the process of dividing the tongue area in the tongue image into multiple partitions, in an example, the tongue area can be divided into multiple fixed-size partitions; specifically, multiple fixed-size image areas in the tongue area can be extracted as individual partitions based on the vertex coordinates of the tongue area or the frame position of the tongue area; it should be noted that the sizes of the extracted partitions can be the same or different, and can have various shapes, such as rectangle, square, circle, ellipse, etc., which are not limited here.

[0028] For example, the lower left corner of the circumscribed rectangle of the original tongue image in step S1 is taken as the origin, the horizontal direction is taken as the X-axis, and the vertical direction is taken as the Y-axis. Then, after the image segmentation is performed by the segmentation network in step S2, the coordinates of the lower left, upper left, lower right, and upper right of the rectangular image block of the tongue area are respectively (t x , t y ), t x , t y + t h ), t x + t w , t y ), t x + t w , t y +t h ); Then the tongue area is divided into a left tongue partition, a right tongue partition, a middle tongue partition and a tongue tip partition; based on the tongue area range described in step S2, the left tongue partition, the right tongue partition, the middle tongue partition and the tongue tip partition are respectively represented as square image blocks extracted with a fixed size blk and having two points represented by corresponding coordinates as the upper left corner and the lower right corner, as shown in Figure 6 As shown; the fixed size blk refers to a pixel data block of a fixed size, and the value range is 0-∞; illustratively, blk=50, which means that each partition is a square of 50x50 pixels.

[0029] in: The coordinates of the upper left corner vertex and the lower right corner vertex of the square image block of the tongue tip partition are: ; The coordinates of the upper left corner vertex and the lower right corner vertex of the square image block of the left tongue partition are: ; The coordinates of the upper left corner vertex and the lower right corner vertex of the square image block of the tongue right partition are: ; The coordinates of the upper left corner vertex and the lower right corner vertex of the square image block in the tongue partition are: .

[0030] In the process of dividing the tongue area in the tongue image into multiple partitions, in another example, the tongue area can be divided into multiple partitions of adaptive sizes; specifically, based on the relationship between each preset partition and the tongue area size, multiple image areas of adaptive sizes in the tongue area can be extracted as each partition; it should be noted that the sizes of the extracted partitions can be the same or different, and can have various shapes, such as rectangle, square, circle, ellipse, etc., which are not limited here.

[0031] Exemplarily, the tongue area is divided into a left tongue partition, a right tongue partition, a middle tongue partition and a tongue tip partition; based on the tongue area range described in step S2, the left tongue partition, the right tongue partition, the middle tongue partition and the tongue tip partition are respectively represented as rectangular image blocks with two points represented by corresponding coordinates as the upper left corner and the lower right corner, such as Figure 7 As shown, where: The coordinates of the upper left corner vertex and the lower right corner vertex of the rectangular image block of the tongue tip partition are: ; The coordinates of the upper left corner vertex and the lower right corner vertex of the rectangular image block of the left tongue partition are: ; The coordinates of the upper left corner vertex and the lower right corner vertex of the rectangular image block of the tongue right partition are: ; The coordinates of the upper left corner vertex and the lower right corner vertex of the rectangular image block of the tongue partition are: .

[0032] The above-mentioned partitioning contents and specific partitioning methods for the tongue area are examples. In actual applications, any partitioning method for the tongue area can be adopted according to specific needs and application scenarios, and no limitation is made here.

[0033] Specifically, in step S4, when extracting the texture features of images of different partitions of the tongue area, it should be noted that the texture features of the surfaces of images of different partitions of the tongue area (image surface roughness, image texture distribution status, image uniformity) are all associated with the features of the tongue area image (tongue coating thickness, tongue coating color, tongue coating moisture, tongue coating greasy and rancid, complete and partial coating, and various physical constitution or diagnostic features corresponding to the tongue coating). Therefore, the image texture features of different partitions of the tongue area can be extracted based on the Gray Level Concurrence Matrix (GLCM) that reflects the image texture roughness and distribution status.

[0034] Taking into account the directionality of the grayscale co-occurrence matrix, the grayscale quantization level of the image, and the distance parameters between pixels, different types of image texture features in different directions in different partitions of the tongue region can be extracted based on the GLCM matrix. Optionally, one or more image texture features such as contrast, correlation, dissimilarity, inverse disparity, angular second-order moment, energy, etc. in different directions in different partitions of the tongue region can be extracted. It should be noted that the selection and application methods for the grayscale co-occurrence matrix and its corresponding image texture features are all examples. In actual applications, any suitable method and category of extracting image texture features can be used, and there is no limitation here.

[0035] Specifically, through the GLCM matrix, the contrast, correlation, dissimilarity, inverse difference, angular second-order moment and energy of different partition images of the tongue area can be extracted respectively. Contrast reflects the clarity of the image and the depth of the texture grooves. The deeper the texture grooves, the greater the contrast and the clearer the visual effect; conversely, if the contrast is small, the grooves are shallow and the image is blurred; in the GLCM matrix, the more pixel pairs with high contrast, the greater the contrast; that is, in the GLCM matrix, the greater the element value away from the diagonal, the greater the contrast.

[0036] Correlation is the degree of similarity of the elements of the GLCM matrix in the row or column direction of the metric space. The value of the correlation reflects the local grayscale correlation in the image; when the values ​​of the GLCM matrix elements are uniformly equal, the value of the correlation is large; on the contrary, if the values ​​of the GLCM matrix elements are very different, the value of the correlation is small. In an example, if there is a horizontal texture in the image, the value of the correlation of the GLCM matrix corresponding to the horizontal direction (such as 0°) will be greater than the value of the correlation of the GLCM matrix corresponding to the remaining directions (such as 90°).

[0037] Dissimilarity refers to the weights taken into account when calculating contrast. Dissimilarity is obtained by changing the contrast from exponential growth to linear growth with the distance of the GLCM matrix elements from the diagonal.

[0038] Homogeneity is a parameter used to measure the local homogeneity of an image. Contrary to the law of contrast or dissimilarity, the value of homogeneity is lower for non-homogeneous images and higher for homogeneous images.

[0039] Angular Second Moment (ASM) refers to the sum of squares of the grayscale co-occurrence matrix element values, reflecting the uniformity of the image grayscale distribution and the coarseness of the texture. If all the values ​​of the grayscale co-occurrence matrix are equal, the ASM value is small; on the contrary, if some values ​​of the grayscale co-occurrence matrix are large and other values ​​are small, the ASM value is large. When the elements in the co-occurrence matrix are concentrated, the ASM value is large. A large ASM value may indicate a more uniform and regularly changing image texture pattern.

[0040] Energy refers to the square root of the second-order angular moment. Energy can also reflect the uniformity of image grayscale distribution and texture coarseness.

[0041] Specifically, in step S5, the tongue image is classified according to the texture features of the images of different partitions of the tongue area using a classifier to obtain a tongue image classification result; the classifier is at least one of a support vector machine, a decision tree, a random forest, a logistic regression, an Adaboost, a perceptron, a K nearest neighbor, a naive Bayes, and a neural network; it should be noted that the above description and enumeration of the classifier are only examples. In actual applications, any corresponding classification model or classification method can be used to classify tongue image features, and there is no limitation here.

[0042] Specifically, the classifier is trained mainly through the following steps: S51: Acquire several tongue images after the image texture features are extracted in step S4; S52: Divide the tongue images into a training set and a test set according to a specified ratio according to the number of tongue images after the image texture features are extracted, use the training set as a training basis, and use a linear kernel function to train and test the classifier.

[0043] It should be noted that in step S52, if the number of tongue images obtained is ≥500, the tongue images are divided into a training set and a test set according to a specified ratio of 9:1; if the number of tongue images obtained is <500, the tongue images are divided into a training set and a test set according to a specified ratio of 7:3 or 6:4; During the training process, the classifier is trained for multiple rounds with the training set as the training basis, and then the classifier obtained in each round of training is immediately verified through the test set; during verification, the average accuracy of the test set (i.e., the average accuracy of the classifier) ​​is used as the classifier performance evaluation parameter. If the average accuracy of the classifier meets the requirements for stopping iterations, the iteration is stopped, and the classifier obtained in this round of training is used as the first candidate classifier; if the average accuracy of the classifier does not meet the requirements for stopping iterations, the training parameters of the classifier obtained in this round of training are iterated and adjusted, and then the obtained classifier is tested again, and the training and testing process is repeated until the average accuracy of the obtained classifier meets the requirements for stopping iterations, and the classifier obtained at this time is used as the first candidate classifier; The stopping iteration requirement is raised and the training process is repeated. If after multiple iterations, the average accuracy of the classifier that meets the stopping iteration requirement can still be obtained, the obtained classifier is used as the second candidate classifier. Then the stopping iteration requirement is raised again and the training process is repeated. After the stopping iteration requirement is raised for the Nth time, after multiple iterations, the average accuracy of the classifier that meets the stopping iteration requirement cannot be obtained, and the classifier cannot be successfully trained. The classifier that meets the stopping iteration requirement for the N-1th time is the final classifier.

[0044] It should be noted that the above-mentioned training method for the classifier is only an example. In practical applications, any training method for the classifier suitable for tongue images can be adopted, and there is no limitation here.

[0045] By using the trained classifier, the image texture features of different partitions extracted from the tongue image to be classified can be input into the classifier to classify the tongue image features, thereby obtaining the corresponding tongue image classification results.

[0046] Specifically, tongue image features mainly include the following categories, and the classifier can classify different types of tongue image features or their combinations, where: In one example, the tongue image feature includes tongue coating thickness, such as thick tongue coating, thin tongue coating; In yet another example, the tongue image features include tongue coating colors, such as white coating, yellow coating, gray coating, and black coating; In yet another example, the tongue image feature includes tongue coating moisture, such as moist tongue coating and dry tongue coating; In yet another example, the tongue image features include greasy tongue coating, such as greasy coating and rotten coating; In yet another example, the tongue image features include full and partial tongue coating, such as full tongue coating and partial tongue coating; In another example, the tongue image features include the constitution corresponding to the tongue coating, such as the peaceful constitution, qi deficiency constitution, yang deficiency constitution, yin deficiency constitution, phlegm-damp constitution, damp-heat constitution, blood stasis constitution, qi stagnation constitution, and special constitution in traditional Chinese medicine constitution; In another example, the tongue image features include corresponding clinical medical indicators or diagnoses obtained from the properties of the tongue coating. Specifically, the concept of traditional Chinese medicine that "the tongue texture can identify the deficiency and excess of the internal organs, and the tongue coating can detect the shallowness and depth of the six pathogens" can be used to obtain modern clinical laboratory indicators or clinical diagnoses that are associated with the internal organs, including: a) liver function indicators: alanine aminotransferase, aspartate aminotransferase, glutamyl transferase, the indicators can be embodied in numerical form, or in two categories (normal indicators, abnormal indicators); b) renal function indicators: blood creatinine, blood urea nitrogen, the indicators can be embodied in numerical form, or in two categories (normal indicators, abnormal indicators); c) blood lipids: triglycerides, cholesterol, the indicators can be embodied in numerical form, or in two categories (normal indicators, abnormal indicators); d) organ or metabolic disease status: chronic gastritis, hypertension, diabetes, etc., the indicators can be embodied in two categories (healthy, sick).

[0047] The above classification of tongue image features is only an example. In practical applications, any category of tongue image features associated with the properties of the tongue coating can be selected as needed, and there is no limitation here.

[0048] According to the tongue image feature classification method of the present invention, the tongue image features can be classified according to the requirements by extracting the image texture features of the tongue area in the tongue image. The tongue image feature classification method of the present invention can provide objective, qualitative or quantitative classification information of tongue image features based on image processing technology without being restricted by various subjective factors, so as to better assist doctors in diagnosis and improve the accuracy of diagnosis.

[0049] The present invention also provides a tongue image feature classification device for implementing the above tongue image feature classification method, comprising: an acquisition unit, a segmentation unit, a division unit, an extraction unit and a classification unit, such as Fig. 9 shown.

[0050] The acquisition unit includes a camera, a data transmission module, and a result output module; the result output module outputs the tongue image to be classified taken by the camera; The segmentation unit includes an image recognition module, an image segmentation module, and a result output module; the result output module outputs a tongue image with the tongue area segmented; The division unit includes an image division module and a result output module; the result output module outputs a tongue image divided into a plurality of tongue partitions of fixed size or adaptive size; The extraction unit includes an image extraction module and a result output module; the result output module outputs the texture features of the tongue image divided into a number of fixed-size or adaptive-size tongue partitions; The classification unit includes an image classification module and a result output module; the result output module outputs the tongue image classification result.

[0051] The acquisition unit uses a camera to collect a tongue image to be classified; the segmentation unit uses a trained segmentation network to perform image segmentation on the tongue image to obtain the tongue area in the tongue image; the division unit uses the tongue diagnosis habit in medical practice to divide the tongue area in the tongue image into a plurality of fixed-size or adaptive-size partitions; the extraction unit extracts texture features of different partition images of the tongue area based on a gray-level co-occurrence matrix (GLCM matrix) that reflects the roughness and distribution state of the image texture; the classification unit classifies the tongue image features according to the texture features of the different partition images of the tongue area using a trained classifier to obtain a classification result.

[0052] The present invention also provides a computer-readable storage medium, on which one or more computer program instructions are stored. The computer program instructions can implement the functions of each unit of the above-mentioned tongue image feature classification device and / or other desired functions, and can execute the above-mentioned tongue image feature classification method.

[0053] It should be understood that the term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information (such as computer readable instructions, data structures, program modules or other data). Computer storage media includes but is not limited to RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer.

[0054] Example 1 This embodiment provides a tongue image feature classification method for tongue coating thickness, comprising the following steps: Step S1: collecting 730 tongue images to be classified through a camera; Step S2: performing image segmentation on the tongue image using a trained segmentation network to obtain a tongue region in the tongue image; Specifically, the segmentation network is trained mainly through the following steps: S21: marking the tongue region in each tongue image acquired in step S1 (i.e. manually outlining the tongue contour in each tongue image); S22: Divide the tongue images into training set and validation set in a ratio of 9:1. Use the training set as the training basis and use the segmentation models pytorch segmentation framework to train multiple decoder automatic segmentation networks. Wherein, in step S22, the decoder is shown in Table 1 below.

[0055]

[0056] In this embodiment, the segmentation network can be trained for multiple rounds using the 5 network frameworks and 28 types of decoders shown in Table 1; First, the requirement for stopping iteration is set to 91% for the segmentation intersection-over-union ratio of the segmentation network (i.e., the IOU of the decoder). When the segmentation network is trained for multiple rounds using the 5 network frameworks and 28 types of decoders shown in Table 1, the segmentation network obtained in each round of training is immediately verified through the validation set. After multiple iterations, if the segmentation intersection-over-union ratio of the obtained segmentation network meets the requirement for stopping iteration, the segmentation network obtained at this time is used as the first candidate segmentation network. Then, the iteration stop requirement is set to the segmentation intersection-over-union ratio of the segmentation network (i.e., the IOU of the decoder) of 92%. After multiple iterations, if the segmentation intersection-over-union ratio of the obtained segmentation network meets the iteration stop requirement, the segmentation network obtained at this time is used as the second candidate segmentation network; The iteration stop requirement is set to 93% of the segmentation intersection-over-union ratio of the segmentation network (i.e., the IOU of the decoder). After multiple iterations, if the segmentation intersection-over-union ratio of the obtained segmentation network meets the iteration stop requirement, the segmentation network obtained at this time is used as the third candidate segmentation network. The iteration stop requirement is continued to be set to the segmentation intersection-over-union ratio of the segmentation network (i.e., the IOU of the decoder) of 94%. After many iterations, the segmentation intersection-over-union ratio that meets the iteration stop requirement cannot be obtained, and the segmentation network cannot be successfully trained.

[0057] The third candidate segmentation network is the final segmentation network. At this time, the decoder efficientnet7 that can meet the third stop iteration requirement (decoder IOU is 93%) among multiple decoders under the segmentation models pytorch segmentation framework is the selected segmentation training model for training.

[0058] Table 2 shows the IOU of the decoder in the tongue image segmentation network in this embodiment.

[0059]

[0060] Step S3: dividing the tongue area in the tongue image into a number of partitions; In this embodiment, the lower left corner of the circumscribed rectangle of the original tongue image in step S1 is taken as the origin, the horizontal direction is taken as the X axis, and the vertical direction is taken as the Y axis. The coordinates of the lower left, upper left, lower right, and upper right of the rectangular image block of the tongue area obtained after image segmentation by the segmentation network in step S2 are (t x , t y ), t x , t y+ t h ), t x + t w , t y ), t x + t w , t y + t h ); In this embodiment, the tongue area is divided into four fixed-size partitions, namely, a left tongue partition, a right tongue partition, a middle tongue partition, and a tongue tip partition; based on the tongue area range described in step S2, the left tongue partition, the right tongue partition, the middle tongue partition, and the tongue tip partition are respectively represented as square image blocks extracted with a fixed size blk, with two points represented by corresponding coordinates as the upper left corner and the lower right corner, as shown in FIG. Figure 6 As shown; among which: The coordinates of the upper left corner vertex and the lower right corner vertex of the square image block of the tongue tip partition are: ; The coordinates of the upper left corner vertex and the lower right corner vertex of the square image block of the left tongue partition are: ; The coordinates of the upper left corner vertex and the lower right corner vertex of the square image block of the tongue right partition are: ; The coordinates of the upper left corner vertex and the lower right corner vertex of the square image block in the tongue partition are: .

[0061] Take the tongue area division of a tongue image as an example, where blk=16: Tongue tip partition, the coordinates of the upper left vertex are (175.00, 216.56), and the coordinates of the lower right vertex are (191.00, 232.56); For the left tongue partition, the coordinates of the upper left vertex are (152.78, 197.89), and the coordinates of the lower right vertex are (168.78, 213.89); For the right tongue partition, the coordinates of the upper left vertex are (197.22, 216.56), and the coordinates of the lower right vertex are (213.22, 232.56); In the middle tongue partition, the coordinates of the upper left vertex are (175.00, 197.89), and the coordinates of the lower right vertex are (191.00, 213.89).

[0062] When blk=32: Tongue tip partition, the coordinates of the upper left vertex are (167.00, 208.56), and the coordinates of the lower right vertex are (199.00, 240.56); For the left tongue partition, the coordinates of the upper left vertex are (144.78, 189.89), and the coordinates of the lower right vertex are (176.78, 221.89); For the right tongue partition, the coordinates of the upper left vertex are (189.22, 208.56), and the coordinates of the lower right vertex are (221.22, 240.56); In the middle tongue partition, the coordinates of the upper left vertex are (167.00, 189.89), and the coordinates of the lower right vertex are (199.00, 221.89).

[0063] When blk=64: Tongue tip partition, the coordinates of the upper left vertex are (151.00, 192.56), and the coordinates of the lower right vertex are (215.00, 256.56); For the left tongue partition, the coordinates of the upper left vertex are (128.78, 173.89), and the coordinates of the lower right vertex are (192.78, 237.89); For the right tongue partition, the coordinates of the upper left vertex are (173.22, 192.56), and the coordinates of the lower right vertex are (237.22, 256.56); In the middle tongue partition, the coordinates of the upper left vertex are (151.00, 173.89), and the coordinates of the lower right vertex are (215.00, 237.89).

[0064] Step S4: extracting texture features of different partition images of the tongue area; After dividing the tongue area into multiple partitions, the texture features of the images of different partitions of the tongue area can be extracted based on the gray-level co-occurrence matrix (GLCM matrix) that reflects the roughness and distribution state of the image texture, including one or more image texture features such as contrast, correlation, dissimilarity, inverse disparity, angular second-order moment, energy, etc. in different directions in different partitions of the tongue area.

[0065] Using 239 tongue images (120 thick tongue coating images and 119 thin tongue coating images) annotated with tongue coating thickness as a tongue image feature, the discrimination of GLCM features was verified for different partitions of the tongue area. Among them, 192 parameter combinations can be formed based on the characteristic parameters of the image grayscale quantization level (6 types) and the distance parameters between two adjacent pixels when calculating GLCM (8 types). For each parameter combination, the scikit-image image processing algorithm library can be used to obtain 24-dimensional image texture feature extraction of six attributes (contrast, correlation, dissimilarity, inverse disparity, angular second-order moment, energy) in four directions (0°, 45°, 90°, 135°) that reflect the image texture features.

[0066] Figure 3 Schematic diagram of different directionality in the gray level co-occurrence matrix of this embodiment; Figure 4is the gray level co-occurrence matrix of the 0° direction in this embodiment; Figure 3 As shown, in the gray level co-occurrence matrix of this embodiment, the gray level co-occurrence matrix can be obtained at 0°, 45°, 90°, and 135° respectively, and the image texture features can be extracted accordingly. Figure 4 Take the gray level co-occurrence matrix of two adjacent points as an example, Figure 4 The values ​​of the GLCM matrix on the right reflect Figure 4 The number of combinations of grayscale values ​​of pixels horizontally adjacent to each other in the 0° direction in I on the left. For example, the value of GLCM (1, 1) is 1, which means Figure 4 There is a pair of pixels with grayscale 1 in I on the left side, which are horizontally adjacent to each other in the 0° direction; the value of GLCM (1, 2) is 2, indicating Figure 4 In the left I, there are two pairs of pixels with grayscales of 1 and 2, which are horizontally adjacent to each other in the 0° direction. It can be seen that the GLCM matrix represents different combinations of pixel grayscale values ​​in the corresponding directions. GLCM (1, 1) is a combination of pixel values ​​1 and 1, and its value corresponds to Figure 4 The number of combinations of pixel values ​​1 and 1 in I on the left; GLCM (4, 5) is the combination of pixel values ​​4 and 5, and its value corresponds to Figure 4 The number of combinations of pixel values ​​4 and 5 in I on the left.

[0067] Accordingly, the 24-dimensional image texture features of six attributes (contrast, correlation, dissimilarity, inverse disparity, angular second moment, and energy) of the above 239 tongue images in four directions (0°, 45°, 90°, and 135°) were extracted through the GLCM matrix.

[0068] Step S5: Based on the texture features of the images of different subareas of the tongue region extracted in step S4, the tongue images are classified using a trained classifier to obtain tongue image classification results.

[0069] Specifically, after extracting GLCM texture features from 239 annotated tongue image data (120 images with thick tongue coating and 119 images with thin tongue coating), these features can be used to perform qualitative analysis of tongue coating thickness based on the classic classifier Support Vector Machine (SVM), and a linear kernel function can be selected for classifier training and testing.

[0070] Specifically, the classifier is trained mainly through the following steps: S51: Acquire 239 tongue images after the GLCM image texture features are extracted in step S4; S52: Divide the tongue images into a training set and a test set according to a ratio of 9:1, First, the stopping iteration requirement is set to 60% of the average accuracy of the classifier. When the classifier is trained for multiple rounds, the classifier obtained in each round of training is verified by the test set. After multiple iterations, if the average accuracy of the classifier meets the stopping iteration requirement, the classifier obtained at this time is used as the first candidate classifier. Then, the stopping iteration requirement is set to the average accuracy of the segmentation network classifier of 70%. After multiple iterations, if the average accuracy of the obtained classifier meets the stopping iteration requirement, the classifier obtained at this time is used as the second candidate classifier; The stopping iteration requirement is continued to be set to the average accuracy of the classifier of the segmentation network is 80%. After many iterations, the average accuracy of the classifier that meets the stopping iteration requirement cannot be obtained, and the classifier cannot be successfully trained.

[0071] Then the second candidate classifier is the final classifier (support vector machine).

[0072] Figure 8 Schematic diagram of classifier accuracy corresponding to different tongue area partitions according to an embodiment of the present invention. Figure 8 The values ​​of the highest accuracy (upper curve), the lowest accuracy (lower curve), and the average accuracy (middle curve) of the classifier corresponding to different feature parameters (image grayscale quantization level g and distance parameter d between two adjacent pixels when calculating GLCM) of the four partitions of the left tongue partition, right tongue partition, tongue tip partition, and tongue middle partition are shown in the figure. The overall curve distribution results of the average classification accuracy (green curve) of the test set of 50 classifiers under the conditions of the partition of the specific tongue area and the image texture feature extraction parameters show that the average accuracy of the trained classifier is about 70%.

[0073] Example 2 This embodiment provides a tongue image feature classification method for tongue coating thickness (the same as that in Embodiment 1), and the steps are the same as those in Embodiment 1, except that in this embodiment, when the tongue image is divided into a plurality of partitions in step S3, the tongue area in the tongue image is divided into four adaptive-size partitions, namely, a left tongue partition, a right tongue partition, a middle tongue partition, and a tongue tip partition; The tongue left partition, tongue right partition, tongue middle partition and tongue tip partition are respectively represented as rectangular image blocks with two points represented by corresponding coordinates as the upper left corner and the lower right corner, such as Figure 7 As shown, where: The coordinates of the upper left corner vertex and the lower right corner vertex of the rectangular image block of the tongue tip partition are: ; The coordinates of the upper left corner vertex and the lower right corner vertex of the rectangular image block of the left tongue partition are: ; The coordinates of the upper left corner vertex and the lower right corner vertex of the rectangular image block of the tongue right partition are: ; The coordinates of the upper left corner vertex and the lower right corner vertex of the rectangular image block of the tongue partition are: .

[0074] Take the tongue area division of a tongue image as an example: Among them, for the tongue tip partition, the coordinates of the upper left vertex are (169.67, 218.33), and the coordinates of the lower right vertex are (196.33, 230.78); For the left tongue partition, the coordinates of the upper left vertex are (151.89, 193.44), and the coordinates of the lower right vertex are (169.67, 218.33); For the right partition of the tongue, the coordinates of the upper left vertex are (196.33, 193.44), and the coordinates of the lower right vertex are (214.11, 218.33); In the middle tongue area, the coordinates of the upper left vertex are (169.67, 187.22), and the coordinates of the lower right vertex are (196.33, 218.33).

[0075] This embodiment classifies the tongue coating thickness features of tongue images (730 images), and the average accuracy of the classifier is about 70%.

[0076] Example 3 This embodiment provides a tongue image for the thickness of the tongue coating ( Figure 2 ), using the segmentation network that has been trained in Example 1 to perform image segmentation on the tongue image to obtain the tongue area in the tongue image; Taking the lower left corner of the circumscribed rectangle of the original tongue image as the origin, the horizontal direction as the X-axis, and the vertical direction as the Y-axis, the coordinates of the lower left, upper left, lower right, and upper right of the rectangular image block of the tongue area obtained after image segmentation by the segmentation network are (t x , t y ), t x , t y + t h ), t x + t w , t y ), t x + t w , t y + t h ); In this embodiment, the tongue area is divided into four fixed-size partitions, namely, a left tongue partition, a right tongue partition, a middle tongue partition, and a tongue tip partition; based on the tongue area range described in step S2, the left tongue partition, the right tongue partition, the middle tongue partition, and the tongue tip partition are respectively represented as square image blocks extracted with a fixed size blk and having two points represented by corresponding coordinates as the upper left corner and the lower right corner, wherein: When blk=16: Tongue tip partition, the coordinates of the upper left vertex are (175.00, 216.56), and the coordinates of the lower right vertex are (191.00, 232.56); For the left tongue partition, the coordinates of the upper left vertex are (152.78, 197.89), and the coordinates of the lower right vertex are (168.78, 213.89); For the right tongue partition, the coordinates of the upper left vertex are (197.22, 216.56), and the coordinates of the lower right vertex are (213.22, 232.56); In the middle tongue partition, the coordinates of the upper left vertex are (175.00, 197.89), and the coordinates of the lower right vertex are (191.00, 213.89).

[0077] The corresponding coordinate system diagram is as follows Fig.10 shown.

[0078] Afterwards, the texture features of the images of different regions of the tongue area are extracted, and the tongue images are classified using the trained classifier to be classified into tongue coating thickness.

[0079] Example 4 This embodiment provides a tongue image feature classification device, including: an acquisition unit, a segmentation unit, a division unit, an extraction unit and a classification unit. Fig. 9 shown.

[0080] The acquisition unit includes a camera, a data transmission module, and a result output module; The segmentation unit includes an image recognition module, an image segmentation module, and a result output module; The division unit includes an image division module and a result output module; The extraction unit includes an image extraction module and a result output module; The classification unit includes an image classification module and a result output module.

[0081] Comparative Example 1 The tongue image feature classification method of this comparative example classifies the 730 tongue images obtained in Example 1. The difference is that when the tongue image is segmented, the Unet segmentation network that has been trained in the existing method is used to perform regional segmentation to obtain tongue root information, tongue middle information, tongue tip information, and tongue edge information. The Unet segmentation network uses full-scale skip connections, and each layer of the Unet segmentation network is provided with Convolution kernel; the remaining steps are the same as those in Example 1.

[0082] This comparative example classifies the tongue images (730 images) features of tongue coating thickness, and the average accuracy of the classifier is about 40%.

[0083] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0084] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A tongue image feature classification method, characterized in that: The following steps are involved: Step S1: obtaining a tongue image to be classified; Step S2: performing image segmentation on the tongue image using a trained segmentation network to obtain a tongue region in the tongue image; Step S3: dividing the tongue area in the tongue image into a plurality of partitions; Step S4: extracting texture features of different partition images of the tongue area; Step S5: classifying the tongue image using a trained classifier according to the texture features of the images of different partitions of the tongue area to obtain a tongue image classification result.

2. The tongue image feature classification method according to claim 1, characterized in that: In step S2, the segmentation network is trained by the following steps: Step 21: obtaining a plurality of tongue images, and marking the tongue region in each tongue image; Step 22: Divide the tongue images into a training set and a validation set according to the specified ratio according to the number of tongue images obtained, use the training set as the training basis, and use the segmentation models pytorch segmentation framework to train multiple decoder automatic segmentation networks; If the number of acquired tongue images is ≥500, the tongue images are divided into a training set and a validation set according to the specified ratio of 9:1; If the number of acquired tongue images is less than 500, the tongue images are divided into a training set and a validation set according to a specified ratio of 7:3 or 6:

4.

3. The tongue image feature classification method according to claim 2, characterized in that: The tongue region division includes dividing the tongue region into a plurality of fixed-size partitions or a plurality of adaptive-size partitions.

4. The tongue image feature classification method according to claim 3, characterized in that: In step S1, the lower left corner of the circumscribed rectangle of the tongue image is taken as the origin, the horizontal direction is taken as the X-axis, and the vertical direction is taken as the Y-axis; The coordinates of the lower left, upper left, lower right, and upper right of the rectangular image block of the tongue area are (t x , t y ), t x , t y + t h ), t x + t w , t y ), t x + t w , t y + t h ); The plurality of fixed-size partitions include a left tongue partition, a right tongue partition, a middle tongue partition and a tongue tip partition, and the left tongue partition, the right tongue partition, the middle tongue partition and the tongue tip partition are all square image blocks; The coordinates of the upper left corner vertex and the lower right corner vertex of the square image block of the tongue tip partition are respectively: ； The coordinates of the upper left corner vertex and the lower right corner vertex of the square image block of the left tongue partition are respectively: ； The coordinates of the upper left corner vertex and the lower right corner vertex of the square image block of the tongue right partition are respectively: ； The coordinates of the upper left corner vertex and the lower right corner vertex of the square image block of the tongue partition are respectively: ； Blk refers to a fixed-size pixel data block.

5. The tongue image feature classification method according to claim 4, characterized in that: The plurality of adaptive size partitions include a left tongue partition, a right tongue partition, a middle tongue partition and a tongue tip partition, and the left tongue partition, the right tongue partition, the middle tongue partition and the tongue tip partition are all rectangular image blocks; The coordinates of the upper left corner vertex and the lower right corner vertex of the rectangular image block of the tongue tip partition are respectively: ； The coordinates of the upper left corner vertex and the lower right corner vertex of the rectangular image block of the left tongue partition are respectively: ； The coordinates of the upper left corner vertex and the lower right corner vertex of the rectangular image block of the tongue right partition are respectively: ； The coordinates of the upper left corner vertex and the lower right corner vertex of the rectangular image block of the tongue partition are respectively: 。 6. The tongue image feature classification method according to claim 5, characterized in that: In step S4, the texture features of the images of different partitions of the tongue region include contrast, correlation, dissimilarity, inverse disparity, angular second-order moment, and energy of different directions of different partitions of the tongue region; The texture features of the tongue region images in different regions are extracted through a gray level co-occurrence matrix.

7. The tongue image feature classification method according to claim 6, characterized in that: In step S5, the classifier is trained by the following steps: S51: Acquire several tongue images after the image texture features are extracted in step S4; S52: dividing the tongue images into a training set and a test set according to a specified ratio according to the number of tongue images after the image texture features are extracted, taking the training set as a training basis, and using a linear kernel function to train and test the classifier; If the number of tongue images obtained is ≥500, the tongue images are divided into a training set and a test set according to a specified ratio of 9:1; if the number of tongue images obtained is <500, the tongue images are divided into a training set and a test set according to a specified ratio of 7:3 or 6:

4.

8. A tongue image feature classification device, used to implement the tongue image feature classification method according to any one of claims 1 to 7, characterized in that: include: Acquisition unit, segmentation unit, division unit, extraction unit and classification unit; The acquisition unit includes a camera, a data transmission module, and a result output module; The segmentation unit includes an image recognition module, an image segmentation module, and a result output module; The division unit includes an image division module and a result output module; The extraction unit includes an image extraction module and a result output module; The classification unit includes an image classification module and a result output module.

9. The tongue image feature classification device according to claim 8, characterized in that: The acquisition unit uses a camera to collect the tongue image to be classified; The segmentation unit uses a trained segmentation network to perform image segmentation on the tongue image to obtain a tongue region in the tongue image; The division unit divides the tongue region in the tongue image into a plurality of partitions of fixed size or adaptive size by using tongue diagnosis habits in medical practice; The extraction unit extracts texture features of different partition images of the tongue region based on a gray level co-occurrence matrix that reflects the roughness and distribution state of the image texture; The classification unit classifies the tongue image features according to the texture features of the images of different partitions of the tongue region using a trained classifier to obtain a classification result.

10. A computer storage medium, characterized in that: The method comprises storing computer executable instructions, wherein the computer executable instructions are used to execute the tongue image feature classification method according to any one of claims 1 to 7.

Images