Tongue feature analysis method and device, electronic equipment and storage medium

Abstract

The invention relates to a tongue feature analysis method and device, electronic equipment and a storage medium. A trained tongue detection model and a trained tongue segmentation model are fused to obtain a target segmentation model, an image data set is segmented through the target segmentation model to obtain a tongue segmentation image data set, and the tongue segmentation image data set is labeled to obtain a tongue feature classification data set. Training a tongue feature classification model based on the tongue feature classification data set to obtain a trained tongue feature classification model, obtaining to-be-detected target image data, inputting the target image data into a target segmentation model to obtain tongue segmentation image data of the to-be-detected target image data, and obtaining the tongue segmentation image data of the to-be-detected target image data. According to the method, the trained tongue feature classification model is used for performing feature analysis on the tongue segmentation image data to obtain a tongue feature analysis result, so that the training efficiency of the model is improved, and the classification stability and reliability are also remarkably improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of feature analysis, and in particular to a tongue feature analysis method, device, electronic device and storage medium. Background Art

[0002] In the field of TCM diagnosis and treatment, observation and analysis of the tongue is an important means of diagnosing diseases. The tongue contains many characteristics such as tongue coating and tongue quality, which can reflect the health status of the body. Traditional tongue analysis mainly relies on the doctor's experience, observing the tongue with the naked eye and making a diagnosis based on experience. However, with the rapid development of computer vision and artificial intelligence technology, tongue analysis has gradually developed from manual diagnosis to automation and intelligence, among which deep learning technology provides strong support for tongue feature analysis.

[0003] At present, tongue feature analysis mainly adopts the following implementation schemes: 1) Traditional machine learning combined with feature engineering methods, through manually designed feature extraction of tongue color, texture and other features, and then input these features into the machine learning classifier for classification; 2) Tongue segmentation and classification model based on deep learning, using convolutional neural network (CNN) to segment tongue images, and combining classification models to classify tongue quality, tongue coating, tongue color and other features. In order to improve classification accuracy, this type of method usually uses an independent classification model for each feature, for example, one model is used for tongue quality classification and another model is used for tongue coating classification; 3) Using improved segmentation algorithm and transfer learning model, first extract the tongue area of ​​the tongue image through the segmentation model, and then use the pre-trained classification model to classify the tongue features.

[0004] Although the above three solutions can achieve the segmentation of tongue area and its feature classification functions to a certain extent, they also have some defects: 1) Solution 1 relies on manually designed feature extraction, such as low-level features such as color and texture. Usually, these features are defined by experts based on experience, and it is difficult to fully and deeply reflect the complex characteristics of the tongue. Therefore, the manually extracted features cannot fully capture the deep information in the tongue image, resulting in limited classification effect and insufficient generalization ability, resulting in poor performance when dealing with different tongue characteristics and individual differences.

[0005] 2) Solution 2 relies on manual separation of tongue coating and tongue quality, and the annotation quality directly affects the performance of the model; secondly, since multiple independent classification models are required, the complexity of the system increases, the coordination and information sharing between models are insufficient, and the detailed information in the tongue image is easily lost, resulting in limited classification effect.

[0006] 3) The effect of the transfer learning of Scheme 3 depends largely on the original data characteristics of the pre-trained model. If the pre-training data is significantly different from the data in the actual application scenario, the performance of the model will be greatly affected. The generalization ability and robustness of the transfer learning model are still insufficient in some complex tasks, especially in the subtle classification of tongue features. Summary of the invention

[0007] In order to solve the above technical problems, the present disclosure provides a tongue feature analysis method, device, electronic device and storage medium.

[0008] In a first aspect, an embodiment of the present disclosure provides a tongue feature analysis method, the method comprising: constructing an image data set, wherein the image data set includes a large amount of image data including tongue images; Based on the image data set, the tongue detection model and the tongue segmentation model are trained respectively to obtain a trained tongue detection model and a trained tongue segmentation model; The trained tongue detection model and the trained tongue segmentation model are integrated to obtain a target segmentation model, and the image data set is segmented by the target segmentation model to obtain a tongue segmentation image data set; Annotating the tongue segmentation image dataset to obtain a tongue feature classification dataset, and training a tongue feature classification model based on the tongue feature classification dataset to obtain a trained tongue feature classification model; Acquire target image data to be detected, input the target image data into the target segmentation model, and obtain tongue segmentation image data of the target image data to be detected; The tongue feature analysis is performed on the tongue segmentation image data using a trained tongue feature classification model to obtain a tongue feature analysis result.

[0009] In a second aspect, an embodiment of the present disclosure provides a tongue feature analysis device, the device comprising: A construction unit, used for constructing an image data set, wherein the image data set includes a large amount of image data including tongue images; A first training unit, configured to train the tongue detection model and the tongue segmentation model based on the image data set to obtain a trained tongue detection model and a trained tongue segmentation model; A first obtaining unit is used to fuse the trained tongue detection model and the trained tongue segmentation model to obtain a target segmentation model, and to perform segmentation processing on the image data set by using the target segmentation model to obtain a tongue segmentation image data set; A second training unit is used to annotate the tongue segmentation image data set to obtain a tongue feature classification data set, and train a tongue feature classification model based on the tongue feature classification data set to obtain a trained tongue feature classification model; A second obtaining unit is used to obtain target image data to be detected, input the target image data into the target segmentation model, and obtain tongue segmentation image data of the target image data to be detected; The feature analysis unit is used to perform feature analysis on the tongue segmentation image data through a trained tongue feature classification model to obtain a tongue feature analysis result.

[0010] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: Memory; Processor; and Computer programs; The computer program is stored in the memory and is configured to be executed by the processor to implement the method as described in the first aspect.

[0011] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the method as described in the first aspect.

[0012] In a fifth aspect, an embodiment of the present disclosure further provides a computer program product, which includes a computer program or instructions, and when the computer program or instructions are executed by a processor, the method described in the first aspect is implemented.

[0013] The tongue feature analysis method, device, electronic device and storage medium provided by the embodiment of the present disclosure, by constructing an image data set, the image data set includes a large amount of image data containing tongue images, respectively training a tongue detection model and a tongue segmentation model based on the image data set to obtain a trained tongue detection model and a trained tongue segmentation model, fusing the trained tongue detection model and the trained tongue segmentation model to obtain a target segmentation model, performing segmentation processing on the image data set by the target segmentation model to obtain a tongue segmentation image data set, annotating the tongue segmentation image data set to obtain a tongue feature classification data set, training a tongue feature classification model based on the tongue feature classification data set to obtain a trained tongue feature classification model, acquiring target image data to be detected, inputting the target image data into the target segmentation model to obtain tongue segmentation image data of the target image data to be detected, and performing feature analysis on the tongue segmentation image data by the trained tongue feature classification model to obtain a tongue feature analysis result. Compared with the prior art, the embodiment of the present disclosure can realize feature sharing and information complementation, thereby improving the accuracy and robustness of feature classification. By directly training on the complete tongue segmentation image, the information loss that may occur in the process of separating the tongue coating and the tongue quality is effectively avoided. The model can more comprehensively learn the complex relationship and detailed information between tongue color, tongue quality, coating color and other features, ensuring that the classification results are more accurate and objective, reducing dependence on manual labeling and erroneous information in separating the tongue coating, which not only improves the training efficiency of the model, but also significantly improves the stability and reliability of the classification.

[0014] The advantages of the present disclosure are derived as follows: Compared with the method of designing a classification model for each feature separately in traditional deep learning, the present disclosure overcomes the problem of information sharing difficulty through a feature sharing strategy. The tongue feature classification model designed in the present disclosure first extracts multi-level features, and merges the features of different levels together through feature splicing operations as the input of each classification task. This feature sharing method enables multiple classification tasks to maintain their respective task independence while using the same set of basic features. Under this information sharing mechanism, different classification tasks can benefit from each other's features, thereby utilizing more comprehensive feature information during classification. Especially in the case of overlap and similarity between tasks (such as tongue coating and tongue quality), this method can better retain the potential connection between different tasks. In addition, by designing an independent classification head for each task, the model can perform fine learning of specific features for each task to ensure the classification accuracy of each task. Therefore, through the combination of the underlying feature extraction network and the information sharing module, the present disclosure realizes information sharing between multiple tasks, reduces the common information loss problem in independent model design, and ensures the integrity and accuracy of the classification results. At the same time, on the basis of realizing information sharing, the present disclosure further reduces redundant calculations, improves the operating efficiency of the model, and makes the training process faster and more efficient.

[0015] The classification methods in traditional machine learning and deep learning usually rely on the image data that is manually annotated after the tongue coating and tongue quality are separated to train and test the model. However, the separation process of the tongue coating and the quality of subsequent manual annotation directly affect the performance of the model. If the separation is inaccurate or the annotation is incorrect, it will lead to the loss of information or the lack of details. Moreover, there is a large similarity between the tongue coating and the tongue quality in color and texture, and it is difficult to have an objective ground truth, so it is very difficult to separate the tongue coating and the tongue quality by manual annotation. The present disclosure abandons this method that needs to rely on tongue coating separation, adopts an automated interactive segmentation strategy, does not perform tongue coating separation, and directly uses large-scale complete tongue segmentation images for training, thereby reducing the dependence on manual annotation and erroneous information in the tongue coating separation process. The tongue feature classification model can learn more complete feature information from large-scale complete tongue segmentation images, including subtle changes in tongue coating and tongue quality, thereby enhancing the accuracy and stability of classification.

[0016] Traditional segmentation methods in machine learning and deep learning usually use raw images for processing. However, raw images often contain a lot of complex background information. Especially when the image size is large, these background noises can easily interfere with the segmentation model, thereby reducing the segmentation accuracy and efficiency of the model. In addition, large-size images will significantly increase the consumption of computing resources during processing, resulting in higher memory requirements and processing time. Especially in applications with high-resolution images, memory overflow is more likely to occur, thus affecting the deployment and use of the model. In contrast, the interactive segmentation strategy accurately locates the tongue area through the tongue detection model, generates a bounding box, and then crops the tongue. This not only effectively filters out the complex background noise in the original image, but also ensures the accuracy of the segmented area. The cropped image size is smaller, which reduces the computational complexity while improving the processing speed, and ensures that the model uses high-quality tongue segmentation image data when performing feature extraction and classification. This method provides clean and accurate data input for subsequent classification and analysis, greatly improves the overall performance of tongue diagnosis and analysis, and also shows higher device adaptability, which is particularly suitable for application scenarios that are sensitive to computing resources, such as mobile terminals or embedded devices.

[0017] Traditional tongue diagnosis relies on the experience of doctors for manual observation, which is easily affected by subjective factors, resulting in poor repeatability and consistency of results. The present disclosure uses a multi-dimensional tongue feature analysis and statistical quantification method that includes qualitative and quantitative results to objectively and quantitatively analyze key features such as tongue color and fur color. Combining RGB and Lab color space conversion, channel analysis, and KS test algorithm, the color distribution in the tongue image can be deeply quantified, and the difference analysis between the distribution of each channel of the test sample and the standard distribution can be analyzed to provide doctors with quantitative tongue color and fur color analysis results. This quantitative analysis method not only improves the scientificity and objectivity of tongue diagnosis, but also provides standardized data references for clinical diagnosis, reduces errors caused by human intervention, and improves the accuracy and repeatability of tongue diagnosis analysis. The present disclosure realizes a fully automated process for tongue feature analysis, from image acquisition, tongue detection, tongue segmentation, tongue classification to statistical analysis, eliminates human intervention, reduces subjective errors, and greatly improves the efficiency and accuracy of intelligent tongue diagnosis. Compared with traditional methods, the present disclosure ensures the consistency of each analysis result through a standardized automated process, is suitable for large-scale data processing and analysis, and has higher reliability and practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A flow chart of a tongue feature analysis method provided in an embodiment of the present disclosure; Figure 2 A schematic diagram of the overall architecture of the tongue feature analysis method provided in an embodiment of the present disclosure; Figure 3 A flow chart of a tongue feature analysis method provided by another embodiment of the present disclosure; Figure 4 A flow chart of a tongue feature analysis method provided by another embodiment of the present disclosure; Figure 5 A diagram of the tongue detection model architecture provided in an embodiment of the present disclosure; Figure 6 A diagram of the tongue segmentation model architecture provided in an embodiment of the present disclosure; Figure 7 A diagram of the tongue feature classification model architecture provided in an embodiment of the present disclosure; Figure 8 A schematic diagram of the workflow of the tongue feature analysis method for interactive segmentation and fusion of a large model provided in an embodiment of the present disclosure; Fig. 9 A schematic diagram of the workflow of the interactive segmentation module provided in the embodiment of the present disclosure; Fig.10 A schematic diagram of the workflow of the statistical analysis module provided in the embodiment of the present disclosure; Fig.11 A schematic diagram of the structure of a tongue feature analysis device provided in an embodiment of the present disclosure; Fig.12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0022] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0023] All actions to obtain information or data in this application are carried out in compliance with the relevant data protection laws and policies of the country where the information or data is located, and with the authorization of the owner of the relevant information or data.

[0024] To address this problem, an embodiment of the present disclosure provides a tongue feature analysis method, which is introduced below in conjunction with a specific embodiment.

[0025] Figure 1 A flow chart of the tongue feature analysis method provided in the embodiment of the present disclosure. The execution subject of the method is an electronic device, which can be a tongue feature analysis device, specifically a tablet computer, a laptop computer; it can also be a personal computer and other devices. The method can be applied to the scenario of analyzing tongue features, can improve the accuracy and efficiency of tongue feature classification, reduce the dependence on manual labeling, and solve the problems of information loss and low efficiency in the classification and segmentation process of traditional methods. It can be understood that the tongue feature analysis method provided in the embodiment of the present disclosure can also be applied in other scenarios.

[0026] Below Figure 1 The tongue feature analysis method shown in the figure is introduced, and the specific steps of the method are as follows: S101. Construct an image dataset, where the image dataset includes a large amount of image data including tongue images.

[0027] like Figure 2 As shown, the user will perform image acquisition, and the electronic device will construct an image data set in response to the user's image acquisition operation, wherein the image data set includes a large amount of image data containing tongue images. Constructing an image data set is the basis for model training. Through systematic and standardized image data acquisition, annotation, and data set production, it is ensured that the model is trained with the support of diverse and high-quality data, thereby providing a solid foundation for subsequent tongue detection models and tongue segmentation models, and can effectively improve the performance and reliability of the model.

[0028] In some embodiments, S101 may include but is not limited to S1011, S1012, S1013, S1014, S1015: S1011. Collect a large amount of image data containing tongue images.

[0029] S1012: Perform color calibration processing on each piece of the image data to obtain each piece of color-calibrated image data.

[0030] In this step, if Figure 8As shown, the electronic device can perform color calibration processing on each of the image data to obtain each color-calibrated image data. Optionally, the color-calibrated image data can be collected by a high-resolution camera with color calibration. Avoid color deviation problems caused by differences in the characteristics of the shooting equipment or changes in ambient lighting conditions. The color-calibrated tongue image serves as the basic data source of the model, providing reliable data support for subsequent analysis, segmentation and classification of tongue features, thereby ensuring the reliability and consistency of the tongue feature analysis results.

[0031] Tongue images of people with different health conditions were collected, and their age composition, gender ratio, and health conditions are shown in Table 1.

[0032] Table 1 Statistics of health status of collectors in different age groups

[0033] Image acquisition not only requires the support of high-resolution camera equipment, but also needs to consider the following key factors: 1) The collected data must cover the tongue characteristics of different individuals and the tongue performance in different health states. Diversified sample data can ensure that the model can cope with various complex tongue image scenarios during training and inference, and improve the generalization ability of the model.

[0034] 2) To ensure data consistency, images need to be collected under standardized conditions, including multiple constant light intensities, fixed shooting distances, etc. This can minimize the impact of external environmental factors on image color and details.

[0035] 3) During the image acquisition process, use a standard colorimetric card to perform image calibration at each shot to correct the color deviation of the camera device. This step can ensure that the color information of the tongue image remains consistent under different devices and different shooting environments.

[0036] The specific sample image acquisition strategy is shown in Table 2.

[0037] Table 2 Sample image acquisition strategy table

[0038] Furthermore, after data cleaning and screening, images with poor quality due to factors such as blurred imaging, overexposure or underexposure were eliminated.

[0039] S1013, cropping each color-calibrated image data to obtain each tongue region image data.

[0040] The color-calibrated image data is cropped. A rectangular frame with a length and width approximately twice that of the tongue area is used to completely surround the tongue area. This cropping method can effectively retain the detailed information around the tongue and provide high-quality training data for the subsequent tongue segmentation model. The cropped image will be used as the input data source for the tongue segmentation model to improve the accuracy and robustness of the model in actual application scenarios.

[0041] S1014. Annotate each color-calibrated image data, annotate the tongue boundary box of each image, obtain a tongue detection data set, and divide the tongue detection data set into a training set, a validation set, and a test set of the tongue detection model based on a preset first ratio.

[0042] After collecting the image data, annotate the collected image data. Use Labelme for annotation. In the specific operation, import these color-calibrated image data into Labelme, use the rectangle tool to annotate the color-calibrated image data, annotate the tongue bounding box of each image, and obtain the tongue detection data set. A stratified sampling strategy is used to ensure that samples of different categories are evenly distributed in each data set, thereby avoiding the problem of class imbalance in the model during training. The tongue detection data set is divided into a training set, a validation set, and a test set in a ratio of 8:1:1. Using a stratified sampling strategy, samples are randomly assigned to the training set, validation set, and test set while maintaining a balanced distribution of different health states, age groups, and genders. This ensures that the model fully learns the characteristics of samples of different categories during training and improves the generalization ability of the model in practical applications.

[0043] S1015. Label each of the tongue area image data, mark the tongue area segmentation frame of each image, obtain a tongue segmentation data set, and divide the tongue segmentation data set into a training set, a validation set, and a test set of the tongue segmentation model based on a preset second ratio.

[0044] After collecting the image data, annotate the collected image data. Use Labelme for annotation. In the specific operation, the tongue bounding box of each image is annotated to obtain the tongue detection dataset, which is imported into Labelme. The cropped tongue area image is annotated using the polygon tool to annotate the tongue area segmentation box of each image to obtain the tongue segmentation dataset. A stratified sampling strategy is used during division to divide the tongue segmentation dataset into training set, validation set, and test set in a ratio of 70:15:15, ensuring that samples of different categories in each dataset are evenly distributed in each dataset to avoid the impact of category imbalance on model performance. Compared with the tongue detection dataset, the verification and test ratios of the tongue segmentation dataset are higher in order to better evaluate the segmentation performance of the model.

[0045] It should be noted that the preset first ratio and the preset second ratio may also be other ratio values, and the embodiments of the present disclosure are only for the convenience of explanation and are not specifically limited.

[0046] S102. Train the tongue detection model and the tongue segmentation model based on the image data set to obtain a trained tongue detection model and a trained tongue segmentation model.

[0047] In this step, if Figure 2 As shown, after constructing the image dataset, the image dataset is used for model training to obtain a trained tongue detection model and a trained tongue segmentation model. Optionally, the image dataset includes a training set, a test set, and a validation set. During the training process, the annotation information of the training set will guide the model to learn how to accurately complete the tongue detection and tongue segmentation tasks. The data of the validation set is used to adjust the hyperparameters of the model to ensure the performance of the model outside the training set and avoid overfitting. After the model training is completed, a performance evaluation is performed based on the test set to confirm the actual detection capability and robustness of the model.

[0048] S103, fusing the trained tongue detection model and the trained tongue segmentation model to obtain a target segmentation model, and performing segmentation processing on the image dataset using the target segmentation model to obtain a tongue segmentation image dataset.

[0049] In this step, the electronic device fuses the trained tongue detection model and the trained tongue segmentation model to obtain a target segmentation model, inputs the image data set into the target segmentation model, and finally generates a high-quality tongue segmentation image data set. In some embodiments, the trained tongue detection model and the trained tongue segmentation model can be fused to construct a unified interactive segmentation module. The core goal of the interactive segmentation module is to quickly locate the approximate area of ​​the tongue through the tongue detection model, and then finely segment the detected tongue area through the tongue segmentation model, thereby improving the segmentation accuracy and efficiency. Fig. 9 As shown in the figure, the specific process of the interactive segmentation module is as follows: 1) First, use the trained tongue detection model to identify the approximate position of the tongue in the input image, and output a bounding box (Bounding Box) containing the tongue area; 2) The tongue area bounding box output by the tongue detection model is extended, and the length and width of the tongue area are enlarged to twice the original bounding box (if the bounding box exceeds the image boundary after enlargement, it is taken as the image boundary value), forming a cropping frame containing the tongue area; 3) The original image is cropped according to the cropping frame, and the area containing the tongue is extracted separately; 4) The cropped tongue area image is input into the tongue segmentation model, and then fine segmentation at the pixel level is performed to output a high-precision tongue segmentation image containing only the tongue.

[0050] S104. Annotate the tongue segmentation image dataset to obtain a tongue feature classification dataset, and train a tongue feature classification model based on the tongue feature classification dataset to obtain a trained tongue feature classification model.

[0051] In this step, after the tongue segmentation image dataset is generated, classification and annotation are performed to obtain a tongue feature classification dataset. Further, a tongue feature classification model is trained based on the tongue feature classification dataset to obtain a trained tongue feature classification model.

[0052] In some embodiments, the tongue feature classification data set can be divided into a training set, a validation set, and a test set of the tongue feature classification model. The specific division strategy is as follows: 1) The data set is divided into a training set, a validation set, and a test set in a ratio of 75:15:10, that is, 75% of the data is used for model training, 15% of the data is used for model verification, and 10% of the data is used for model testing; 2) In order to ensure that the distribution of tongue feature categories in each data set is relatively balanced, a stratified sampling strategy needs to be adopted during division to ensure that the sample ratios of each tongue feature category in each data set are as consistent as possible to avoid the impact of category imbalance on model performance.

[0053] S105 , obtaining target image data to be detected, and inputting the target image data into the target segmentation model to obtain tongue segmentation image data of the target image data to be detected.

[0054] In this step, the electronic device acquires the target image data to be detected, which includes the tongue image, and inputs the target image data into the target segmentation model to obtain the tongue segmentation image data of the target image data to be detected.

[0055] S106. Perform feature analysis on the tongue segmentation image data using a trained tongue feature classification model to obtain a tongue feature analysis result.

[0056] In this step, after obtaining the tongue segmentation image data of the target image data to be detected, the tongue segmentation image data is input into the trained tongue feature classification model to obtain the tongue feature analysis result. Figure 8 As shown, the tongue feature analysis results include but are not limited to tongue color prediction results, tongue coating color prediction results, tongue quality prediction results, tongue coating prediction results, and overall prediction results.

[0057] The disclosed embodiment constructs an image data set, the image data set includes a large number of image data containing tongue images, trains a tongue detection model and a tongue segmentation model based on the image data set, obtains a trained tongue detection model and a trained tongue segmentation model, fuses the trained tongue detection model and the trained tongue segmentation model, obtains a target segmentation model, performs segmentation processing on the image data set through the target segmentation model, obtains a tongue segmentation image data set, annotates the tongue segmentation image data set, obtains a tongue feature classification data set, trains a tongue feature classification model based on the tongue feature classification data set, obtains a trained tongue feature classification model, obtains target image data to be detected, inputs the target image data into the target segmentation model, obtains tongue segmentation image data of the target image data to be detected, performs feature analysis on the tongue segmentation image data through the trained tongue feature classification model, and obtains a tongue feature analysis result. Compared with the prior art, the disclosed embodiment can realize feature sharing and information complementarity, thereby improving the accuracy and robustness of feature classification. By directly training on the complete tongue segmentation image, the information loss that may occur in the process of separating the tongue coating and the tongue quality is effectively avoided. The model can more comprehensively learn the complex relationship and detailed information between tongue color, tongue quality, coating color and other features, ensuring that the classification results are more accurate and objective, reducing dependence on manual labeling and erroneous information in separating the tongue coating, which not only improves the training efficiency of the model, but also significantly improves the stability and reliability of the classification.

[0058] Figure 3A flow chart of a tongue feature analysis method provided by another embodiment of the present disclosure is as follows: Figure 3 As shown, the method includes the following steps: S201. Construct an image dataset, where the image dataset includes a large amount of image data including tongue images.

[0059] Specifically, the implementation process and principle of S201 and S101 are the same, and will not be repeated here.

[0060] S202: construct a tongue detection model and a tongue segmentation model.

[0061] In order to accurately locate the tongue area in a complex tongue image and minimize the consumption of computing resources while ensuring model performance, the present invention designs a lightweight model architecture with good detection performance. Figure 5 As shown in the figure, the detection model uses a convolutional neural network structure MobileNetV3 as the backbone network (Backbone), using deep separable convolution to reduce the number of parameters and calculations while maintaining high detection accuracy. At the same time, the spatial attention module and channel attention module are introduced before the final detection head to improve the model's ability to express tongue area features.

[0062] exist Figure 5 In , Conv_BN_HSwish is a combined operation module, which mainly includes convolution layer, batch normalization layer and Hard-Swish activation function. CBAM module is a combination of attention mechanisms, which combines spatial attention module and channel attention module, can improve the model's ability to capture spatial and channel information, let the model focus more on important feature areas and channels, and improve the overall performance of the model. C2f is an optimized convolution module used to better capture the multi-scale feature information of feature maps, while keeping the amount of calculation and parameters relatively small, and improve the performance of the model. C stands for "Conv", that is, convolution operation; 2 represents that two different feature extraction paths are used in this module; f stands for "Focus", indicating that this module better focuses on and captures features of different scales through a new structure. Upsample is used to increase the spatial resolution (height and width) of the input feature map to a higher resolution. Concat is used to splice multiple tensors along a specific dimension (axis) to merge them into a new tensor.

[0063] In order to accurately segment the tongue area, Figure 6As shown in the figure, the segmentation model structure is based on a multi-layer convolutional neural network. The CBAM attention mechanism combination module and the residual block are introduced in the encoder part to enable the model to focus on more important tongue feature areas and improve the model's gradient flow and feature learning capabilities. Multi-scale feature fusion is introduced at the decoder end to better capture the detailed information of the tongue edge.

[0064] exist Figure 6 In the paper, the residual convolution module (R-ConvBlock) consists of two convolutional layers (Conv), a batch normalization layer (BatchNorm, BN), a Relu activation function, and a residual connection. Each R-ConvBlock module is responsible for extracting image features. The residual module used can effectively alleviate the gradient vanishing problem in deep networks. The residual convolution module is used to extract features, and then the feature representation is further optimized through the attention module. The layers are connected through downsampling to reduce the spatial resolution of the feature map and increase the receptive field of the feature. Downsampling is used to reduce the spatial resolution or sampling rate of the input feature map or signal, and to extract higher-level features by reducing the spatial size of the data, while reducing the amount of calculation and memory usage. After each downsampling, the feature map is upsampled (Upsample) and concatenated (Concat) with the high-level feature map for multi-scale fusion, and finally the output tongue segmentation mask (Mask) is generated through the convolution layer. This network structure is very suitable for fine tongue segmentation tasks and can maintain high tongue segmentation accuracy in more complex hospital scenarios.

[0065] S203. Train the tongue detection model based on the tongue detection data set, and update the model parameters of the tongue detection model. If the tongue detection result of each image output by the updated tongue detection model has converged to the annotated tongue boundary box of each image, a trained tongue detection model is obtained.

[0066] S204. Train the tongue segmentation model based on the tongue segmentation data set, and update the model parameters of the tongue segmentation model. If the tongue segmentation result of each image output by the updated tongue segmentation model has converged to the annotated tongue area segmentation box of each image, a trained tongue segmentation model is obtained.

[0067] In S203-S204, the tongue detection model and the tongue segmentation model are jointly trained based on the training set and the validation set in their respective data sets, and the ultimate goal is to train a high-precision tongue detection model and a tongue segmentation model. First, data enhancement technology is applied to the training set in the detection and segmentation data sets to increase the generalization ability of the model, mainly including the following operations: 1) Rotation: Randomly rotate the image in the angle range of [-30°, 30°].

[0068] 2) Translation: Randomly translate in width and height, ranging from [-10%, 10%].

[0069] 3) Flip: Randomly flip the image horizontally or vertically.

[0070] 4) Brightness / Contrast Adjustment: Randomly adjust the brightness and contrast in the range of [0.8, 1.2].

[0071] 5) Noise: Add random noise to the image.

[0072] For the training set in the detection and segmentation dataset, one or more of the above operations will be randomly selected to perform data augmentation operations on the original tongue image, and the training set data will be expanded to twice the original training set data.

[0073] The training process of the tongue detection model and the tongue segmentation model uses a series of carefully designed hyperparameters to achieve the training effect and final performance of the model. Reasonable hyperparameters can significantly improve the detection and segmentation performance of the model on tongue images, ensuring that the model converges quickly during training and has good generalization ability. Table 3 lists in detail the hyperparameters used in the training process of the tongue detection model and the tongue segmentation model, including several key factors such as learning rate, batch size, optimizer type, loss function, and data augmentation strategy. The specific hyperparameters are shown in Table 3 below: Table 3. Hyper parameters of tongue detection model and tongue segmentation model training

[0074] Cosine Annealing in Table 3 above is a learning rate scheduling strategy that controls the model training process by adjusting the learning rate change curve. Unlike the traditional learning rate decay strategy, the learning rate change trend of cosine annealing is a smooth decline along the cosine curve, which helps the model to be fine-tuned at a lower learning rate in the later stage of training, effectively avoiding the overfitting problem in the training process and promoting the rapid convergence of the model. The specific formula is as follows:

[0075] in, is the current learning rate; is the current minimum learning rate; is the maximum learning rate (initial learning rate); is the current training round; is the maximum number of annealing cycles.

[0076] Step Decay is another learning rate scheduling strategy that is used to gradually reduce the learning rate during training. It reduces the learning rate by a certain proportion when the preset number of training iterations (epochs) reaches a certain value. The learning rate adjustment process of the step decay strategy can be expressed as the following formula:

[0077] in, is the current learning rate; is the initial learning rate; is the decay coefficient (usually less than 1, such as 0.1); s is the decay step size, which indicates how many epochs are used for decay; t is the current training round.

[0078] In some embodiments, after respectively training the tongue detection model and the tongue segmentation model based on the image data set to obtain the trained tongue detection model and the trained tongue segmentation model, the method further includes step A, step B, and step C: Step A: Use respective test sets to test the trained tongue detection model and the trained tongue segmentation model respectively, and obtain the tongue detection output result and the tongue segmentation output result of the test set.

[0079] In this step, after the model training is completed, in order to comprehensively evaluate the performance of the tongue detection model and the tongue segmentation model, the trained tongue detection model and the trained tongue segmentation model are tested using the test set in the tongue detection dataset and the tongue segmentation dataset to obtain the tongue detection output result and the tongue segmentation output result of the test set. This process is intended to verify the performance of the model in the actual application environment and ensure that the model can run stably and accurately in a variety of scenarios.

[0080] Step B: using preset test indicators to evaluate the tongue detection output results and the tongue segmentation output results of the test set to obtain the test results of the tongue detection model and the test results of the tongue segmentation model.

[0081] Optionally, the evaluation indicators of the tongue detection model include: Mean Average Precision (mAP), Recall, Precision, F1-Score. The evaluation indicators of the tongue segmentation model include: Dice Coefficient, Intersection over Union (IoU), Pixel Accuracy, etc.

[0082] Optionally, Dice Coefficient is an evaluation metric used in image segmentation tasks to measure the similarity between the model segmentation result and the true segmentation area. The value of Dice coefficient ranges from 0 to 1, where 1 means that the segmentation result is completely consistent with the true area, and 0 means that there is no intersection at all. The calculation formula of Dice coefficient is as follows:

[0083] Among them, A is the segmentation area predicted by the model; B is the actual segmentation area; is the total number of pixels in the model prediction area; is the total number of pixels in the real segmentation area; is the number of pixels at the intersection of the model prediction area and the true segmentation area; TP is the number of pixels that are actually positive examples and predicted by the model as positive examples; FP is the number of pixels that are actually negative examples but predicted by the model as positive examples; FN is the number of pixels that are actually positive examples but predicted by the model as negative examples.

[0084] Pixel Accuracy (PA) is an indicator used to evaluate the performance of models in image segmentation tasks. It is used to measure the accuracy of the model's classification on all pixels. It evaluates the overall segmentation accuracy of the model by calculating the ratio of all correctly classified pixels to the total pixels. The calculation formula for pixel accuracy is as follows:

[0085] Among them, TP (True Positive) is the number of pixels correctly classified as the target category; TN (True Negative) is the number of pixels correctly classified as the background; FP (False Positive) is the number of pixels incorrectly classified as the target category; FN (False Negative) is the number of pixels incorrectly classified as the background.

[0086] Step C: When the test results of the tongue detection model and the test results of the tongue segmentation model both meet the target index values, it is determined that the training of the tongue detection model and the tongue segmentation model is completed.

[0087] In this step, if the test results of the tongue detection model and the test results of the tongue segmentation model both meet the target index values, it is determined that the training of the tongue detection model and the tongue segmentation model is completed, that is, the tongue detection model and the tongue segmentation model can be used.

[0088] S205 , fusing the trained tongue detection model and the trained tongue segmentation model to obtain a target segmentation model, and performing segmentation processing on the image dataset using the target segmentation model to obtain a tongue segmentation image dataset.

[0089] Specifically, the implementation process and principle of S205 and S103 are the same, and will not be repeated here.

[0090] S206: Eliminate the image data that does not meet the preset image quality requirement in the tongue segmentation image data set to obtain a tongue segmentation image data set after elimination.

[0091] In this step, the images in the tongue segmentation image dataset are reviewed, and samples that are confirmed to have serious problems during the evaluation process (such as blurred segmentation boundaries, incomplete cropping of the tongue area, etc.) are eliminated to ensure that the samples entering the tongue segmentation image dataset are all high-quality samples. Finally, tongue segmentation images that meet the image quality requirements are included in the dataset.

[0092] S207. In combination with a preset tongue feature classification standard, feature annotation is performed on the tongue segmentation image dataset after elimination, and the tongue features of each image are annotated to obtain a tongue feature classification dataset, wherein the tongue features include tongue color, tongue quality, tongue coating color, tongue coating quality and overall features.

[0093] After completing the data cleaning, according to the preset feature classification standards, the tongue segmentation images that meet the image quality requirements are finely classified and annotated to produce a tongue feature classification data set. The tongue features include tongue color, tongue quality, coating color, coating quality and overall features. Specifically, the tongue color feature classification includes: light red tongue, light white tongue, red tongue, crimson tongue, cyan tongue, purple tongue; the tongue quality feature classification includes: fat tongue, thin and thin tongue, cracked tongue, tooth-marked tongue, dotted tongue, thorn tongue; the coating color feature classification includes: light white coating, dark white coating, light yellow coating, medium yellow coating, dark yellow coating, gray coating, yellow-black coating, burnt black coating, light green coating, dark green coating; the coating quality feature classification includes: thin coating, thick coating, moist coating, dry coating, greasy coating, rough coating, rotten coating, peeling coating; the overall feature classification includes all categories of tongue color, tongue quality, coating color and coating quality.

[0094] S208. Training a tongue feature classification model based on the tongue feature classification data set to obtain a trained tongue feature classification model.

[0095] S209, acquiring target image data to be detected, and inputting the target image data into the target segmentation model to obtain tongue segmentation image data of the target image data to be detected.

[0096] Specifically, the implementation process and principle of S209 and S105 are the same, and will not be repeated here.

[0097] S210 , performing feature analysis on the tongue segmentation image data using a trained tongue feature classification model to obtain a tongue feature analysis result.

[0098] Specifically, the implementation process and principle of S210 and S106 are the same, and will not be repeated here.

[0099] The disclosed embodiment constructs an image data set, which includes a large amount of image data containing tongue images, constructs a tongue detection model and a tongue segmentation model, trains the tongue detection model based on the tongue detection data set, and updates the model parameters of the tongue detection model. If the tongue detection result of each image output by the updated tongue detection model has converged to the annotated tongue boundary box of each image, a trained tongue detection model is obtained. The tongue segmentation model is trained based on the tongue segmentation data set, and the model parameters of the tongue segmentation model are updated. If the tongue segmentation result of each image output by the updated tongue segmentation model has converged to the annotated tongue area segmentation box of each image, a trained tongue segmentation model is obtained. Furthermore, the trained tongue detection model and the trained tongue segmentation model are fused to obtain a target segmentation model, and the image data set is segmented and processed by the target segmentation model to obtain a tongue segmentation image data set, and the image data that does not meet the preset image quality requirements in the tongue segmentation image data set is eliminated to obtain a tongue segmentation image data set after elimination, and the tongue segmentation image data set after elimination is feature-labeled in combination with the preset tongue feature classification standard, and the tongue features of each image are labeled to obtain a tongue feature classification data set, and the tongue features include tongue color, tongue quality, tongue coating color, tongue coating quality and overall features. Next, the tongue feature classification model is trained based on the tongue feature classification data set to obtain a trained tongue feature classification model, and the target image data to be detected is obtained, and the target image data is input into the target segmentation model to obtain the tongue segmentation image data of the target image data to be detected. Then, the tongue segmentation image data is feature analyzed by the trained tongue feature classification model to obtain the tongue feature analysis results. Through this method, the accuracy and robustness of feature classification can be improved, and the reliance on manual labeling and erroneous information in separating tongue coating can be reduced. This not only improves the training efficiency of the model, but also significantly improves the stability and reliability of classification.

[0100] Figure 4 A flow chart of a tongue feature analysis method provided by another embodiment of the present disclosure is as follows: Figure 4 As shown, the method includes the following steps: S301, constructing an image data set, wherein the image data set includes a large amount of image data including tongue images.

[0101] Specifically, the implementation process and principle of S301 and S101 are the same, and will not be repeated here.

[0102] S302. Train the tongue detection model and the tongue segmentation model based on the image data set to obtain a trained tongue detection model and a trained tongue segmentation model.

[0103] Specifically, the implementation process and principle of S302 and S102 are the same, and will not be repeated here.

[0104] S303: Fusing the trained tongue detection model and the trained tongue segmentation model to obtain a target segmentation model, and performing segmentation processing on the image dataset using the target segmentation model to obtain a tongue segmentation image dataset.

[0105] Specifically, the implementation process and principle of S303 and S103 are the same, and will not be repeated here.

[0106] S304. Annotate the tongue segmentation image dataset to obtain a tongue feature classification dataset, and train a tongue feature classification model based on the tongue feature classification dataset to obtain a trained tongue feature classification model.

[0107] Specifically, the implementation process and principle of S304 and S104 are the same, and will not be repeated here.

[0108] In some embodiments, in S304, the tongue feature classification model is trained based on the tongue feature classification data set to obtain a trained tongue feature classification model, which may include but is not limited to S3041 and S3042: S3041, constructing a tongue feature classification model; like Figure 7 As shown in the figure, the tongue feature classification model adopts a multi-task learning (MTL) architecture to build different tongue feature classification tasks into a unified fusion model. The overall architecture of the model can be divided into the following main parts: 1) Underlying feature extraction network The underlying feature extraction network uses EfficientNet as the basic module to extract low-level and high-level features of tongue images. EfficientNet is a convolutional neural network architecture that improves model performance while minimizing computational costs through automated model expansion. The underlying feature extraction network adopts a composite expansion strategy, which simultaneously expands the depth, width, and resolution of the network, thereby achieving a good balance between model size, computational complexity, and accuracy. It can capture information such as the edge, texture, and color of tongue images, and provide rich tongue feature representations for subsequent tasks.

[0109] 2) Task branch module The task branch module is used to classify and predict tongue color, tongue quality, tongue coating color, tongue coating quality and overall features. The task branch module is a network structure consisting of multi-layer feature extraction, skip layer connection, full connection layer and activation function. The output layer uses the Softmax activation function to output the prediction probabilities of different categories.

[0110] 3) Feature Sharing After the underlying feature extraction network, a shared feature layer is used to achieve information sharing and feature complementarity between different tasks. The shared feature layer is designed as a multi-layer convolutional neural network structure to integrate feature information from multiple tasks at different levels. Its main function is to fuse and re-encode the low-level and high-level features output by the underlying feature extraction network to provide rich semantic features for each task branch.

[0111] 4) Loss function and optimization strategy Each task branch has an independent loss function. Tongue color, tongue quality, fur color, fur quality and overall feature branches all use the cross-entropy loss function to optimize the classification output. In order to better balance the learning process of each task, the total loss function is the weighted sum of the losses of each task.

[0112] 5) Model output The model output uses multi-task classification and simultaneously outputs classification predictions of tongue color, tongue quality, tongue coating color, tongue coating quality and overall feature results. Each classification task outputs confidence scores of different categories, and for each classification task, a comprehensive evaluation is performed using indicators such as accuracy, precision, and recall to comprehensively measure the performance of the model in multi-task classification.

[0113] exist Figure 7In , Dropout is a regularization technique to prevent overfitting of deep neural networks. It reduces the network's over-reliance on certain features by randomly discarding (i.e., disconnecting) some neurons and their connections during training, thereby improving the generalization ability of the model. Global Average Pooling (GAP) is a pooling operation that reduces the spatial dimensions (i.e., height and width) of each feature map to 1, so that the average value of the entire feature map is used as the output. The Fully Connected Layer (FC) is a basic layer type in a neural network. Each of its input nodes is connected to each output node, so each output node is the weighted sum of all input nodes, which is used to integrate the extracted features and ultimately used for classification. Softmax is an activation function used in the output layer of multi-classification problems.

[0114] exist Figure 7 In the example, Conv3_BN_Swish is a module for feature extraction and nonlinear transformation. It consists of a 3*3 convolution kernel, a batch normalization layer, and a Swish activation function in series. The 3*3 convolution kernel can balance the computational complexity and feature extraction capabilities, and is used to capture local spatial information. Conv1_BN_Swish mainly includes a 1*1 convolution kernel, followed by a batch normalization layer, and finally a Swish activation function. The 1*1 convolution kernel is used to change the number of channels of the feature map without changing the spatial size of the input feature map. It is used to compress the channel dimension, which can fuse the information of multiple channels together and increase the nonlinear expression ability of the model.

[0115] exist Figure 7 In MBConv1, MBConv1 is a convolution module, which is an extended inverted bottleneck structure, mainly including a deep convolution layer, two batch normalization layers, a point convolution layer, a residual connection, and a ReLU6 activation function. In MBConv1, the expansion factor is 1, so there is no expansion layer, and the number of input channels and output channels remain the same. MBConv1 with a residual structure can provide high performance while maintaining model computational efficiency, and is particularly suitable for deep learning applications in mobile and embedded devices.

[0116] exist Figure 7In the , MBConv6_SE is a convolutional module based on MBConv1, with an additional expansion layer and SiLU replacing ReLU. It increases the number of input channels by 6 times. The purpose is to increase the number of channels so that subsequent deep convolutional layers can learn more features in high-dimensional space. SE means re-weighting the features. It first performs global average pooling on the feature map in the spatial dimension, then calculates the channel attention through two fully connected layers, and finally scales the original features. Through the SE module operation, the model can adaptively focus on and select specific channel features, improving the feature selectivity and classification ability of the model.

[0117] S3042. Train the tongue feature classification model based on the tongue feature classification data set, and update the model parameters of the tongue feature classification model. If the tongue feature analysis results of each image output by the updated tongue feature classification model have converged to the annotated tongue features of each image, a trained tongue feature classification model is obtained.

[0118] The tongue feature classification model is trained using the divided tongue feature classification data set. In order to ensure that the model can fully and accurately capture the multi-dimensional features of the tongue, this step adopts the multi-task learning method, which aims to simultaneously optimize and train multiple tasks such as tongue color, tongue quality, tongue coating color, tongue coating quality, and overall features.

[0119] First, the cross-entropy loss function is used to optimize each classification task. The cross-entropy loss function has good convergence in multi-classification problems and can effectively measure the deviation between the model's predicted value and the true value, thereby guiding the model to continuously improve the prediction accuracy during the training process. In order to achieve the best performance of the model on multiple tasks, the total loss function is used as the weighted sum of the losses of each task. Then the total loss function It can be expressed as:

[0120] in, is the loss function of the i-th task (such as the cross entropy loss function); is the weight of the i-th task, indicating the importance of the task in the total loss; N is the total number of tasks (in this paper, N=5, i.e., tongue color, tongue quality, tongue coating color, tongue coating quality and overall features).

[0121] The loss weight of each task is adjusted according to its importance in the overall model to ensure that each task can be reasonably optimized during the training process. Finally, a tongue feature classification model is obtained.

[0122] The hyperparameters used in the training process of the tongue feature classification model are listed in Table 4, including several key factors such as learning rate, batch size, optimizer type, loss function, etc.

[0123] Table 4. Training hyperparameters of tongue feature classification model

[0124] In Table 4 above, Xavier weight initialization is a method for weight initialization in deep neural networks, which aims to keep the variance of the input and output of each layer consistent when the network is initialized, thereby avoiding the gradient vanishing or gradient exploding problem and promoting the rapid convergence of the neural network. Assuming that the input of a layer is X and the output is Y, Xavier hopes that the initialization of the weight matrix W can ensure the following variance relationship:

[0125] In order to make the variance of the output Y as close as possible to the variance of the input X, Xavier initialization samples the elements of the weight matrix W from the following uniform distribution:

[0126] in, is the number of input nodes (number of neurons) of the current layer; is the number of output nodes (neurons) of the current layer.

[0127] In some embodiments, after S304, the method further includes step a, step b, and step c: Step a, using a test set in the tongue feature classification data set to test the trained tongue feature classification model, and obtaining a tongue feature classification output result of the test set; Step b, using preset test indicators to evaluate the tongue feature classification output results of the test set to obtain the test results of the tongue feature classification model; Step c: when the test result of the tongue feature classification model meets the target index value, it is determined that the training of the tongue feature classification model is completed.

[0128] In this embodiment, a comprehensive performance test will be conducted on the trained tongue feature classification model. This step is intended to verify the actual performance of the model in different tongue feature classification tasks, and to evaluate its reliability and generalization ability in actual application scenarios. The test data set consists of tongue segmentation image data that does not participate in model training. Each image in the test set is input into the trained tongue feature classification model. The model will predict the five classification tasks of tongue color, tongue quality, tongue coating color, tongue coating quality and overall features respectively, and output the prediction results and confidence scores of each task one by one. In order to comprehensively measure the comprehensive performance of the model under multi-task classification, the accuracy (Accuracy), precision (Precision) and recall rate (Recall) will be used to evaluate the real effect and obtain the test results of the tongue feature classification model. When the test results of the tongue feature classification model meet the target index value, it is determined that the tongue feature classification model training is completed.

[0129] S305 , integrating the trained tongue feature classification model, color space conversion algorithm, channel analysis algorithm, and verification algorithm to obtain a statistical analysis module.

[0130] like Fig.10 As shown in the figure, the trained tongue feature classification model is combined with the color space conversion algorithm, channel analysis algorithm and test algorithm to construct a comprehensive statistical analysis module. This module can receive tongue segmentation images, perform multi-dimensional tongue feature analysis and statistical tests, and output detailed tongue feature classification prediction results and various channel analysis data.

[0131] S306: Input the tongue segmentation image data into a statistical analysis module, perform feature analysis on the tongue segmentation image data using a trained tongue feature classification model, and obtain the credibility of each feature category.

[0132] The preprocessed tongue segmentation image is input into the tongue feature classification model, and the model will perform multi-task feature classification inference on the input image. The model will output the prediction information of tongue color, tongue quality, coating color, coating quality and overall feature classification, and provide the confidence score of each prediction, so as to provide a credibility reference for the prediction results of each feature category.

[0133] S307 , using a color space conversion algorithm to convert the tongue segmentation image data into color opposition space channel image data.

[0134] In order to further analyze the color characteristics of the image, such as Figure 8As shown in the figure, the RGB channel image, i.e. the original tongue segmentation image data, is converted into the color opposition space Lab color space using the color space conversion algorithm. The brightness information (L channel) and color information (a, b channels) in the tongue image are separated, which is of great significance for the subsequent tongue color and tongue coating color analysis.

[0135] S308, input the tongue segmentation image data and the color opposition space channel image data into the channel analysis algorithm respectively for channel analysis, and obtain red, green and blue channel analysis results and color opposition space channel analysis results.

[0136] like Figure 8 As shown in the figure, the RGB and converted Lab channel images are input into the channel analysis algorithm respectively, and detailed statistical analysis is performed on the RGB and Lab channels respectively. The analysis content includes the mean and standard deviation of each channel, as well as the distance between the channel space mean and the eight vertices. These statistics can help us quantify the color distribution characteristics of the tongue image, thereby providing richer reference data for the characteristic analysis of the tongue.

[0137] S309 , performing distribution inspection on the tongue segmentation image data and the color opposition space channel image data by using an inspection algorithm to obtain inspection values ​​of the red, green and blue channels and inspection values ​​of the color opposition space channel.

[0138] The RGB and Lab channel images are input into the KS test algorithm to test the spatial distribution of the image channels and the standard distribution, and the KS test p value of each channel is calculated. This p value can be used to evaluate the significant differences in tongue characteristics in different health states, thereby providing a statistical basis for the results of intelligent tongue diagnosis.

[0139] Finally, the statistical analysis module will output multi-dimensional tongue feature analysis data, including the prediction results and confidence scores of tongue color, tongue quality, coating color, coating quality and overall characteristics, various statistics of RGB and Lab channels, KS test p-value and other detailed information. These analysis data will be presented in a structured form for medical analysts and researchers to better understand and analyze the relationship between different tongue features and their indicative role in health status.

[0140] The Kolmogorov-Smirnov (KS) test is a nonparametric statistical method used to compare the distributions of two samples, or the difference between a sample and a reference distribution. and , then their cumulative distribution functions (CDFs) are and , and its calculation formula is as follows:

[0141] In the above formula, X represents a given random variable.

[0142] From the above CDF formula, we can see that CDF It represents the probability that the random variable X takes a value less than or equal to x. As x increases, It is monotonically increasing, and the CDF value range is (0, 1). When x tends to negative infinity tends to 0, as x tends to positive infinity Tends to 1. For two samples with cumulative distribution functions, the calculation formula for the p-value output by the KS test algorithm is as follows:

[0143]

[0144] in, Indicates the sample The cumulative distribution function of Indicates the sample The cumulative distribution function of .

[0145] In some embodiments, various tongue feature analysis and channel statistical test results are extracted from the statistical analysis module inference results, and a structured statistical analysis report is output. The specific output content is as follows: 1) Multi-task feature classification prediction results. Including the prediction values ​​and confidence scores of multiple tasks such as tongue color, tongue quality, tongue coating color, tongue coating quality and overall features, and corresponding charts are given.

[0146] 2) Color channel statistics. Including the mean and standard deviation of RGB and Lab channels, and the distance between the channel space mean and the eight vertices.

[0147] 3) KS test statistic. The KS test p value of the tongue image distribution and the standard distribution is used to determine the significant difference between the distribution of each color channel and the standard distribution, and then output the corresponding KS test statistic chart.

[0148] The disclosed embodiment constructs an image data set, which includes a large amount of image data containing tongue images. Based on the image data set, a tongue detection model and a tongue segmentation model are trained respectively to obtain a trained tongue detection model and a trained tongue segmentation model. Then, the trained tongue detection model and the trained tongue segmentation model are fused to obtain a target segmentation model, the image data set is segmented by the target segmentation model to obtain a tongue segmentation image data set, the tongue segmentation image data set is annotated to obtain a tongue feature classification data set, and the tongue feature classification model is trained based on the tongue feature classification data set to obtain a trained tongue feature classification model. Next, the trained tongue feature classification model, the color space conversion algorithm, the channel analysis algorithm, and the inspection algorithm are integrated to obtain a statistical analysis module, the tongue segmentation image data is input into the statistical analysis module, and the tongue segmentation image data is feature analyzed by the trained tongue feature classification model to obtain the credibility of each feature category. Furthermore, the color space conversion algorithm is used to convert the tongue segmentation image data into color opposition space channel image data, and the tongue segmentation image data and the color opposition space channel image data are respectively input into the channel analysis algorithm for channel analysis to obtain the red, green, and blue channel analysis results and the color opposition space channel analysis results. Then, the tongue segmentation image data and the color opposition space channel image data are distributed tested by the test algorithm to obtain the test values ​​of the red, green, and blue channels and the test values ​​of the color opposition space channel. Through this method, key features such as tongue color and fur color can be quantitatively analyzed. Through channel analysis, the model can deeply extract and quantify the color features in the tongue image, and evaluate the significant differences between samples through statistical methods such as tests. The analysis results can provide doctors with detailed quantitative data of tongue color, fur color and other features, and provide strong data support for clinical applications.

[0149] Fig.11 The structure diagram of the tongue feature analysis device provided in the embodiment of the present disclosure. The tongue feature analysis device may be the electronic device as described in the above embodiment, or the tongue feature analysis device may be a component or assembly in the electronic device. The tongue feature analysis device provided in the embodiment of the present disclosure may execute the processing flow provided in the tongue feature analysis method embodiment, such as Fig.11As shown, the tongue feature analysis device 40 includes: a construction unit 41, a first training unit 42, a first obtaining unit 43, a second training unit 44, a second obtaining unit 45, and a feature analysis unit 46; wherein the construction unit 41 is used to construct an image data set, and the image data set includes a large amount of image data containing tongue images; the first training unit 42 is used to train the tongue detection model and the tongue segmentation model respectively based on the image data set to obtain a trained tongue detection model and a trained tongue segmentation model; the first obtaining unit 43 is used to fuse the trained tongue detection model and the trained tongue segmentation model to obtain a target segmentation model, and the target segmentation model is obtained by the target segmentation The model performs segmentation processing on the image data set to obtain a tongue segmentation image data set; the second training unit 44 is used to annotate the tongue segmentation image data set to obtain a tongue feature classification data set, and train the tongue feature classification model based on the tongue feature classification data set to obtain a trained tongue feature classification model; the second obtaining unit 45 is used to obtain the target image data to be detected, input the target image data into the target segmentation model, and obtain the tongue segmentation image data of the target image data to be detected; the feature analysis unit 46 is used to perform feature analysis on the tongue segmentation image data through the trained tongue feature classification model to obtain a tongue feature analysis result.

[0150] Optionally, when the construction unit 41 constructs an image dataset, it is specifically used to: collect a large amount of image data containing tongue images; perform color calibration on each of the image data to obtain each color-calibrated image data; crop each of the color-calibrated image data to obtain each tongue area image data; annotate each of the color-calibrated image data to mark the tongue boundary box of each image to obtain a tongue detection dataset, and divide the tongue detection dataset into a training set, a verification set, and a test set of the tongue detection model based on a preset first ratio; annotate each of the tongue area image data to mark the tongue area segmentation box of each image to obtain a tongue segmentation dataset, and divide the tongue segmentation dataset into a training set, a verification set, and a test set of the tongue segmentation model based on a preset second ratio.

[0151] Optionally, the image dataset includes a tongue detection dataset and a tongue segmentation dataset; The first training unit 42 trains the tongue detection model and the tongue segmentation model based on the image data set respectively, and when a trained tongue detection model and a trained tongue segmentation model are obtained, it is specifically used to: construct a tongue detection model and a tongue segmentation model; train the tongue detection model based on the tongue detection data set, and update the model parameters of the tongue detection model, and if the tongue detection result of each image output by the updated tongue detection model has converged to the annotated tongue boundary box of each image, then a trained tongue detection model is obtained; train the tongue segmentation model based on the tongue segmentation data set, and update the model parameters of the tongue segmentation model, and if the tongue segmentation result of each image output by the updated tongue segmentation model has converged to the annotated tongue area segmentation box of each image, then a trained tongue segmentation model is obtained.

[0152] Optionally, after the tongue detection model and the tongue segmentation model are trained based on the image data set to obtain the trained tongue detection model and the trained tongue segmentation model, the tongue feature analysis device 40 also includes: a first testing unit 47; the first testing unit 47 is used to test the trained tongue detection model and the trained tongue segmentation model using their respective test sets to obtain the tongue detection output results and the tongue segmentation output results of the test sets; the tongue detection output results and the tongue segmentation output results of the test set are evaluated using preset test indicators to obtain the test results of the tongue detection model and the test results of the tongue segmentation model; when the test results of the tongue detection model and the test results of the tongue segmentation model both meet the target indicator values, it is determined that the training of the tongue detection model and the tongue segmentation model is completed.

[0153] Optionally, when the second training unit 44 annotates the tongue segmentation image dataset to obtain a tongue feature classification dataset, it is specifically used to: eliminate image data in the tongue segmentation image dataset that does not meet preset image quality requirements to obtain a tongue segmentation image dataset after elimination; and perform feature annotation on the tongue segmentation image dataset after elimination in combination with a preset tongue feature classification standard to annotate the tongue features of each image to obtain a tongue feature classification dataset, wherein the tongue features include tongue color, tongue quality, tongue coating color, tongue coating quality and overall features.

[0154] Optionally, the second training unit 44 trains the tongue feature classification model based on the tongue feature classification data set to obtain the trained tongue feature classification model, which is specifically used to: construct a tongue feature classification model; train the tongue feature classification model based on the tongue feature classification data set, and update the model parameters of the tongue feature classification model, if the tongue feature analysis results of each image output by the updated tongue feature classification model have converged to the annotated tongue features of each image, then the trained tongue feature classification model is obtained; After the tongue feature classification model is trained based on the tongue feature classification data set to obtain the trained tongue feature classification model, the tongue feature analysis device 40 also includes: a second testing unit 48; the second testing unit 48 is used to test the trained tongue feature classification model using the test set in the tongue feature classification data set to obtain the tongue feature classification output result of the test set; the tongue feature classification output result of the test set is evaluated using preset test indicators to obtain the test result of the tongue feature classification model; when the test result of the tongue feature classification model meets the target indicator value, it is determined that the training of the tongue feature classification model is completed.

[0155] Optionally, after the tongue feature classification model is trained based on the tongue feature classification data set to obtain the trained tongue feature classification model, the tongue feature analysis device 40 further includes: an integration unit 49; the integration unit 49 is used to integrate the trained tongue feature classification model, the color space conversion algorithm, the channel analysis algorithm, and the verification algorithm to obtain a statistical analysis module; the tongue segmentation image data is input into the statistical analysis module, and feature analysis is performed on the tongue segmentation image data using the trained tongue feature classification model to obtain the credibility of each feature category; the color space conversion algorithm is used to convert the tongue segmentation image data into color opposition space channel image data; the tongue segmentation image data and the color opposition space channel image data are respectively input into the channel analysis algorithm for channel analysis to obtain red, green, and blue channel analysis results and color opposition space channel analysis results; the tongue segmentation image data and the color opposition space channel image data are distributed tested by the verification algorithm to obtain the test values ​​of the red, green, and blue channels and the test values ​​of the color opposition space channels.

[0156] Fig.11 The tongue feature analysis device of the illustrated embodiment can be used to execute the technical solution of the above-mentioned method embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0157] Fig.12 FIG. 1 is a schematic diagram of the structure of an electronic device in an embodiment of the present disclosure, and the electronic device may be a tongue feature analysis device. Fig.12, which shows a structural schematic diagram of an electronic device 600 suitable for implementing the embodiments of the present disclosure. Fig.12 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0158] like Fig.12 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603 to implement the tongue feature analysis method of the embodiment described in the present disclosure. In RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604. Optionally, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0159] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0160] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0161] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device can execute the tongue feature analysis method described in the above embodiments.

[0162] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0163] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0164] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A tongue feature analysis method, characterized in that: The method comprises: constructing an image data set, wherein the image data set includes a large amount of image data including tongue images; Based on the image data set, the tongue detection model and the tongue segmentation model are trained respectively to obtain a trained tongue detection model and a trained tongue segmentation model; The trained tongue detection model and the trained tongue segmentation model are integrated to obtain a target segmentation model, and the image data set is segmented by the target segmentation model to obtain a tongue segmentation image data set; Annotating the tongue segmentation image dataset to obtain a tongue feature classification dataset, and training a tongue feature classification model based on the tongue feature classification dataset to obtain a trained tongue feature classification model; Acquire target image data to be detected, input the target image data into the target segmentation model, and obtain tongue segmentation image data of the target image data to be detected; The tongue feature analysis is performed on the tongue segmentation image data using a trained tongue feature classification model to obtain a tongue feature analysis result.

2. The method according to claim 1, characterized in that The constructing of the image dataset comprises: Collect a large amount of image data containing tongue images; Performing color calibration processing on each piece of the image data to obtain each piece of color-calibrated image data; Cropping each color-calibrated image data to obtain each tongue region image data; Annotating each of the color-calibrated image data, annotating the tongue boundary box of each image, obtaining a tongue detection data set, and dividing the tongue detection data set into a training set, a validation set, and a test set of the tongue detection model based on a preset first ratio; Each of the tongue area image data is labeled, and the tongue area segmentation frame of each image is labeled to obtain a tongue segmentation data set, and the tongue segmentation data set is divided into a training set, a validation set and a test set of the tongue segmentation model based on a preset second ratio.

3. The method according to claim 1, characterized in that The image dataset includes a tongue detection dataset and a tongue segmentation dataset; The tongue detection model and the tongue segmentation model are trained based on the image data set to obtain a trained tongue detection model and a trained tongue segmentation model, including: Build tongue detection model and tongue segmentation model; Training a tongue detection model based on the tongue detection data set, updating model parameters of the tongue detection model, and obtaining a trained tongue detection model if the tongue detection result of each image output by the updated tongue detection model has converged to the annotated tongue boundary box of each image; The tongue segmentation model is trained based on the tongue segmentation data set, and the model parameters of the tongue segmentation model are updated. If the tongue segmentation result of each image output by the updated tongue segmentation model has converged to the annotated tongue area segmentation box of each image, a trained tongue segmentation model is obtained.

4. The method according to claim 1, characterized in that: After respectively training the tongue detection model and the tongue segmentation model based on the image data set to obtain the trained tongue detection model and the trained tongue segmentation model, the method further includes: Using respective test sets to test the trained tongue detection model and the trained tongue segmentation model respectively, to obtain tongue detection output results and tongue segmentation output results of the test sets; Using preset test indicators to evaluate the tongue detection output results and the tongue segmentation output results of the test set, to obtain the test results of the tongue detection model and the test results of the tongue segmentation model; When the test results of the tongue detection model and the test results of the tongue segmentation model both meet the target index values, it is determined that the training of the tongue detection model and the tongue segmentation model is completed.

5. The method according to claim 1, characterized in that The step of labeling the tongue segmentation image dataset to obtain a tongue feature classification dataset includes: Eliminating image data that does not meet preset image quality requirements in the tongue segmentation image data set to obtain a tongue segmentation image data set after elimination; Combined with the preset tongue feature classification standard, the tongue segmentation image dataset after elimination is feature-labeled, the tongue features of each image are labeled, and a tongue feature classification dataset is obtained. The tongue features include tongue color, tongue quality, tongue coating color, tongue coating quality and overall features.

6. The method according to claim 1, characterized in that The step of training the tongue feature classification model based on the tongue feature classification data set to obtain a trained tongue feature classification model includes: Construct a tongue feature classification model; Training a tongue feature classification model based on the tongue feature classification data set, updating model parameters of the tongue feature classification model, and obtaining a trained tongue feature classification model if the tongue feature analysis result of each image output by the updated tongue feature classification model has converged to the annotated tongue features of each image; After the tongue feature classification model is trained based on the tongue feature classification data set to obtain a trained tongue feature classification model, the method further includes: Using the test set in the tongue feature classification data set to test the trained tongue feature classification model, and obtaining the tongue feature classification output result of the test set; Using a preset test index to evaluate the tongue feature classification output result of the test set to obtain a test result of the tongue feature classification model; When the test result of the tongue feature classification model meets the target index value, it is determined that the training of the tongue feature classification model is completed.

7. The method according to claim 1, characterized in that After the tongue feature classification model is trained based on the tongue feature classification data set to obtain a trained tongue feature classification model, the method further includes: The trained tongue feature classification model, color space conversion algorithm, channel analysis algorithm, and inspection algorithm are integrated to obtain a statistical analysis module; Inputting the tongue segmentation image data into a statistical analysis module, performing feature analysis on the tongue segmentation image data using a trained tongue feature classification model, and obtaining the credibility of each feature category; The tongue segmentation image data is converted into color-opponent space channel image data using a color space conversion algorithm; The tongue segmentation image data and the color opposition space channel image data are respectively input into the channel analysis algorithm for channel analysis to obtain the red, green and blue channel analysis results and the color opposition space channel analysis results; The tongue segmentation image data and the color opposition space channel image data are subjected to distribution inspection by means of the inspection algorithm, and the inspection values ​​of the red, green and blue channels and the inspection values ​​of the color opposition space channel are obtained.

8. A tongue feature analysis device, characterized in that: include: A construction unit, used for constructing an image data set, wherein the image data set includes a large amount of image data including tongue images; A first training unit, configured to train the tongue detection model and the tongue segmentation model based on the image data set to obtain a trained tongue detection model and a trained tongue segmentation model; A first obtaining unit is used to fuse the trained tongue detection model and the trained tongue segmentation model to obtain a target segmentation model, and to perform segmentation processing on the image data set by using the target segmentation model to obtain a tongue segmentation image data set; A second training unit is used to annotate the tongue segmentation image data set to obtain a tongue feature classification data set, and train a tongue feature classification model based on the tongue feature classification data set to obtain a trained tongue feature classification model; A second obtaining unit is used to obtain target image data to be detected, input the target image data into the target segmentation model, and obtain tongue segmentation image data of the target image data to be detected; The feature analysis unit is used to perform feature analysis on the tongue segmentation image data through a trained tongue feature classification model to obtain a tongue feature analysis result.

9. An electronic device, characterized in that: include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and is configured to be executed by the processor to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images