Traditional Chinese medicine tongue diagnosis intelligent analysis method based on visual language model
By constructing a tongue image-tongue diagnosis aligned dataset and training it with a visual language model, combined with a large language model and review by TCM experts, the problem of traditional tongue diagnosis relying on experience was solved. This enabled the automated recognition and structured description of tongue image features, improving the objectivity and consistency of tongue diagnosis.
Patent Information
- Application Number
- CN202511833243.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-08
- Publication Date
- 2026-03-20
AI Technical Summary
Traditional tongue diagnosis relies heavily on physician experience, resulting in unstable diagnostic results that lack objectivity and standardization. This makes it difficult to meet the intelligent and standardized needs of modern medical systems. Furthermore, existing research struggles to capture the complex, multidimensional features of the tongue and cannot generate interpretive tongue diagnosis descriptions.
A tongue image-tongue diagnosis aligned dataset was constructed, trained using a visual language model such as Qwen-VL, and combined with a large language model and review by TCM experts to generate a corresponding dataset of tongue image images and structured descriptions of tongue diagnosis. A fine-tuning mechanism guided by tongue diagnosis knowledge was introduced to achieve automatic recognition and natural language description of tongue image features.
It achieves automation, objectivity, and interpretability in tongue diagnosis analysis, improves diagnostic consistency and reliability, and generates structured tongue diagnosis results to support TCM syndrome differentiation and treatment.
Smart Images
Figure CN121709160A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of artificial intelligence and intelligent diagnosis of traditional Chinese medicine, and in particular to a traditional Chinese medicine tongue diagnosis intelligent analysis method based on a vision language model. BACKGROUND
[0002] Tongue diagnosis is one of the most important components of the four diagnostic methods (inspection, auscultation and olfaction, interrogation, and pulse-taking) in traditional Chinese medicine. By observing the shape, color, and thickness of the tongue and tongue fur, the function and pathological changes of the organs of the body can be determined, which has important clinical reference value in the process of traditional Chinese medicine syndrome differentiation and treatment. However, traditional tongue diagnosis is highly dependent on the experience of physicians and is greatly influenced by subjective judgment and differences in knowledge background, resulting in unstable diagnosis results, lack of objective and standardized basis, and difficulty in meeting the needs of intelligent and standardized modern medical systems.
[0003] With the development of medical artificial intelligence, image recognition and deep learning technologies have achieved remarkable results in medical image analysis. For example, in various image diagnoses, convolutional neural networks (CNN), Transformer models, and generative large models have been widely used to achieve intelligent auxiliary diagnosis functions for traditional Chinese medicine tongue diagnosis. However, research on traditional Chinese medicine tongue image is still in its early stages, and most studies rely on traditional image features for tongue image recognition, making it difficult to capture complex multi-dimensional features in tongue images. Moreover, most studies are limited to classification tasks and cannot generate explanatory tongue diagnosis descriptions or directly empower clinical syndrome differentiation and treatment.
[0004] In recent years, with the rapid development of large language models, vision language models (VLM) such as Qwen-VL and Gpt-o3 have shown strong capabilities in cross-modal representation learning, multi-modal understanding, and natural language generation, enabling joint modeling of image features and text semantics, making it possible to automatically generate structured information and natural language descriptions based on images. However, there is currently a lack of mature research on applying vision language models to the field of traditional Chinese medicine tongue diagnosis, especially a lack of high-quality tongue image and tongue diagnosis text alignment datasets and intelligent analysis methods for clinical tongue diagnosis tasks.
[0005] To address the problems of existing tongue diagnosis analysis relying on artificial experience, lack of diagnostic consistency, and lack of standardized expression, there is an urgent need for an intelligent tongue diagnosis analysis method that can automatically identify key features of tongue images and output structured indicators and explanatory descriptions, to improve the objectivity, standardization, and explainability of the tongue diagnosis process, and to provide reliable data basis and decision-making basis for traditional Chinese medicine syndrome differentiation and treatment and intelligent diagnosis and treatment systems. SUMMARY
[0006] The application provides a traditional Chinese medicine tongue diagnosis intelligent analysis method based on a visual language model, to solve the problems of existing tongue diagnosis analysis relying on artificial experience, lack of standardization of feature description, and insufficient diagnosis consistency. The specific method comprises:
[0007] A tongue image - tongue diagnosis alignment dataset is constructed, traditional Chinese medicine tongue images are obtained, data preprocessing is performed on the tongue images, including tongue image quality screening and tongue body region segmentation, and a large language model is used to automatically identify and label the tongue body and tongue fur features, and then a traditional Chinese medicine expert reviews and corrects the automatic labeling results to form a corresponding dataset of tongue image and tongue diagnosis structured description;
[0008] Based on the tongue image - tongue diagnosis alignment dataset, a visual language model is trained, a Qwen-VL model is used as a basic model, and a tongue diagnosis knowledge guided fine-tuning mechanism is introduced to make the model learn the mapping relationship between the tongue body and tongue fur visual features and the tongue diagnosis natural language description;
[0009] The tongue image is input into the trained visual language model, and a structured JSON result containing tongue color, fur color, fur quality and other tongue diagnosis indicators is output, and a natural language tongue diagnosis description corresponding to the structured result is automatically generated, realizing automatic identification and structured expression of tongue diagnosis features.
[0010] Optionally, the tongue image quality screening specifically comprises:
[0011] The sharpness of the tongue image is detected by using an image blurriness evaluation algorithm, and low-quality images meeting a preset blur threshold are removed; the image brightness and contrast are evaluated by using an exposure analysis algorithm, and images that are overexposed or underexposed are removed; and the candidate images are preprocessed based on color balance and illumination correction technology to obtain qualified tongue images.
[0012] Optionally, the tongue body region segmentation specifically comprises:
[0013] The tongue body region in the tongue image is automatically segmented based on an open source tongue body segmentation model TongueSAM; a minimum circumscribed rectangular region containing the tongue body region is generated according to the segmentation result, and the rectangular region is cropped into a square region, if the cropped region is not a square, black pixel filling is performed on the missing edge region, so that the tongue body region forms a standardized square image; finally, the square image is scaled to a preset uniform size of 512x512 for subsequent model training and analysis.
[0014] Optionally, the large language model is used to automatically identify and label the tongue body and tongue fur features, and then a traditional Chinese medicine expert reviews and corrects the automatic labeling results to form a corresponding dataset of tongue image and tongue diagnosis structured description, specifically comprising:
[0015] A large model (such as Gpt-o3) is used to extract features from the segmented tongue images and generate preliminary textual descriptions of tongue diagnosis, including tongue shape, color, coating color, and texture. The automatically generated tongue diagnosis descriptions are then submitted to TCM clinical experts for manual review. Based on the review comments, erroneous, missing, or inconsistent features are revised and standardized labeled text is generated. An image-text aligned dataset is constructed based on the revised standardized tongue diagnosis descriptions and used for training and optimizing the tongue diagnosis visual language model. This significantly reduces the workload of manual labeling and improves the consistency and accuracy of labeling.
[0016] Optionally, the training of the visual language model and the introduction of a fine-tuning mechanism guided by tongue diagnosis knowledge, enabling the model to learn the mapping relationship between the visual features of the tongue body and tongue coating and the natural language description of tongue diagnosis, specifically includes:
[0017] The visual language model was pre-trained and supervised fine-tuned using the tongue image-tongue diagnosis text alignment dataset. This enabled the model to learn the mapping relationship between tongue images and tongue diagnosis descriptions, resulting in a large-scale TCM tongue diagnosis model capable of automatically generating structured descriptions of tongue diagnosis features based on tongue images. During the training and fine-tuning process, a Qwen-VL-based visual language model framework was employed to unify the tasks of tongue image understanding and tongue diagnosis generation. TCM tongue diagnosis knowledge guidance was added during the fine-tuning stage to enhance the model's ability to recognize and generate key tongue diagnosis semantics such as tongue color, coating color, and coating texture, thereby improving the interpretability and clinical usability of the tongue diagnosis analysis results.
[0018] Optionally, the step of inputting the tongue image into a trained visual language model, outputting a structured JSON result containing tongue diagnosis indicators such as tongue color, coating color, and coating texture, and automatically generating a natural language tongue diagnosis description corresponding to the structured result, specifically includes:
[0019] The tongue image is input into the TCM tongue diagnosis model. Based on the tongue and tongue coating features identified by the model, a structured output containing tongue diagnosis indicators such as tongue shape, tongue color, tongue coating color, and tongue coating texture is generated. Based on this, the tongue diagnosis analysis results in JSON format and the corresponding natural language description are automatically generated, realizing the structured presentation of the tongue image analysis results.
[0020] Compared to existing technologies, the beneficial effects of this invention are as follows: This invention proposes an intelligent analysis method for TCM tongue diagnosis based on a visual language model. By introducing a data construction mechanism that combines tongue image processing, automatic annotation, and expert review, a large-scale, high-quality aligned dataset of tongue image and tongue diagnosis structured descriptions is constructed. The hybrid process of automatic annotation using a large model and verification by TCM experts significantly reduces the workload of manual annotation and improves the consistency and accuracy of annotation. Based on this, this invention proposes a large-scale TCM tongue diagnosis model framework based on Qwen-VL, unifying tongue image visual understanding and tongue diagnosis natural language generation tasks within the same model. A knowledge-guided fine-tuning mechanism is designed, introducing knowledge prompts from the tongue diagnosis domain to enhance the model's ability to recognize and express key tongue diagnosis semantics such as tongue color, coating color, and coating texture. This invention can automatically generate structured natural language tongue diagnosis descriptions in JSON format based on the input tongue image, achieving automation, objectivity, and interpretability in tongue diagnosis analysis. Compared with traditional tongue diagnosis methods that rely on experience-based judgment, this invention achieves automatic extraction, quantitative analysis, and unified expression of tongue features through a large visual language model. This not only significantly improves the objectivity and consistency of tongue diagnosis results but also presents the basis for tongue diagnosis in a structured manner, thereby enhancing the reliability and scalability of clinical applications. Attached Figure Description
[0021] The accompanying drawings of this invention are incorporated in and constitute a part of this specification, serving to illustrate embodiments of the invention and to explain the principles of the invention in conjunction with the specification.
[0022] Figure 1 A flowchart illustrating an intelligent analysis method for TCM tongue diagnosis based on a visual language model, provided in an embodiment of the present invention;
[0023] Figure 2 This is an overall flowchart of image quality screening provided in an embodiment of the present invention;
[0024] Figure 3 This is an overall flowchart of tongue region segmentation provided in an embodiment of the present invention;
[0025] Figure 4 This is a schematic diagram of the large-scale model of tongue diagnosis in traditional Chinese medicine according to the present invention. Detailed Implementation
[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be described in detail below with reference to the accompanying drawings. Obviously, the embodiments described are only some, not all, implementations of this invention. Other implementations obtained by those skilled in the art based on these embodiments without creative effort are also within the scope of protection of this invention.
[0027] The technical solution of the present invention and the method for solving the technical problem are described in detail below through specific embodiments. The embodiments of the present invention will be described below with reference to the accompanying drawings.
[0028] Figure 1 This is a flowchart of a TCM tongue diagnosis intelligent analysis method based on a visual language model provided in an embodiment of the present invention, such as... Figure 1 As shown in the figure, this embodiment illustrates an intelligent analysis method for TCM tongue diagnosis based on a visual language model, including:
[0029] S1: Obtain images of tongue appearance in Traditional Chinese Medicine and perform image quality screening to remove low-quality images such as blurry or abnormally exposed images.
[0030] For example, clinical tongue images are collected or tongue data is obtained through intelligent tongue diagnosis devices. During the collection process, the patient's tongue is ensured to be extended and flat. The collected data may include tongue images of different ages, genders, physical conditions, and types of tongue diseases to build a diverse and comprehensive tongue image database. The collected images are then screened for quality, and low-quality images that are blurry, abnormally exposed, or unevenly lit are removed to obtain high-quality tongue images.
[0031] Optional, specific methods for image quality screening include:
[0032] An image blur evaluation algorithm is used to detect the sharpness of tongue images, eliminating low-quality images that meet a preset blur threshold; an exposure analysis algorithm is used to evaluate image brightness and contrast, eliminating overexposed or underexposed images; and candidate images are preprocessed based on color equalization and illumination correction techniques to obtain tongue images of acceptable quality. Figure 2 As shown.
[0033] S2: Based on the tongue segmentation model, the tongue region in the tongue image is automatically extracted to obtain a standard tongue image.
[0034] For example, such as Figure 3 As shown, an open-source tongue segmentation model (such as TongueSAM) is used to automatically identify and segment the tongue region in the high-quality image filtered by S1. During the segmentation process, a tongue mask can be generated, and the tongue region is cropped to its minimum bounding rectangle. To ensure consistency in subsequent processing, if the cropped region is not square, black pixels can be used to fill the missing edges, ultimately obtaining a standardized square tongue image with a uniform size of 512×512, which is used as input to the visual language model.
[0035] S3: Utilize a large model to automatically identify and label the features of the tongue body and tongue coating, generate tongue diagnosis feature description text, and have TCM experts review and correct the automatic labeling results to construct a dataset that aligns tongue images with tongue diagnosis descriptions.
[0036] For example, standardized tongue images obtained through S2 are used to automatically identify key features such as tongue shape, color, coating color, and texture using a large model (e.g., Gpt-o3), generating corresponding tongue diagnosis text descriptions. Subsequently, the automatically generated tongue diagnosis descriptions are submitted to TCM clinical experts for review and correction, adjusting any incorrectly labeled, missing, or inconsistent features to form a high-quality tongue image-tongue diagnosis text aligned dataset. This dataset is used for model training and also serves as a standardized reference for subsequent clinical auxiliary analysis, significantly reducing the workload of manual annotation and improving the consistency and accuracy of annotations.
[0037] S4: Based on the tongue image-tongue diagnosis aligned dataset, the visual language model is trained, using the Qwen-VL model as the base model, and a fine-tuning mechanism guided by tongue diagnosis knowledge is introduced to enable the model to learn the mapping relationship between the visual features of the tongue body and tongue coating and the natural language description of tongue diagnosis.
[0038] For example, a high-quality tongue image-tongue diagnosis text alignment dataset constructed using S3 is used to pre-train and supervisedly fine-tune a visual language model. This allows the model to learn the mapping relationship between tongue images and tongue diagnosis descriptions, resulting in a large-scale TCM tongue diagnosis model capable of automatically generating structured descriptions of tongue diagnosis features based on tongue images. During the training and fine-tuning process, a Qwen-VL-based visual language model framework is employed to unify the tasks of tongue image understanding and tongue diagnosis generation. In the fine-tuning stage, TCM tongue diagnosis knowledge guidance is added to enhance the model's ability to recognize and generate key tongue diagnosis semantics such as tongue color, coating color, and coating texture, thereby improving the interpretability and clinical usability of the tongue diagnosis analysis results. Figure 4 As shown, a tongue image is input into the TCM tongue diagnosis model. Based on the tongue and tongue coating features identified by the TCM tongue diagnosis model, a structured result containing tongue diagnosis indicators such as tongue shape, tongue color, tongue coating color, and tongue coating texture is generated. A structured natural language tongue diagnosis description in JSON format is automatically generated to realize the tongue image analysis result.
[0039] S5: Input the tongue image into the trained visual language model, output a structured JSON result containing tongue diagnosis indicators such as tongue color, coating color, and coating texture, and automatically generate a natural language tongue diagnosis description corresponding to the structured result, thereby realizing the automated recognition and structured expression of tongue diagnosis features.
[0040] For example, such as Figure 4 As shown, by inputting a tongue image into the large-scale TCM tongue diagnosis model trained in step S4, the model can automatically generate a structured JSON output covering tongue diagnosis indicators such as tongue color, coating color, and coating texture, and further generate a consistent and standardized natural language tongue diagnosis description based on this structured content, thereby realizing the automation and structuring of tongue image feature recognition.
[0041] The above describes the implementation of the intelligent analysis method for TCM tongue diagnosis based on a visual language model as described in this specification. However, those skilled in the art can make various modifications and improvements without departing from the principle of this invention, all of which fall within the protection scope of this invention.
Claims
1. A method for intelligent analysis of tongue diagnosis in Traditional Chinese Medicine based on a visual language model, characterized in that, include: A tongue image-tongue diagnosis aligned dataset is constructed. Traditional Chinese medicine tongue images are obtained, and the tongue images are preprocessed, including tongue image quality screening and tongue body region segmentation. The tongue body and tongue coating features are automatically identified and labeled using a large language model. The automatic labeling results are then reviewed and corrected by traditional Chinese medicine experts to form a corresponding dataset of tongue image images and structured descriptions of tongue diagnosis. Based on the tongue image-tongue diagnosis aligned dataset, a visual language model is trained, using the Qwen-VL model as the base model, and a fine-tuning mechanism guided by tongue diagnosis knowledge is introduced to enable the model to learn the mapping relationship between the visual features of the tongue body and tongue coating and the natural language description of tongue diagnosis. The tongue image is input into a trained visual language model, which outputs a structured JSON result containing tongue diagnosis indicators such as tongue color, coating color, and coating texture. Natural language tongue diagnosis descriptions corresponding to the structured result are automatically generated, realizing the automated recognition and structured expression of tongue diagnosis features.
2. The intelligent analysis method for TCM tongue diagnosis based on a visual language model as described in claim 1, characterized in that, The tongue image quality screening specifically includes: The clarity of tongue images is detected using an image blur evaluation algorithm, and low-quality images that meet the preset blur threshold are eliminated. The brightness and contrast of images are evaluated using an exposure analysis algorithm, and overexposed or underexposed images are eliminated. Finally, candidate images are preprocessed based on color balance and illumination correction techniques to obtain qualified tongue images.
3. The intelligent analysis method for TCM tongue diagnosis based on a visual language model as described in claim 1, characterized in that, The segmentation of the tongue region specifically includes: The open-source tongue segmentation model TongueSAM is used to automatically segment the tongue region in a tongue image. Based on the segmentation results, the minimum bounding rectangle containing the tongue region is generated and cropped into a square region. If the cropped region is not a square, the missing edge region is filled with black pixels to make the tongue region form a standardized square image. Finally, the square image is scaled to a preset uniform size of 512×512 for subsequent model training and analysis.
4. The intelligent analysis method for TCM tongue diagnosis based on a visual language model as described in claim 1, characterized in that, The process involves automatically identifying and labeling tongue and tongue coating features using a large language model, followed by review and correction of the automatic labeling results by TCM experts, resulting in a dataset corresponding to tongue images and structured descriptions of tongue diagnosis. Specifically, this includes: A large model (such as Gpt-o3) is used to extract features from the segmented tongue images and generate a structured text description of the tongue diagnosis, including tongue color, coating color, and coating texture. The automatically generated tongue diagnosis description is submitted to TCM clinical experts for manual review. Based on the review comments, erroneous, missing, or inconsistent features are revised and standardized labeled text is formed. Based on the standardized tongue diagnosis description after review and revision, an image-text aligned dataset is constructed for the training and optimization of the tongue diagnosis visual language model.
5. The intelligent analysis method for TCM tongue diagnosis based on a visual language model as described in claim 1, characterized in that, The tongue image-tongue diagnosis aligned dataset is used to train a visual language model. The Qwen-VL model is used as the base model, and a fine-tuning mechanism guided by tongue diagnosis knowledge is introduced. This enables the model to learn the mapping relationship between the visual features of the tongue body and tongue coating and the natural language description of tongue diagnosis, specifically including: The visual language model was pre-trained and supervised fine-tuned using the tongue image-tongue diagnosis text alignment dataset. This enabled the model to learn the mapping relationship between tongue images and tongue diagnosis descriptions, resulting in a large-scale TCM tongue diagnosis model capable of automatically generating structured descriptions of tongue diagnosis features based on tongue images. During the training and fine-tuning process, a Qwen-VL-based visual language model framework was employed to unify the tasks of tongue image understanding and tongue diagnosis generation. TCM tongue diagnosis knowledge guidance was added during the fine-tuning stage to enhance the model's ability to recognize and generate key tongue diagnosis semantics such as tongue color, coating color, and coating texture. A structured output mechanism was used to achieve natural language interpretable descriptions of tongue diagnosis features in JSON format, thereby improving the interpretability and clinical usability of the tongue diagnosis analysis results.
6. The intelligent analysis method for TCM tongue diagnosis based on a visual language model as described in claim 1, characterized in that, The process involves inputting a tongue image into a trained visual-language model, outputting a structured JSON result containing tongue diagnosis indicators such as tongue color, coating color, and coating texture, and automatically generating a natural language tongue diagnosis description corresponding to the structured result. Specifically, this includes: The preprocessed tongue image is input into the large-scale tongue diagnosis model. Based on the tongue body and tongue coating features identified by the large-scale tongue diagnosis model, a structured result containing key tongue diagnosis indicators such as tongue color, coating color, and coating texture is automatically generated. The corresponding natural language tongue diagnosis analysis description is output, realizing the interpretable and automated output of tongue diagnosis results.