A multi-modal traditional Chinese medicine comprehensive diagnosis system

Abstract

The present application relates to the technical field of traditional Chinese medicine intelligent diagnosis, and discloses a multi-modal traditional Chinese medicine comprehensive diagnosis system, comprising a four-examination information acquisition module, an edge calculation module, a cloud analysis module and a diagnosis result output module. The system can not only realize the synchronous acquisition and standardized processing of tongue diagnosis, face diagnosis, pulse diagnosis, voice and inquiry information, reduce the influence of human factors on the diagnosis result, but also introduce the collaborative analysis mechanism of traditional Chinese medicine knowledge graph and multi-modal large model, break through the limitation of traditional Chinese medicine intelligent diagnosis system which only relies on rule matching or shallow classification, significantly enhance the knowledge reasoning ability and comprehensive judgment ability of the system, and output structured and interpretable diagnosis results, provide reliable auxiliary decision support for traditional Chinese medicine clinical diagnosis and treatment, and improve the objectivity, consistency and practicality of traditional Chinese medicine diagnosis.

Description

Technical Field

[0001] This invention relates to the field of intelligent diagnostic technology in traditional Chinese medicine, and in particular to a multimodal comprehensive diagnostic system for traditional Chinese medicine. Background Technology

[0002] Traditional Chinese medicine (TCM) diagnosis and treatment relies on the four diagnostic methods of "inspection, auscultation and olfaction, inquiry, and palpation" as its core approach. It emphasizes a comprehensive assessment of the body's organ function and pathological changes through observation of external manifestations combined with inquiry and palpation. However, in traditional TCM diagnosis and treatment, the information obtained through these four methods largely depends on the physician's personal experience, making the diagnostic results susceptible to the influence of the physician's experience level, environmental conditions, and subjective judgment.

[0003] Specifically, in tongue diagnosis, the characteristics of the tongue, such as its color, shape, coating color, thickness, moisture, and sublingual veins, are usually observed visually, lacking standardized imaging conditions and quantitative standards. In facial diagnosis, changes in complexion and color in local areas such as the glabella, cheeks, and nose rely heavily on experience, making objective recording and long-term comparison difficult. In auscultation, changes in tone, volume, clarity, and speed of speech are difficult to quantitatively collect using traditional methods. In palpation, changes in pulse depth, speed, strength, and waveform depend heavily on the physician's tactile experience, making standardized preservation and reproduction difficult.

[0004] With the development of sensor and computer vision technologies, TCM auxiliary diagnostic devices such as tongue and pulse image acquisition devices have emerged. However, these devices are mostly designed for a single diagnostic method, and data from tongue, face, and pulse diagnosis are typically collected separately. The devices lack a unified structural layout and data interfaces, making it impossible to collaboratively collect information from multiple diagnostic methods within the same testing process. Furthermore, existing TCM intelligent diagnostic systems often employ simple feature matching or rule-based judgment methods, which are insufficiently supportive of the complex relationships between TCM syndromes. This makes it difficult to fully utilize information from multiple diagnostic methods for comprehensive analysis, resulting in a need to improve the accuracy and stability of diagnostic results. Summary of the Invention

[0005] This invention provides a multimodal TCM comprehensive diagnostic system, which solves at least one of the technical problems of existing TCM diagnostic processes, such as reliance on physicians' personal experience, scattered collection of information from the four diagnostic methods, low degree of data standardization and the inability to perform only simple syndrome matching, and lack of deep knowledge reasoning ability.

[0006] To achieve the above objectives, this invention provides a multimodal TCM comprehensive diagnostic system. Guided by the TCM theories of "inspection, auscultation, inquiry, and palpation," the system simultaneously acquires multi-source diagnostic information, including tongue diagnosis, facial diagnosis, voice, pulse diagnosis, and inquiry, through a unified hardware acquisition platform. Relying on a system architecture that combines edge computing and cloud analysis, the system performs layered processing and in-depth fusion analysis on the acquired data, thereby generating auxiliary diagnostic results and treatment suggestions that conform to the TCM theoretical system.

[0007] The multimodal TCM comprehensive diagnostic system provided in this invention includes a four-diagnostic-information acquisition module, an edge computing module, a cloud analysis module, and a diagnostic result output module. The four diagnostic methods information acquisition module is used to collect multi-source data, including tongue diagnosis, facial diagnosis, voice, questioning and pulse diagnosis data. The edge computing module is used to preprocess, perform quality control and feature extraction on the multi-source data, and encapsulate it into standardized multimodal feature data. The cloud-based analysis module is used to perform syndrome reasoning on the multimodal feature data based on a preset TCM knowledge graph and a preset TCM big language model, and to generate auxiliary diagnostic results. The diagnostic result output module is used to generate a structured TCM diagnostic report and output it to the terminal device.

[0008] The beneficial effects of this invention are as follows: This invention provides a multimodal TCM comprehensive diagnostic system. By constructing a unified multimodal TCM comprehensive diagnostic system, it achieves synchronous collection and standardized processing of tongue diagnosis, facial diagnosis, pulse diagnosis, voice, and consultation information, reducing the impact of human factors on diagnostic results. By introducing a collaborative analysis mechanism of TCM knowledge graph and multimodal large model, it breaks through the limitations of traditional TCM intelligent diagnostic systems that rely solely on rule matching or shallow classification, significantly enhancing the system's knowledge reasoning ability and comprehensive judgment ability. By outputting structured and interpretable diagnostic results, it provides reliable auxiliary decision support for TCM clinical diagnosis and treatment, improving the objectivity, consistency, and practicality of TCM diagnosis.

[0009] To make the above-mentioned objects, features and advantages of the invention more apparent and understandable, preferred embodiments of the invention are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1This is a user interaction diagram of the multimodal TCM comprehensive diagnostic system provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of the multimodal TCM comprehensive diagnostic system according to an embodiment of the present invention. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0013] It should be noted that, unless otherwise specified, the various features in the embodiments of this invention can be combined with each other, all of which are within the protection scope of this invention. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the device or the order in the flowchart. Moreover, the terms "first," "second," and "third" used in this invention do not limit the data or execution order, but only distinguish identical or similar items with essentially the same function and effect.

[0014] Figure 1 This is a user interaction diagram of the multimodal TCM comprehensive diagnostic system provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of a multimodal TCM comprehensive diagnostic system, such as... Figure 1 , Figure 2 As shown, it includes a four-diagnostic-information acquisition module 100, an edge computing module 200, a cloud analysis module 300, and a diagnostic result output module 400. The four diagnostic methods information acquisition module 100 is used to collect multi-source data, including tongue diagnosis, facial diagnosis, voice, questioning and pulse diagnosis data. The edge computing module 200 is used to preprocess, perform quality control and feature extraction on the multi-source data, and encapsulate it into standardized multimodal feature data; The cloud-based analysis module 300 is used to perform syndrome reasoning on the multimodal feature data based on a preset TCM knowledge graph and a preset TCM big language model, and to generate auxiliary diagnostic results. The diagnostic result output module 400 is used to generate a structured TCM diagnostic report and output it to the terminal device.

[0015] The above embodiments provide a multimodal TCM comprehensive diagnostic system. It synchronously acquires multi-source diagnostic information, including tongue diagnosis, facial diagnosis, voice, pulse diagnosis, and patient inquiry, through a unified hardware acquisition platform. Relying on a system architecture combining edge computing and cloud analytics, it performs edge preprocessing, feature extraction, cloud knowledge graph retrieval, and multimodal large-scale model inference on the acquired data to generate structured TCM syndrome differentiation and diagnosis reports and personalized treatment / conditioning suggestions. For example, the modules exchange standardized JSON-formatted feature vectors and metadata via secure communication protocols (such as HTTPS). The specific user interaction process and overall system architecture are as follows: Figure 1 , Figure 2 As shown, the relevant modules are first given an overall description in this manual, and then explained in detail below with the help of formulas and principles.

[0016] For example, in a preferred embodiment, the four diagnostic methods information acquisition module 100 includes a pose adjustment unit, a parameter adjustment unit, and an information acquisition unit. The posture adjustment unit is used to adjust the position and angle of the preset fixed component to adjust the patient's current posture to the target posture; The parameter adjustment unit is used to adjust the parameters of each acquisition device according to the current operating environment. The information acquisition unit is used to acquire images of the patient's tongue and face through an image acquisition device, acquire the patient's pulse wave signal through a finger clip or wristband pulse sensor, and acquire the patient's standard voice sample and structured consultation data through a voice acquisition device. The consultation data includes the patient's basic information, chief complaint symptoms, medical history and / or relevant physical signs.

[0017] In the specific deployment process, the hardware acquisition end is equipped with an adjustable head positioning bracket, chin rest, and camera lighting components, facilitating the simultaneous acquisition of tongue, facial, pulse, and voice information with a single body position adjustment. The camera uses a high-resolution color camera (example resolution 2160×3840), coupled with ring LED lighting, to ensure the accurate reproduction of tongue and facial color and texture information. The pulse diagnosis sensing unit uses a finger-clip photoplethysmography sensor (dual-band: red light 660nm and infrared 940nm), transmitting the raw pulse wave (PPG) to the edge computing module in real time via USB or serial port at a fixed sampling rate (example 100Hz). Voice acquisition uses a standard microphone array and performs environmental noise detection before acquisition to ensure signal quality, thereby extracting tongue, facial, voice, and pulse features based on the acquired information.

[0018] In a preferred embodiment, the edge computing module 200 includes a tongue diagnosis feature extraction unit. Exemplarily, the tongue diagnosis features include one or more of the following: tongue color, tongue shape, tongue coating color, tongue coating texture, tongue posture, tongue condition (whether it is healthy or dry), and sublingual vein features. The tongue diagnosis feature extraction unit includes at least: The first segmentation unit is used to preprocess the patient's tongue surface image and perform tongue body localization and tongue coating segmentation based on a preset deep learning model; The first feature extraction unit is used to extract the current color feature vector of the tongue color or tongue coating color in a unified color space, calculate the similarity with the reference tongue color or reference tongue coating color, and take the reference color with the highest similarity as the target tongue color or target tongue coating color. The second feature extraction unit is used to extract tongue contour parameters and generate tongue shape features using morphological methods, and to extract tongue coating texture using texture analysis methods, and to calculate the pixel coverage, thickness estimation index and / or smoothness of the coating area, and generate coating quality and tongue health status features.

[0019] Specifically, the acquired tongue surface color image is first preprocessed in the edge calculation module, including white balance correction, artifact removal (such as saliva highlights), and gamma correction. To ensure consistency in color discrimination under different devices and light sources, the preferred embodiment adopts a standard color space transformation and linearization process from sRGB to CIE XYZ. The standardization steps can refer to the sRGB linearization and XYZ transformation formula: first, normalize R, G, B to [0,1] and perform inverse gamma transformation to obtain r′, g′, b′, then perform matrix transformation to obtain X, Y, Z, thereby calculating tongue color features under a unified color gamut.

[0020] For example, a lightweight YOLOv8 detection model or its variants can be used for tongue detection and ROI localization. A detection confidence threshold is used to filter high-quality image samples for constructing a training set. Then, for the pixel set obtained after tongue segmentation, the following main features are extracted and quantified: tongue color (e.g., pale red, pale white, red, crimson, bluish-purple), tongue shape (large, thin, teeth marks, cracks, old / young tongue, etc.), tongue posture (normal, weak, stiff, crooked, trembling, etc.), tongue coating color (white, yellow, grayish-black), coating texture (thin, thick, greasy, peeled, rotten, etc.), and sublingual veins (varicose, thin, purplish, etc.). During model training, keyword-driven web crawlers can be used to obtain tongue images, and a pre-trained YOLO model can be used to filter non-tongue images or low-quality samples, forming a labeled dataset for subsequent model training.

[0021] Tongue color quantification employs a method of statistically analyzing the channel mean, variance, and distribution histogram of pixels in a unified color space (e.g., CIE-Lab or YCrCb) to form a current color feature vector. Then, the cosine similarity or Euclidean distance between the current color feature vector and the reference tongue body color or reference tongue coating color is calculated. The reference color with the highest similarity is selected as the target tongue color or target tongue coating color. The specific formula for calculating cosine similarity is as follows: , in This similarity score is used for semantic-level near-synonym matching and symptom mapping in knowledge graph retrieval or classifier post-processing stages.

[0022] For example, the quantification of tongue shape and tongue coating employs morphological and textural analysis methods. Geometric quantities such as the aspect ratio, perimeter, area, convexity, and edge curvature of the tongue outline are used to determine its thickness, teeth marks, cracks, etc. The texture of the tongue coating can be extracted using Gabor wavelets or Local Binary Patterns (LBP), and the pixel coverage and thickness estimation indices of the coating area are calculated, such as the contrast between the grayscale or brightness distribution of the coating area and the tongue base. For the quantification of smoothness, the embodiment uses the following difference product formula to evaluate the difference between pixels and their neighborhoods and calculate the smoothness index D:

[0023] Where f(x,y) is the gray value of pixel (x,y), this index is sensitive to the difference between roughness and smoothness and can be used to judge the dryness or wetness of the tongue coating and the luster of the tongue body (tongue gloss). The above features can be mapped to clinical labels and used for training a tongue diagnosis classifier.

[0024] For example, in a preferred embodiment, the edge computing module 200 further includes a facial diagnosis feature extraction unit, which includes at least: The second segmentation unit is used to segment the diagnostic region of the patient's facial image and label the segmentation results; The third feature extraction unit is used to convert the patient's facial image to several color spaces, establish the color feature vector corresponding to each diagnostic region in each color space, calculate the distance between each diagnostic region and different reference face colors in each color space, and generate the total distance between each diagnostic region and each reference face color by weighted summation to determine the closest target face color for each diagnostic region, thereby generating facial features.

[0025] Specifically, when processing facial images, the edge computing module first performs face detection and key point localization. For example, it locates key areas such as the forehead, cheeks, nose, and jaw based on a facial key point model. Then, it calculates the channel mean, standard deviation, and color histogram for each local area in multiple color spaces, such as RGB, HSV, YCrCb, and Lab, forming a local color feature vector. To achieve the classification of facial colors into red, white, yellow, and black according to traditional Chinese medicine, it calculates the Euclidean distance between the features of the area to be tested and the four baseline facial color feature vectors in multiple color spaces, and then performs a weighted summation to determine the closest facial color category. That is, it sums the distances in each color space to obtain the total distance D and selects the facial color category corresponding to the minimum distance. Thus, the color distribution of each facial region and its correspondence with the internal organs in traditional Chinese medicine are input into a knowledge graph for semantic-level syndrome matching.

[0026] For example, in a preferred embodiment, the edge computing module 200 further includes a speech feature extraction unit, which includes at least: The speech processing unit is used to perform environmental noise detection and speech activity detection on the standard speech samples, and to extract time-frequency features after framing and window function processing of the effective speech segments; The fourth feature extraction unit is used to generate corresponding tone, timbre and speech rate features based on the numerical range of the time-frequency features, and to calculate the loudness using a preset formula.

[0027] Specifically, firstly, standard speech samples or natural speech samples of the subjects reading a preset text are collected. Then, environmental noise detection and voice activity detection (VAD) are performed. After framing and window function processing of the effective speech segments, time-frequency features are extracted, including fundamental frequency (F0), mean / variance of volume, spectral envelope, zero-crossing rate, Mel-frequency cepstral coefficients (MFCC), and speech rate (number of words / syllables per unit time). Among these, fundamental frequency, mean / variance of volume, and speech rate correspond to tone, timbre, and speech rate in traditional Chinese medicine auscultation. Combined with the loudness index and consultation information, these can be mapped to sound diagnosis indicators such as qi deficiency, excess heat, and phlegm dampness. For example, to obtain the "loudness" index, audio sampling points are... Calculate energy-based metrics (example formula): , in is the instantaneous sound pressure at the i-th audio sampling point, L is the loudness value, k is the preset weighting coefficient, N is the number of sampling points, and P is the reference sound pressure, such as 20 μPa. This "loudness" index is combined with individualized baselines such as age / gender to infer the clinical significance of weak or loud voice.

[0028] For example, in a preferred embodiment, the edge computing module 200 further includes a pulse feature extraction unit. First, the acquired pulse waveform undergoes baseline removal, bandpass filtering, and artifact (motion) suppression at the edges. Then, a sliding window peak detection algorithm is used to locate the pulse peaks, thereby obtaining the time interval (i.e., the RR interval sequence) and peak amplitude sequence between two adjacent pulse peaks. From this, the following can be calculated: (1) The pulse rate per minute, used to characterize the beat, is calculated using the following formula: , in, Pulse rate per minute For time window, for Detected within seconds Each pulse peak.

[0029] (2) The root mean square of the time interval sequence, used to characterize the pulse rhythm, is calculated using the following formula: , in, This indicates the time interval between two adjacent pulse peaks. The sequence length is given.

[0030] (3) The average peak pulse value, used to characterize pulse strength, is calculated using the following formula: , in, Indicates the detected number Each pulse peak.

[0031] (4) The upward slope of the ascending branch is used to characterize the changes in tension and / or fluency. The calculation formula is as follows: , in, for The amplitude value of the rising branch waveform at time 1. for The amplitude value of the rising branch waveform corresponding to the given moment.

[0032] The above feature set can quantify the dimensions described in traditional Chinese medicine, such as "pulse number, pulse rhythm, pulse strength, fluency, and tension". The system uses these numerical features as structured inputs to feed them into a lightweight feedforward neural network (MLP) classifier to output pulse categories, such as normal pulse, floating pulse, deep pulse, slow pulse, rapid pulse, weak pulse, and strong pulse. An example of the MLP structure is: input layer (4-dimensional features) → fully connected (64) → ReLU → Dropout (20%) → fully connected (32) → ReLU → Dropout (20%) → output (7) → Softmax.

[0033] For example, in a preferred embodiment, the edge computing module 200 can be located in the integrated data acquisition device or a local computing device connected to it. This allows it to perform image quality screening, noise suppression, region localization, and data normalization locally. It also allows for feature extraction of each modality's data locally, encapsulating it into a unified format, such as a standardized JSON feature vector, and uploading it to the cloud analysis module 300. The JSON feature vector includes at least patient ID, timestamp, modality type, feature key-value pairs, and a quality score, which is used for weighting during multimodal fusion in the cloud. In other embodiments, based on diagnostic needs or the hardware environment, it can also determine whether to perform lightweight inference, including preliminary classification results of tongue, facial, and / or pulse images, thereby sending the lightweight inference results to the cloud analysis module, reducing cloud computing pressure and improving the overall system response efficiency.

[0034] For example, in a preferred embodiment, the cloud analytics module 300 includes: Model building unit 301 is used to construct a Traditional Chinese Medicine (TCM) knowledge graph and a TCM large-scale language model. The TCM knowledge graph draws on authoritative TCM classics such as the *Huangdi Neijing* and *Shanghan Lun*, medical case studies by renowned physicians, a TCM standard terminology database, and related academic materials. It stores the relationships between TCM syndromes, symptoms, prescriptions, treatments, and tongue, facial, vocal, and pulse characteristics. Through unified encoding of TCM terminology, standardization of symptom descriptions, and modeling of syndrome relationships, it achieves a systematic expression of TCM knowledge and supports multi-hop reasoning and semantic retrieval based on entity relationships, providing knowledge support for in-depth analysis of multimodal diagnostic information. The TCM large-scale language model adopts a general language model architecture and is adapted and trained using TCM domain corpora. It can understand and synthesize the intrinsic connections between unstructured information such as tongue, pulse, vocal, and facial features and TCM theoretical knowledge. During diagnosis, it not only provides diagnostic conclusions but also simultaneously generates supporting evidence and auxiliary treatment suggestions, improving the interpretability and clinical reference value of diagnostic results.

[0035] The feature fusion unit 302 is used to obtain weighting coefficients based on the quality score, and to perform cross-modal weighted calculations after mapping the multimodal feature data to a unified dimension based on the attention mechanism to generate a fused feature vector. The knowledge retrieval unit 303 is used to calculate the similarity between the modality type label of each modality and the embedding vector of the TCM knowledge graph, and to perform multi-hop relationship expansion to generate syndrome / prescription options; The reasoning unit 304 is used to obtain the fused feature vector and the syndrome / prescription options, and call the TCM big language model to perform semantic reasoning based on the retrieval enhancement generation strategy to generate structured diagnostic results.

[0036] Specifically, the model building unit 301 includes a first building unit 3011 and a second building unit 3012. The first construction unit 3011 is used to construct a TCM knowledge graph dataset, which contains several triplet data, covering various entity and relation types such as "symptom-medicine", "syndrome-symptom", "medicine-dosage" and "medicine-property". The second construction unit 3012 is used to model and visualize the triplet data using the Neo4j graph database, and import entities and corresponding relationships into the graph database through the Cypher query language, so that each entity node can be connected with other semantically related nodes through edges, and an embedded vector index is established to support semantic retrieval, including vectorized embedding, approximate nearest neighbor retrieval and cosine similarity matching, etc. In this way, structured relationship retrieval and multi-hop reasoning are realized through knowledge graph import and visualization and Cypher query, thus building a bridge between multimodal features and traditional Chinese medicine theory.

[0037] Multimodal fusion and reasoning adopts a two-stage strategy. The first stage is based on the attention mechanism to achieve feature-level fusion, that is, after mapping the features of tongue, face, pulse, voice and consultation to a unified dimension, the weighted representation is calculated by cross-modal attention layer. Then, the fused feature vector and the syndrome / prescription options returned by the TCM knowledge graph are input into the multimodal big model for semantic reasoning and natural language generation.

[0038] In a preferred embodiment, to improve the applicability of the large model in the field of Traditional Chinese Medicine (TCM), a specialized TCM corpus can be used for fine-tuning. For example, LoRA can be used to adapt the large model to the domain, or a Prompt project can be used to annotate the prompt templates with TCM expert annotations. Exemplarily, the system also includes a model optimization unit for building a training set and replacing some linear layer parameters of the basic large model, such as the DeepSeek model, with low-rank matrix combinations using LoRA. The training set is then used to train the low-rank matrix combinations to enhance the model's semantic understanding and generation capabilities for TCM terminology, symptom descriptions, and / or pulse characteristics. The training set contains several TCM question-and-answer pairs, covering TCM syndrome types, treatment suggestions, and prescription analysis.

[0039] In other preferred embodiments, when selecting a large model, multiple domestic / open source large models can be compared and scored horizontally using an automated evaluation framework to select the best inference engine. To ensure the long-term availability of the model, it is recommended to establish a model version management, data feedback mechanism and a regular retraining strategy.

[0040] For example, the cloud-based large model incorporates a Retrieval Enhanced Generation (RAG) strategy during inference, using retrieved knowledge graph subgraphs and typical cases as contextual prompts to enhance the interpretability and factuality of the reasoning. In the knowledge retrieval and syndrome matching stages, the system calculates the similarity between the classification labels of each modality and the embedding vectors of knowledge graph entities (e.g., using cosine similarity), and performs multi-hop relationship expansion to discover possible syndrome linkages (e.g., rules like "red tongue + yellow greasy coating + wiry pulse → liver and gallbladder damp-heat syndrome"). Based on these candidate syndromes, the large model further generates structured diagnostic results, such as JSON fields including pathogenesis analysis, syndrome type judgment, recommended prescriptions, dietary / lifestyle suggestions, and medical advice, and outputs a diagnostic credibility score and supporting evidence, i.e., supported tongue / facial / pulse features and knowledge graph retrieval paths, for physician review and secondary judgment.

[0041] In a preferred embodiment, the system also employs engineering measures to improve reliability and compliance during model training and fine-tuning. Specifically, tongue and facial image data undergo medical expert annotation and quality review before training, and pulse data is collected from large-scale samples across different ages, genders, and body positions to enhance generalization. The training process utilizes cross-entropy loss, the Adam optimizer (with an initial learning rate of 0.001), L2 regularization, and Dropout to reduce the risk of overfitting. Both the pulse diagnosis MLP and the tongue / facial image model have their precision / recall monitored on independent validation sets, and clinical controlled trials are completed before deployment to evaluate their diagnostic value.

[0042] In a further preferred embodiment, the diagnostic result output module 400 is used to convert the structured diagnostic results into a visual report and / or input them into an electronic medical record, and to provide a review and / or editing interface according to user permissions; the visual report includes patient basic information, acquisition time, device ID, original images / waveforms and feature inference results of each modality, diagnostic results, main and secondary syndrome types, diagnostic basis, diagnostic confidence level, risk warnings, and treatment plans (recommended prescriptions or conditioning suggestions), etc., thereby facilitating physician review and long-term management.

[0043] This invention constructs a unified multimodal TCM comprehensive diagnostic system to achieve synchronous collection and standardized processing of information from tongue diagnosis, facial diagnosis, auscultation and olfaction, pulse diagnosis, and medical history taking, thereby reducing the impact of human factors on diagnostic results. By introducing a collaborative analysis mechanism of TCM knowledge graph and multimodal large model, it breaks through the limitations of traditional TCM intelligent diagnostic systems that rely solely on rule matching or shallow classification, significantly enhancing the system's knowledge reasoning and comprehensive judgment capabilities. By outputting structured and interpretable diagnostic results, it provides reliable auxiliary decision support for TCM clinical diagnosis and treatment, thereby improving the objectivity, consistency, and practicality of TCM diagnosis.

[0044] The parameters, models, and thresholds listed in the above embodiments are merely exemplary implementations. Those skilled in the art can make improvements and changes without departing from the technical solution of this invention, and the improved solutions should still fall within the protection scope of this invention.

[0045] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0046] Those skilled in the art will recognize that the units of the various examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of the invention.

[0047] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / terminal device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0048] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0049] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0050] The present invention is not limited to the description in the specification and embodiments, and thus other advantages and modifications can be readily realized by those skilled in the art. Therefore, the present invention is not limited to the specific details, representative devices and illustrated examples shown and described herein without departing from the spirit and scope of the general concept as defined by the claims and their equivalents.

Claims

1. A multimodal TCM comprehensive diagnostic system, characterized in that, It includes a four-diagnostic-information collection module, an edge computing module, a cloud analysis module, and a diagnostic result output module. The four diagnostic methods information acquisition module is used to collect multi-source data, including tongue diagnosis, facial diagnosis, voice, questioning and pulse diagnosis data. The edge computing module is used to preprocess, perform quality control and feature extraction on the multi-source data, and encapsulate it into standardized multimodal feature data. The cloud-based analysis module is used to perform syndrome reasoning on the multimodal feature data based on a preset TCM knowledge graph and a preset TCM big language model, and to generate auxiliary diagnostic results. The diagnostic result output module is used to generate a structured TCM diagnostic report and output it to the terminal device.

2. The multimodal TCM comprehensive diagnostic system according to claim 1, characterized in that, The four diagnostic methods information acquisition module includes a pose adjustment unit, a parameter adjustment unit, and an information acquisition unit. The posture adjustment unit is used to adjust the position and angle of the preset fixed component to adjust the patient's current posture to the target posture; The parameter adjustment unit is used to adjust the parameters of each acquisition device according to the current operating environment. The information acquisition unit is used to acquire images of the patient's tongue and face through an image acquisition device, acquire the patient's pulse wave signal through a finger clip or wristband pulse sensor, and acquire the patient's standard voice sample and structured consultation data through a voice acquisition device. The consultation data includes the patient's basic information, chief complaint symptoms, medical history and / or relevant physical signs.

3. The multimodal TCM comprehensive diagnostic system according to claim 1, characterized in that, The edge computing module includes a tongue diagnosis feature extraction unit, wherein the tongue diagnosis features include one or more of the following: tongue color, tongue shape, tongue coating color, tongue coating texture, tongue posture, tongue condition, and sublingual vein features; the tongue diagnosis feature extraction unit includes at least: The first segmentation unit is used to preprocess the patient's tongue surface image and perform tongue body localization and tongue coating segmentation based on a preset deep learning model; The first feature extraction unit is used to extract the current color feature vector of the tongue color or tongue coating color in a unified color space, calculate the similarity with the reference tongue color or reference tongue coating color, and take the reference color with the highest similarity as the target tongue color or target tongue coating color. The second feature extraction unit is used to extract tongue contour parameters and generate tongue shape features using morphological methods, and to extract tongue coating texture using texture analysis methods, and to calculate the pixel coverage, thickness estimation index and / or smoothness of the coating area, and generate coating quality and tongue health status features. The tongue contour parameters include one or more of the following: aspect ratio, perimeter, area, convexity, and edge curvature.

4. The multimodal TCM comprehensive diagnostic system according to claim 1, characterized in that, The edge computing module further includes a facial diagnosis feature extraction unit, which includes at least: The second segmentation unit is used to segment the diagnostic region of the patient's facial image and label the segmentation results; The third feature extraction unit is used to convert the patient's facial image to several color spaces, establish the color feature vector corresponding to each diagnostic region in each color space, calculate the distance between each diagnostic region and different reference face colors in each color space, and generate the total distance between each diagnostic region and each reference face color by weighted summation to determine the closest target face color for each diagnostic region, thereby generating facial features.

5. The multimodal TCM comprehensive diagnostic system according to claim 1, characterized in that, The edge computing module further includes a speech feature extraction unit, which includes at least: The speech processing unit is used to perform environmental noise detection and speech activity detection on the standard speech samples, and to extract time-frequency features after framing and window function processing of the effective speech segments; The fourth feature extraction unit is used to generate corresponding tone, timbre, and speech rate features based on the numerical range of the time-frequency features, and to calculate the loudness using a preset formula, which is as follows: ， Where L is the loudness, k is the preset weighting coefficient, N is the number of sampling points, and P is the reference sound pressure level. Let be the instantaneous sound pressure at the i-th audio sampling point.

6. The multimodal TCM comprehensive diagnostic system according to any one of claims 1-5, characterized in that, The edge computing module extracts features from the data of each modality locally and encapsulates them into standardized JSON feature vectors. It also determines whether to perform lightweight inference based on diagnostic needs or hardware environment, and sends the JSON feature vectors and / or lightweight inference results to the cloud analysis module. The JSON feature vector contains at least the patient ID, timestamp, modality type, feature key-value pairs, and quality score; the lightweight inference results include preliminary classification results of tongue appearance, facial appearance, and / or pulse appearance.

7. The multimodal TCM comprehensive diagnostic system according to claim 6, characterized in that, The cloud-based analytics module includes: Model building unit, used to build TCM knowledge graph and TCM big language model; The feature fusion unit is used to obtain weighting coefficients based on the quality score, and to perform cross-modal weighted calculations after mapping the multimodal feature data to a unified dimension based on an attention mechanism to generate a fused feature vector. The knowledge retrieval unit is used to calculate the similarity between the modality type label of each modality and the embedding vector of the TCM knowledge graph, and to perform multi-hop relationship expansion to generate syndrome / prescription options; The reasoning unit is used to obtain the fused feature vector and the syndrome / prescription options, and call the TCM big language model to perform semantic reasoning based on the retrieval enhancement generation strategy to generate structured diagnostic results.

8. The multimodal TCM comprehensive diagnostic system according to claim 7, characterized in that, The model building unit specifically includes: The first construction unit is used to construct a TCM knowledge graph dataset. The dataset contains several triplet data, covering various entity and relation types such as "symptom-medicine", "syndrome-symptom", "medicine-dosage" and "medicine-property". The second building unit is used to model and visualize the triple data using the Neo4j graph database, and to import entities and corresponding relationships into the graph database using the Cypher query language, so that each entity node can establish a connection with other semantically related nodes through edges.

9. The multimodal TCM comprehensive diagnostic system according to claim 7, characterized in that, The model building unit also includes: The model optimization unit is used to build a training set and replace some linear layer parameters of the basic DeepSeek model with low-rank matrix combinations using LoRA. The low-rank matrix combinations are then trained using the training set to enhance the model's semantic understanding and generation capabilities for TCM terminology, symptom descriptions, and / or pulse characteristics. The training set contains several TCM question-and-answer pairs, covering TCM syndrome types, treatment suggestions, and prescription analysis.

10. The multimodal TCM comprehensive diagnostic system according to claim 7, characterized in that, The diagnostic result output module is used to convert structured diagnostic results into visual reports and / or input them into electronic medical records, and to provide review and / or editing interfaces according to user permissions; The visualization report includes the acquisition time, device ID, original images / waveforms and feature inference results for each modality, diagnostic results, diagnostic basis, diagnostic confidence level, and treatment plan.

Images