Traditional Chinese medicine intelligent four-diagnosis auxiliary diagnosis system

Abstract

The invention discloses a traditional Chinese medicine intelligent four-diagnosis auxiliary diagnosis system, and relates to the technical field of medical equipment, and the system comprises a patient feature collection module which is used for collecting features of different modes of a patient; the plurality of single-mode diagnosis modules are respectively used for diagnosing the characteristics of different modes to obtain corresponding preliminary diagnosis results; and the multi-modal fusion reasoning module fuses the preliminary diagnosis results and performs reasoning according to the fusion result to obtain a target diagnosis result of the patient. According to the method, the feature diagnosis modules oriented to different modes are constructed respectively, independent extraction and preliminary analysis of the features of each diagnosis method are achieved, and on the basis, deep fusion and joint reasoning are performed on the features of different modes obtained by the multi-source heterogeneous diagnosis method by using the multi-mode fusion reasoning module, so that the multi-source heterogeneous diagnosis method is obtained. Therefore, intelligent dialectical analysis of'four diagnosis methods' is realized, and the diagnosis accuracy is remarkably improved.

Description

Technical Field

[0001] This application relates to the field of medical device technology, and in particular to a traditional Chinese medicine intelligent four diagnostic auxiliary system. Background Technology

[0002] Traditional Chinese medicine (TCM) diagnosis primarily relies on the four diagnostic methods of "inspection, auscultation and olfaction, inquiry, and palpation." However, this approach suffers from several significant problems in practice: First, the diagnostic process is highly subjective; different physicians may arrive at different conclusions regarding the same patient's diagnostic characteristics, leading to substantial discrepancies in diagnostic results. Second, the lack of a standardized evaluation system hinders the quantification and objective recording of diagnostic features, affecting the consistency and repeatability of diagnoses. Third, the efficiency of diagnosis and treatment is low; experienced physicians require extensive training, and the diagnostic process for each patient is time-consuming, failing to meet the efficiency requirements of modern medicine. These problems severely restrict the standardization and large-scale development of TCM diagnosis and treatment. Summary of the Invention

[0003] The main objective of this application is to propose a TCM intelligent four-diagnosis auxiliary diagnostic system to improve the accuracy of diagnosis.

[0004] To achieve the above objectives, this application proposes a traditional Chinese medicine intelligent four-diagnostic auxiliary system, the system comprising: The patient feature acquisition module is used to collect features of patients in different modalities. Multiple single-modal diagnostic modules are used to diagnose the features of different modalities and obtain corresponding preliminary diagnostic results; The multimodal fusion reasoning module fuses the preliminary diagnostic results from each of them and infers the patient's target diagnostic result based on the fusion result.

[0005] In some embodiments, the patient feature acquisition module includes a facial feature acquisition module, a tongue feature acquisition module, and a pulse feature acquisition module; The facial feature acquisition module is used to acquire the patient's facial features. The tongue image feature acquisition module is used to acquire the patient's tongue image features; The pulse feature acquisition module is used to acquire the pulse features of patients.

[0006] In some embodiments, the single-modal diagnostic module includes a facial diagnosis module, a tongue diagnosis module, and a pulse diagnosis module; The facial image diagnosis module is used to diagnose facial image results based on the facial image features; The tongue diagnosis module is used to diagnose tongue appearance based on the tongue appearance characteristics to obtain a tongue appearance diagnosis result; The pulse diagnosis module is used to diagnose pulse characteristics and obtain pulse diagnosis results.

[0007] In some embodiments, the facial feature acquisition module includes: The first image acquisition unit is used to acquire images of the patient's face; A region localization unit is used to locate the face region in the face image using a feature point-based deformation model; The feature-point-based deformation model is as follows: ; in, The total energy function measures the degree of matching between the model and the feature points. Represents the coordinates of the feature point. The deformation parameter vector controls the shape and attitude transformation of the model. For deformation transformation function, Control the regularization strength; A region segmentation unit is used to segment the face region based on morphological methods; wherein, the face region includes the glabella, left eye bag, right eye bag, left cheek, right cheek, tip of nose, philtrum, and chin; A region selection unit is used to select several target regions from each of the face regions; A facial feature extraction unit is used to extract features of the target region as facial features.

[0008] In some embodiments, the facial feature extraction unit includes: The face feature extraction subunit is used to put each of the target regions into the three color spaces of HSV, RGB and YCrCb respectively for color comparison, and then output the face features of four categories: red, white, black and yellow; wherein, the features of each space are reduced in dimensionality by principal component analysis. Among them, a multi-color space fusion strategy is used to analyze color features. The multi-color space fusion strategy is as follows: ; in, These are color feature vectors extracted from the HSV, RGB, and YCrCb color spaces, respectively. This is the fused color feature vector; The classification decision function for the facial features is: ; in, This is the weight matrix. For bias vectors, This is the final prediction result.

[0009] In some embodiments, the tongue diagnosis module includes: The second image acquisition unit is used to acquire images of the patient's tongue; An image preprocessing unit is used to perform standardization operations on the tongue image so that the tongue diagnosis module can adapt to tongue images of different resolutions and formats; wherein, the standardization operations include size uniform adjustment, image format conversion, and pixel value normalization; The detection box localization unit is used to capture multi-scale feature representations of the tongue image from low-level texture to high-level semantics using a deep convolutional neural network through a multi-level structure; generate multiple candidate detection boxes based on the multi-scale feature representations; wherein each candidate detection box contains position coordinates and a corresponding confidence score; in the target localization stage, the coordinates and size of each candidate detection box are regressed through a regression network to obtain the position of the tongue region; The detection box post-processing unit is used to filter overlapping boxes in the candidate detection boxes using a non-maximum suppression algorithm, and retain the candidate detection box with the highest confidence as the target detection box; wherein, the target detection box is output in the format (x1, y1, x2, y2), which correspond to the coordinates of the upper left corner and the lower right corner of the tongue region, respectively; The tongue image feature extraction unit is used to extract the tongue image features of the patient based on the target detection box.

[0010] In some embodiments, the tongue feature extraction unit is used to extract the patient's tongue features based on the target detection box using an improved YOLOv8 model; The improved YOLOv8 model includes an improved Backbone, an improved Neck, and an improved prediction head. The improved Backbone part extends the original CSPDarknet53 of the YOLOv8 model to a CSPDarknet65 structure by adding residual connection blocks. The feature extraction formula of the improved Backbone part includes: ; in, For the first The output feature map of the layer, This is the weight matrix. and These are convolution transformations of 3×3 and 1×1, respectively; The improved Neck section integrates a hybrid attention mechanism module, including both spatial attention and channel attention sub-networks, for focusing on the tongue diagnosis region; the feature extraction formula of the improved Neck section includes: ; in, and These represent spatial and channel attention maps, respectively. For the sigmoid function, Indicates global average pooling. Feature maps from different sources extracted from Backbone; The improved prediction head combines a shared feature layer and a dedicated prediction head, expressed as follows: ; in, Cross-entropy loss representing seven types of features, Dynamic weights guided by prior knowledge of Traditional Chinese Medicine. Controlling the regularization strength, This is the set of all weight parameters for the model.

[0011] In some embodiments, the pulse feature acquisition module includes a finger clip sensor, which is used to acquire the patient's infrared and red pulse waveform data. The pulse diagnosis module includes: The pulse waveform preprocessing unit is used to suppress motion artifacts and high-frequency noise in the pulse waveform data by using a sliding window buffer and combining digital filtering; and to perform baseline correction and dynamic range compression on the unnormalized pulse waveform. The pulse feature extraction unit is used to extract pulse features from the pulse waveform data from two dimensions: time domain and frequency domain. The time domain features include peak interval, waveform area, rise slope, and perfusion index. The frequency domain features are obtained by calculating the main frequency component and energy distribution through fast Fourier transform. The visualization unit is used to dynamically draw the normalized pulse waveform curve and display blood oxygen, pulse rate and perfusion index in real time. The report generation unit is used to generate a comprehensive analysis report based on historical data; wherein the comprehensive analysis report includes the average value of parameters, health risk warnings, and the pulse diagnosis results.

[0012] In some embodiments, the multimodal fusion inference module includes: A vector construction unit is used to unify the various preliminary diagnostic results into a traditional Chinese medicine representation vector. The reasoning unit is used to input the TCM representation vector into a multimodal large model for deep fusion and semantic reasoning, and output the target diagnostic result; The interaction unit obtains user questions about the target diagnostic results and generates answers to the questions using the multimodal large model.

[0013] In some embodiments, the system further includes: The results display module is used to display the features of different modalities; A storage module is used to store the features of different modalities into a database.

[0014] The embodiments of this application include at least the following beneficial effects: This application provides a TCM intelligent four-diagnosis auxiliary diagnostic system. The system includes a patient feature acquisition module for collecting features from different patient modalities; multiple single-modal diagnostic modules for obtaining preliminary diagnostic results based on the features of different modalities; and a multi-modal fusion reasoning module for fusing the preliminary diagnostic results and reasoning based on the fusion results to obtain the patient's target diagnostic result. This application's solution achieves independent extraction and preliminary analysis of features from each diagnostic method by constructing feature diagnostic modules for different modalities. Based on this, the multi-modal fusion reasoning module deeply fuses and jointly reasons the features from different modalities obtained from multi-source heterogeneous diagnostic methods, thereby realizing intelligent diagnostic analysis based on the integration of the four diagnostic methods, significantly improving diagnostic accuracy. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of the structure of a traditional Chinese medicine intelligent four diagnostic methods auxiliary diagnostic system provided in an embodiment of this application; Figure 2 This is a flowchart of face recognition processing provided in an embodiment of this application; Figure 3 This is a flowchart of tongue image recognition processing provided in an embodiment of this application; Figure 4 A structural diagram of the improved YOLOv8 model provided in the embodiments of this application; Figure 5 A flowchart illustrating the workflow of a TCM intelligent four diagnostic methods auxiliary diagnostic system provided in this application embodiment; Figure 6 This is a hardware platform structure diagram of a traditional Chinese medicine intelligent four diagnostic auxiliary diagnostic system provided in an embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit the scope of this application.

[0019] Before providing a detailed description of the embodiments of this application, some related technologies involved in the embodiments of this application will be described first, as follows: While some TCM diagnostic devices, such as tongue and pulse diagnostic instruments, have achieved digital data acquisition for single diagnostic methods, significant shortcomings remain. Firstly, existing technologies lack effective multimodal data fusion capabilities, resulting in isolated analysis of data from each diagnostic method and failing to achieve true "integrated diagnosis and treatment," thus violating the fundamental principle of holistic TCM thinking. Secondly, the systems' built-in TCM knowledge reserves are insufficient; most devices only possess basic syndrome matching functions and lack in-depth knowledge reasoning capabilities, limiting diagnostic effectiveness. More importantly, existing systems fail to leverage the advantages of current large-scale modeling technologies, exhibiting shortcomings in natural language processing, multimodal feature fusion, and intelligent decision-making, thus failing to achieve truly intelligent diagnosis. These limitations prevent existing devices from playing a substantial auxiliary role in clinical practice, resulting in a significant gap between application effects and expectations.

[0020] To address the aforementioned problems, the contributions of this application's embodiments are mainly reflected in the following aspects: First, a deep learning-based analysis model for facial diagnosis, tongue diagnosis, and lightweight pulse diagnosis was proposed, enabling automated extraction and preliminary quantification of key features of facial, tongue, and pulse characteristics. The facial and tongue diagnosis components employ high-precision detection and segmentation algorithms for accurate localization and multi-dimensional feature analysis of the face and tongue. The pulse diagnosis component utilizes temporal analysis of the pulse signal waveform and rhythm to extract key time-frequency domain indicators, achieving a digital description of the pulse.

[0021] Secondly, this paper innovatively introduces large-scale model technology as the core for the fusion and inference of multimodal diagnostic data, realizing the intelligent practice of "integrated diagnosis and treatment of the four diagnostic methods." Based on a multimodal large-scale model, this paper constructs a unified architecture capable of simultaneously processing images, text, and sequential signals. This model can not only deeply integrate multi-source information such as facial diagnosis images, tongue diagnosis images, pulse diagnosis curves, and patient consultation texts, but also perform cross-modal semantic alignment and joint inference based on the holistic concept of traditional Chinese medicine, truly realizing the transformation from "isolated multi-diagnosis" to "synergistic diagnosis and treatment of the four diagnostic methods."

[0022] Third, a complete intelligent TCM diagnostic system with four diagnostic methods was designed, encompassing both software workflows and hardware platforms. In terms of workflow, the system achieves fully automated processing from patient registration, synchronous multimodal data acquisition, feature extraction and fusion analysis to diagnostic report generation. The system integrates a high-resolution tongue imaging module, a finger-clip pulse acquisition device, and a structured consultation terminal to construct a multimodal TCM data acquisition platform.

[0023] Reference Figure 1 This application provides a TCM intelligent four-diagnostic auxiliary diagnostic system, the system comprising: The patient feature acquisition module is used to collect features of patients in different modalities. Multiple single-modal diagnostic modules are used to diagnose the features of different modalities and obtain corresponding preliminary diagnostic results; The multimodal fusion reasoning module fuses the preliminary diagnostic results from each of them and infers the patient's target diagnostic result based on the fusion result.

[0024] Optionally, the patient feature acquisition module includes a facial feature acquisition module, a tongue feature acquisition module, and a pulse feature acquisition module; The facial feature acquisition module is used to acquire the patient's facial features. The tongue image feature acquisition module is used to acquire the patient's tongue image features; The pulse feature acquisition module is used to acquire the pulse features of patients.

[0025] Optionally, the single-modal diagnostic module includes a facial diagnosis module, a tongue diagnosis module, and a pulse diagnosis module; The facial image diagnosis module is used to diagnose facial image results based on the facial image features; The tongue diagnosis module is used to diagnose tongue appearance based on the tongue appearance characteristics to obtain a tongue appearance diagnosis result; The pulse diagnosis module is used to diagnose pulse characteristics and obtain pulse diagnosis results.

[0026] Optionally, the facial feature acquisition module includes: The first image acquisition unit is used to acquire images of the patient's face; A region localization unit is used to locate the face region in the face image using a feature point-based deformation model; The feature-point-based deformation model is as follows: ; in, The total energy function measures the degree of matching between the model and the feature points. Represents the coordinates of the feature point. The deformation parameter vector controls the shape and attitude transformation of the model. For deformation transformation function, Control the regularization strength; A region segmentation unit is used to segment the face region based on morphological methods; wherein, the face region includes the glabella, left eye bag, right eye bag, left cheek, right cheek, tip of nose, philtrum, and chin; A region selection unit is used to select several target regions from each of the face regions; A facial feature extraction unit is used to extract features of the target region as facial features.

[0027] Optionally, the facial feature extraction unit includes: The face feature extraction subunit is used to put each of the target regions into the three color spaces of HSV, RGB and YCrCb respectively for color comparison, and then output the face features of four categories: red, white, black and yellow; wherein, the features of each space are reduced in dimensionality by principal component analysis. Among them, a multi-color space fusion strategy is used to analyze color features. The multi-color space fusion strategy is as follows: ; in, These are color feature vectors extracted from the HSV, RGB, and YCrCb color spaces, respectively. This is the fused color feature vector; The classification decision function for the facial features is: ; in, This is the weight matrix. For bias vectors, This is the final prediction result.

[0028] Optionally, the tongue diagnosis module includes: The second image acquisition unit is used to acquire images of the patient's tongue; An image preprocessing unit is used to perform standardization operations on the tongue image so that the tongue diagnosis module can adapt to tongue images of different resolutions and formats; wherein, the standardization operations include size uniform adjustment, image format conversion, and pixel value normalization; The detection box localization unit is used to capture multi-scale feature representations of the tongue image from low-level texture to high-level semantics using a deep convolutional neural network through a multi-level structure; generate multiple candidate detection boxes based on the multi-scale feature representations; wherein each candidate detection box contains position coordinates and a corresponding confidence score; in the target localization stage, the coordinates and size of each candidate detection box are regressed through a regression network to obtain the position of the tongue region; The detection box post-processing unit is used to filter overlapping boxes in the candidate detection boxes using a non-maximum suppression algorithm, and retain the candidate detection box with the highest confidence as the target detection box; wherein, the target detection box is output in the format (x1, y1, x2, y2), which correspond to the coordinates of the upper left corner and the lower right corner of the tongue region, respectively; The tongue image feature extraction unit is used to extract the tongue image features of the patient based on the target detection box.

[0029] Optionally, the tongue image feature extraction unit is used to extract the patient's tongue image features based on the target detection box using an improved YOLOv8 model; The improved YOLOv8 model includes an improved Backbone, an improved Neck, and an improved prediction head. The improved Backbone part extends the original CSPDarknet53 of the YOLOv8 model to a CSPDarknet65 structure by adding residual connection blocks. The feature extraction formula of the improved Backbone part includes: ; in, For the first The output feature map of the layer, This is the weight matrix. and These are convolution transformations of 3×3 and 1×1, respectively; The improved Neck section integrates a hybrid attention mechanism module, including both spatial attention and channel attention sub-networks, for focusing on the tongue diagnosis region; the feature extraction formula of the improved Neck section includes: ; in, and These represent spatial and channel attention maps, respectively. For the sigmoid function, Indicates global average pooling. Feature maps from different sources extracted from Backbone; The improved prediction head combines a shared feature layer and a dedicated prediction head, expressed as follows: ; in, Cross-entropy loss representing seven types of features, Dynamic weights guided by prior knowledge of Traditional Chinese Medicine. Controlling the regularization strength, This is the set of all weight parameters for the model.

[0030] Optionally, the pulse feature acquisition module includes a finger clip sensor, which is used to acquire the patient's infrared and red light pulse waveform data; The pulse diagnosis module includes: The pulse waveform preprocessing unit is used to suppress motion artifacts and high-frequency noise in the pulse waveform data by using a sliding window buffer and combining digital filtering; and to perform baseline correction and dynamic range compression on the unnormalized pulse waveform. The pulse feature extraction unit is used to extract pulse features from the pulse waveform data from two dimensions: time domain and frequency domain. The time domain features include peak interval, waveform area, rise slope, and perfusion index. The frequency domain features are obtained by calculating the main frequency component and energy distribution through fast Fourier transform. The visualization unit is used to dynamically draw the normalized pulse waveform curve and display blood oxygen, pulse rate and perfusion index in real time. The report generation unit is used to generate a comprehensive analysis report based on historical data; wherein the comprehensive analysis report includes the average value of parameters, health risk warnings, and the pulse diagnosis results.

[0031] Optionally, the multimodal fusion inference module includes: A vector construction unit is used to unify the various preliminary diagnostic results into a traditional Chinese medicine representation vector. The reasoning unit is used to input the TCM representation vector into a multimodal large model for deep fusion and semantic reasoning, and output the target diagnostic result; The interaction unit obtains user questions about the target diagnostic results and generates answers to the questions using the multimodal large model.

[0032] Optionally, the system further includes: The results display module is used to display the features of different modalities; A storage module is used to store the features of different modalities into a database.

[0033] The following sections will provide a detailed description and explanation of some optional embodiments of this application, using specific application examples.

[0034] This application proposes a TCM intelligent four-diagnostic auxiliary system. First, dedicated diagnostic models for facial appearance, tongue appearance, and pulse appearance are constructed, enabling independent extraction and preliminary analysis of features for each diagnostic method. Based on this, multimodal large-scale model technology is introduced to deeply fuse and jointly infer multi-source heterogeneous diagnostic data, thereby achieving intelligent diagnostic analysis through the integration of the four diagnostic methods. Finally, a complete system architecture is designed, encompassing hardware and software integration solutions and end-to-end workflows, providing an operable and scalable auxiliary decision support platform for TCM clinical diagnosis.

[0035] Specifically, the embodiments of this application include the following solutions: 1. Facial Recognition: For objective research on facial diagnosis, computer image recognition technology is used to locate and segment facial images, obtain key parts of the facial images, extract information features, and realize automatic recognition and analysis of facial information features. Figure 2 This is a flowchart of the face recognition process.

[0036] 1.1 Facial region segmentation: Morphological methods such as mathematical morphology-based segmentation, region-based segmentation, and threshold-based segmentation were used to segment the facial region. Since different facial parts are closely related to the state of corresponding internal organs, eight regions were segmented after obtaining the facial image: glabella, left eye bag, right eye bag, left cheek, right cheek, tip of nose, philtrum, and chin. Five regions were ultimately selected as local facial features for extraction. A feature point-based deformation model was used for locating key facial regions. ; in, The total energy function measures the degree of matching between the model and the feature points. Represents the coordinates of the feature point. The deformation parameter vector controls the shape and attitude transformation of the model. For deformation transformation function, Control the regularization intensity. Focus on extracting the corresponding areas of five internal organs: glabella (heart), left cheek (liver), nose (spleen), right cheek (lung), and jaw (kidney).

[0037] 1.2 Facial Feature Extraction and Recognition: The segmented key facial regions are placed into different color spaces for color comparison. They are placed into HSV, RGB, and YCrCb color spaces for color comparison, and finally color classification is achieved.

[0038] Color feature analysis employs a multi-color space fusion strategy: ; in, These are color feature vectors extracted from the HSV, RGB, and YCrCb color spaces, respectively. This is the fused color feature vector; The spatial features are dimensionality reduced using principal component analysis. The final output includes four face features: red, white, black, and yellow, with the following classification decision function: ; in, This is the weight matrix. For bias vectors, This is the final prediction result.

[0039] 2. Tongue image recognition: This module employs a deep learning framework, deeply integrating traditional Chinese medicine diagnostic theories to construct an intelligent analysis process for multi-dimensional tongue feature interpretation. First, a target detection model is used to locate the tongue region. Then, a semantic segmentation network is used to finely divide the tongue surface, tongue coating, and sublingual veins. Based on this, a multi-task classification module is used to extract seven categories of features in parallel: tongue color, tongue shape, tongue coating color, tongue coating texture, tongue posture, tongue condition, and sublingual veins. Figure 3 This is a flowchart of tongue image recognition processing.

[0040] 2.1 Tongue region localization: The tongue region localization employs an end-to-end deep learning detection workflow, primarily comprising four core stages: image preprocessing, feature extraction, target localization, and bounding box post-processing. In the image preprocessing stage, the input image is first standardized, including resizing, image format conversion, and pixel value normalization, ensuring the model can adapt to raw inputs of different resolutions and formats, thus improving the robustness of the localization process.

[0041] The feature extraction stage relies on a deep convolutional neural network, capturing multi-scale feature representations from low-level texture to high-level semantics through a multi-level structure. Based on the rich extracted features, the model further generates a series of candidate detection boxes, each containing location coordinates and a corresponding confidence score. In the target localization stage, a regression network performs fine regression on the coordinates and dimensions of each candidate box to obtain a more accurate location of the tongue region.

[0042] In the post-processing stage of the detection boxes, the Non-Maximum Suppression (NMS) algorithm is used to filter overlapping boxes, retaining the detection box with the highest confidence as the final localization result. The detection boxes are output in the format (x1, y1, x2, y2), corresponding to the coordinates of the upper left and lower right corners of the tongue region, respectively, providing a stable and accurate spatial localization foundation for subsequent tongue feature extraction and recognition.

[0043] 2.2 Tongue Feature Extraction and Recognition: To achieve automated extraction and accurate identification of seven major categories of features—tongue color, tongue shape, tongue coating color, tongue coating texture, tongue posture, tongue condition, and sublingual veins—the model's basic architecture adopts YOLOv8, and structural optimizations adapted to the characteristics of tongue image analysis are introduced in several key modules. Figure 4 This is a structural diagram of the improved YOLOv8 model in the embodiments of this application.

[0044] In the backbone section, the original CSPDarknet53 structure is expanded to a deeper CSPDarknet65 structure. By adding residual connectives, the ability to capture subtle features of the tongue image is significantly improved, such as tongue surface fissures and punctate marks—pathological features that are difficult to detect. Its main formula is: ; in, For the first The output feature map of the layer, This is the weight matrix. and The convolutional transformations are 3×3 and 1×1, respectively. This cross-layer connection effectively captures both local details and global features of the tongue image.

[0045] The Neck section integrates a hybrid attention mechanism module, including spatial attention and channel attention subnetworks, which can automatically focus on key areas of the tongue, such as the tip and sides, which are crucial for diagnosis. Its main formula is: ; in, and These represent spatial and channel attention maps, respectively. For the sigmoid function, Indicates global average pooling. These are feature maps extracted from Backbone from different sources.

[0046] For the prediction head, a multi-task learning framework was designed, which achieves efficient collaboration by combining a shared feature layer and a dedicated prediction head. ; in, Cross-entropy loss representing seven types of features, Dynamic weights guided by prior knowledge of Traditional Chinese Medicine. Controlling the regularization strength, This is the set of all weight parameters for the model.

[0047] Specifically, the features include: 1) Tongue color grading: Covering pale red tongue, pale white tongue, dry white tongue, red tongue, crimson tongue, red tongue tip, red tongue edges and tip, bluish-purple tongue, pale purple tongue, ecchymotic tongue, and petechiae tongue. The system identifies 11 tongue color types, including pale red, pale white, and crimson, which correspond to different physiological and pathological characteristics such as Qi and blood status and cold / heat attributes.

[0048] 2) Tongue shape classification: including scalloped tongue, fissured tongue, swollen tongue, thin tongue, prickly tongue, tongue bleeding, tongue sores, etc. Each shape is related to the dysfunction of specific internal organs.

[0049] 3) Classification of moss color: covering white moss, yellow moss, gray-black moss, etc., reflecting the nature of diseases such as exterior and interior, cold and heat.

[0050] 4) Tongue texture assessment: including thin tongue, thick tongue, greasy tongue, putrid tongue, peeled tongue, mirror tongue, etc., used to judge the strength of stomach qi and the depth of pathogenic factors.

[0051] 5) Tongue posture and spirit: Tongue posture and spirit mainly observe the movement function of the tongue and the color of the tongue to judge the existence of vital energy.

[0052] 6) Sublingual veins: The analysis focuses on detecting changes in the shape and color of the veins, which serve as an important basis for judging the state of Qi and blood circulation.

[0053] 3. Pulse identification: Pulse recognition is based on dual-channel infrared (IR) and red light (RED) pulse waveform data acquired by a finger-clip sensor. It integrates signal processing and artificial intelligence methods to construct an intelligent diagnostic system. After the raw data is transmitted to the host computer via serial port, it is buffered through a sliding window and combined with digital filtering (including median filtering and low-pass filtering) to suppress motion artifacts and high-frequency noise. Furthermore, baseline correction and dynamic range compression are performed on the unnormalized waveform to improve signal stability and provide a reliable foundation for subsequent analysis.

[0054] The feature extraction and real-time analysis module extracts key pulse features from both the time and frequency domains: time-domain features include peak interval (pulse rate PR), waveform area, rise slope, and perfusion index (PI); frequency-domain features are calculated using Fast Fourier Transform (FFT) to determine the dominant frequency component and energy distribution. The system can dynamically adjust feature weights based on a preset pressure-signal relationship to adapt to different measurement conditions and enhance calibration robustness. Feature extraction is triggered every 500 accumulated sampling points (sampling rate 100Hz, corresponding to 5 seconds of data), achieving real-time processing under low latency constraints.

[0055] The visualization and report generation section dynamically plots normalized waveform curves, displaying parameters such as blood oxygen (SpO2), pulse rate (PR), and perfusion index (PI) in real time. Based on historical data, it generates comprehensive analysis reports, including parameter averages, health risk warnings (such as low blood oxygen alerts), and preliminary diagnostic conclusions, providing a comprehensive basis for assessment.

[0056] 4. Multimodal large-scale model fusion inference: In the multimodal large-scale model fusion reasoning module, after calling professional models of facial appearance, tongue appearance, and pulse appearance, preliminary diagnostic results for each modality are obtained. Subsequently, based on the multimodal diagnostic information, a unified TCM representation vector is constructed and input into the multimodal large-scale model for deep fusion and semantic reasoning. Within this framework, users can further engage in multiple rounds of interactive question-and-answer sessions with the large-scale model based on the preliminary diagnostic conclusions, such as inquiring in detail about the etiology and pathogenesis of specific symptoms, obtaining personalized TCM conditioning suggestions, or requesting in-depth interpretation of the diagnostic results, etc., to achieve intelligent reasoning based on the integration of the four diagnostic methods.

[0057] 5. Construction of a Traditional Chinese Medicine Intelligent Four Diagnostic Methods-Assisted Diagnosis System: The workflow of the system in this application embodiment is as follows: Figure 5 As shown, the process begins with the user registering as a patient using a traditional Chinese medicine diagnostic equipment. The front end immediately sends the patient registration information to the multimodal data acquisition module. Subsequently, this module sends a data request to the backend multimodal data processing system to obtain multimodal diagnostic data related to the patient. After receiving the request, the backend system integrates and performs preliminary analysis on the multi-source data, and generates a structured response to return to the acquisition module.

[0058] The data acquisition module transmits the processed data to the patient condition analysis module, which performs a comprehensive analysis of the patient's condition based on multimodal input, extracting key pathological features and diagnostic elements. The analysis results are further transmitted to the intelligent diagnosis generation module, which automatically generates a structured diagnostic report based on preset TCM diagnostic rules and algorithm models.

[0059] Finally, the diagnostic report is sent back to the front-end interface via the patient condition analysis module and presented to the user.

[0060] The system hardware platform consists of modules such as a camera, a mechanical support, a photoelectric sensor finger clip, and a host computer control unit. Among them, the camera (1) is fixed on the top of the high-precision mechanical support (2), and the optical lens is installed through a standard C interface to ensure that the imaging optical axis is perpendicular to the measurement plane, effectively reducing the distortion of the viewing angle. The support is made of aluminum alloy, which has excellent rigidity and shock absorption performance, and can realize the fine-tuning and positioning of the camera in three-dimensional space. The finger clip photoelectric sensor (3) is integrated into the bottom platform of the support and is used to collect physiological signals such as finger pulse and blood oxygen. Its clamping mechanism is equipped with elastic pads to adapt to different finger sizes and ensure contact stability. The host computer (4) is connected to the camera and sensor through a USB 3.0 interface and is responsible for synchronously controlling image acquisition, physiological signal acquisition and data fusion processing to realize real-time acquisition and analysis of multimodal information.

[0061] For example, Figure 6 This is a hardware platform structure diagram of a traditional Chinese medicine intelligent four diagnostic auxiliary system.

[0062] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0063] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer technical solutions than shown, or a combination of certain technical solutions, or different technical solutions.

[0064] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0065] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0066] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A TCM intelligent four-diagnostic auxiliary system, characterized in that, The system includes: The patient feature acquisition module is used to collect features of patients in different modalities; Multiple single-modal diagnostic modules are used to diagnose the features of different modalities and obtain corresponding preliminary diagnostic results; The multimodal fusion reasoning module fuses the preliminary diagnostic results from each of them and infers the patient's target diagnostic result based on the fusion result.

2. The intelligent four-diagnostic auxiliary system for traditional Chinese medicine according to claim 1, characterized in that, The patient feature acquisition module includes a facial feature acquisition module, a tongue feature acquisition module, and a pulse feature acquisition module. The facial feature acquisition module is used to acquire the patient's facial features. The tongue image feature acquisition module is used to acquire the patient's tongue image features; The pulse feature acquisition module is used to acquire the pulse features of patients.

3. The TCM intelligent four-diagnostic auxiliary system according to claim 2, characterized in that, The single-modal diagnostic module includes a facial diagnosis module, a tongue diagnosis module, and a pulse diagnosis module. The facial image diagnosis module is used to diagnose facial image results based on the facial image features; The tongue diagnosis module is used to diagnose tongue appearance based on the tongue appearance characteristics to obtain a tongue appearance diagnosis result; The pulse diagnosis module is used to diagnose pulse characteristics and obtain pulse diagnosis results.

4. The TCM intelligent four-diagnostic auxiliary system according to claim 2, characterized in that, The facial feature acquisition module includes: The first image acquisition unit is used to acquire images of the patient's face; A region localization unit is used to locate the face region in the face image using a feature point-based deformation model; The feature-point-based deformation model is as follows: ； in, The total energy function measures the degree of matching between the model and the feature points. Represents the coordinates of the feature point. The deformation parameter vector controls the shape and attitude transformation of the model. For deformation transformation function, Control the regularization strength; A region segmentation unit is used to segment the face region based on morphological methods; wherein, the face region includes the glabella, left eye bag, right eye bag, left cheek, right cheek, tip of nose, philtrum, and chin; A region selection unit is used to select several target regions from each of the face regions; A facial feature extraction unit is used to extract features of the target region as facial features.

5. The TCM intelligent four-diagnostic auxiliary system according to claim 4, characterized in that, The facial feature extraction unit includes: The face feature extraction subunit is used to put each of the target regions into the three color spaces of HSV, RGB and YCrCb respectively for color comparison, and then output the face features of four categories: red, white, black and yellow; wherein, the features of each space are reduced in dimensionality by principal component analysis. Among them, a multi-color space fusion strategy is used to analyze color features. The multi-color space fusion strategy is as follows: ； in, These are color feature vectors extracted from the HSV, RGB, and YCrCb color spaces, respectively. This is the fused color feature vector; The classification decision function for the facial features is: ； in, This is the weight matrix. For bias vectors, This is the final prediction result.

6. The TCM intelligent four-diagnostic auxiliary system according to claim 3, characterized in that, The tongue diagnosis module includes: The second image acquisition unit is used to acquire images of the patient's tongue; An image preprocessing unit is used to perform standardization operations on the tongue image so that the tongue diagnosis module can adapt to tongue images of different resolutions and formats; wherein, the standardization operations include size uniform adjustment, image format conversion, and pixel value normalization; The detection box localization unit is used to capture multi-scale feature representations of the tongue image from low-level texture to high-level semantics using a deep convolutional neural network through a multi-level structure; generate multiple candidate detection boxes based on the multi-scale feature representations; wherein each candidate detection box contains position coordinates and a corresponding confidence score; in the target localization stage, the coordinates and size of each candidate detection box are regressed through a regression network to obtain the position of the tongue region; The detection box post-processing unit is used to filter overlapping boxes in the candidate detection boxes using a non-maximum suppression algorithm, and retain the candidate detection box with the highest confidence as the target detection box; wherein, the target detection box is output in the format (x1, y1, x2, y2), which correspond to the coordinates of the upper left corner and the lower right corner of the tongue region, respectively; The tongue image feature extraction unit is used to extract the tongue image features of the patient based on the target detection box.

7. The TCM intelligent four-diagnostic auxiliary system according to claim 6, characterized in that, The tongue image feature extraction unit is used to extract the patient's tongue image features based on the target detection box using an improved YOLOv8 model; The improved YOLOv8 model includes an improved Backbone, an improved Neck, and an improved prediction head. The improved Backbone part extends the original CSPDarknet53 of the YOLOv8 model to a CSPDarknet65 structure by adding residual connection blocks. The feature extraction formula of the improved Backbone part includes: ； in, For the first The output feature map of the layer, This is the weight matrix. and These are convolution transformations of 3×3 and 1×1, respectively; The improved Neck section integrates a hybrid attention mechanism module, including both spatial attention and channel attention sub-networks, for focusing on the tongue diagnosis region; the feature extraction formula of the improved Neck section includes: ； in, and These represent spatial and channel attention maps, respectively. For the sigmoid function, Indicates global average pooling. Feature maps from different sources extracted from Backbone; The improved prediction head combines a shared feature layer and a dedicated prediction head, expressed as follows: ； in, Cross-entropy loss representing seven types of features, Dynamic weights guided by prior knowledge of Traditional Chinese Medicine. Controlling the regularization strength, This is the set of all weight parameters for the model.

8. The TCM intelligent four-diagnostic auxiliary system according to claim 3, characterized in that, The pulse feature acquisition module includes a finger clip sensor, which is used to acquire the patient's infrared and red pulse waveform data. The pulse diagnosis module includes: The pulse waveform preprocessing unit is used to suppress motion artifacts and high-frequency noise in the pulse waveform data by using a sliding window buffer and combining digital filtering; and to perform baseline correction and dynamic range compression on the unnormalized pulse waveform. The pulse feature extraction unit is used to extract pulse features from the pulse waveform data from two dimensions: time domain and frequency domain. The time domain features include peak interval, waveform area, rise slope, and perfusion index. The frequency domain features are obtained by calculating the main frequency component and energy distribution through fast Fourier transform. The visualization unit is used to dynamically draw the normalized pulse waveform curve and display blood oxygen, pulse rate and perfusion index in real time. The report generation unit is used to generate a comprehensive analysis report based on historical data; wherein the comprehensive analysis report includes the average value of parameters, health risk warnings, and the pulse diagnosis results.

9. The TCM intelligent four-diagnostic auxiliary system according to claim 1, characterized in that, The multimodal fusion inference module includes: A vector construction unit is used to unify the various preliminary diagnostic results into a traditional Chinese medicine representation vector. The reasoning unit is used to input the TCM representation vector into a multimodal large model for deep fusion and semantic reasoning, and output the target diagnostic result; The interaction unit obtains user questions about the target diagnostic results and generates answers to the questions using the multimodal large model.

10. A TCM intelligent four-diagnostic auxiliary system according to any one of claims 1 to 9, characterized in that, The system also includes: The results display module is used to display the features of different modalities; A storage module is used to store the features of different modalities into a database.

Images