A four-diagnosis information adaptive fusion syndrome differentiation system based on a cross-modal attention mechanism

By employing cross-modal attention mechanisms and adaptive fusion technology, a deep, complementary semantic association of information from the four diagnostic methods in Traditional Chinese Medicine (TCM) is achieved. This addresses the issues of superficial fusion of information from the four diagnostic methods and insufficient adaptive capabilities in existing technologies, thereby improving the accuracy and robustness of diagnosis and providing interpretable diagnostic results.

CN123072689APending Publication Date: 2026-11-10SHANGHAI MAISHEN HEALTH TECHNOLOGY CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202611429125.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-15
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing technologies for intelligent integration and diagnosis of TCM diagnostic information suffer from superficial integration methods, lack of cross-modal attention mechanisms, inability to fully explore deep complementary semantic relationships between heterogeneous modalities, insufficient adaptive capabilities, poor interpretability of diagnosis results, and inability to cope with real-world data acquisition scenarios such as modality loss and quality differences.

Method used

An adaptive fusion system for the four diagnostic methods based on a cross-modal attention mechanism is adopted. Through a four diagnostic methods information acquisition module, a single-modal feature encoding module, a cross-modal attention fusion module, an adaptive gating fusion module, and a syndrome differentiation output module, the system achieves fine-grained interactive alignment of the four diagnostic methods information in a unified feature space, and dynamically adjusts the fusion weights according to modal quality to output interpretable syndrome differentiation results.

Benefits of technology

It achieves fine-grained interactive alignment of deep complementary semantic associations of the four diagnostic methods, improves the accuracy and robustness of syndrome differentiation, has adaptive fusion capabilities for modality loss and quality differences, and provides interpretable syndrome differentiation results, meeting the requirements of TCM clinical auxiliary decision-making.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123072689A_ABST
    Figure CN123072689A_ABST
Patent Text Reader

Abstract

The application discloses a four-diagnosis information adaptive fusion syndrome differentiation system based on a cross-modal attention mechanism, and relates to the cross field of artificial intelligence and traditional Chinese medicine diagnosis. The system comprises a four-diagnosis information acquisition module, a single-modal feature coding module, a cross-modal attention fusion module, an adaptive gating fusion module and a syndrome differentiation output module. The acquisition module acquires tongue appearance, face appearance, voice, question and answer text and pulse appearance four-diagnosis information; the coding module extracts modal feature vectors; the cross-modal attention fusion module performs multi-head cross-modal attention operation with a learnable syndrome differentiation center vector as a query and each modal feature as a key and a value; the adaptive gating fusion module dynamically calculates gating weights for weighted fusion according to the quality and confidence of each modal, replaces missing modal with a placeholder vector and suppresses the weight; and the syndrome differentiation output module outputs a constitution type, a syndrome type and a pathogenesis explainable description. The application realizes cross-modal alignment and adaptive fusion of four-diagnosis heterogeneous information, and improves the accuracy and explainability of syndrome differentiation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of interdisciplinary technology of artificial intelligence and traditional Chinese medicine diagnosis, and in particular to a diagnostic system based on the adaptive fusion of four diagnostic methods based on cross-modal attention mechanism. Background Technology

[0002] The four diagnostic methods of Traditional Chinese Medicine (TCM) – observation, auscultation, inquiry, and palpation – are the core method and fundamental characteristic of TCM's syndrome differentiation and treatment. The "four diagnostic methods combined" refers to the physician's comprehensive use of clinical information obtained through observation (observing tongue appearance, complexion, shape, and demeanor), auscultation (listening to voice and breath), inquiry (inquiring about symptoms, medical history, and daily life), and palpation (feeling the pulse). This information is then cross-referenced, supplemented, and comprehensively analyzed to ultimately arrive at a conclusion regarding the pathogenesis and syndrome. However, the traditional four diagnostic methods rely heavily on the physician's individual clinical experience, resulting in significant problems such as strong subjectivity, low standardization, poor repeatability, and difficulty in large-scale implementation. With the accelerating pace of deep aging, there is a severe shortage of professional TCM practitioners in communities, grassroots levels, elderly care, and senior citizen settings, hindering the provision of standardized, large-scale, and high-quality diagnostic services. Therefore, there is an urgent need to digitize, objectify, and intelligentize the TCM four diagnostic methods process through artificial intelligence technology.

[0003] To address the aforementioned issues, existing technologies have proposed several intelligent diagnostic schemes for the four diagnostic methods of traditional Chinese medicine based on artificial intelligence.

[0004] Prior art 1, Chinese invention patent application CN107242857A, discloses an intelligent TCM comprehensive diagnosis and treatment system based on deep learning, which includes a subsystem for observation, auscultation and olfaction, inquiry, pulse diagnosis, and syndrome differentiation analysis. This solution achieves digital acquisition of information from the four diagnostic methods (inspection, auscultation and olfaction, inquiry, and palpation) and processes the information acquired by each subsystem separately through deep learning. However, the subsystems in this solution are relatively independent, lacking an effective deep fusion mechanism between the four diagnostic methods. It can only perform isolated analysis of each modality, failing to truly achieve collaborative interaction and cross-verification of the four diagnostic methods at the feature level. It cannot restore the TCM diagnostic logic of "combining the four diagnostic methods and inferring internal from external symptoms," thus limiting the accuracy and robustness of diagnosis.

[0005] Prior art two, Chinese invention patent application CN117334324A, discloses a method and apparatus for TCM syndrome differentiation analysis by fusing prior knowledge with information from the four diagnostic methods. This method acquires datasets from the four diagnostic methods and syndrome differentiation, and then fuses these datasets, using prior knowledge for TCM syndrome differentiation analysis. While this approach introduces the concept of fusing information from the four diagnostic methods, its fusion method primarily relies on the splicing, mapping, and shallow fusion of prior knowledge. It fails to fully explore the deep, fine-grained cross-modal semantic relationships between the information from the four diagnostic methods and does not consider the alignment relationships of different modalities in the semantic space. This results in a lack of adaptability in the fusion process, making it difficult to address common problems in real-world data acquisition scenarios such as modality loss and modality quality differences.

[0006] The third prior art, Chinese invention patent application CN119169656A, discloses a method and device for TCM constitution identification based on multimodal features of pulse and tongue images. It acquires raw pulse signals and tongue images of individuals with different TCM constitutions, preprocesses them, extracts features, and then fuses them to achieve TCM constitution identification. This scheme fuses pulse and tongue images, representing an improvement over single-modal schemes. However, it only utilizes these two modalities, neglecting information from the other four diagnostic methods (face diagnosis, auscultation and olfaction, inquiry, and palpation), resulting in insufficient information utilization. Furthermore, this scheme lacks a cross-modal attention interaction mechanism and adaptive fusion capabilities for modal missingness, modal noise, and modal conflicts. When a modality is missing or the acquisition quality is poor, the accuracy and stability of the identification results significantly degrade.

[0007] The fourth prior art, Chinese invention patent application CN116543891A, discloses a TCM constitution analysis system, method, terminal, and medium based on the fusion of tongue image features and a simplified TCM constitution scale. This system achieves constitution analysis by fusing tongue image features with information from a simplified TCM constitution scale. However, this approach is also limited to only two information sources: tongue image and scale. Its modal coverage is incomplete, and the fusion method is traditional feature splicing or shallow fusion, lacking cross-modal deep interaction and adaptive weighting mechanisms.

[0008] Prior art five, Chinese invention patent application CN115512110A, discloses a medical image tumor segmentation method involving a cross-modal attention mechanism. This method achieves precise tumor segmentation by applying the cross-modal attention mechanism to the interaction between PET and CT images. While this solution successfully applies the cross-modal attention mechanism to cross-modal interaction in Western medicine images, its application is limited to medical image segmentation using only two image modalities: PET and CT. It does not address the heterogeneous multimodal scenario of traditional Chinese medicine's four diagnostic methods, which simultaneously includes images (tongue and face), temporal signals (pulse), speech (auscultation), and text (inquiry). Furthermore, this solution lacks adaptive fusion capabilities for modal missingness, modal quality differences, and modal conflicts, and it does not establish an interpretable mapping relationship between cross-modal interaction and traditional Chinese medicine pathogenesis.

[0009] In summary, existing technologies for intelligent fusion of information from the four diagnostic methods in Traditional Chinese Medicine (TCM) suffer from the following main technical deficiencies: First, the fusion methods for the four diagnostic methods are mostly simple feature splicing, prior knowledge mapping, or shallow fusion, lacking fine-grained interactive alignment capabilities based on cross-modal attention mechanisms. This makes it difficult to fully explore the deep, complementary semantic relationships between heterogeneous modalities such as tongue appearance, facial appearance, pulse appearance, auscultation and olfaction, and inquiry, and thus cannot effectively restore the inherent diagnostic logic of TCM's "combining the four diagnostic methods to infer internal conditions from external manifestations." Second, existing fusion schemes generally lack adaptive capabilities and cannot... The fusion weights are dynamically adjusted based on the acquisition quality, confidence, and completeness of each modality, but the robustness is insufficient in real acquisition scenarios such as modality missing, modality quality differences, and modality information conflicts. Third, the interpretability of the existing scheme is poor, and it is impossible to intuitively correspond the attention weights in the cross-modal fusion process with the pathogenesis and syndromes of traditional Chinese medicine, which makes it difficult to meet the trust requirements of clinical prudence and decision support. Fourth, although the cross-modal attention mechanism has been applied in Western medicine medical image fusion, it has not been effectively transferred to the heterogeneous multimodal syndrome differentiation scenario of the four diagnostic methods of traditional Chinese medicine.

[0010] Therefore, how to effectively introduce cross-modal attention mechanisms and adaptive fusion technology into the intelligent fusion and diagnosis process of TCM four diagnostic methods, while fully exploring the deep correlation between heterogeneous modalities of the four diagnostic methods, to achieve adaptive and robust fusion for modal missing and quality differences, and to provide interpretable diagnosis results, has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0011] To address the shortcomings of existing technologies, the purpose of this invention is to provide an adaptive fusion and diagnostic system for the four diagnostic methods (inspection, auscultation and olfaction, inquiry, and palpation) based on a cross-modal attention mechanism. This invention aims to solve the following technical problems existing in the intelligent fusion and diagnostic system for the four diagnostic methods in traditional Chinese medicine: a lack of fine-grained interactive alignment between heterogeneous modal information based on a cross-modal attention mechanism; a superficial fusion method that fails to fully reproduce the diagnostic logic of "combining the four diagnostic methods and inferring internal from external symptoms" in traditional Chinese medicine; a lack of adaptability in the fusion process, making it difficult to cope with real-world data acquisition scenarios such as modality loss, modality quality differences, and modal information conflicts; and insufficient interpretability of the diagnostic results, failing to establish an intuitive correspondence between the fusion process and the pathogenesis in traditional Chinese medicine.

[0012] The above-mentioned objective of this invention is achieved through the following technical solutions:

[0013] A diagnostic system based on cross-modal attention mechanism that adaptively fuses diagnostic information from the four diagnostic methods (inspection, auscultation and olfaction, inquiry, and palpation) is provided. The system includes a four diagnostic information acquisition module, a single-modal feature encoding module, a cross-modal attention fusion module, an adaptive gating fusion module, and a diagnostic output module.

[0014] The four diagnostic information acquisition module is used to collect the four diagnostic information of the tested object. The four diagnostic information includes at least four types of modal data: inspection information, auscultation information, inquiry information, and palpation information. Among them, the inspection information includes tongue image and facial image, the auscultation information includes voice signal, the inquiry information includes structured question and answer text, and the palpation information includes pulse time sequence signal.

[0015] The single-modal feature encoding module is used to encode the features of each modality data in the four diagnostic methods information to obtain the single-modal feature vector corresponding to each modality; the single-modal feature encoding module includes a tongue image feature encoder, a facial image feature encoder, a pulse image feature encoder, a questioning feature encoder, and an auscultation feature encoder.

[0016] The cross-modal attention fusion module is used to project the single-modal feature vectors of each modality onto a unified feature space, and use the learnable dialectical center vector as the query vector, and the single-modal feature vectors of each modality as the key vector and value vector, to perform multi-head cross-modal attention operation to obtain the cross-modal enhanced feature vectors corresponding to each modality.

[0017] The adaptive gating fusion module is used to calculate the adaptive gating weights of each modality based on the acquisition quality information and feature confidence of each modality, and to perform weighted fusion of the cross-modal enhanced feature vectors of each modality according to the adaptive gating weights to obtain a global dialectical feature vector; wherein, when any modality data is missing, the feature of the missing modality is replaced by a learnable placeholder vector, and the adaptive gating weight corresponding to the missing modality is constrained to approach zero.

[0018] The diagnostic output module is used to perform diagnostic classification based on the global diagnostic feature vector, output the constitution type and syndrome type of the tested object, and output interpretability weights to characterize the degree of contribution of each modality to the diagnostic results.

[0019] Furthermore, in the cross-modal attention fusion module, the multi-head cross-modal attention operation includes: taking the query matrix obtained by projecting the dialectical center vector through the query and the key matrix obtained by projecting the single-modal feature vectors of each modality through the key, scaling and normalizing the result to obtain the cross-modal attention weight matrix; performing a weighted summation of the cross-modal attention weight matrix and the value matrix obtained by projecting the single-modal feature vectors of each modality through the value to obtain the cross-modal enhancement feature vectors corresponding to each modality; the cross-modal attention fusion module also performs inter-modal interactive attention operation on the cross-modal enhancement feature vectors between different modalities to capture complementary information between modalities.

[0020] Furthermore, in the adaptive gating fusion module, the adaptive gating weights of each modality are calculated as follows: the cross-modal enhanced feature vectors of each modality are concatenated with the corresponding modality confidence vectors, mapped through a gating network and activation function, and then normalized to obtain the adaptive gating weights of each modality; the modality confidence vector is determined based on the acquisition quality index, feature completeness index, and single-modality classification confidence index of the modality data.

[0021] Furthermore, when any modality data is missing, the adaptive gating fusion module replaces the cross-modal enhanced feature vector corresponding to the missing modality with a learnable placeholder vector, and applies sparsity regularization constraints to the adaptive gating weights corresponding to the missing modality during training, so that the weights approach zero, thereby suppressing the interference of the missing modality on the global dialectical feature vector.

[0022] Furthermore, the syndrome differentiation output module is also used to: generate an attention contribution heatmap from each modality to each syndrome type based on the cross-modal attention weight matrix output by the cross-modal attention fusion module, and associate the attention contribution heatmap with a pre-constructed TCM pathogenesis knowledge graph to output a pathogenesis interpretability description for explaining the syndrome differentiation results.

[0023] Furthermore, the tongue image feature encoder is used to sequentially perform tongue body segmentation and tongue coating separation processing on the tongue image, and extract multi-dimensional tongue image features including tongue color, coating color, tongue shape, tongue texture, peeled coating, greasy coating, purple tongue, cracks, enlargement, and teeth marks; the pulse image feature encoder is used to perform three-segment segmentation processing of the pulse time sequence signal at the cun, guan, and chi levels, and extract pulse image features under three pulse pressures of light, medium, and heavy pulse taking, and output single pulse image and combined pulse image features accordingly.

[0024] Furthermore, the facial feature encoder is used to extract facial features including face color, shape, eye color, and lip color, and to perform color gradation distribution analysis on facial skin color in the CIExyY color space; the consultation feature encoder is used to perform semantic encoding on structured question and answer text and extract semantic features related to symptoms and medical history.

[0025] Furthermore, the diagnostic output module outputs nine constitution types of the tested object, including balanced constitution, qi deficiency constitution, yang deficiency constitution, yin deficiency constitution, phlegm-dampness constitution, damp-heat constitution, blood stasis constitution, qi stagnation constitution, and special constitution; the syndrome types include visceral syndrome conclusions based on visceral differentiation.

[0026] Furthermore, the system also includes a model training module, which is used to construct a training set with the sample four diagnostic information and its corresponding constitution type label and syndrome type label, and to perform end-to-end joint training on the single-modal feature encoding module, cross-modal attention fusion module, adaptive gating fusion module and syndrome output module by constructing a joint loss function with syndrome differentiation loss, modality consistency loss and attention sparsity regularization term.

[0027] The core technical features and contributions of this invention compared to existing technologies are as follows: First, compared to the independent and deeply integrated four diagnostic subsystems in existing technology one (CN107242857A), this invention achieves fine-grained interactive alignment of heterogeneous modal information of the four diagnostic methods in a unified feature space through a cross-modal attention fusion module. This allows the five modal features of tongue appearance, facial appearance, pulse appearance, auscultation and olfaction, and inquiry to cross-verify around the diagnostic center vector, truly restoring the diagnostic logic of "combining the four diagnostic methods" in traditional Chinese medicine. Second, compared to the shallow splicing and fusion relying on prior knowledge in existing technology two (CN117334324A) and the traditional feature splicing and fusion in existing technology four (CN116543891A), this invention uses a learnable diagnostic center vector as the query and each modal feature as the key and value in a multi-head cross-modal attention mechanism. This mechanism can automatically learn the importance distribution of each modal internal feature dimension to the diagnostic result, rather than relying on manually set prior mapping rules, resulting in a deeper fusion method. Thirdly, compared with the existing technology three (CN119169656A), which only utilizes two modalities, pulse and tongue appearance and lacks adaptive capability, this invention covers five modalities of information in four categories: observation, auscultation, inquiry, and palpation. It also dynamically adjusts the fusion weights according to the acquisition quality and feature confidence of each modality through an adaptive gating fusion module, which can effectively cope with real acquisition scenarios such as modality missing, modality quality differences, and modality information conflicts, and significantly improves robustness. Fourthly, compared with the existing technology five (CN115512110A), which only uses cross-modal attention for tumor segmentation of two homogeneous image modalities, PET and CT, does not involve heterogeneous multimodal scenarios of the four diagnostic methods of traditional Chinese medicine and does not have interpretable mapping, this invention applies the cross-modal attention mechanism to the heterogeneous multimodal diagnosis scenario of the four diagnostic methods of traditional Chinese medicine for the first time, which simultaneously includes images, time-series signals, speech and text. It also establishes the correlation between cross-modal attention weights and the knowledge graph of traditional Chinese medicine pathogenesis through the pathogenesis interpretable mapping submodule, so that the diagnosis results have interpretability. The aforementioned distinguishing technical features are not simple combinations or conventional improvements made by those skilled in the art based on existing technologies. Rather, they require creative work to address technical challenges such as semantic space alignment of heterogeneous multimodal approaches in the four diagnostic methods of traditional Chinese medicine, robust handling of modality loss, and interpretable mapping of pathogenesis.

[0028] In summary, compared with the prior art, the present invention has at least one of the following beneficial technical effects:

[0029] Firstly, this invention introduces a multi-head cross-modal attention mechanism that uses a learnable diagnostic center vector as the query vector and heterogeneous modal features of the four diagnostic methods as key and value vectors. This mechanism enables fine-grained interactive alignment of information from the four diagnostic methods in a unified feature space. It can fully explore the deep and complementary semantic relationships between heterogeneous modalities such as tongue appearance, facial appearance, pulse appearance, auscultation and olfaction, and inquiry, effectively restoring the inherent diagnostic logic of "combining the four diagnostic methods and inferring the internal from the external" in traditional Chinese medicine, and significantly improving the accuracy of diagnosis.

[0030] Secondly, this invention uses an adaptive gating fusion module to dynamically calculate the fusion weight based on the acquisition quality information and feature confidence of each modality, thereby achieving adaptive robust fusion for modality missing, modality quality differences and modality information conflicts. When a modality is missing, its interference is suppressed by a learnable placeholder vector and sparsity regularization constraints, ensuring the stability and reliability of the system in real acquisition scenarios.

[0031] Third, by associating cross-modal attention weights with the TCM pathogenesis knowledge graph, this invention outputs an interpretable description of the diagnostic results, enabling the fusion process to form an intuitive correspondence with the TCM pathogenesis, thus meeting the requirements of prudence and trustworthiness for TCM clinical auxiliary decision-making.

[0032] Fourth, this invention achieves a fully integrated process from the collection of information from the four diagnostic methods, single-modal coding, cross-modal attention fusion, adaptive gating fusion to syndrome differentiation output. It can be deployed on hardware platforms such as tongue surface analyzers, pulse diagnostic instruments, and smart terminals, providing standardized and scalable TCM intelligent syndrome differentiation services for scenarios such as communities, grassroots, elderly care, and senior citizens. Attached Figure Description

[0033] Figure 1 A schematic diagram of the overall architecture of a diagnostic system based on cross-modal attention mechanism for adaptive fusion of four diagnostic methods is provided in an embodiment of the present invention.

[0034] Figure 2 This is a schematic diagram of the structure of a single-modal feature encoding module provided in an embodiment of the present invention;

[0035] Figure 3 This is a schematic diagram of the cross-modal attention fusion module provided in an embodiment of the present invention;

[0036] Figure 4 This is a schematic diagram of the adaptive gating fusion module provided in an embodiment of the present invention;

[0037] Figure 5 This is a schematic diagram of the structure of the dialectical output module and the interpretable mapping of pathogenesis provided in the embodiments of the present invention. Detailed Implementation

[0038] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0039] This invention provides an adaptive fusion and diagnostic system based on cross-modal attention mechanisms for the four diagnostic methods (inspection, auscultation, inquiry, and palpation) in Traditional Chinese Medicine. The core idea is to address the heterogeneous, heterogeneous, and complementary characteristics of these four diagnostic methods by constructing a cross-modal attention mechanism. This mechanism uses a learnable diagnostic center vector as the query and heterogeneous modal features of the four diagnostic methods as keys and values, achieving fine-grained interactive alignment of the information in a unified semantic space. Simultaneously, an adaptive gating fusion mechanism is constructed to address modal missingness and quality differences, dynamically adjusting the fusion weights of each modality. Ultimately, the system outputs interpretable diagnostic results for constitution type and syndrome type.

[0040] Example 1: System Overall Architecture

[0041] like Figure 1 As shown in the figure, the four diagnostic methods information adaptive fusion diagnostic system based on cross-modal attention mechanism provided by the present invention is composed of a four diagnostic methods information acquisition module, a single-modal feature encoding module, a cross-modal attention fusion module, an adaptive gating fusion module and a diagnostic output module connected in sequence.

[0042] The four diagnostic methods information acquisition module is located at the front end of the system. It is responsible for collecting the four diagnostic methods information of the tested object and outputting the collected four types of modal data respectively. Among them, inspection information includes tongue and facial images acquired by the image acquisition device; auscultation information includes speech signals acquired by the sound pickup device; inquiry information includes structured question and answer text acquired by the interactive question and answer interface; and palpation information includes pulse timing signals acquired by the multi-channel array flexible pulse sensor. The four types of modal data output by the four diagnostic methods information acquisition module are respectively sent to the single-modal feature encoding module.

[0043] The unimodal feature encoding module receives four types of modal data and encodes them using tongue image feature encoder, facial image feature encoder, pulse image feature encoder, medical history feature encoder, and auscultation and olfaction feature encoder, respectively, to obtain the unimodal feature vector corresponding to each modality. The unimodal feature vectors output by the unimodal feature encoding module are then fed into the cross-modal attention fusion module.

[0044] The cross-modal attention fusion module receives single-modal feature vectors from each modality. First, it projects these vectors into a unified feature space through their respective fully connected projection layers, resulting in modal representation vectors of a unified dimension. Then, the module introduces a learnable dialectical center vector, using it as the query vector and the modal representation vectors from each modality as the key and value vectors, respectively, to perform multi-head cross-modal attention operations, yielding cross-modal enhanced feature vectors for each modality. The cross-modal enhanced feature vectors output by the cross-modal attention fusion module are then fed into the adaptive gating fusion module.

[0045] The adaptive gating fusion module receives cross-modal enhanced feature vectors from each modality and calculates the adaptive gating weights for each modality by combining the acquisition quality information and feature confidence scores. Subsequently, it performs a weighted summation of the cross-modal enhanced feature vectors based on the adaptive gating weights of each modality to obtain the global dialectical feature vector. When any modality data is missing, the adaptive gating fusion module replaces the missing modality's features with a learnable placeholder vector and constrains the corresponding adaptive gating weights of the missing modality to near zero. The global dialectical feature vector output by the adaptive gating fusion module is then fed into the dialectical output module.

[0046] The syndrome differentiation output module receives the global syndrome differentiation feature vector, performs syndrome differentiation classification through the syndrome differentiation classification head, and outputs the constitution type and syndrome type of the tested object. At the same time, the syndrome differentiation output module generates the attention contribution heatmap from each modality to each syndrome type based on the cross-modal attention weight matrix output by the cross-modal attention fusion module, and associates it with the pre-constructed TCM pathogenesis knowledge graph to output an interpretable description of the pathogenesis.

[0047] Through the collaborative work of the above five modules, this embodiment of the invention realizes the integration of the entire process from the collection of information from the four diagnostic methods to the output of interpretable syndrome differentiation. It can effectively restore the syndrome differentiation logic of "combining the four diagnostic methods and inferring the internal from the external" in traditional Chinese medicine, and has adaptive and robust fusion capabilities for modality loss and quality differences.

[0048] Example 2: Single-modal feature encoding module

[0049] like Figure 2 As shown, the single-modal feature encoding module includes a tongue image feature encoder, a facial image feature encoder, a pulse image feature encoder, a questioning feature encoder, and an auscultation feature encoder. The five encoders encode the features of the inspection information (tongue image image, facial image image), the palpation information (pulse time sequence signal), the questioning information (structured question and answer text), and the auscultation information (speech signal), respectively, and output the single-modal feature vector of the corresponding modality.

[0050] Specifically, the tongue image feature encoder receives a tongue image and first uses a semantic segmentation network to segment the tongue body, separating the tongue region from the oral cavity background. Then, it performs tongue coating separation processing on the tongue region, decoupling the tongue body layer from the tongue coating layer. Based on this, a convolutional neural network backbone network extracts multidimensional tongue image features of the tongue body and tongue coating. These multidimensional tongue image features include, but are not limited to, ten dimensions: tongue color, coating color, tongue shape, tongue body, peeling coating, greasy coating, purple tongue, cracks, enlargement, and teeth marks. The tongue image feature encoder concatenates and projects these multidimensional tongue image features to obtain a single-modal tongue image feature vector.

[0051] The facial feature encoder receives facial images and extracts facial features such as skin tone, complexion, shape, eye color, and lip color by detecting facial key points and aligning them with facial regions. Specifically, the facial skin tone features undergo color gradation distribution analysis in the CIExyY color gamut. By mapping the facial skin tone to the CIExyY color gamut map, the color gradation distribution statistics of the facial skin tone are calculated and incorporated as part of the facial features. The facial feature encoder then concatenates and projects these facial features to obtain a single-modal facial feature vector.

[0052] The pulse feature encoder receives the pulse temporal signal and first performs filtering, denoising, and normalization preprocessing on the pulse temporal signal. Then, it performs three-segment processing on the pulse temporal signal according to the Cun, Guan, and Chi positions in traditional Chinese medicine pulse diagnosis, dividing the pulse temporal signal into three segments: Cun, Guan, and Chi. Based on this, pulse features are extracted under three pulse-taking pressures: light, medium, and heavy. Furthermore, a temporal neural network (such as a gated recurrent unit or a temporal convolutional network) is used to perform temporal modeling of the above pulse features, outputting single pulse features and combined pulse features. The single pulse features can cover eighteen types, including wiry, thin, slippery, rapid, slow, floating, deep, weak, full, surging, faint, tight, slow, soft, weak, scattered, hollow, and leathery. The combined pulse features are composed of multiple single pulse features, covering up to one hundred and seventy-eight types of combined pulse features. The pulse feature encoder concatenates and projects the above pulse features to obtain a pulse single-modal feature vector.

[0053] The consultation feature encoder receives structured question-and-answer text. First, it performs word segmentation and stop word removal on the text. Then, it uses a pre-trained text encoding model (such as a bidirectional encoder based on a Transformer structure) to semantically encode the text, extracting semantic features related to symptoms, medical history, and daily life, resulting in a single-modal feature vector for consultation. The structured question-and-answer text can come from a dynamic intelligent consultation engine. This engine abandons fixed questions and dynamically generates subsequent questions based on the real-time responses of the test subject, completing the collection of consultation information through full-process voice interaction.

[0054] The auscultation feature encoder receives the speech signal, first performs framing, windowing and acoustic feature extraction (such as Mel frequency cepstral coefficients) on the speech signal, and then encodes the acoustic features through the acoustic encoder to extract the acoustic features related to speech and breath, thus obtaining the auscultation single-modal feature vector.

[0055] It should be noted that the dimensions of the single-modal feature vectors output by each single-modal feature encoder may be different, but before being fed into the cross-modal attention fusion module, they must all be projected to a unified feature dimension through their respective fully connected projection layers to ensure the consistency of subsequent cross-modal attention operations.

[0056] Example 3: Cross-modal attention fusion module

[0057] like Figure 3 As shown, the cross-modal attention fusion module includes a modal projection layer, a dialectical center vector, a multi-head cross-modal attention layer, and an inter-modal interaction attention layer.

[0058] The modal projection layer is used to project the single-modal feature vectors of each modality onto a unified feature space. Let h_m be the single-modal feature vector of the m-th modality, then its modal representation vector z_m in the unified feature space is calculated using the following formula:

[0059] z_m=W_m^{proj}·h_m+b_m^{proj};

[0060] Where W_m^{proj} is the projection matrix of the m-th mode, b_m^{proj} is the projection bias of the m-th mode, and m takes one of the five modes: tongue appearance, facial appearance, pulse appearance, inquiry, and auscultation.

[0061] The diagnostic center vector is a learnable parameter vector, denoted as z_0. This diagnostic center vector can be understood as a learnable "diagnostic query anchor point," which automatically learns how to perform a global "parameter-based" query on the four diagnostic methods during training, thereby guiding the cross-modal attention mechanism to focus on the modal features most valuable for diagnostic purposes.

[0062] The multi-head cross-modal attention layer uses the dialectical center vector z_0 as the query and the modal representation vectors z_m of each modality as the key and value to perform multi-head cross-modal attention operations. For the i-th attention head, the operation process is as follows:

[0063] First, project the query, key, and value through their respective projection matrices:

[0064] Q_i=W_i^Q·z_0, K_{m,i}=W_i^K·z_m, V_{m,i}=W_i^V·z_m;

[0065] Where W_i^Q, W_i^K, and W_i^V are the query projection matrix, key projection matrix, and value projection matrix of the i-th attention head, respectively.

[0066] Secondly, calculate the attention score between the query and the key, and then scale and normalize it:

[0067] A_{m,i}=softmax(Q_i·K_{m,i}^T / √d_k);

[0068] Where d_k is the dimension of the key vector, √d_k is the scaling factor, and softmax is the normalization function. A_{m,i} is the cross-modal attention weight of the dialectical center vector under the i-th attention head to the m-th modality, which represents the importance of each feature dimension of the m-th modality to the current dialectical query during the dialectical process.

[0069] Next, the value vector V_{m,i} is weighted and summed with attention weights A_{m,i} to obtain the output of the i-th attention head:

[0070] O_{m,i}=A_{m,i}·V_{m,i};

[0071] Finally, the outputs of each attention head are concatenated and projected to obtain the cross-modal enhancement feature vector of the m-th modality:

[0072] z_m^{cross}=W^O·Concat(O_{m,1},O_{m,2},...,O_{m,H});

[0073] Where H is the number of attention heads, W^O is the output projection matrix, and Concat represents the concatenation operation.

[0074] In a specific implementation, the dimension d of the unified feature space can be 512, the number of attention heads H can be 8, and the dimensions d_k (key vector) and d_v (value vector) can both be 64, i.e., d = H × d_k = 8 × 64 = 512. To ensure training stability, the multi-head cross-modal attention layer sets layer normalization before and after the attention operation, and superimposes residual connections after the output projection, i.e., z_m^{cross} = LayerNorm(z_m + W^O·Concat(O_{m,1},O_{m,2},...,O_{m,H})), where LayerNorm represents the layer normalization operation. Simultaneously, to prevent overfitting, dropout can be introduced into the attention weights and each feedforward network layer, with a dropout probability of 0.1. The above hyperparameter values ​​are merely illustrative examples; those skilled in the art can make appropriate adjustments according to the actual task and data scale, without departing from the scope of protection of this invention.

[0075] The intermodal interaction attention layer is used to capture complementary information between different modalities. It uses the cross-modal enhanced feature vector of one modality as a query and the cross-modal enhanced feature vector of another modality as the key and value, performing pairwise interaction attention operations to further explore the cross-verification relationships between modal pairs such as tongue and pulse, tongue and facial features, and pulse and medical history. Finally, it outputs the cross-modal enhanced feature vectors of each modality after intermodal interaction enhancement. Specifically, for any modal pair (m,n), the intermodal interaction attention operation is as follows: using the cross-modal enhanced feature vector z_m^{cross} of modality m as a query and the cross-modal enhanced feature vector z_n^{cross} of modality n as the key and value, the interaction attention weight of modality m on modality n is calculated, and the features of modality n are weighted and aggregated accordingly to obtain the interaction enhanced features of modality m; subsequently, the original cross-modal enhanced feature vector of modality m and its interaction enhanced features are concatenated using residuals to obtain the final cross-modal enhanced feature vector of modality m. By traversing all modality pairs, the intermodal attention layer enables each modality to absorb complementary information from other modalities, thereby further enhancing the cross-verification effect of the four diagnostic methods.

[0076] Through the aforementioned cross-modal attention fusion module, this invention achieves fine-grained interactive alignment of heterogeneous modal information of the four diagnostic methods in a unified semantic space. Compared with the simple splicing or shallow fusion in the prior art, it can more fully explore the deep complementary semantic relationships between the four diagnostic methods.

[0077] Example 4: Adaptive Gated Fusion Module

[0078] like Figure 4 As shown, the adaptive gating fusion module includes a modality confidence assessment submodule, a gating weight calculation submodule, a missing modality processing submodule, and a weighted fusion submodule.

[0079] The modality confidence assessment submodule is used to evaluate the confidence level of each modality. Specifically, for each modality, the modality confidence vector c_m is determined jointly based on the acquisition quality index, feature completeness index, and single-modality classification confidence index of the modality data. The acquisition quality index includes image sharpness, illumination uniformity, and signal-to-noise ratio; the feature completeness index reflects the completeness of feature extraction for that modality; and the single-modality classification confidence index is the classification confidence level output by a single-modality classifier for the features of that modality. These three indices are normalized and concatenated to obtain the modality confidence vector c_m. In a specific implementation, image sharpness can be quantified using image gradient energy or Laplacian operator variance, illumination uniformity can be quantified using image block brightness standard deviation, and signal-to-noise ratio can be estimated by the ratio of signal power to noise power. After the above quantification indicators are normalized to the [0,1] interval, they are concatenated with the single-modal classification confidence to form the modal confidence vector c_m. Its dimension can be different from the dimension of the cross-modal enhancement feature vector z_m^{cross}. Before being fed into the gate control network, a linear projection layer can be used to align the two to the same dimension.

[0080] The gating weight calculation submodule is used to calculate adaptive gating weights based on the cross-modal enhancement feature vectors and modal confidence vectors of each modality. Specifically, the cross-modal enhancement feature vector z_m^{cross} of the m-th modality is concatenated with the modal confidence vector c_m, mapped through a gating network and activation function, and then normalized to obtain the adaptive gating weights of the m-th modality. The calculation process can be represented by the following formula:

[0081] g_m=σ(W_g·Concat(z_m^{cross},c_m)+b_g);

[0082] α_m = g_m / Σ_ng_n;

[0083] Where W_g is the weight matrix of the gating network, b_g is the bias, σ is the activation function (such as the sigmoid function), Concat represents concatenation, g_m is the intermediate gating value of the m-th mode, α_m is the normalized adaptive gating weight of the m-th mode, and Σ_ng_n is the sum of the intermediate gating values ​​of all modes. Through the above normalization process, the sum of the adaptive gating weights of each mode is 1.

[0084] The missing modality processing submodule handles the case of missing modalities. When the m-th modality data is missing, the missing modality processing submodule replaces the cross-modal augmented feature vector z_m^{cross} corresponding to the missing modality with a learnable placeholder vector z_{pad}. During training, it applies a sparsity regularization constraint to the adaptive gating weight α_m corresponding to the missing modality, making α_m approach zero, thereby suppressing the interference of the missing modality on the global dialectical feature vector. The sparsity regularization constraint can use an L1 norm regularization term or other regularization methods that promote weight sparsity. Specifically, the placeholder vector z_{pad} is a learnable parameter vector with the same dimension as the cross-modal augmented feature vector, which is updated during training to occupy a neutral position in the feature space that does not contribute to the dialectical result. During the training phase, a certain proportion of modalities are randomly masked in the training samples (e.g., randomly masking one or more modalities with a probability of 10% to 30%), enabling the model to learn the correct way to handle missing modalities. During the inference phase, when the system detects that a modality is missing due to equipment failure, user refusal to cooperate, or acquisition failure, it automatically replaces the features of that modality with placeholder vectors and suppresses the corresponding gating weights to near zero based on what has been learned during training, thereby ensuring that the fusion result is not contaminated by missing modalities. Compared to existing technologies that directly discard missing modalities or pad them with zeros, this invention, through the synergy of learnable placeholder vectors and sparsity regularization constraints, can achieve robust suppression of missing modalities while maintaining consistent input dimensions and avoiding destruction of information structure.

[0085] The weighted fusion submodule is used to perform a weighted summation of the cross-modal enhanced feature vectors of each modality based on the adaptive gating weights of each modality, to obtain the global dialectical feature vector. Its calculation process can be represented by the following formula:

[0086] z_fusion=Σ_mα_m·z_m^{cross};

[0087] Where z_fusion is the global dialectical feature vector, and Σ_m represents the summation over all modalities.

[0088] Through an adaptive gating fusion module, this invention can dynamically adjust the fusion weights based on the actual acquisition quality and confidence level of each modality: when a modality has high quality and high confidence, its weight increases accordingly, thus playing a greater role in the fusion result; when a modality is missing or of poor quality, its weight decreases accordingly or even approaches zero, thereby avoiding interference with the fusion result. This mechanism enables the system to have good robustness and stability in real-world acquisition scenarios.

[0089] Example 5: Mapping the Diagnostic Output Module to the Explainable Pathogenesis

[0090] like Figure 5 As shown, the syndrome differentiation output module includes a syndrome differentiation classification head, a constitution classification output, a syndrome classification output, and a pathogenesis interpretable mapping submodule.

[0091] The syndrome differentiation and classification head receives the global syndrome differentiation feature vector z_fusion and performs syndrome differentiation and classification through one or more fully connected layers and a softmax activation function. Specifically, the constitution classification output corresponds to a classifier that identifies nine constitution types of the tested object, including balanced constitution, qi deficiency constitution, yang deficiency constitution, yin deficiency constitution, phlegm-dampness constitution, damp-heat constitution, blood stasis constitution, qi stagnation constitution, and special constitution. The syndrome classification output corresponds to a classifier that identifies the syndrome type of the tested object, including visceral syndrome conclusions based on visceral differentiation, such as liver blood deficiency syndrome, spleen qi deficiency syndrome, and kidney yang deficiency syndrome.

[0092] Specifically, the body constitution classification output can be represented by the following formula:

[0093] p_tizhi=softmax(W_t·z_fusion+b_t);

[0094] Where W_t is the weight matrix for body constitution classification, b_t is the bias, and p_tizhi is the probability distribution of the nine body constitution types. The body constitution type with the highest probability is selected as the body constitution type of the tested subject.

[0095] The pathogenesis interpretable mapping submodule is used to generate interpretable descriptions of the diagnostic results. Specifically, based on the cross-modal attention weight matrix A_{m,i} output by the cross-modal attention fusion module, it generates an attention contribution heatmap from each modality to each syndrome type by aggregating the weights of each attention head. Subsequently, this attention contribution heatmap is associated with a pre-constructed TCM pathogenesis knowledge graph. The TCM pathogenesis knowledge graph stores the associations between each modality feature, each syndrome type, and the pathogenesis, such as "pale tongue—blood deficiency—liver blood deficiency syndrome", "wiry and thready pulse—liver blood deficiency—liver blood deficiency syndrome", "sallow complexion—spleen deficiency—spleen qi deficiency syndrome", etc. By matching and associating high-weight modal features in the attention contribution heatmap with pathogenesis nodes in the knowledge graph, the pathogenesis interpretable mapping submodule outputs an interpretable description of the pathogenesis used to explain the diagnosis results, such as "The diagnosis results are mainly based on the tongue appearance (weight 0.42) and pulse appearance (weight 0.35), among which the pale tongue color and the wiry and thin pulse both point to the pathogenesis of liver blood deficiency".

[0096] The construction process of the TCM pathogenesis knowledge graph is as follows: First, extract the "syndrome-symptom-pathogenesis" tripartite from TCM classics, textbooks, and clinical medical records. The TCM classics include, but are not limited to, classic works such as *Huangdi Neijing*, *Shanghan Zabing Lun*, *Jinkui Yaolue*, and *Wenbing Tiaobian*, as well as various editions of TCM diagnostic textbooks. Second, perform entity alignment and disambiguation on the syndrome entities, symptom entities, and pathogenesis entities in the extracted tripartite to construct a knowledge graph containing syndrome nodes, symptom nodes, and pathogenesis nodes. Third, assign type labels and weights to the association edges between the above nodes. The type labels include relationship types such as "manifests as", "pathogenesis is", "concurrent", and "transformation". Finally, map each modal feature (such as tongue color, coating color, pulse, etc.) to symptom nodes in the knowledge graph, thereby establishing an interpretable association path from modal features to syndromes and pathogenesis. Through the aforementioned knowledge graph, the pathogenesis-explainable mapping submodule can provide a complete explanatory chain of "which four diagnostic features and pathogenesis pathways led to the diagnosis" while outputting the diagnosis results, significantly improving the credibility and clinical acceptability of the diagnosis results.

[0097] Through the syndrome differentiation output module and the pathogenesis interpretable mapping submodule, this invention not only outputs the constitution type and syndrome type, but also outputs the interpretability basis of the syndrome differentiation process, which meets the requirements of prudence and trustworthiness for TCM clinical auxiliary decision-making, and facilitates physicians to review and verify the syndrome differentiation results.

[0098] Example 6: Model Training Method

[0099] The system provided in this embodiment of the invention also includes a model training module. The model training module constructs a training set using the sample's four diagnostic methods information and its corresponding constitution type label and syndrome type label, and uses a joint loss function consisting of syndrome differentiation loss, modality consistency loss and attention sparsity regularization term to perform end-to-end joint training on the single-modal feature encoding module, cross-modal attention fusion module, adaptive gating fusion module and syndrome differentiation output module.

[0100] Specifically, the joint loss function can be expressed by the following formula:

[0101] L=L_cls+λ_1·L_cons+λ_2·L_sparse;

[0102] Wherein, L_cls is the syndrome classification loss, which can be the sum of the cross-entropy losses of constitution classification and syndrome classification; L_cons is the modality consistency loss, used to constrain the consistency of each modality before and after cross-modal fusion, so that the feature representations of different modalities for the same syndrome result tend to be consistent, and can be the contrast loss or mean squared error loss; L_sparse is the attention sparsity regularization term, used to make the cross-modal attention weights and adaptive gating weights tend to be sparse, so that the model focuses on the modal features most critical to the syndrome, while suppressing the weights of missing modalities; λ_1 and λ_2 are balance coefficients, used to adjust the weights between the loss terms, and the typical value range of λ_1 and λ_2 is 0.01 to 0.5, and the specific values ​​can be determined through validation set tuning.

[0103] During training, training samples can cover samples with different modalities, including complete samples of the four diagnostic methods, samples with missing single modalities, samples with missing multimodalities, and samples with differences in modal quality. This allows the model to fully learn adaptive fusion capabilities for modal missingness and quality differences. In a specific implementation, model training can use the Adam optimizer with an initial learning rate of 1×10^-4 and a cosine annealing learning rate decay strategy. The batch size can be 32, and the number of training epochs can be 100 to 200. An early stopping strategy is also used, i.e., training stops when the validation set loss no longer decreases within a certain number of consecutive epochs, in order to select the model with the best generalization ability. To improve the generalization and robustness of the model, data augmentation techniques can also be used during the training phase, including random rotation, flipping, brightness perturbation, and color jittering of tongue and facial images, random amplitude scaling and time shifting of pulse time-series signals, and synonym replacement of question-and-answer text. The values ​​of the hyperparameters and training strategies mentioned above are illustrative examples, and those skilled in the art can make adaptive adjustments according to the actual data scale and computing resources.

[0104] Example 7: System Deployment and Hardware Implementation

[0105] The system provided in this invention can be deployed on hardware platforms such as tongue analyzers, pulse diagnostic instruments, and AI-powered TCM intelligent terminals. In a specific deployment embodiment, the system is deployed on an AI-powered TCM four diagnostic intelligent terminal. This terminal integrates a 15.6-inch interactive screen, a 12-megapixel image acquisition device, a multi-channel array flexible pulse sensor, and a sound pickup device to acquire tongue images, facial images, pulse timing signals, and voice signals, respectively. The pulse sensor uses a multi-channel array flexible sensor to acquire pulse images at the cun, guan, and chi positions through three-segment pressure application, reproducing three pulse-taking methods: light, medium, and heavy pressure. The terminal supports full-process voice interaction and dynamic consultation, making it suitable for self-service operation by elderly individuals.

[0106] After collecting the four diagnostic methods (inspection, auscultation and olfaction) information of the tested subject, the system performs single-modal feature encoding, cross-modal attention fusion, adaptive gating fusion, and syndrome differentiation output in the cloud or local edge computing unit. Finally, it generates a syndrome differentiation report containing interpretable descriptions of constitution type, syndrome type, and pathogenesis, which is then displayed on an interactive screen. The entire syndrome differentiation process can be completed within minutes, achieving standardization, objectivity, and intelligence in the integrated application of the four diagnostic methods in Traditional Chinese Medicine.

[0107] Example 8: Overall System Workflow

[0108] This invention also provides an adaptive fusion diagnostic method for four diagnostic methods based on cross-modal attention mechanism. This method is executed by the aforementioned system and includes the following steps:

[0109] Step S1: Collect the four diagnostic information of the tested object. The four diagnostic information includes at least four types of modal data: inspection information, auscultation information, inquiry information and palpation information. Among them, inspection information includes tongue image and facial image, auscultation information includes voice signal, inquiry information includes structured question and answer text, and palpation information includes pulse time sequence signal.

[0110] Step S2: Each modal data in the four diagnostic information is feature-encoded to obtain the single modal feature vector corresponding to each modality; wherein, after tongue body segmentation and tongue coating separation, the tongue image extracts multi-dimensional tongue features such as tongue color, coating color, tongue shape, tongue texture, peeled coating, greasy coating, purple tongue, cracks, enlargement, and teeth marks; after the pulse time sequence signal is processed by three-segment segmentation of cun, guan, and chi, single pulse and combined pulse features under three pulse pressures of light, medium, and heavy pulse are extracted respectively.

[0111] Step S3: Project the single-modal feature vectors of each modality onto a unified feature space, and use the learnable dialectical center vector as the query vector, and the single-modal feature vectors of each modality as the key vector and value vector, to perform multi-head cross-modal attention operation to obtain the cross-modal enhancement feature vectors corresponding to each modality.

[0112] Step S4: Calculate the adaptive gating weights of each modality based on the acquisition quality information and feature confidence of each modality, and perform weighted fusion of the cross-modal enhanced feature vectors of each modality according to the adaptive gating weights to obtain the global dialectical feature vector; wherein, when any modality data is missing, the feature of the missing modality is replaced by a learnable placeholder vector, and the adaptive gating weights corresponding to the missing modality are constrained to approach zero.

[0113] Step S5: Perform syndrome differentiation and classification based on the global syndrome differentiation feature vector, output the constitution type and syndrome type of the tested object, and generate the attention contribution heat map from each modality to each syndrome type based on the cross-modal attention weight matrix obtained by the multi-head cross-modal attention operation. Associate the attention contribution heat map with the TCM pathogenesis knowledge graph and output the interpretable description of the pathogenesis.

[0114] The steps S1 to S5 described above correspond one-to-one with the functions of each module of the aforementioned system. Those skilled in the art will understand that each step of the method can be executed by the corresponding module or device in the aforementioned embodiments. The specific implementation details can be referred to the descriptions of the aforementioned embodiments one to seven, which will not be repeated here.

[0115] The embodiments described herein are preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape, and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A diagnostic system based on cross-modal attention mechanism that adaptively fuses information from the four diagnostic methods, characterized in that, include: The four diagnostic information acquisition module is used to collect the four diagnostic information of the tested object. The four diagnostic information includes at least four types of modal data: inspection information, auscultation information, inquiry information, and palpation information. Among them, the inspection information includes tongue image and facial image, the auscultation information includes voice signal, the inquiry information includes structured question and answer text, and the palpation information includes pulse time sequence signal. The single-modal feature encoding module is used to encode the features of each type of modality data in the four diagnostic information to obtain the single-modal feature vector corresponding to each modality; The cross-modal attention fusion module is used to project the single-modal feature vectors of each modality onto a unified feature space, and use the learnable dialectical center vector as the query vector, and the single-modal feature vectors of each modality as the key vector and value vector to perform multi-head cross-modal attention operation to obtain the cross-modal enhanced feature vectors corresponding to each modality. The adaptive gating fusion module is used to calculate the adaptive gating weights of each modality based on the acquisition quality information and feature confidence of each modality, and to perform weighted fusion of the cross-modal enhanced feature vectors of each modality according to the adaptive gating weights to obtain a global dialectical feature vector; wherein, when any modality data is missing, the features of the missing modality are replaced by a learnable placeholder vector, and the adaptive gating weights corresponding to the missing modality are constrained to approach zero; The dialectical output module is used to perform dialectical classification based on the global dialectical feature vector, output the constitution type and syndrome type of the tested object, and output interpretability weights to characterize the degree of contribution of each modality to the dialectical result.

2. The diagnostic system for four diagnostic methods based on cross-modal attention mechanism according to claim 1, characterized in that, In the cross-modal attention fusion module, the multi-head cross-modal attention operation includes: taking the query matrix obtained by projecting the dialectical center vector through the query and the key matrix obtained by projecting the single-modal feature vectors of each modality through the key, scaling and normalizing the result to obtain the cross-modal attention weight matrix; performing a weighted summation of the cross-modal attention weight matrix and the value matrix obtained by projecting the single-modal feature vectors of each modality through the value to obtain the cross-modal enhancement feature vectors corresponding to each modality; the cross-modal attention fusion module also performs inter-modal interactive attention operation on the cross-modal enhancement feature vectors between different modalities to capture complementary information between modalities.

3. The diagnostic system for four diagnostic methods based on cross-modal attention mechanism according to claim 1, characterized in that, In the adaptive gating fusion module, the adaptive gating weights of each modality are calculated as follows: the cross-modal enhanced feature vectors of each modality are concatenated with the corresponding modality confidence vectors, mapped through a gating network and activation function, and then normalized to obtain the adaptive gating weights of each modality; the modality confidence vectors are determined based on the acquisition quality index, feature completeness index, and single-modality classification confidence index of the modality data.

4. The diagnostic system based on cross-modal attention mechanism for adaptive fusion of four diagnostic methods according to claim 1, characterized in that, When any modality data is missing, the adaptive gating fusion module replaces the cross-modal enhanced feature vector corresponding to the missing modality with a learnable placeholder vector, and applies sparsity regularization constraints to the adaptive gating weights corresponding to the missing modality during training, so that the weights approach zero, thereby suppressing the interference of the missing modality on the global dialectical feature vector.

5. The diagnostic system for four diagnostic methods based on cross-modal attention mechanism according to claim 1, characterized in that, The diagnostic output module is further configured to: generate an attention contribution heatmap from each modality to each syndrome type based on the cross-modal attention weight matrix output by the cross-modal attention fusion module, and associate the attention contribution heatmap with a pre-constructed TCM pathogenesis knowledge graph to output a pathogenesis interpretability description for explaining the diagnostic results.

6. The diagnostic system based on cross-modal attention mechanism for adaptive fusion of four diagnostic methods according to claim 1, characterized in that, The single-modal feature encoding module includes a tongue image feature encoder, a facial image feature encoder, a pulse image feature encoder, a medical history feature encoder, and an auscultation and olfaction feature encoder. The tongue image feature encoder is used to sequentially perform tongue body segmentation and tongue coating separation processing on the tongue image image, and extract multi-dimensional tongue image features including tongue color, coating color, tongue shape, tongue texture, peeled coating, greasy coating, purple tongue, cracks, enlargement, and teeth marks.

7. A diagnostic system based on cross-modal attention mechanism for adaptive fusion of four diagnostic methods, as described in claim 6, is characterized in that... The pulse feature encoder is used to perform three-segment processing of the pulse time sequence signal (cun, guan, chi) to extract pulse features under three pulse pressures (light, medium, and heavy), and output single pulse features and combined pulse features accordingly. The facial feature encoder is used to extract facial features including complexion, shape, eye color, and lip color, and perform color gradation distribution analysis on facial skin in the CIExyY color space. The consultation feature encoder is used to perform semantic encoding on structured question and answer text to extract semantic features related to symptoms and medical history.

8. The diagnostic system for four diagnostic methods based on cross-modal attention mechanism according to claim 1, characterized in that, The diagnostic output module outputs nine constitution types of the tested object, including balanced constitution, qi deficiency constitution, yang deficiency constitution, yin deficiency constitution, phlegm-dampness constitution, damp-heat constitution, blood stasis constitution, qi stagnation constitution, and special constitution; the syndrome types include the syndrome conclusions of the viscera based on the differentiation of syndromes of the viscera.

9. A diagnostic system based on cross-modal attention mechanism for adaptive fusion of four diagnostic methods, as described in claim 1, is characterized in that... The system also includes a model training module, which is used to construct a training set with sample four diagnostic information and their corresponding constitution type labels and syndrome type labels, and to perform end-to-end joint training on the single-modal feature encoding module, cross-modal attention fusion module, adaptive gating fusion module and syndrome output module by constructing a joint loss function with syndrome differentiation loss, modality consistency loss and attention sparsity regularization term.

10. A diagnostic system based on cross-modal attention mechanism for adaptive fusion of four diagnostic methods, as described in claim 1, is characterized in that, The four diagnostic information acquisition module is integrated into a smart terminal, which includes an interactive screen, an image acquisition device, a multi-channel array flexible pulse sensor, and a sound pickup device. The multi-channel array flexible pulse sensor acquires pulse images in three segments (cun, guan, chi) under pressure to reproduce three pulse-taking methods: light, medium, and heavy. The smart terminal supports full-process voice interaction and dynamic consultation.

Citation Information

Patent Citations

  • Intelligent traditional Chinese medicine comprehensive diagnosis and treatment system based on deep learning

    CN107242857A

  • Medical image tumor segmentation method involving cross-modal attention mechanism

    CN115512110A

  • Traditional Chinese medicine constitution analysis system and method based on fusion tongue picture characteristics and simplified scale, terminal and medium

    CN116543891A

  • Traditional Chinese medicine syndrome differentiation analysis method and device based on fusion of four diagnosis information and prior knowledge

    CN117334324A

  • Traditional Chinese medicine constitution identification method and device based on multi-modal characteristics of pulse condition and tongue condition

    CN119169656A