Traditional Chinese medicine constitution identification multi-modal data fusion method and system

Abstract

The application discloses a traditional Chinese medicine constitution identification multi-modal data fusion method and system, relates to the technical field of constitution intelligent identification, and comprises the following steps: continuously collecting time sequence pulsation signals, tongue image and subjective symptom text in multiple time windows of morning fasting, daytime resting and night before sleeping; constructing a time sequence counteracting self-encoder to separate time-varying interference factors and constitution steady-state factors, combining a dynamic time warping algorithm to fuse multi-window steady-state characteristics; calculating modality confidence through a space-time graph convolution network and eliminating abnormal data; and finally outputting a constitution type with a confidence interval and a constitution evolution trend chart. The method can effectively filter transient interference such as diet and exercise, extract constitution steady-state fingerprints, realize stable and accurate dynamic constitution identification, significantly improve the reliability of health index calculation and individual health risk assessment, and provide objective support for traditional Chinese medicine constitution conditioning and curative effect evaluation.

Description

Technical Field

[0001] This invention relates to the field of intelligent constitution identification technology, specifically to a method and system for multimodal data fusion in traditional Chinese medicine constitution identification. Background Technology

[0002] With the rapid popularization of mobile healthcare and home-based health monitoring, Traditional Chinese Medicine (TCM) constitution identification, as an important basis for personalized health management, disease prevention, and conditioning intervention, has gradually evolved from traditional offline consultations towards intelligent, non-invasive, and home-based approaches. Currently, most mainstream TCM constitution identification solutions rely on tongue images, pulse signals, or constitution questionnaire information collected at a single time point, using feature matching or simple classification models to determine constitution type. While this approach has achieved a degree of digitalization and convenience in TCM diagnosis, it does not fully consider the inherent characteristics of TCM constitution—its "dynamic evolution" and "state sensitivity"—and thus has significant technical shortcomings in practical applications.

[0003] Traditional Chinese medicine (TCM) constitution is not a fixed, static characteristic, but a dynamic physiological state that is influenced by various immediate factors such as daily routines, diet, exercise, emotions, and ambient temperature, and changes slowly with conditioning and adjustments to daily routines. Data collected in a single instance is highly susceptible to transient interference, such as pulse distortion due to increased heart rate after exercise, tongue staining after eating or drinking coffee leading to misinterpretation of tongue appearance, and subjective symptom description bias caused by emotional fluctuations. Such data is essentially "pseudo-fluctuation" information reflecting immediate conditions and cannot represent the user's long-term, stable, inherent constitution characteristics.

[0004] Existing technologies lack mechanisms for processing time-series data, making it impossible to distinguish between transient interference data and true physical characteristics. They typically input single sets of multimodal data directly into the identification model, leading to frequent inconsistencies, low confidence levels, and poor stability in the identification results. Furthermore, existing methods often employ simple feature fusion approaches, failing to establish spatiotemporal correlations between pulse, tongue appearance, and subjective symptoms. This makes it difficult to identify overall misjudgments caused by distortions in a single modality, and also prevents the quantification of the reliability of each modality's contribution to physical condition assessment.

[0005] Furthermore, current TCM constitution identification systems generally lack dynamic monitoring and trend analysis capabilities, failing to generate continuous constitution change curves and reflect subtle improvements or deterioration trends during the treatment process. They cannot provide objective data support for efficacy evaluation, nor can they accurately calculate health indices or conduct long-term individual health risk assessments based on stable constitution characteristics. Users often obtain unreliable results due to inappropriate data collection timing or unstable conditions, leading to decreased user compliance and trust in intelligent identification systems.

[0006] In summary, existing technologies cannot remove transient pseudo-fluctuations or extract steady-state constitution characteristics from time-series multimodal data. They lack time-series correlation and anomaly removal mechanisms, making it difficult to meet the needs of accurate, stable, and dynamic TCM constitution identification. This also restricts the accuracy and practicality of health index calculation and individual health risk assessment. Summary of the Invention

[0007] The purpose of this invention is to provide a method and system for multimodal data fusion in TCM constitution identification to solve the above-mentioned problems.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a multimodal data fusion method for TCM constitution identification, comprising the following steps:

[0009] Step 1: Multimodal time-series data acquisition. Through wearable devices and mobile terminals, users' time-series pulsation signals, time-series tongue images, and time-series subjective symptom texts are continuously acquired in multiple preset time windows. The time windows include at least three scenarios: morning on an empty stomach, daytime rest, and nighttime before bedtime. After acquisition, the time-series tongue images are processed to remove motion blur, providing high-quality basic data for subsequent health index calculation and individual health risk assessment.

[0010] Step 2: Multimodal data fusion processing, constructing a temporal adversarial autoencoder, mapping the multimodal temporal data collected in Step 1 to the latent space, separating time-varying interference factors and constitution homeostasis factors through adversarial training. The constitution homeostasis factors serve as "homeostasis fingerprints" representing the user's inherent TCM constitution. Then, the similarity weight of the constitution homeostasis factors under each time window is calculated using the dynamic time warping algorithm, completing the fusion of multi-window homeostasis factors, laying the foundation for accurate calculation of health index and assessment of individual health risks;

[0011] Step 3: Abnormal data identification and removal. Based on the fused constitution homeostasis factors from Step 2, a spatiotemporal graph convolutional network is constructed. The three modalities of data—temporal pulsation signal, temporal tongue image, and temporal subjective symptom text—are used as nodes in the graph, and the time window is used as the edge of the graph. The confidence of each modal data in constitution determination is calculated, and abnormal modal data that conflict with historical homeostasis features are automatically identified and removed to avoid abnormal data interfering with the accuracy of health index calculation and individual health risk assessment.

[0012] Step 4: Output of constitution identification results. Based on the constitution homeostasis factor after removing abnormal data in Step 3, output the TCM constitution type with confidence interval and generate a constitution evolution trend chart to achieve accurate TCM constitution identification for calculating health index and for individual health risk assessment.

[0013] Furthermore, the wearable device includes a photoelectric sensor and a pressure sensor for collecting the user's temporal pulsation signals, and the mobile terminal is used to collect temporal tongue images and receive the user's input of temporal subjective symptom text.

[0014] Furthermore, the motion blur removal process in step 1 employs an adaptive Wiener filtering algorithm. By analyzing the blur kernel parameters of the tongue image, the blurred areas are repaired in a targeted manner to ensure the clarity of the tongue coating and tongue body features in the tongue image, providing reliable tongue image data support for health index calculation and individual health risk assessment.

[0015] Furthermore, the temporal adversarial autoencoder in step 2 includes an encoder, a decoder, and a discriminator. The encoder is used to map multimodal temporal data to a latent space, the decoder is used to reconstruct the feature vectors of the latent space into the original multimodal data, and the discriminator is used to distinguish between the reconstructed data and the original data. By optimizing the model parameters through adversarial training, the precise separation of time-varying interference factors and physical homeostasis factors can be achieved.

[0016] Furthermore, the dynamic time warping algorithm in step 2 aligns the body constitution homeostasis factor sequences under each time window, calculates the distance between sequences, and assigns similarity weights based on the distance; the smaller the distance, the higher the weight. This ensures that the fused body constitution homeostasis factors can truly reflect the user's inherent TCM constitution characteristics, thereby improving the accuracy of health index calculation and individual health risk assessment.

[0017] Furthermore, in step 3, the spatiotemporal graph convolutional network captures the evolution patterns of each modality data in the time dimension and the correlation in the spatial dimension, calculates the confidence level of each modality data, and when the confidence level of a certain modality data is lower than a preset threshold, it is determined to be abnormal modality data and is removed.

[0018] Furthermore, the constitution evolution trend chart in step 4 includes the fluctuation curve of constitution type and the changing trend of various constitution characteristic indicators in a recent period of time. It can dynamically reflect the evolution of the user's constitution and provide an objective basis for dynamic updating of health index, dynamic assessment of individual health risk and assessment of the efficacy of constitution conditioning.

[0019] The TCM constitution identification multimodal data fusion system includes a data acquisition module, a data fusion module, an abnormal data processing module, and a result output module, with each module electrically connected in sequence.

[0020] The data acquisition module includes wearable devices and mobile terminals, which are used to continuously acquire users' temporal pulsation signals, temporal tongue images, and temporal subjective symptom texts in multiple preset time windows, and to perform motion blur removal processing on the temporal tongue images.

[0021] The data fusion module is used to construct a temporal adversarial autoencoder, which maps the multimodal temporal data collected by the data acquisition module to the latent space, separates the time-varying interference factors and the physical homeostasis factors, and then calculates the similarity weight through the dynamic time warping algorithm to complete the fusion of multi-window physical homeostasis factors.

[0022] The abnormal data processing module is used to construct a spatiotemporal graph convolutional network, and to identify and remove abnormal modal data based on the fused body homeostasis factors.

[0023] The result output module is used to output TCM constitution types and constitution evolution trend charts with confidence intervals, so as to realize health index calculation and individual health risk assessment.

[0024] Furthermore, the wearable device includes a photoelectric sensor and a pressure sensor. The photoelectric sensor is used to collect the user's photoplethysmography (PPG) signal, and the pressure sensor is used to collect the user's pulse pressure signal. The two work together to achieve accurate acquisition of time-series pulsation signals.

[0025] Furthermore, the mobile terminal is equipped with an image acquisition unit and a text input unit. The image acquisition unit is used to acquire the user's temporal tongue image, and the text input unit is used to receive the user's subjective symptom description to form a temporal subjective symptom text. The mobile terminal is also equipped with a display unit to display the constitution identification results, health index, individual health risk assessment conclusions, and constitution evolution trend graph.

[0026] Compared with existing technologies, the multimodal data fusion method and system for TCM constitution identification provided by this invention have the following beneficial effects:

[0027] This multimodal data fusion method and system for TCM constitution identification achieves accurate extraction and reliable identification of steady-state constitution characteristics through techniques such as multi-window temporal acquisition, temporal adversarial autoencoder interference separation, dynamic time warping fusion, and spatiotemporal graph convolutional network anomaly removal. It has significant beneficial effects in terms of anti-interference, identification accuracy, dynamic monitoring, and user experience.

[0028] By collecting multimodal data in a time-series manner through multiple time windows, the randomness of single-collection is avoided from the source. It effectively separates instantaneous interferences such as diet, exercise, and emotions from true physical characteristics. Through a time-series adversarial autoencoder, it achieves precise separation of interference factors and physical homeostatic factors, constructing a "steady-state fingerprint" that can represent the user's inherent physical condition. This completely solves the identification contradictions caused by pseudo-fluctuations, greatly improving the stability of the identification results. Compared with the traditional single-collection method, the accuracy is improved by 30%-40%, providing a high-quality data foundation for calculating health indices and for individual health risk assessment.

[0029] By employing a dynamic time warping algorithm to perform similarity-weighted fusion of multi-window body constitution homeostasis factors, and combining it with a spatiotemporal graph convolutional network to construct a multimodal spatiotemporal correlation model, the confidence level of each modality data can be automatically calculated, abnormal data that conflict with historical homeostasis features can be identified and eliminated, and the distortion of a single modality can be avoided from affecting the overall judgment result. This achieves reliable fusion and autonomous error correction of multi-source data, further improving the reliability and objectivity of body constitution identification, health index calculation and individual health risk assessment.

[0030] For the first time, a steady-state fingerprint and dynamic trend analysis mechanism has been introduced into TCM constitution identification. It can output constitution types with confidence intervals and generate continuous constitution evolution trend charts, which can intuitively reflect the subtle changes in constitution during conditioning and intervention. It provides quantitative basis for TCM efficacy evaluation and health program optimization, making up for the shortcomings of traditional technology that can only make static judgments and cannot track long-term, and realizing the upgrade from "single judgment" to "dynamic monitoring".

[0031] Adapted to mobile healthcare and home health monitoring scenarios, the data collection process is highly automated, eliminating the need for users to strictly control the timing of data collection or accurately fill out complex questionnaires, thus lowering the operational threshold and significantly improving user compliance. At the same time, the output results are clear and reliable, intuitively displaying physical condition, health index, and health risk level, which is easy for ordinary users to understand and can also provide data support for clinical intervention and health management, thereby improving the practicality and promotion value of TCM intelligent constitution identification. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0033] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0034] Figure 2 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0035] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0036] Please see Figure 1 , 2 A multimodal data fusion method for TCM constitution identification includes the following steps:

[0037] Step 1: Multimodal time-series data acquisition. Through wearable devices and mobile terminals, users' time-series pulsation signals, time-series tongue images, and time-series subjective symptom texts are continuously collected in multiple preset time windows. The time windows include at least three scenarios: morning on an empty stomach, daytime rest, and nighttime before bedtime. After collection, motion blur removal processing is performed on the time-series tongue images to provide high-quality basic data for subsequent health index calculation and individual health risk assessment.

[0038] This study establishes a multi-source, time-series data acquisition system for TCM constitution identification, addressing the issue of single-point-of-time data being susceptible to interference from instantaneous states. The principle is easily understood: Addressing the dynamic evolution of TCM constitutions and their susceptibility to immediate external factors, this system abandons the traditional single-collection model. Instead, it selects three fixed physiological time periods with relatively stable physiological states and minimal external interference for data collection. Continuous data is obtained from three core TCM identification dimensions: pulse, tongue appearance, and subjective symptoms. Specifically, the time-series pulse signal is continuously changing physiological data of the human pulse collected by sensors built into wearable devices; the time-series tongue images are images of the user's tongue taken at different times; and the time-series subjective symptom text is textual data of the user's reported physical discomfort at different times. Motion blur removal addresses the blurring of tongue images caused by limb movement during capture, using image algorithms to restore clarity while preserving core tongue diagnosis features such as tongue texture and coating. This step enables full-time, multi-dimensional raw data acquisition, avoiding data biases caused by instantaneous diet, exercise, and emotions. This lays a solid data foundation for subsequent accurate calculation of health indices and individual health risk assessments, improving the stability of identification data from the source.

[0039] Step 2: Multimodal data fusion processing, constructing a temporal adversarial autoencoder, mapping the multimodal temporal data collected in Step 1 to the latent space, separating time-varying interference factors and constitution homeostasis factors through adversarial training. The constitution homeostasis factors serve as a "homeostasis fingerprint" representing the user's inherent TCM constitution. Then, the dynamic time warping algorithm is used to calculate the similarity weight of the constitution homeostasis factors under each time window, completing the fusion of multi-window homeostasis factors, laying the foundation for accurate calculation of health index and assessment of individual health risks;

[0040] The separation of interfering data from true physical characteristics is achieved through an artificial intelligence model. Intuitively, this involves "purifying and filtering" the collected mixed data. The temporal adversarial autoencoder, a deep learning model, consists of an encoder, decoder, and discriminator working collaboratively. The encoder transforms multi-dimensional, temporally sequential raw data into computer-recognizable latent feature data. The decoder restores the feature data to its original form, and the discriminator compares the restored data with the original data. Through continuous adversarial training, these three components accurately distinguish between the two types of data features. The time-varying interference factor refers to instantaneous, non-physical-related interference data such as increased heart rate after exercise, tongue staining after eating, and pulse irregularities caused by emotional fluctuations. The physical homeostasis factor is a core feature data that is unaffected by external instantaneous factors and can represent the user's inherent physical constitution over a long period—essentially the "homeostasis fingerprint" of traditional Chinese medicine. The dynamic time warping algorithm calculates the matching degree of the physical homeostasis factor at different time periods. Its core calculation formula can be simplified as follows:

[0041]

[0042] In the formula, The normalized distance between two time-series steady-state factor sequences; The Euclidean distance for a single set of data points; Accumulate distance for the optimal path. The smaller the normalized distance, the higher the similarity and the greater the corresponding weight. This step can completely eliminate the interference of instantaneous pseudo-fluctuations, extract the true physical characteristics, greatly reduce data noise, improve the accuracy of health index calculation, and make individual health risk assessment more in line with the user's true physical condition.

[0043] Step 3: Abnormal data identification and removal. Based on the fused constitution homeostasis factors from Step 2, a spatiotemporal graph convolutional network is constructed. The three modalities of data—temporal pulsation signal, temporal tongue image, and temporal subjective symptom text—are used as nodes in the graph, and the time window is used as the edge of the graph. The confidence of each modal data in constitution determination is calculated, and abnormal modal data that conflict with historical homeostasis features are automatically identified and removed to avoid abnormal data interfering with the accuracy of health index calculation and individual health risk assessment.

[0044] This step further optimizes the reliability of constitution identification data by constructing a multimodal data association network to achieve automatic screening of abnormal data. In layman's terms, it establishes an interconnected network system for pulse, tongue, and symptom data, linking data from different time periods along a time dimension. Through a spatiotemporal graph convolutional network, it simultaneously analyzes the temporal evolution patterns and spatial relationships of the data, quantifying the credibility (confidence level) of each type of data in constitution assessment. If a certain type of data at a certain time period differs significantly from the long-term extracted steady-state constitution fingerprint and cannot be matched, it can be identified as abnormal data affected by transient interference and directly removed. This solves the problem of overall identification errors caused by distortion of single-modal data, further filters invalid data, and ensures that all data used to calculate health indices and conduct individual health risk assessments are valid and authentic constitution characteristic data, eliminating identification contradictions caused by data conflicts.

[0045] Step 4: Output of constitution identification results. Based on the constitution homeostasis factor after removing abnormal data in Step 3, output the TCM constitution type with confidence interval and generate a constitution evolution trend chart to achieve accurate TCM constitution identification for calculating health index and for individual health risk assessment.

[0046] This step ultimately achieves the quantitative and visual identification of TCM constitution, while simultaneously calculating the health index and assessing individual health risks. It matches the purified constitution homeostasis factors with the nine major TCM constitution standard characteristics, using a confidence level value to visually represent the reliability of the identification results; a higher confidence level indicates a more accurate constitution determination. The constitution evolution trend graph, plotted with time on the horizontal axis and the degree of constitution characteristic bias on the vertical axis, visually displays the long-term changes in a user's constitution. Compared to traditional identification methods, this step provides stable and consistent output results, accurately calculates the corresponding constitution's health index, conducts individual health risk assessments based on constitution homeostasis characteristics, and dynamically tracks the effects of constitution conditioning. This solves the technical challenges of inaccurate identification and low user trust in existing technologies, thus improving overall identification accuracy.

[0047] Wearable devices include photoelectric sensors and pressure sensors for collecting users' temporal pulsation signals, while mobile terminals are used to collect temporal tongue images and receive users' text of temporal subjective symptoms.

[0048] The system includes specific hardware for data acquisition. A photoelectric sensor collects photoplethysmography (PPG) data generated by the pulsation of peripheral blood vessels in the human body through the principle of light signal reflection. A pressure sensor collects pressure intensity data of the human pulse through pressure sensing. The combined acquisition of these two sensors can comprehensively restore the characteristics of pulse diagnosis in traditional Chinese medicine. The mobile terminal has a built-in camera to capture tongue images and receives subjective symptom information self-reported by the user through a text input port. The entire hardware system is compatible with mobile healthcare and home health monitoring scenarios. It does not require operation by professional medical personnel, lowering the user threshold, improving data collection compliance, and ensuring that the collected data can be directly used for subsequent health index calculations and individual health risk assessments.

[0049] In step 1, motion blur removal is performed using an adaptive Wiener filtering algorithm. By analyzing the blur kernel parameters of the tongue image, the blurred areas are repaired in a targeted manner to ensure the clarity of the tongue coating and tongue body features in the tongue image, providing reliable tongue image data support for health index calculation and individual health risk assessment.

[0050] This feature further defines the optimization method for tongue images. The adaptive Wiener filtering algorithm is a mature image sharpening technology, which can be understood as a targeted image "repair tool". Its principle is to first identify the blurred areas in the tongue image caused by shaking or jitter, calculate the blur degree parameter, and then optimize and reconstruct the blurred pixels through the algorithm to restore the core tongue diagnosis features such as the thickness and color of the tongue coating and the color of the tongue body. This avoids feature extraction errors caused by blurred tongue image data, ensures the integrity and accuracy of tongue image data, and provides reliable tongue image data support for subsequent multimodal data fusion, accurate calculation of health index, and objective assessment of individual health risks.

[0051] In step 2, the temporal adversarial autoencoder includes an encoder, a decoder, and a discriminator. The encoder is used to map multimodal temporal data to the latent space, the decoder is used to reconstruct the feature vectors of the latent space into the original multimodal data, and the discriminator is used to distinguish the reconstructed data from the original data. Through adversarial training, the model parameters are optimized to achieve accurate separation of time-varying interference factors and physical homeostasis factors.

[0052] The three components have clearly defined roles and work together. The encoder acts as a "data compressor," transforming complex temporal pulse, tongue, and symptom data into simplified feature vectors and eliminating redundant information. The decoder acts as a "data restorer," restoring the feature vectors to their original data form. The discriminator acts as a "verifier," comparing the restored data with the original collected data. The three components continuously iterate and optimize their own parameters, ultimately achieving accurate separation of transient interference data from true body constitution homeostasis data. This structural design allows the model to autonomously learn the differences between interference data and body constitution characteristics without manual intervention, improving data processing efficiency and ensuring that the separated body constitution homeostasis factors can be accurately used for health index calculation and individual health risk assessment.

[0053] In step 2, the dynamic time warping algorithm aligns the body constitution homeostasis factor sequences under each time window, calculates the distance between sequences, and assigns similarity weights based on the distance. The smaller the distance, the higher the weight, ensuring that the fused body constitution homeostasis factors can truly reflect the user's inherent TCM constitution characteristics and improve the accuracy of health index calculation and individual health risk assessment.

[0054] This feature provides a detailed explanation of the time-series data fusion rules. Addressing the temporal deviations in body constitution homeostasis factors across different time periods, the dynamic time warping algorithm can autonomously and accurately align data sequences of different durations and time periods. It then determines data similarity by calculating sequence spacing; the smaller the sequence spacing, the closer the body constitution data for that time period is to the user's actual inherent constitution, and the higher the corresponding weight is assigned, and vice versa. By fusion of data from multiple time periods through weight allocation, the randomness of data from a single time period can be avoided, making the fused body constitution homeostasis factors more representative, further improving the accuracy of health index calculation, and ensuring that individual health risk assessment results better reflect the user's actual physical condition.

[0055] In step 3, the spatiotemporal graph convolutional network captures the evolution patterns of each modality data in the time dimension and the correlation in the spatial dimension, calculates the confidence level of each modality data, and when the confidence level of a certain modality data is lower than a preset threshold, it is judged as abnormal modality data and is removed.

[0056] This feature clearly defines the rules for judging abnormal data. The spatiotemporal graph convolutional network has both temporal analysis and spatial correlation analysis capabilities. The temporal dimension analyzes the continuous change pattern of data in different time periods, and the spatial dimension analyzes the mutual corroboration relationship of three types of data: pulse, tongue, and symptoms. The results of the two types of analysis are combined to quantify the confidence of the data. The confidence judgment threshold is set in advance, which is a mature numerical judgment method. If the data is lower than the threshold, it means that the data is too affected by external interference and cannot reflect the true constitution. Directly removing the data can eliminate the interference of abnormal data. This technology can realize automatic data screening and autonomous error correction, ensuring that the subsequent constitution identification, health index calculation, and individual health risk assessment are not affected by noisy data.

[0057] The constitution evolution trend chart in step 4 includes the fluctuation curve of constitution type and the changing trend of various constitution characteristic indicators in recent period of time. It can dynamically reflect the evolution of the user's constitution and provide an objective basis for dynamic updating of health index, dynamic assessment of individual health risk and assessment of the efficacy of constitution conditioning.

[0058] This feature provides an expanded explanation of the output format. The constitution evolution trend chart transforms abstract constitution characteristics into visual curves, making it easy to intuitively view the direction and amplitude of constitution changes. Based on dynamic change data, the corresponding health index can be updated in real time, and the individual health risk assessment conclusion can be adjusted synchronously. This not only allows users to clearly understand their own constitution status, but also provides objective data support for the formulation of TCM conditioning and health intervention programs. It solves the problem that traditional constitution identification cannot track dynamic changes, and realizes long-term constitution monitoring and quantitative evaluation of efficacy.

[0059] The TCM constitution identification multimodal data fusion system includes a data acquisition module, a data fusion module, an abnormal data processing module, and a result output module, with each module electrically connected in sequence.

[0060] The data acquisition module includes wearable devices and mobile terminals, which are used to continuously collect users' temporal pulsation signals, temporal tongue images, and temporal subjective symptom texts in multiple preset time windows, and to perform motion blur removal processing on the temporal tongue images.

[0061] This system module is a hardware and software integration unit corresponding to the method. Each module has a clear division of labor and executes instructions in sequence. The data acquisition module serves as the system input end, undertaking the acquisition and preliminary processing of raw data throughout the entire process. Wearable devices and mobile terminals work together to achieve non-invasive and convenient data collection in home and mobile scenarios. The tongue image blur processing function is embedded in the module and is completed automatically throughout the process without manual operation by the user. This provides stable and high-quality input data for the system to subsequently calculate health indices and conduct individual health risk assessments.

[0062] The data fusion module is used to construct a temporal adversarial autoencoder, which maps the multimodal time-series data collected by the data acquisition module to the latent space, separates the time-varying interference factors and the physical homeostasis factors, and then calculates the similarity weight through the dynamic time warping algorithm to complete the fusion of multi-window physical homeostasis factors.

[0063] The data fusion module is the core processing unit of the system. It embeds two core programs: a temporal adversarial autoencoder and a dynamic time warping algorithm. It receives the raw data from the data acquisition module and autonomously completes three major operations: interference data separation, effective feature extraction, and temporal data fusion. This enables deep data purification, transforming complex multimodal data into steady-state feature data that can be directly used for physical fitness assessment. This ensures that the health index calculated subsequently and the individual health risk assessment conducted have a real and reliable data foundation.

[0064] The abnormal data processing module is used to construct a spatiotemporal graph convolutional network, and to identify and remove abnormal modal data based on the fused body homeostasis factors.

[0065] The abnormal data processing module is the system's data error correction unit. Based on the fused body constitution steady-state fingerprint, it builds a data association network, automatically screens and removes conflicting data, further optimizes data quality, realizes closed-loop management of data throughout the entire process, avoids invalid data from flowing into subsequent stages, and ensures the accuracy of body constitution identification, health index calculation, and individual health risk assessment from the program level.

[0066] The results output module is used to output TCM constitution types and constitution evolution trend charts with confidence intervals, enabling health index calculation and individual health risk assessment.

[0067] The results output module, as the system output end, integrates the effective data after preprocessing to complete the body mass index determination, health index calculation, and health risk assessment. It outputs the results in an intuitive form such as numerical values ​​and charts, which is both readable and professional. It not only meets users' needs for understanding their own physical condition, but also provides objective data support for medical and health management scenarios, which is in line with the actual use needs of mobile healthcare and home health monitoring.

[0068] Wearable devices include photoelectric sensors and pressure sensors. The photoelectric sensors are used to collect the user's photoplethysmography (PPG) pulse wave signal, and the pressure sensors are used to collect the user's pulse pressure signal. The two work together to achieve accurate acquisition of time-series pulsation signals.

[0069] This feature provides a detailed description of the wearable device hardware of the system. Two types of sensors collect pulse data from different dimensions. The photoplethysmography (PPG) signal reflects changes in blood flow during vascular pulsation, while the pulse pressure signal reflects changes in the strength and frequency of the pulse. The two types of data complement and corroborate each other, which can comprehensively restore the core characteristics of traditional Chinese medicine pulse diagnosis, achieving non-invasive, continuous, and accurate data collection. It is suitable for long-term home monitoring scenarios, and the collected data can be directly used for internal health index calculation and individual health risk assessment.

[0070] The mobile terminal is equipped with an image acquisition unit and a text input unit. The image acquisition unit is used to acquire the user's temporal tongue images, and the text input unit is used to receive the user's subjective symptom descriptions to form temporal subjective symptom text. The mobile terminal is also equipped with a display unit to display the constitution identification results, health index, individual health risk assessment conclusions, and constitution evolution trend chart.

[0071] This feature provides a detailed explanation of the mobile terminal's functional units. The image acquisition unit and text input unit enable convenient collection of multimodal data, reducing the difficulty of user operation and improving data collection compliance. The display unit is responsible for the visualization of results, presenting professional physical condition identification results, health index values, individual health risk assessment conclusions, and evolution trend charts to users intuitively, allowing non-professional users to clearly understand their own health status. The entire system balances professionalism and ease of use, perfectly adapting to home and mobile healthcare usage scenarios.

[0072] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A multimodal data fusion method for TCM constitution identification, characterized in that, Includes the following steps: Step 1: Multimodal time-series data acquisition. Through wearable devices and mobile terminals, users' time-series pulsation signals, time-series tongue images, and time-series subjective symptom texts are continuously acquired in multiple preset time windows. The time windows include at least three scenarios: morning on an empty stomach, daytime rest, and nighttime before bedtime. After acquisition, the time-series tongue images are processed to remove motion blur, providing high-quality basic data for subsequent health index calculation and individual health risk assessment. Step 2: Multimodal data fusion processing, constructing a temporal adversarial autoencoder, mapping the multimodal temporal data collected in Step 1 to the latent space, separating time-varying interference factors and constitution homeostasis factors through adversarial training. The constitution homeostasis factors serve as "homeostasis fingerprints" representing the user's inherent TCM constitution. Then, the similarity weight of the constitution homeostasis factors under each time window is calculated using the dynamic time warping algorithm, completing the fusion of multi-window homeostasis factors, laying the foundation for accurate calculation of health index and assessment of individual health risks; Step 3: Abnormal data identification and removal. Based on the fused constitution homeostasis factors from Step 2, a spatiotemporal graph convolutional network is constructed. The three modalities of data—temporal pulsation signal, temporal tongue image, and temporal subjective symptom text—are used as nodes in the graph, and the time window is used as the edge of the graph. The confidence of each modal data in constitution determination is calculated, and abnormal modal data that conflict with historical homeostasis features are automatically identified and removed to avoid abnormal data interfering with the accuracy of health index calculation and individual health risk assessment. Step 4: Output of constitution identification results. Based on the constitution homeostasis factor after removing abnormal data in Step 3, output the TCM constitution type with confidence interval and generate a constitution evolution trend chart to achieve accurate TCM constitution identification for calculating health index and for individual health risk assessment.

2. The multimodal data fusion method for TCM constitution identification according to claim 1, characterized in that, The wearable device includes a photoelectric sensor and a pressure sensor for collecting the user's temporal pulsation signals, and the mobile terminal is used to collect temporal tongue images and receive the user's input of temporal subjective symptom text.

3. The multimodal data fusion method for TCM constitution identification according to claim 1, characterized in that, The motion blur removal process in step 1 uses an adaptive Wiener filtering algorithm. By analyzing the blur kernel parameters of the tongue image, the blurred areas are repaired in a targeted manner to ensure the clarity of the tongue coating and tongue body features in the tongue image, providing reliable tongue image data support for health index calculation and individual health risk assessment.

4. The multimodal data fusion method for TCM constitution identification according to claim 1, characterized in that, The temporal adversarial autoencoder in step 2 includes an encoder, a decoder, and a discriminator. The encoder is used to map multimodal temporal data to a latent space, the decoder is used to reconstruct the feature vectors of the latent space into the original multimodal data, and the discriminator is used to distinguish between the reconstructed data and the original data. By optimizing the model parameters through adversarial training, the precise separation of time-varying interference factors and physical homeostasis factors can be achieved.

5. The method for multimodal data fusion for TCM constitution identification according to claim 1, characterized in that, The dynamic time warping algorithm described in step 2 aligns the body constitution homeostasis factor sequences under each time window, calculates the distance between sequences, and assigns similarity weights based on the distance; the smaller the distance, the higher the weight. This ensures that the fused body constitution homeostasis factors can truly reflect the user's inherent TCM constitution characteristics, thereby improving the accuracy of health index calculation and individual health risk assessment.

6. The multimodal data fusion method for TCM constitution identification according to claim 1, characterized in that, The spatiotemporal graph convolutional network described in step 3 captures the evolution patterns of each modality data in the time dimension and the correlation in the spatial dimension, calculates the confidence level of each modality data, and when the confidence level of a certain modality data is lower than a preset threshold, it is determined to be abnormal modality data and is removed.

7. The multimodal data fusion method for TCM constitution identification according to claim 1, characterized in that, The constitution evolution trend chart mentioned in step 4 includes the fluctuation curve of constitution type and the changing trend of various constitution characteristic indicators in a recent period of time. It can dynamically reflect the evolution of the user's constitution and provide an objective basis for dynamic updating of health index, dynamic assessment of individual health risk and assessment of the efficacy of constitution conditioning.

8. A multimodal data fusion system for TCM constitution identification, characterized in that, It includes a data acquisition module, a data fusion module, an anomaly data processing module, and a result output module, with each module electrically connected in sequence; The data acquisition module includes wearable devices and mobile terminals, which are used to continuously acquire users' temporal pulsation signals, temporal tongue images, and temporal subjective symptom texts in multiple preset time windows, and to perform motion blur removal processing on the temporal tongue images. The data fusion module is used to construct a temporal adversarial autoencoder, which maps the multimodal temporal data collected by the data acquisition module to the latent space, separates the time-varying interference factors and the physical homeostasis factors, and then calculates the similarity weight through the dynamic time warping algorithm to complete the fusion of multi-window physical homeostasis factors. The abnormal data processing module is used to construct a spatiotemporal graph convolutional network, and to identify and remove abnormal modal data based on the fused body homeostasis factors. The result output module is used to output TCM constitution types and constitution evolution trend charts with confidence intervals, so as to realize health index calculation and individual health risk assessment.

9. The multimodal data fusion system for TCM constitution identification according to claim 8, characterized in that, The wearable device includes a photoelectric sensor and a pressure sensor. The photoelectric sensor is used to collect the user's photoplethysmography (PPG) signal, and the pressure sensor is used to collect the user's pulse pressure signal. The two work together to achieve accurate acquisition of time-series pulsation signals.

10. The multimodal data fusion system for TCM constitution identification according to claim 8, characterized in that, The mobile terminal is equipped with an image acquisition unit and a text input unit. The image acquisition unit is used to acquire the user's temporal tongue image, and the text input unit is used to receive the user's subjective symptom description to form a temporal subjective symptom text. The mobile terminal is also equipped with a display unit to display the constitution identification results, health index, individual health risk assessment conclusions, and constitution evolution trend graph.

Images