Traditional Chinese medicine health management intelligent interaction system and method based on multi-modal data identification

Abstract

The invention relates to the cross technical field of artificial intelligence and digital medical treatment, in particular to a multi-modal data identification-based traditional Chinese medicine health management intelligent interaction system and method, which adopts a five-layer distributed architecture and comprises a perception interaction layer, a core processing layer, a learning evolution layer, a safety compliance layer and an output presentation layer, user voice, tongue image and face image data are collected through the multi-modal sensing module, biological characteristic analysis is carried out through the voiceprint recognition module and the emotion recognition module, dialectical reasoning is carried out in combination with a traditional Chinese medicine knowledge graph constructed based on the Huangdi Neijing, and a personalized health management scheme is generated. According to the method, continuous model optimization is realized by adopting a technology of combining federal learning and adaptive learning, data privacy and medical specification compliance are ensured through a full-process safety compliance mechanism, and finally an interaction result is output through a multi-terminal visual interface, so that precise, natural and active services of personalized traditional Chinese medicine health management are realized.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence and digital healthcare, and in particular to a smart interactive system and method for TCM health management based on multimodal data recognition. Background Technology

[0002] With the rapid development of the digital health industry, traditional Chinese medicine (TCM) health management has evolved from paper-based records to electronic storage, and then to preliminary intelligent assistance. In recent years, artificial intelligence (AI) technology has been introduced into this field, resulting in health management systems with basic question-and-answer capabilities and simple data recording functions. This marks a new stage in the digital exploration of TCM health management. These systems attempt to initially combine traditional Chinese medicine theory with modern technology, laying a foundation for the digitalization of health management.

[0003] However, existing technological systems still have many significant shortcomings. In terms of interactive experience, systems generally lack natural and smooth full-duplex dialogue capabilities, forcing users to adapt to a mechanical interaction rhythm rather than a humanized communication experience. Regarding user understanding, the system's assessment of user health status often remains superficial, failing to deeply integrate multi-dimensional biometric information such as voiceprint features and emotional state, resulting in insufficient comprehensiveness and accuracy in health assessments. Service models are mostly limited to passive responses, lacking proactive intervention mechanisms based on dynamic monitoring, making it difficult to achieve a true closed-loop health management system. Simultaneously, existing solutions have significant deficiencies in compliance assurance, failing to meet the medical regulations of different regions and lacking necessary risk warning mechanisms. Language barriers also become a bottleneck restricting its international development; insufficient real-time translation capabilities severely hinder the global dissemination of traditional Chinese medicine culture. From a system architecture perspective, existing solutions are mostly single-function independent modules, lacking a unified architecture that organically integrates multiple advanced technologies, resulting in poor technological synergy and limited overall efficiency. Summary of the Invention

[0004] The purpose of this invention is to overcome the above-mentioned problems and provide an intelligent interactive system and method for TCM health management based on multimodal data recognition. To achieve the above objective, this invention adopts the following technical solution:

[0005] The intelligent interactive system for TCM health management based on multimodal data recognition includes a perception and interaction layer, a core processing layer, a learning and evolution layer, a security and compliance layer, and an output presentation layer.

[0006] The perception and interaction layer communicates with the core processing layer, transmitting collected user data and interaction commands to the core processing layer, and receiving interaction response requests from the core processing layer. The core processing layer communicates with the learning and evolution layer, the security and compliance layer, and the output presentation layer, transmitting user time-series health data and solution feedback information to the learning and evolution layer, transmitting the generated health management solution to the security and compliance layer, and transmitting health status assessment results to the output presentation layer. The learning and evolution layer transmits model adjustment signals and health status monitoring results to the core processing layer, and updates the user-personalized model parameters of the core processing layer. The security and compliance layer transmits compliance check results and risk warning information to the core processing layer, and performs compliance verification on the health management solution output by the core processing layer. The output presentation layer receives interaction response information from the perception and interaction layer, health status assessment results from the core processing layer, and reminder signals from the learning and evolution layer, providing a user-accessible interactive interface.

[0007] Furthermore, the perception and interaction layer includes a multimodal perception module, a voiceprint recognition module, an emotion recognition module, a full-duplex dialogue module, and a real-time speech translation and TTS module;

[0008] The multimodal perception module collects at least one of the user's voice, tongue image, and facial image data, and configures a voice interaction interface to receive user input information;

[0009] The voiceprint recognition module uses a deep neural network model based on x-vector to extract voiceprint features, including fundamental frequency contour, formant structure, prosodic pattern, and vocal tract length, for identity authentication and personalized services.

[0010] The emotion recognition module extracts prosodic features and deep acoustic features from the user's speech signal, and combines them with text sentiment analysis based on the TCM emotion dictionary to output confidence scores for joy, anger, worry, thought, grief, fear, and surprise.

[0011] The full-duplex dialogue module is equipped with an LSTM-based dialogue state tracker, which supports real-time voice activity detection and interruption handling, maintains dialogue history memory, performs smooth handling of topic switching and continuation, and supports multi-turn dialogue context management.

[0012] The real-time speech translation and TTS module adopts an end-to-end speech translation model, supports real-time translation between multiple languages, and performs voiceprint-driven timbre adaptation and emotion-aware prosody generation based on neural speech synthesis technology to carry out cross-language interaction and personalized speech synthesis.

[0013] Furthermore, the core processing layer includes a digital module for the Yellow Emperor's Inner Classic framework, an algorithmic module for TCM theory, a TCM knowledge reasoning module, a health plan generation module, a dream diagnosis and organ mapping module, and a dynamic user profile module;

[0014] The digital module of the Yellow Emperor's Inner Classic framework transforms the theoretical system of Yin-Yang and Five Elements, Zang-Xiang and Meridians, Qi, Blood and Body Fluids, and Etiology and Pathogenesis in the Yellow Emperor's Inner Classic into a computable knowledge graph containing entities and relationships.

[0015] The TCM theory algorithm module is connected to the Huangdi Neijing framework digital module, which transforms the logical process of TCM syndrome differentiation and treatment into an executable algorithm model, including a knowledge graph-based symbolic reasoning algorithm, a health data-based statistical learning algorithm, and a neural network reasoning algorithm for discovering potential correlations.

[0016] The TCM knowledge reasoning module is built on the knowledge system of the "Huangdi Neijing". It receives data transmitted from the perception and interaction layer, performs TCM syndrome differentiation reasoning, and outputs the syndrome differentiation reasoning results.

[0017] The health plan generation module receives the reasoning results from the TCM knowledge reasoning module and generates a personalized TCM health management plan.

[0018] The Dream Diagnosis and Organ Mapping Module receives and understands the dream information described by the user, performs inference based on the mapping relationship between dreams and organ health in traditional Chinese medicine theory, assesses the user's organ function status, and generates corresponding health risk warnings.

[0019] The dynamic user profile module integrates multi-dimensional information to construct a three-dimensional user profile that evolves over time. It builds a five-dimensional profile model that includes a basic attribute layer, a behavioral pattern layer, a psychological trait layer, a physiological state layer, and a social relationship layer. It uses an LSTM network to capture the profile's changing trends and performs visualization through radar charts, trend lines, and heat maps.

[0020] Furthermore, the learning evolution layer includes an adaptive learning module, a federated learning collaboration module, and an active monitoring and reminder module;

[0021] The adaptive learning module is deployed on the user terminal, receiving the user's time-series health data and feedback data on historical health management plans, and dynamically adjusting the user-personalized model stored locally.

[0022] The federated learning collaboration module includes a central server and multiple federated clients corresponding to user terminals. It coordinates multiple federated clients to jointly train and optimize the global model managed by the central server without the local data leaving the terminal. It adopts privacy-preserving aggregation technology, supports heterogeneous federated learning and personalized federated learning, and performs balanced optimization of the global model and the local model.

[0023] The proactive monitoring and alert module defines and detects health event patterns based on a complex event processing engine. It adopts dynamic threshold adjustment based on personal baselines and supports multi-channel alerts, including voice, messaging, telephone, and smart device linkage. It also features anti-harassment mechanisms and importance classification functions.

[0024] Furthermore, the security and compliance layer includes a data security protection module, a compliance inspection module, and a risk warning module;

[0025] The data security protection module employs end-to-end encryption, homomorphic encryption, differential privacy, and secure multi-party computation technologies, covering the entire process of data transmission, storage, and processing across the perception and interaction layer, core processing layer, learning and evolution layer, and output presentation layer.

[0026] The compliance check module has a built-in configurable regulatory rule base, which checks the compliance of system recommendations output by the core processing layer in real time. It supports graphical rule configuration, rule verification before execution decisions, and has regional adaptation and rule version management functions, and complies with HIPAA and GDPR requirements.

[0027] The risk warning module pre-sets standardized risk warning information for different health recommendations, stores them according to the type of health recommendation, and enforces output for high-risk health recommendations.

[0028] Furthermore, the output presentation layer includes a dynamic health visualization module and a multi-terminal application module;

[0029] The dynamic health visualization module receives the health status assessment results transmitted from the core processing layer, transforms the health status assessment results into personalized, interactive visualization charts and outputs them. The health status assessment results include constitution, syndrome, organ balance status and health risk warning. The visualization charts include meridian diagrams, five elements mutual generation and restraint diagrams, radar charts, trend lines and heat maps.

[0030] The multi-terminal application module supports access from multiple terminals, including the Web, mobile applications, smart speakers, and in-vehicle systems. It provides accessibility design in accordance with the WCAG 2.1AA standard, which includes voice-first interaction, high-contrast interface themes, and full support for screen readers.

[0031] Furthermore, a microservice architecture is adopted, with the perception and interaction layer, core processing layer, learning and evolution layer, security and compliance layer, and output presentation layer communicating through an API gateway, supporting horizontal scaling and high availability deployment.

[0032] Furthermore, the intelligent interactive method for TCM health management based on multimodal data recognition comprises the following steps:

[0033] S1, the perception and interaction layer performs data acquisition and interaction processing:

[0034] Step S11: Collect at least one of the user's voice, tongue image, and facial image data through the multimodal perception module;

[0035] Step S12: The voiceprint recognition module uses a deep neural network model based on x-vector to extract voiceprint feature dimensions including fundamental frequency contour, formant structure, prosodic pattern, and vocal tract length, and performs identity authentication and personalized services based on the user's voice features.

[0036] Step S13: Extract prosodic features and deep acoustic features from the user's speech signal through the emotion recognition module, and combine them with text sentiment analysis based on the TCM emotion dictionary to output confidence scores for joy, anger, worry, thought, grief, fear, and surprise.

[0037] Step S14: Through the full-duplex dialogue module, an LSTM-based dialogue state tracker is used to support real-time voice activity detection and interruption processing, maintain dialogue history memory, perform smooth processing of topic switching and continuation, and support multi-turn dialogue context management.

[0038] Step S15: Using an end-to-end speech translation model through real-time speech translation and TTS module, it supports real-time translation between multiple languages. Based on neural speech synthesis technology, it performs voiceprint-driven timbre adaptation and emotion perception prosodic generation to carry out cross-language interaction and personalized speech synthesis, and outputs interactive response information.

[0039] S2. The core processing layer performs TCM reasoning and solution generation:

[0040] Step S21: Transform the TCM theoretical system into a computable knowledge graph through the digital module of the Huangdi Neijing framework;

[0041] Step S22: Through the algorithmic module of traditional Chinese medicine theory, based on knowledge graph and user data, perform algorithmic calculation of dialectical reasoning;

[0042] Step S23: Based on the classical theoretical knowledge system of traditional Chinese medicine, the TCM knowledge reasoning module receives the data transmitted by the perception and interaction layer, performs TCM syndrome differentiation reasoning, and outputs the syndrome differentiation reasoning results.

[0043] Step S24: Generate a personalized TCM health management plan based on the reasoning results of the TCM knowledge reasoning module through the health plan generation module;

[0044] Step S25: Receive and understand the dream information described by the user through the dream diagnosis and organ mapping module, perform reasoning based on the mapping relationship between dreams and organ health in traditional Chinese medicine theory, assess the user's organ function status and generate corresponding health risk warnings;

[0045] Step S26: By integrating multi-dimensional information through the dynamic user profile module, a five-dimensional profile model is constructed, including a basic attribute layer, a behavior pattern layer, a psychological trait layer, a physiological state layer, and a social relationship layer. An LSTM network is used to capture the profile change trend, and a visualization is performed through radar charts, trend lines, and heat maps to construct and update a three-dimensional user profile that evolves over time.

[0046] S3. Learn and optimize the evolutionary layer model:

[0047] Step S31: Receive the user's time-series health data and feedback data on historical health management plans through the adaptive learning module, and dynamically adjust the locally stored user-personalized model.

[0048] Step S32: Through the central server of the federated learning collaboration module, coordinate multiple federated clients corresponding to user terminals, and use privacy-preserving aggregation technology to support heterogeneous federated learning and personalized federated learning, while ensuring that local data does not leave the terminal, jointly train and optimize the global model managed by the central server, and perform balanced optimization of the global model and the local model.

[0049] Step S33: Transfer the optimized global model parameters to the adaptive learning module to update the locally stored user-personalized model;

[0050] S4. Security and compliance layer implements security and compliance controls:

[0051] Step S41: The data security protection module employs end-to-end encryption, homomorphic encryption, differential privacy, and secure multi-party computation technologies to provide full security protection for the data transmission, storage, and processing of the perception interaction layer, core processing layer, learning evolution layer, and output presentation layer.

[0052] Step S42: The compliance check module calls the built-in configurable regulatory rule library to check the compliance of the health management plan output by the core processing layer in real time, perform rule verification before decision-making, and, based on the regional adaptation and rule version management functions, check the results in accordance with the relevant requirements of HIPAA and GDPR.

[0053] Step S43: Based on the different health recommendations output by the core processing layer, the risk warning module retrieves the preset standardized risk warning information and transmits it to the core processing layer, and performs forced output for high-risk health recommendations;

[0054] S5. Output the execution results of the presentation layer:

[0055] Step S51: Receive the health status assessment results transmitted from the core processing layer through the dynamic health visualization module, and convert the health status assessment results into personalized interactive visualization charts including meridian diagrams, five elements mutual generation and restraint diagrams, radar charts, trend lines, and heat maps.

[0056] Step S52: Receive the interactive response information output by the perception interaction layer and the charts output by the dynamic health visualization module through the multi-terminal application module, and provide access interfaces for Web, mobile applications, smart speakers, and in-vehicle systems, supporting barrier-free operation including voice-first interaction, high-contrast interface themes, and full support for screen readers.

[0057] Furthermore, a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of a smart interactive method for TCM health management based on multimodal data recognition, the storage medium including a solid-state drive, a USB flash drive, a portable hard drive, a read-only memory, a random access memory, a magnetic disk, or an optical disk.

[0058] The advantages of this invention are:

[0059] 1. This invention constructs an intelligent perception layer that integrates voiceprint biometric recognition, multimodal emotion computing, and full-duplex voice interaction. Combined with a TCM diagnostic reasoning engine based on the theoretical system of the Yellow Emperor's Inner Classic, it achieves in-depth cognition and dynamic tracking of users' health status. With the help of dynamic user profiling and five-dimensional modeling technology, it continuously captures the unique physiological rhythms and behavioral patterns of individuals, thereby generating a health management plan that truly matches the individual's physical characteristics. This allows the traditional TCM diagnostic and treatment ideas to be accurately mapped and personalized in the digital space.

[0060] 2. This invention utilizes a dialogue management system with real-time interruption processing and multi-turn context memory capabilities, combined with an intelligent voice interaction channel that supports multilingual translation, to construct a natural and fluent human-computer communication environment. Through voiceprint-driven personalized speech synthesis and emotion-aware prosodic generation technology, it breaks down language and cultural barriers, allowing users with different cognitive habits to manage their health in the most instinctive way, significantly improving the inclusiveness and accessibility of digital health services.

[0061] 3. This invention constructs a continuously optimized health management ecosystem by deploying a dual evolutionary system that integrates a federated learning framework and an adaptive learning mechanism. Under the premise of strictly ensuring data privacy and security, it can continuously improve its reasoning model and knowledge system through distributed intelligent collaboration. At the same time, it uses a complex event processing engine to realize the service paradigm shift from passive response to proactive care, enabling health management services to have the vitality of self-evolution.

[0062] 4. This invention achieves reliable protection for medical data throughout its entire lifecycle by embedding a compliant security gateway with a rule engine throughout the entire process and constructing a defense-in-depth system using end-to-end encryption and privacy computing technologies. The system ensures that all health advice undergoes strict compliance verification through a built-in configurable regulatory rule base and a mandatory risk warning mechanism, providing personalized services while fully complying with international medical data security standards.

[0063] 5. This invention deeply integrates the visual charts with traditional Chinese medicine characteristics with cross-terminal accessibility design, creating an intuitive and vivid way of presenting health status. The system uses traditional medical visualization elements such as meridian diagrams and the five elements mutual generation and restraint diagrams, combined with an interactive interface that meets international accessibility standards, to transform complex health data into an easy-to-understand visual language, allowing users with different usage habits to equally access health information services. Attached Figure Description

[0064] The accompanying drawings, which form part of this application, are used to provide a further understanding of the application and to make other features, objects, and advantages of the application more apparent. The illustrative embodiments and descriptions of this application are used to explain the application and do not constitute an undue limitation of the application.

[0065] In the attached diagram:

[0066] Figure 1 This is a system framework diagram of the intelligent interactive system for TCM health management based on multimodal data recognition in Example 1.

[0067] Figure 2 This is a flowchart of the intelligent interactive method for TCM health management based on multimodal data recognition in Example 1.

[0068] Figure 3 This is a flowchart of the perception and interaction layer of the intelligent interactive system for TCM health management based on multimodal data recognition in Example 1.

[0069] Figure 4 This is a flowchart of the core processing layer of the intelligent interactive system for TCM health management based on multimodal data recognition in Example 1.

[0070] Figure 5 This is a flowchart of the learning evolution layer of the intelligent interactive system for TCM health management based on multimodal data recognition in Example 1.

[0071] Figure 6 This is a flowchart of the security and compliance layer of the intelligent interactive system for TCM health management based on multimodal data recognition in Example 1.

[0072] Figure 7This is a flowchart of the output presentation layer of the intelligent interactive system for TCM health management based on multimodal data recognition in Example 1. Detailed Implementation

[0073] The present invention will now be described in detail and specifically through specific embodiments to enable a better understanding of the invention. However, the following embodiments do not limit the scope of protection of the present invention.

[0074] Example 1

[0075] like Figure 1 As shown, the intelligent interactive system for TCM health management based on multimodal data recognition includes a perception and interaction layer, a core processing layer, a learning and evolution layer, a security and compliance layer, and an output presentation layer.

[0076] The perception and interaction layer communicates with the core processing layer, transmitting collected user data and interaction commands to the core processing layer, and receiving interaction response requests from the core processing layer. The core processing layer communicates with the learning and evolution layer, the security and compliance layer, and the output presentation layer, transmitting user time-series health data and solution feedback information to the learning and evolution layer, transmitting the generated health management solution to the security and compliance layer, and transmitting health status assessment results to the output presentation layer. The learning and evolution layer transmits model adjustment signals and health status monitoring results to the core processing layer, and updates the user-personalized model parameters of the core processing layer. The security and compliance layer transmits compliance check results and risk warning information to the core processing layer, and performs compliance verification on the health management solution output by the core processing layer. The output presentation layer receives interaction response information from the perception and interaction layer, health status assessment results from the core processing layer, and reminder signals from the learning and evolution layer, providing a user-accessible interactive interface.

[0077] In a specific embodiment, the system adopts a layered architecture design. The perception and interaction layer collects user voice data through the microphone array of a smart speaker and obtains physiological indicators with the help of medical-grade sensors. The core processing layer is deployed on a GPU server cluster equipped with NVIDIA A100 and relies on the PyTorch and TensorFlow frameworks to realize real-time inference computation of the knowledge graph. The learning and evolution layer coordinates multiple terminal devices through a federated learning mechanism to complete model iteration and updates while protecting data privacy. The security and compliance layer uses homomorphic encryption and differential privacy technology to build a comprehensive protection system to ensure that medical data complies with international standards. The output presentation layer generates an interactive visual interface based on the WebGL engine and realizes multi-terminal adaptation through the ReactNative framework. Each functional layer is deployed in Docker containers and uses Kubernetes orchestration tools to realize resource scheduling. At the same time, Kafka message queues are used to ensure the reliability of inter-system communication, ultimately forming a complete health management closed loop.

[0078] Furthermore, such as Figure 3As shown, the perception and interaction layer includes a multimodal perception module, a voiceprint recognition module, an emotion recognition module, a full-duplex dialogue module, and a real-time speech translation and TTS module;

[0079] The multimodal perception module collects at least one of the user's voice, tongue image, and facial image data, and configures a voice interaction interface to receive user input information;

[0080] In a specific embodiment, the multimodal perception module collects voice data through the microphone array of the smart speaker, while simultaneously integrating a tongue image acquisition device and a facial analysis instrument to obtain visual information. The system is configured with a dedicated voice interaction interface to receive user input, and uses multi-source data fusion technology to process heterogeneous information. In the specific implementation, a distributed message queue is used to coordinate the data streams of each sensor to ensure the synchronous acquisition and real-time processing of multimodal data.

[0081] The voiceprint recognition module uses a deep neural network model based on x-vector to extract voiceprint features, including fundamental frequency contour, formant structure, prosodic pattern, and vocal tract length, for identity authentication and personalized services.

[0082] In a specific embodiment, the voiceprint recognition module constructs a feature extraction model based on the x-vector deep neural network architecture, obtains acoustic feature parameters through the MFCC feature extractor, processes the fundamental frequency profile and formant structure in conjunction with the prosody analysis module, and uses a pre-trained xvector_400d model to process audio input. In actual deployment, data is collected through a four-microphone array, and the original audio is converted into a 400-dimensional feature vector using an end-to-end feature extraction process.

[0083] The emotion recognition module extracts prosodic features and deep acoustic features from the user's speech signal, and combines them with text sentiment analysis based on the TCM emotion dictionary to output confidence scores for joy, anger, worry, thought, grief, fear, and surprise.

[0084] In a specific embodiment, the emotion recognition module adopts a multimodal fusion strategy. It extracts 88-dimensional prosodic features from the speech signal using the OpenSmile toolkit and performs acoustic emotion analysis by combining 256-dimensional deep acoustic features. The text emotion analysis is based on a fine-tuned BERT model and combines a traditional Chinese medicine emotion dictionary to classify the seven emotions. The system fuses acoustic and text features through a cross-modal Transformer architecture and finally outputs confidence scores for seven emotions: joy, anger, worry, contemplation, grief, fear, and surprise.

[0085] The full-duplex dialogue module is equipped with an LSTM-based dialogue state tracker, which supports real-time voice activity detection and interruption handling, maintains dialogue history memory, performs smooth handling of topic switching and continuation, and supports multi-turn dialogue context management.

[0086] In a specific embodiment, the full-duplex dialogue module builds a dialogue state tracker based on an LSTM network and uses a finite state machine to control the dialogue flow. The system supports real-time voice activity detection and interruption processing, maintains 128 rounds of dialogue history to ensure contextual coherence, converts the audio stream into a text sequence through real-time speech recognition during actual interaction, selects the optimal response strategy in conjunction with the dialogue policy manager, and finally generates a dialogue response that conforms to the current context through the natural language generation module.

[0087] The real-time speech translation and TTS module adopts an end-to-end speech translation model, supports real-time translation between multiple languages, and performs voiceprint-driven timbre adaptation and emotion-aware prosody generation based on neural speech synthesis technology to carry out cross-language interaction and personalized speech synthesis.

[0088] In a specific embodiment, the real-time speech translation and TTS module adopts an end-to-end speech translation architecture, integrating the SpeechTransformer model to achieve real-time translation between multiple languages. The speech synthesis system is based on Tacotron2 and WaveNet technologies, using voiceprint features to drive timbre adaptation, and combining the emotion perception module to generate speech output with appropriate rhythm. The system has specially designed a TCM terminology protection mechanism to automatically identify and retain professional terms during the translation process. The end-to-end latency of the entire processing flow is controlled at the millisecond level.

[0089] Furthermore, such as Figure 4 As shown, the core processing layer includes a digitization module of the Yellow Emperor's Inner Classic framework, an algorithmization module of traditional Chinese medicine theory, a traditional Chinese medicine knowledge reasoning module, a health plan generation module, a dream diagnosis and organ mapping module, and a dynamic user profile module;

[0090] The digital module of the Yellow Emperor's Inner Classic framework transforms the theoretical system of Yin-Yang and Five Elements, Zang-Xiang and Meridians, Qi, Blood and Body Fluids, and Etiology and Pathogenesis in the Yellow Emperor's Inner Classic into a computable knowledge graph containing entities and relationships.

[0091] The Huangdi Neijing (Yellow Emperor's Inner Classic) framework digitization module is used to construct a computable knowledge framework for Traditional Chinese Medicine (TCM) theory. This module transforms concepts such as Yin-Yang, Five Elements, Zang-Fu organs, meridians, Qi, blood, body fluids, etiology, and pathogenesis, along with their interrelationships, from classic texts like the Huangdi Neijing into a structured knowledge graph by defining TCM domain ontology. In practice, the Neo4j graph database is used for storage and management, constructing a knowledge network containing Zang-Fu entities such as "liver," "heart," and "spleen," syndrome entities such as "Qi stagnation" and "blood deficiency," and relationship types such as "mutual generation," "mutual restraint," and "manifestations." Its scale can reach hundreds of thousands of entities and millions of relationships, providing a structured TCM theoretical data foundation for the entire system.

[0092] The TCM theory algorithm module is connected to the Huangdi Neijing framework digital module, which transforms the logical process of TCM syndrome differentiation and treatment into an executable algorithm model, including a knowledge graph-based symbolic reasoning algorithm, a health data-based statistical learning algorithm, and a neural network reasoning algorithm for discovering potential correlations.

[0093] In a specific embodiment, the TCM theory algorithm module is tightly coupled with the aforementioned digitization module to provide computational capabilities for TCM syndrome differentiation and treatment. This module is specifically implemented as a hybrid inference engine, integrating three core algorithms:

[0094] (1) Symbolic reasoning algorithm, which performs logical deduction based on predefined rules in the knowledge graph provided by the digital module; such as "liver qi stagnation → hypochondriac distending pain";

[0095] (2) Neural network inference algorithm, which uses deep learning model to mine the complex nonlinear mapping relationship between user multimodal features and health status;

[0096] (3) Statistical learning algorithm, which performs probability modeling and correlation analysis based on historical health data. This module is the core of algorithmic execution of TCM syndrome differentiation logic.

[0097] The TCM knowledge reasoning module is built on the knowledge system of the "Huangdi Neijing". It receives data transmitted from the perception and interaction layer, performs TCM syndrome differentiation reasoning, and outputs the syndrome differentiation reasoning results.

[0098] In a specific embodiment, the TCM knowledge reasoning module, acting as an application-oriented coordinator, invokes and integrates the capabilities of the two modules mentioned above. It receives data from the perception and interaction layer, first using the knowledge graph of the digitization module to semantically associate and standardize symptoms and signs, then scheduling the hybrid reasoning engine in the algorithmic module for comprehensive analysis and calculation, ultimately outputting the diagnostic reasoning result. For example, it uses the Cypher query language to quickly associate symptoms in the knowledge graph and utilizes the hybrid reasoning engine to fuse rules and data evidence to form a diagnostic conclusion for syndromes such as "liver stagnation and spleen deficiency syndrome."

[0099] The health plan generation module receives the reasoning results from the TCM knowledge reasoning module and generates a personalized TCM health management plan.

[0100] In a specific embodiment, after receiving the dialectical reasoning results, the health plan generation module analyzes physical characteristics and environmental factors based on the constraint satisfaction algorithm, uses a multi-objective optimization algorithm to balance the effectiveness and feasibility of the conditioning plan, and combines a personalized federated learning model to adjust the plan parameters to generate a complete health management plan that includes diet, exercise and emotional conditioning.

[0101] The Dream Diagnosis and Organ Mapping Module receives and understands the dream information described by the user, performs inference based on the mapping relationship between dreams and organ health in traditional Chinese medicine theory, assesses the user's organ function status, and generates corresponding health risk warnings.

[0102] In a specific embodiment, the dream diagnosis and organ mapping module parses the dream text described by the user through a natural language processing flow, uses a symbol classifier to extract dream symbols such as natural elements, animal imagery, and emotional states, performs pathogenesis inference based on the dream organ mapping rules established in the Inner Canon of Medicine, assesses the functional status of the organs in combination with the user's health context, and generates corresponding early warning information.

[0103] The dynamic user profile module integrates multi-dimensional information to construct a three-dimensional user profile that evolves over time. It builds a five-dimensional profile model that includes a basic attribute layer, a behavioral pattern layer, a psychological trait layer, a physiological state layer, and a social relationship layer. It uses an LSTM network to capture the profile's changing trends and performs visualization through radar charts, trend lines, and heat maps.

[0104] In a specific embodiment, the dynamic user profile module integrates data from five dimensions: basic attributes, behavioral patterns, psychological traits, physiological state, and social relationships. It uses an LSTM network to model the temporal variation of profile parameters, displays the relative relationships of multidimensional features through radar charts, tracks the trajectory of health indicator changes using trend lines, and visualizes the distribution of behavioral patterns using heatmaps, thereby constructing a continuously evolving three-dimensional user profile.

[0105] Furthermore, such as Figure 5 As shown, the learning evolution layer includes an adaptive learning module, a federated learning collaboration module, and an active monitoring and reminder module;

[0106] The adaptive learning module is deployed on the user terminal, receiving the user's time-series health data and feedback data on historical health management plans, and dynamically adjusting the user-personalized model stored locally.

[0107] In a specific embodiment, the adaptive learning module integrates the linear UCB algorithm, the neural gambling machine model, and the Thompson sampling method to construct a contextual multi-arm gambling machine system. It combines online support vector machines, incremental neural networks, and decaying memory models to form an incremental learning framework. Through a dynamic learning rate adjustment mechanism, it processes user time-series health data and historical scheme feedback information to achieve continuous optimization and parameter updates of the local personalized model.

[0108] The federated learning collaboration module includes a central server and multiple federated clients corresponding to user terminals. It coordinates multiple federated clients to jointly train and optimize the global model managed by the central server without the local data leaving the terminal. It adopts privacy-preserving aggregation technology, supports heterogeneous federated learning and personalized federated learning, and performs balanced optimization of the global model and the local model.

[0109] In a specific embodiment, the federated learning collaboration module constructs a distributed training architecture that includes a central server and multiple federated clients. It uses the FedAvg algorithm to perform model aggregation operations, sets the differential privacy noise scale to 1.0, selects 10% of the clients to participate in each round of training, processes client parameter updates through a secure aggregation protocol, uses an adaptive weighted average algorithm to calculate the global model update amount, and finally uses model fusion technology to solve the parameter conflict problem, thereby achieving collaborative optimization of the global model and the local personalized model.

[0110] The proactive monitoring and alert module defines and detects health event patterns based on a complex event processing engine. It adopts dynamic threshold adjustment based on personal baselines and supports multi-channel alerts, including voice, messaging, telephone, and smart device linkage. It also features anti-harassment mechanisms and importance classification functions.

[0111] In a specific embodiment, the proactive monitoring and alert module defines health event patterns based on a complex event processing engine, uses the isolated forest algorithm to implement anomaly detection, generates personalized early warning baselines through a dynamic threshold adjustment mechanism, supports a multi-channel notification system including voice broadcast, message push, telephone reminders, and smart device linkage, is equipped with an anti-harassment mechanism based on frequency control and time period management, and implements an importance-based push strategy according to the health risk level.

[0112] Furthermore, such as Figure 6 As shown, the security and compliance layer includes a data security protection module, a compliance inspection module, and a risk warning module;

[0113] The data security protection module employs end-to-end encryption, homomorphic encryption, differential privacy, and secure multi-party computation technologies, covering the entire process of data transmission, storage, and processing across the perception and interaction layer, core processing layer, learning and evolution layer, and output presentation layer.

[0114] In a specific embodiment, the data security protection module uses the TLS 1.3 transport layer protocol and the AES-256 storage layer algorithm to build a basic encryption system. It combines homomorphic encryption technology to support data processing operations in ciphertext state, uses differential privacy mechanism to add controllable noise to protect individual privacy, and realizes cross-organizational data collaboration through a secure multi-party computation protocol. These security technologies together cover the entire data flow process from perception and interaction to output presentation at all levels.

[0115] The compliance check module has a built-in configurable regulatory rule base, which checks the compliance of system recommendations output by the core processing layer in real time. It supports graphical rule configuration, rule verification before execution decisions, and has regional adaptation and rule version management functions, and complies with HIPAA and GDPR requirements.

[0116] In a specific embodiment, the compliance check module has a built-in configurable regulatory rule base containing HIPAA and GDPR provisions. It supports the visual configuration of rule conditions through a graphical interface, uses a real-time compliance checker to verify the compliance of health advice before decision generation, and is equipped with a rule adaptation mechanism based on regional characteristics and a version management system to maintain different versions of rules, ensuring that the system output complies with the medical regulatory requirements of multiple countries.

[0117] The risk warning module pre-sets standardized risk warning information for different health recommendations, stores them according to the type of health recommendation, and enforces output for high-risk health recommendations.

[0118] In a specific embodiment, the risk warning module establishes a standardized warning information database categorized by type, such as traditional Chinese medicine conditioning plans, exercise suggestions, and dietary recommendations. It adopts a hierarchical storage architecture to manage warning content of different risk levels and ensures that high-risk health advice must be accompanied by corresponding warning information through a mandatory triggering mechanism. This mechanism works in conjunction with the compliance inspection module to form a complete risk prevention and control system.

[0119] Furthermore, such as Figure 7 As shown, the output presentation layer includes a dynamic health visualization module and a multi-terminal application module;

[0120] The dynamic health visualization module receives the health status assessment results transmitted from the core processing layer, transforms the health status assessment results into personalized, interactive visualization charts and outputs them. The health status assessment results include constitution, syndrome, organ balance status and health risk warning. The visualization charts include meridian diagrams, five elements mutual generation and restraint diagrams, radar charts, trend lines and heat maps.

[0121] In a specific embodiment, the dynamic health visualization module builds a TCM-featured chart library based on the WebGL high-performance rendering engine. It dynamically displays the circulation of Qi and blood through meridian diagrams, presents the interrelationships between the internal organs using the Five Elements mutual generation and restraint diagram, displays constitution characteristics from multiple dimensions using radar charts, depicts the trajectory of health indicator changes using trend lines, and visualizes the distribution of behavioral patterns using heat maps. It transforms abstract constitution identification data, syndrome classification results, organ balance status, and health risk warnings into intuitive and interactive visual elements.

[0122] The multi-terminal application module supports access from multiple terminals, including the Web, mobile applications, smart speakers, and in-vehicle systems. It provides accessibility design in accordance with the WCAG 2.1AA standard, which includes voice-first interaction, high-contrast interface themes, and full support for screen readers.

[0123] In a specific embodiment, the multi-terminal application module adopts the React Native cross-platform framework to implement a unified code library, supporting multi-terminal adaptation to web browsers, iOS / Android mobile applications, smart speakers and in-vehicle systems. It reduces the operation threshold through voice-first interaction design, provides high-contrast interface themes to meet the needs of visually impaired users, improves screen reader support functions to ensure accessible information, and the overall interface design strictly follows the perceptibility, operability, comprehensibility and robustness requirements of the WCAG 2.1AA accessibility standard.

[0124] Furthermore, a microservice architecture is adopted, with the perception and interaction layer, core processing layer, learning and evolution layer, security and compliance layer, and output presentation layer communicating through an API gateway, supporting horizontal scaling and high availability deployment.

[0125] In a specific embodiment, the system adopts a microservice architecture based on Docker containers. Each functional layer establishes a communication link through a unified API gateway. The TCM cognition microservice is configured with 8GB of memory and 4 CPU cores, the multimodal fusion microservice is allocated 4GB of memory and 2 CPU cores, and the dialogue management microservice is configured with 2GB of memory and 1 CPU core. These microservices are uniformly orchestrated and managed through a Kubernetes cluster. The system dynamically adjusts the number of replicas between 5 and 50 based on CPU utilization using a horizontal Pod auto-scaling mechanism. At the same time, a multi-region disaster recovery solution is provided to ensure the high availability of the system services.

[0126] Furthermore, such as Figure 2 As shown, the intelligent interactive method for TCM health management based on multimodal data recognition has the following steps:

[0127] S1, the perception and interaction layer performs data acquisition and interaction processing:

[0128] Step S11: Collect at least one of the user's voice, tongue image, and facial image data through the multimodal perception module;

[0129] Step S12: The voiceprint recognition module uses a deep neural network model based on x-vector to extract voiceprint feature dimensions including fundamental frequency contour, formant structure, prosodic pattern, and vocal tract length, and performs identity authentication and personalized services based on the user's voice features.

[0130] Step S13: Extract prosodic features and deep acoustic features from the user's speech signal through the emotion recognition module, and combine them with text sentiment analysis based on the TCM emotion dictionary to output confidence scores for joy, anger, worry, thought, grief, fear, and surprise.

[0131] Step S14: Through the full-duplex dialogue module, an LSTM-based dialogue state tracker is used to support real-time voice activity detection and interruption processing, maintain dialogue history memory, perform smooth processing of topic switching and continuation, and support multi-turn dialogue context management.

[0132] Step S15: Using an end-to-end speech translation model through real-time speech translation and TTS module, it supports real-time translation between multiple languages. Based on neural speech synthesis technology, it performs voiceprint-driven timbre adaptation and emotion perception prosodic generation to carry out cross-language interaction and personalized speech synthesis, and outputs interactive response information.

[0133] In a specific embodiment, during the perception and interaction stage, user voice signals and visual image data are collected synchronously through multiple source sensors. The x-vector deep neural network architecture is used to extract voiceprint biometric features to complete identity authentication. The prosodic features obtained by the OpenSmile toolkit and the BERT model enhanced by the TCM emotional dictionary are combined to output the seven emotion state vectors. The LSTM dialogue state tracker is used to maintain the multi-turn dialogue context. The SpeechTransformer architecture is used to realize low-latency cross-language interaction and generate voiceprint-driven personalized voice feedback.

[0134] S2. The core processing layer performs TCM reasoning and solution generation:

[0135] Step S21: Transform the TCM theoretical system into a computable knowledge graph through the digital module of the Huangdi Neijing framework;

[0136] Step S22: Through the algorithmic module of traditional Chinese medicine theory, based on knowledge graph and user data, perform algorithmic calculation of dialectical reasoning;

[0137] Step S23: Based on the classical theoretical knowledge system of traditional Chinese medicine, the TCM knowledge reasoning module receives the data transmitted by the perception and interaction layer, performs TCM syndrome differentiation reasoning, and outputs the syndrome differentiation reasoning results.

[0138] Step S24: Generate a personalized TCM health management plan based on the reasoning results of the TCM knowledge reasoning module through the health plan generation module;

[0139] Step S25: Receive and understand the dream information described by the user through the dream diagnosis and organ mapping module, perform reasoning based on the mapping relationship between dreams and organ health in traditional Chinese medicine theory, assess the user's organ function status and generate corresponding health risk warnings;

[0140] Step S26: By integrating multi-dimensional information through the dynamic user profile module, a five-dimensional profile model is constructed, including a basic attribute layer, a behavior pattern layer, a psychological trait layer, a physiological state layer, and a social relationship layer. An LSTM network is used to capture the profile change trend, and a visualization is performed through radar charts, trend lines, and heat maps to construct and update a three-dimensional user profile that evolves over time.

[0141] In a specific embodiment, during the TCM reasoning stage, dialectical analysis is performed based on a knowledge graph with a scale of 100,000 entities. Personalized health plans are generated through constraint satisfaction algorithms. Natural language processing technology is used to analyze dream text and map the functional state of internal organs. Five-dimensional feature data is integrated to construct a dynamic user profile. An LSTM network is used to model the temporal change pattern and display the health trend through multi-dimensional charts.

[0142] S3. Learn and optimize the evolutionary layer model:

[0143] Step S31: Receive the user's time-series health data and feedback data on historical health management plans through the adaptive learning module, and dynamically adjust the locally stored user-personalized model.

[0144] Step S32: Through the central server of the federated learning collaboration module, coordinate multiple federated clients corresponding to user terminals, and use privacy-preserving aggregation technology to support heterogeneous federated learning and personalized federated learning, while ensuring that local data does not leave the terminal, jointly train and optimize the global model managed by the central server, and perform balanced optimization of the global model and the local model.

[0145] Step S33: Transfer the optimized global model parameters to the adaptive learning module to update the locally stored user-personalized model;

[0146] In a specific embodiment, during the model optimization stage, the local model is adaptively updated through the contextual multi-armed gambling machine algorithm. Relying on the federated learning framework, multiple terminals are coordinated to complete the global model training under the premise of privacy protection. The secure aggregation protocol is used to handle parameter updates and resolve model conflicts, forming a continuously evolving health management model system.

[0147] S4. Security and compliance layer implements security and compliance controls:

[0148] Step S41: The data security protection module employs end-to-end encryption, homomorphic encryption, differential privacy, and secure multi-party computation technologies to provide full security protection for the data transmission, storage, and processing of the perception interaction layer, core processing layer, learning evolution layer, and output presentation layer.

[0149] Step S42: The compliance check module calls the built-in configurable regulatory rule library to check the compliance of the health management plan output by the core processing layer in real time, perform rule verification before decision-making, and, based on the regional adaptation and rule version management functions, check the results in accordance with the relevant requirements of HIPAA and GDPR.

[0150] Step S43: Based on the different health recommendations output by the core processing layer, the risk warning module retrieves the preset standardized risk warning information and transmits it to the core processing layer, and performs forced output for high-risk health recommendations;

[0151] In a specific embodiment, a data protection system covering the entire process is built during the security and compliance phase. Homomorphic encryption and differential privacy technologies are used to ensure data processing security. A configurable rule base is used to verify the compliance of health recommendations in real time. A tiered early warning mechanism is used to forcibly output high-risk alerts to ensure that the system meets international medical standards.

[0152] S5. Output the execution results of the presentation layer:

[0153] Step S51: Receive the health status assessment results transmitted from the core processing layer through the dynamic health visualization module, and convert the health status assessment results into personalized interactive visualization charts including meridian diagrams, five elements mutual generation and restraint diagrams, radar charts, trend lines, and heat maps.

[0154] Step S52: Receive the interactive response information output by the perception interaction layer and the charts output by the dynamic health visualization module through the multi-terminal application module, and provide access interfaces for Web, mobile applications, smart speakers, and in-vehicle systems, supporting barrier-free operation including voice-first interaction, high-contrast interface themes, and full support for screen readers.

[0155] In a specific embodiment, during the results presentation stage, health assessment data is transformed into visual charts with TCM characteristics. Interactive meridian diagrams and five-element relationship diagrams are rendered using a WebGL engine. A cross-platform framework is adopted to achieve multi-terminal adaptive display. Combined with voice-first interaction and high-contrast interface design, it meets the needs of barrier-free operation, forming a complete closed loop of health management services.

[0156] Furthermore, a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of a smart interactive method for TCM health management based on multimodal data recognition, the storage medium including a solid-state drive, a USB flash drive, a portable hard drive, a read-only memory, a random access memory, a magnetic disk, or an optical disk.

[0157] In a specific embodiment, a computer-readable storage medium carries the complete program code of the TCM health management intelligent interactive system. This code includes x-vector model parameters of the voiceprint recognition module, TCM emotion dictionary data of the emotion analysis module, Neo4j graph database files of the knowledge reasoning module, and model aggregation algorithms of the federated learning platform. In actual deployment, the system uses a solid-state drive to store 100,000 entity relationship data in the knowledge graph, quickly imports system initialization configuration via USB flash drive, completes offline backup of health record data with the help of a portable hard drive, uses read-only memory to solidify the core dialectical reasoning algorithm, uses random access memory to maintain real-time dialogue status information, uses magnetic disk to store historical health record archives, and uses optical disc media for long-term preservation of the system image. These storage media together constitute the data infrastructure for stable system operation.

[0158] The specific embodiments of the present invention have been described in detail above, but they are merely examples, and the present invention is not equivalent to the specific embodiments described above. For those skilled in the art, any equivalent modifications and substitutions to the present invention are also within the scope of the present invention. Therefore, all equivalent transformations and modifications made without departing from the spirit and scope of the present invention should be covered within the scope of the present invention.

Claims

1. A smart interactive system for TCM health management based on multimodal data recognition, characterized in that: It includes a perception and interaction layer, a core processing layer, a learning and evolution layer, a security and compliance layer, and an output presentation layer; The perception and interaction layer communicates with the core processing layer, transmits collected user data and interaction instructions to the core processing layer, and receives interaction response requests from the core processing layer. The core processing layer is connected to the learning and evolution layer, the security and compliance layer, and the output presentation layer, respectively, and transmits user time-series health data and solution feedback information to the learning and evolution layer, transmits the generated health management solution to the security and compliance layer, and transmits health status assessment results to the output presentation layer. The learning and evolution layer transmits model adjustment signals and health status monitoring results to the core processing layer, and updates the user-personalized model parameters of the core processing layer. The security and compliance layer transmits compliance check results and risk warning information to the core processing layer, and performs compliance verification on the health management plan output by the core processing layer. The output presentation layer receives interactive response information from the perception and interaction layer, health status assessment results from the core processing layer, and reminder signals from the learning and evolution layer, providing a user-accessible interactive interface.

2. The intelligent interactive system for TCM health management based on multimodal data recognition according to claim 1, characterized in that, The perception and interaction layer includes a multimodal perception module, a voiceprint recognition module, an emotion recognition module, a full-duplex dialogue module, and a real-time speech translation and TTS module. The multimodal perception module collects at least one of the user's voice, tongue image, and facial image data, and configures a voice interaction interface to receive user input information; The voiceprint recognition module uses a deep neural network model based on x-vector to extract voiceprint features including fundamental frequency contour, formant structure, prosodic pattern, and vocal tract length for identity authentication and personalized services. The emotion recognition module extracts prosodic features and deep acoustic features from the user's voice signal, and combines them with text sentiment analysis based on a traditional Chinese medicine emotion dictionary to output confidence scores for joy, anger, worry, thought, grief, fear, and surprise. The full-duplex dialogue module is equipped with an LSTM-based dialogue state tracker, which supports real-time voice activity detection and interruption processing, maintains dialogue history memory, performs smooth processing of topic switching and continuation, and supports multi-turn dialogue context management. The real-time speech translation and TTS module adopts an end-to-end speech translation model, supports real-time translation between multiple languages, and performs voiceprint-driven timbre adaptation and emotion perception prosodic generation based on neural speech synthesis technology to carry out cross-language interaction and personalized speech synthesis.

3. The intelligent interactive system for TCM health management based on multimodal data recognition according to claim 2, characterized in that, The core processing layer includes a digital module for the Yellow Emperor's Inner Classic framework, a module for algorithmizing traditional Chinese medicine theories, a module for reasoning traditional Chinese medicine knowledge, a module for generating health plans, a module for dream diagnosis and organ mapping, and a dynamic user profile module; The aforementioned digital module of the Yellow Emperor's Inner Classic framework transforms the theoretical system of Yin-Yang and Five Elements, Zang-Xiang and Meridians, Qi, Blood and Body Fluids, and Etiology and Pathogenesis in the Yellow Emperor's Inner Classic into a computable knowledge graph containing entities and relationships. The TCM theory algorithm module is connected to the Huangdi Neijing framework digital module, which transforms the logical process of TCM syndrome differentiation and treatment into an executable algorithm model, including a knowledge graph-based symbolic reasoning algorithm, a health data-based statistical learning algorithm, and a neural network reasoning algorithm for discovering potential correlations. The TCM knowledge reasoning module is built on the knowledge system of "Huangdi Neijing". It receives data transmitted from the perception and interaction layer, performs TCM syndrome differentiation reasoning, and outputs the syndrome differentiation reasoning results. The health plan generation module receives the reasoning results from the TCM knowledge reasoning module and generates a personalized TCM health management plan. The dream diagnosis and organ mapping module receives and understands the dream information described by the user, performs reasoning based on the mapping relationship between dreams and organ health in traditional Chinese medicine theory, assesses the user's organ function status and generates corresponding health risk warnings. The dynamic user profile module integrates multi-dimensional information to construct a three-dimensional user profile that evolves over time. It constructs a five-dimensional profile model including a basic attribute layer, a behavior pattern layer, a psychological trait layer, a physiological state layer, and a social relationship layer. It uses an LSTM network to capture the profile's changing trends and performs visualization through radar charts, trend lines, and heat maps.

4. The intelligent interactive system for TCM health management based on multimodal data recognition according to claim 3, characterized in that, The learning evolution layer includes an adaptive learning module, a federated learning collaboration module, and an active monitoring and reminder module; The adaptive learning module is deployed on the user terminal, receives the user's time-series health data and feedback data on historical health management plans, and dynamically adjusts the locally stored user-personalized model. The federated learning collaboration module includes a central server and multiple federated clients corresponding to user terminals. It coordinates multiple federated clients to jointly train and optimize the global model managed by the central server without leaving the terminal, adopts privacy-preserving aggregation technology, supports heterogeneous federated learning and personalized federated learning, and performs balanced optimization of the global model and the local model. The proactive monitoring and alert module defines and detects health event patterns based on a complex event processing engine, adopts dynamic threshold adjustment based on personal baselines, and supports multi-channel alerts, including voice, messages, phone calls, and smart device linkage. It also features an anti-harassment mechanism and importance classification function.

5. The intelligent interactive system for TCM health management based on multimodal data recognition according to claim 4, characterized in that, The security and compliance layer includes a data security protection module, a compliance inspection module, and a risk warning module. The data security protection module employs end-to-end encryption, homomorphic encryption, differential privacy, and secure multi-party computation technologies, covering the entire process of data transmission, storage, and processing across the perception and interaction layer, core processing layer, learning and evolution layer, and output presentation layer. The compliance check module has a built-in configurable regulatory rule base, checks the compliance of system recommendations output by the core processing layer in real time, supports graphical rule configuration, rule verification before execution decisions, has regional adaptation and rule version management functions, and complies with HIPAA and GDPR requirements. The risk warning module pre-sets standardized risk warning information for different health recommendations, stores them according to the type of health recommendation, and forces the output of high-risk health recommendations.

6. The intelligent interactive system for TCM health management based on multimodal data recognition according to claim 5, characterized in that, The output presentation layer includes a dynamic health visualization module and a multi-terminal application module; The dynamic health visualization module receives the health status assessment results transmitted from the core processing layer, transforms the health status assessment results into personalized, interactive visualization charts and outputs them. The health status assessment results include constitution, syndrome, organ balance status and health risk warning. The visualization charts include meridian charts, five elements mutual generation and restraint charts, radar charts, trend lines and heat maps. The multi-terminal application module supports multi-terminal access from Web, mobile applications, smart speakers, and in-vehicle systems, and provides WCAG compliance. 2.1AA standard accessibility design, which includes voice-first interaction, high-contrast interface themes and full screen reader support.

7. The intelligent interactive system for TCM health management based on multimodal data recognition according to claim 6, characterized in that, The system adopts a microservice architecture, in which the perception and interaction layer, core processing layer, learning and evolution layer, security and compliance layer, and output presentation layer communicate through an API gateway, supporting horizontal scaling and high availability deployment.

8. A TCM health management intelligent interaction method based on multimodal data recognition, executing a TCM health management intelligent interaction system based on multimodal fusion as described in any one of claims 1-7, characterized in that, The steps are as follows: S1, the perception and interaction layer performs data acquisition and interaction processing: Step S11: Collect at least one of the user's voice, tongue image, and facial image data through the multimodal perception module; Step S12: The voiceprint recognition module uses a deep neural network model based on x-vector to extract voiceprint feature dimensions including fundamental frequency contour, formant structure, prosodic pattern, and vocal tract length, and performs identity authentication and personalized services based on the user's voice features. Step S13: Extract prosodic features and deep acoustic features from the user's speech signal through the emotion recognition module, and combine them with text sentiment analysis based on the TCM emotion dictionary to output confidence scores for joy, anger, worry, thought, grief, fear, and surprise. Step S14: Through the full-duplex dialogue module, an LSTM-based dialogue state tracker is used to support real-time voice activity detection and interruption processing, maintain dialogue history memory, perform smooth processing of topic switching and continuation, and support multi-turn dialogue context management. Step S15: Using an end-to-end speech translation model through real-time speech translation and TTS module, it supports real-time translation between multiple languages. Based on neural speech synthesis technology, it performs voiceprint-driven timbre adaptation and emotion perception prosodic generation to carry out cross-language interaction and personalized speech synthesis, and outputs interactive response information. S2. The core processing layer performs TCM reasoning and solution generation: Step S21: Transform the TCM theoretical system into a computable knowledge graph through the digital module of the Huangdi Neijing framework; Step S22: Through the algorithmic module of traditional Chinese medicine theory, based on knowledge graph and user data, perform algorithmic calculation of dialectical reasoning; Step S23: Based on the classical theoretical knowledge system of traditional Chinese medicine, the TCM knowledge reasoning module receives the data transmitted by the perception and interaction layer, performs TCM syndrome differentiation reasoning, and outputs the syndrome differentiation reasoning results. Step S24: Generate a personalized TCM health management plan based on the reasoning results of the TCM knowledge reasoning module through the health plan generation module; Step S25: Receive and understand the dream information described by the user through the dream diagnosis and organ mapping module, perform reasoning based on the mapping relationship between dreams and organ health in traditional Chinese medicine theory, assess the user's organ function status and generate corresponding health risk warnings; Step S26: By integrating multi-dimensional information through the dynamic user profile module, a five-dimensional profile model is constructed, including a basic attribute layer, a behavior pattern layer, a psychological trait layer, a physiological state layer, and a social relationship layer. An LSTM network is used to capture the profile change trend, and a visualization is performed through radar charts, trend lines, and heat maps to construct and update a three-dimensional user profile that evolves over time. S3. Learn and optimize the evolutionary layer model: Step S31: Receive the user's time-series health data and feedback data on historical health management plans through the adaptive learning module, and dynamically adjust the locally stored user-personalized model. Step S32: Through the central server of the federated learning collaboration module, coordinate multiple federated clients corresponding to user terminals, and use privacy-preserving aggregation technology to support heterogeneous federated learning and personalized federated learning, while ensuring that local data does not leave the terminal, jointly train and optimize the global model managed by the central server, and perform balanced optimization of the global model and the local model. Step S33: Transfer the optimized global model parameters to the adaptive learning module to update the locally stored user-personalized model; S4. Security and compliance layer implements security and compliance controls: Step S41: The data security protection module employs end-to-end encryption, homomorphic encryption, differential privacy, and secure multi-party computation technologies to provide full security protection for the data transmission, storage, and processing of the perception interaction layer, core processing layer, learning evolution layer, and output presentation layer. Step S42: The compliance check module calls the built-in configurable regulatory rule library to check the compliance of the health management plan output by the core processing layer in real time, perform rule verification before decision-making, and, based on the regional adaptation and rule version management functions, check the results in accordance with the relevant requirements of HIPAA and GDPR. Step S43: Based on the different health recommendations output by the core processing layer, the risk warning module retrieves the preset standardized risk warning information and transmits it to the core processing layer, and performs forced output for high-risk health recommendations; S5. Output the execution results of the presentation layer: Step S51: Receive the health status assessment results transmitted from the core processing layer through the dynamic health visualization module, and convert the health status assessment results into personalized interactive visualization charts including meridian diagrams, five elements mutual generation and restraint diagrams, radar charts, trend lines, and heat maps. Step S52: Receive the interactive response information output by the perception interaction layer and the charts output by the dynamic health visualization module through the multi-terminal application module, and provide access interfaces for Web, mobile applications, smart speakers, and in-vehicle systems, supporting barrier-free operation including voice-first interaction, high-contrast interface themes, and full support for screen readers.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the intelligent interactive method for TCM health management based on multimodal data recognition as described in claim 8, wherein the storage medium comprises a solid-state drive, a USB flash drive, a portable hard drive, a read-only memory, a random access memory, a magnetic disk, or an optical disk.

Images