Traditional Chinese medicine intelligent diagnosis system based on multi-modal data fusion and diagnosis method thereof

Abstract

The invention relates to a multi-modal data fusion-based traditional Chinese medicine intelligent diagnosis system and a diagnosis method thereof, and the system comprises a user terminal which is integrated with a mirror surface assembly which is used for providing face image feedback for a user; the image acquisition module is used for acquiring a tongue image of a user; the bioelectricity sensing module is used for collecting a bioelectricity signal of a hand of a user; the remote server is connected with the user terminal through a communication network, and the feature extraction module is used for carrying out deep learning analysis on the tongue image data to extract tongue image feature vectors; and the multi-modal data fusion diagnosis module is used for normalizing the tongue picture feature vector and the bioelectricity feature vector and then inputting the tongue picture feature vector and the bioelectricity feature vector into a pre-trained traditional Chinese medicine syndrome type classification model. The method has the advantages that a complete traditional Chinese medicine intelligent diagnosis solution is constructed, and the method has remarkable advantages in the aspects of technical advancement, practicability, industrialization value and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to a traditional Chinese medicine intelligent diagnosis system based on multi-modal data fusion and a diagnosis method thereof, and belongs to the field of traditional Chinese medicine intelligent diagnosis. BACKGROUND

[0002] The core of traditional Chinese medicine diagnosis lies in the comprehensive analysis of the four diagnostic methods of observation, listening, questioning and palpation. Among them, tongue diagnosis, as an important part of observation, judges the zang-fu organs by observing the changes of tongue and tongue fur; and pulse diagnosis, as a part of palpation, understands the state of qi and blood by sensing the changes of pulse. However, the traditional diagnosis method highly depends on the personal experience and subjective judgment of the doctor, and lacks unified objective quantitative standard. The observation of tongue image is easily affected by light conditions and environmental factors, and the pulse diagnosis skill needs years of clinical practice to master. This strong experience-dependent and low-standard situation seriously restricts the inheritance and development of traditional Chinese medicine.

[0003] With the progress of science and technology, digital tongue image collection equipment and electronic pulse instrument and other traditional Chinese medicine diagnosis instruments have appeared in the market. These devices realize the digital recording of diagnosis information to some extent, but most of the products have single function and can only collect single type data such as tongue image or pulse, and cannot realize the synchronous collection and comprehensive analysis of multi-modal information. At the same time, the existing devices are usually complex to operate and need to be operated by professional personnel, and the data collection and diagnosis analysis are disconnected, lacking intelligent data analysis model support. More importantly, these devices are usually designed with strong professional nature, and the user experience is not good, which is difficult to integrate into daily health monitoring scene and cannot meet the needs of modern people for convenient and intelligent traditional Chinese medicine health management.

[0004] In summary, the existing traditional Chinese medicine diagnosis technology has obvious limitations: on the one hand, the traditional diagnosis method is highly subjective and lacks standardization; on the other hand, the existing digital equipment has single function, complex operation and low intelligent degree. Therefore, it is urgent to develop a traditional Chinese medicine intelligent diagnosis system which can integrate multi-modal data collection, realize intelligent fusion analysis and has good user experience, so as to promote the development of traditional Chinese medicine diagnosis towards objectivity, standardization and intelligence. SUMMARY

[0005] In order to overcome the defects of the prior art, the present application provides a traditional Chinese medicine intelligent diagnosis system based on multi-modal data fusion and a diagnosis method thereof. The technical scheme of the present application is as follows: A traditional Chinese medicine intelligent diagnosis system based on multi-modal data fusion, comprising: A user terminal, wherein the user terminal is integrated with: A mirror assembly for providing facial image feedback to the user; An image acquisition module arranged near the mirror assembly for collecting tongue images of the user; A bioelectric sensing module is disposed in the grip part of the user terminal and is used to collect bioelectric signals from the user's hand. The terminal processing and communication module is used to control the image acquisition module and the bioelectric sensing module to perform synchronous data acquisition, and to send the acquired tongue image data and bioelectric signal data to diagnostic data. A remote server, connected to the user terminal via a communication network, includes: The communication receiving module is used to receive diagnostic data from the user terminal; The feature extraction module is used to perform deep learning analysis on the tongue image data to extract tongue image feature vectors, and to perform time-frequency domain analysis on the bioelectric signal data to extract bioelectric feature vectors. The multimodal data fusion diagnostic module is used to normalize the tongue image feature vector and the bioelectric feature vector and then input them into a pre-trained TCM syndrome classification model. The TCM syndrome classification model calculates the joint contribution probability of the tongue image feature vector and the bioelectric feature vector to different TCM syndromes through a weighted fusion algorithm, and outputs the final TCM syndrome diagnosis result and health advice.

[0006] The mirror component of the user terminal is a semi-transparent and semi-reflective mirror located in front of the tablet computer screen. The image acquisition module is the front-facing camera of the tablet computer. The terminal processing and communication module is implemented by the processor and network module of the tablet computer.

[0007] The feature extraction module performs deep learning analysis on tongue image data, specifically by using a convolutional neural network to segment the tongue image, obtain the tongue body and tongue coating regions, and extract their color, texture, and morphological features respectively, which together constitute the tongue image feature vector.

[0008] The feature extraction module performs time-frequency domain analysis of bioelectric signal data, specifically including: performing wavelet transform on the bioelectric signal, extracting the energy distribution of a specific frequency band as features, and calculating the sample entropy or complexity of the signal, which together constitute the bioelectric feature vector.

[0009] The weighted fusion algorithm is implemented based on an attention mechanism, which specifically includes: Calculate attention weights for the tongue image feature vector and the bioelectric feature vector respectively; The attention weights are calculated using a single-layer neural network, and the calculation process is as follows: α_t = softmax(W_a * V_tongue), α_b = softmax(W_a * V_bio); Where α_t represents the attention weight vector of tongue image features, α_b represents the attention weight vector of bioelectrical features, W_a represents the attention weight matrix, V_tongue represents the normalized tongue image feature vector, and V_bio represents the normalized bioelectrical feature vector. softmax represents the normalized exponential function; The original feature vector is weighted using the calculated attention weights: V_t' = α_t·V_tongue; V_b' = α_b·V_bio Where V_t' represents the weighted tongue image feature vector and V_b' represents the weighted bioelectric feature vector. The weighted feature vectors are concatenated and input into a fully connected layer for classification, and the probability distribution of each TCM syndrome is calculated.

[0010] It also includes a feedback guidance module, which overlays the ideal tongue outline onto the user's image in the mirror during the data acquisition process at the user terminal, so as to guide the user to adjust the tongue position to the optimal acquisition area.

[0011] The remote server also includes a model optimization module, which is used to dynamically adjust the weight matrix W_a in the attention mechanism based on the historical data and feedback continuously uploaded by the user, thereby realizing the personalized evolution of the diagnostic model.

[0012] A TCM diagnostic method based on the aforementioned multimodal data fusion-based intelligent TCM diagnostic system includes the following steps: S1. Synchronously collect tongue images and hand bioelectrical signals through the user terminal; S2. Send the data to the remote server; S3. The server extracts the tongue image feature vector and the bioelectric feature vector respectively; S4. Calculate the joint contribution probability of tongue appearance and bioelectrical features to different TCM syndromes using a weighted fusion algorithm based on attention mechanism. S5. Determine the final TCM syndrome diagnosis result based on the highest probability and generate a suggestion; S6. Feedback the diagnostic results to the user terminal.

[0013] The advantages of this invention are: Breaking through the limitations of traditional single diagnostic modes, this method synchronously collects and deeply integrates visual information of the tongue image with bioelectrophysiological signals. Employing a weighted fusion algorithm based on an attention mechanism, it can adaptively evaluate the contribution of different modalities of data to the diagnostic results. Through joint contribution probability calculation, it significantly improves the accuracy and reliability of TCM syndrome identification and solves the problem of strong subjectivity in traditional diagnosis. The solution, which combines a semi-transparent mirror and a tablet computer, seamlessly integrates diagnostic equipment into everyday life scenarios. Augmented reality guidance ensures standardized data collection and effectively eliminates interference from environmental factors. The overall design makes professional TCM diagnosis simple and easy to use, greatly reducing the user's operational threshold.

[0014] The system achieves automatic and accurate extraction of tongue features based on deep learning convolutional neural networks. The personalized model optimization module enables the diagnostic system to adapt to the evolution of individual user characteristics over time through a continuous learning mechanism, forming a complete closed loop from data collection, feature extraction, intelligent diagnosis to personalized services.

[0015] By making full use of existing mature hardware platforms (tablets, cameras, etc.), effectively controlling manufacturing costs, and possessing good scalability, it can be easily integrated into application scenarios such as smart healthcare and family health management, providing a practical and feasible technical path for the standardization, objectification, and intelligentization of TCM diagnosis.

[0016] It effectively solves the problem of reliance on experience in the inheritance of traditional Chinese medicine, provides a standardized data collection and analysis platform for scientific research in traditional Chinese medicine, and enables high-quality traditional Chinese medicine diagnostic services to benefit a wider range of people. Attached Figure Description

[0017] Figure 1 This is a block diagram of the main structure of the present invention.

[0018] Figure 2 yes Figure 1 A structural block diagram of a Chinese user terminal.

[0019] Figure 3 yes Figure 1 Structure diagram of a medium-to-long-range server. Detailed Implementation

[0020] The present invention will be further described below with reference to specific embodiments, and the advantages and features of the present invention will become clearer as a result. However, these embodiments are merely exemplary and do not constitute any limitation on the scope of the present invention. Those skilled in the art should understand that modifications or substitutions can be made to the details and form of the technical solutions of the present invention without departing from the spirit and scope of the present invention, but all such modifications and substitutions fall within the protection scope of the present invention.

[0021] See Figures 1 to 3 This invention relates to a TCM intelligent diagnostic system based on multimodal data fusion, comprising: User terminal 1, wherein the user terminal integrates: Mirror component 11 is used to provide facial image feedback to the user; Image acquisition module 12 is located near the mirror component and is used to acquire images of the user's tongue; The bioelectric sensing module 13 is disposed on the grip part of the user terminal and is used to collect bioelectric signals from the user's hand. The terminal processing and communication module 14 is used to control the image acquisition module and the bioelectric sensing module to perform synchronous data acquisition, and to send the acquired tongue image data and bioelectric signal data to diagnostic data. Remote server 2 is connected to the user terminal via a communication network and includes: The communication receiving module 21 is used to receive diagnostic data from the user terminal; The feature extraction module 22 is used to perform deep learning analysis on the tongue image data to extract tongue image feature vectors, and to perform time-frequency domain analysis on the bioelectric signal data to extract bioelectric feature vectors. The multimodal data fusion diagnostic module 23 is used to normalize the tongue image feature vector and the bioelectric feature vector and then input them into a pre-trained TCM syndrome classification model. The TCM syndrome classification model calculates the joint contribution probability of the tongue image feature vector and the bioelectric feature vector to different TCM syndromes through a weighted fusion algorithm, and outputs the final TCM syndrome diagnosis result and health advice.

[0022] By simultaneously acquiring tongue images and hand bioelectric signals, the traditional Chinese medicine diagnostic methods of observation and palpation are organically combined. Using an attention-based weighted fusion algorithm, the contribution of different modal features is adaptively balanced, effectively overcoming the diagnostic limitations of a single data source and significantly improving the accuracy of TCM syndrome identification.

[0023] The integrated design of the mirror assembly and image acquisition module allows the diagnostic process to be naturally integrated into daily mirror examination scenarios; the separate handle design ensures stable acquisition of bioelectrical signals. The overall solution greatly reduces the operating threshold, making professional TCM diagnosis simple and easy to use.

[0024] Based on a cloud-based collaborative architecture, the system achieves accurate feature extraction through deep learning and signal processing technologies in the feature extraction module. Combined with the gradient descent algorithm in the model optimization module, the system dynamically adjusts the weight parameters, enabling the diagnostic system to continuously evolve as it is used, thus achieving personalized and accurate diagnosis.

[0025] The mirror component of the user terminal 1 is a semi-transparent, semi-reflective mirror positioned in front of the tablet computer's display screen. The image acquisition module is the front-facing camera of the tablet computer, and the terminal processing and communication module is implemented by the tablet computer's processor and network module. By covering the tablet computer's display screen with the semi-transparent, semi-reflective mirror, electronic display and optical reflection are cleverly integrated, allowing the user to simultaneously complete tongue image acquisition while seeing their own facial image in the mirror. This design naturally integrates the professional diagnostic process into the user's daily mirror-viewing action, utilizing the tablet computer's mature camera, processor, and network module to greatly reduce hardware costs, while completely eliminating the operational burden and psychological pressure brought by traditional medical equipment, achieving truly "diagnosis completed in the mirror"—a truly unobtrusive data acquisition.

[0026] The feature extraction module 22 performs deep learning analysis on tongue image data, specifically by using a convolutional neural network to segment the tongue image, obtaining the tongue body and tongue coating regions, and extracting their color, texture, and morphological features to form the tongue image feature vector. By performing pixel-level segmentation of the tongue image using a convolutional neural network, the tongue body and tongue coating regions can be accurately distinguished, and their multi-dimensional features such as color, texture, and morphology can be extracted. This transforms traditional Chinese medicine's empirical descriptions such as "pale red tongue, thin white coating" into objective data features, providing a reliable data foundation for subsequent intelligent diagnosis.

[0027] The feature extraction module 22 performs time-frequency domain analysis of the bioelectric signal data, specifically including: performing wavelet transform on the bioelectric signal, extracting the energy distribution of specific frequency bands as features, and calculating the sample entropy or complexity of the signal, which together constitute the bioelectric feature vector. Extracting the energy distribution of different frequency bands through wavelet transform can reflect the rhythmic characteristics of human physiological states; simultaneously, combining nonlinear dynamic indicators such as sample entropy quantifies the complexity of the signal, thereby more comprehensively capturing bioelectric information related to TCM syndrome types and providing a deeper basis for diagnosis.

[0028] The weighted fusion algorithm is implemented based on an attention mechanism, which specifically includes: Calculate attention weights for the tongue image feature vector and the bioelectric feature vector respectively; The attention weights are calculated using a single-layer neural network, and the calculation process is as follows: α_t = softmax(W_a * V_tongue), α_b = softmax(W_a * V_bio); Where α_t represents the attention weight vector of tongue image features, α_b represents the attention weight vector of bioelectrical features, W_a represents the attention weight matrix, V_tongue represents the normalized tongue image feature vector, and V_bio represents the normalized bioelectrical feature vector. softmax represents the normalized exponential function; The original feature vector is weighted using the calculated attention weights: V_t' = α_t·V_tongue; V_b' = α_b·V_bio; where V_t' represents the weighted tongue image feature vector and V_b' represents the weighted bioelectrical feature vector. The weighted feature vectors are concatenated and input into a fully connected layer for classification, calculating the probability distribution of each TCM syndrome. Through an attention mechanism, the algorithm can dynamically evaluate and assign importance weights to specific tongue image and bioelectrical signal features in different syndrome judgments, thereby simulating the weighing of various symptoms and signs by TCM experts when considering the four diagnostic methods, significantly improving the rationality and accuracy of diagnostic decisions.

[0029] It also includes a feedback guidance module, which overlays an ideal tongue outline onto the user's image in the mirror during data acquisition at the user's terminal, guiding the user to adjust the tongue position to the optimal acquisition area. Overlaying the ideal tongue outline onto the user's mirror image in real time provides the user with an intuitive positional reference, effectively solving problems such as inaccurate tongue placement and improper acquisition distance caused by the lack of a reference point. This ensures the standardization and consistency of the acquired data from the source, laying a reliable foundation for subsequent accurate diagnosis.

[0030] The remote server also includes a model optimization module. This module dynamically adjusts the weight matrix W_a in the attention mechanism based on continuously uploaded historical data and feedback from users, using a gradient descent algorithm to achieve personalized evolution of the diagnostic model. By dynamically adjusting the attention weights based on users' historical data, the system can gradually learn and adapt to each user's individual physiological characteristics, allowing the diagnostic model to become increasingly personalized with increased usage frequency, ultimately providing each user with tailored TCM diagnostic services.

[0031] A TCM diagnostic method based on the aforementioned multimodal data fusion-based intelligent TCM diagnostic system includes the following steps: S1. Synchronously collect tongue images and hand bioelectrical signals through the user terminal; S2. Send the data to the remote server; S3. The server extracts the tongue image feature vector and the bioelectric feature vector respectively; S4. Calculate the joint contribution probability of tongue appearance and bioelectrical features to different TCM syndromes using a weighted fusion algorithm based on attention mechanism. S5. Determine the final TCM syndrome diagnosis result based on the highest probability and generate a suggestion; S6. Feedback the diagnostic results to the user terminal.

[0032] Through a standardized process of synchronous data collection, cloud processing, intelligent fusion, and real-time feedback, the traditional Chinese medicine process of relying on manual diagnosis and treatment of the four diagnostic methods is transformed into an efficient, objective, and replicable automated diagnostic system. While preserving the essence of TCM syndrome differentiation and treatment, it achieves standardization of the diagnostic process and quantification of results.

[0033] This invention, based on multimodal data fusion technology, achieves intelligent and standardized TCM diagnosis through a deep integration of hardware innovation and intelligent algorithms. Its working principle is implemented in the following steps: Step 1: User preparation and data synchronization The user stands in front of the user terminal 1, facing the mirror component 11 to observe their own facial image. After the system starts, the user adjusts the tongue position according to the augmented reality contour displayed by the feedback guidance module, while simultaneously holding the bioelectric sensing module 13. The terminal processing and communication module 14 controls the image acquisition module 12 and the bioelectric sensing module 13 to perform synchronous data acquisition, acquiring the tongue image and hand bioelectric signals at the same time.

[0034] Step 2: Data Transmission and Preprocessing The acquired tongue image data and bioelectrical signal data are processed by the terminal and packaged into diagnostic data by the communication module 14, and then transmitted to the remote server 2 via the communication network. After receiving the data, the communication receiving module 21 performs preliminary format conversion and verification to ensure the integrity and usability of the data.

[0035] Step 3: Multimodal Feature Extraction Feature extraction module 22 performs parallel processing on the received data: For tongue image data, a convolutional neural network is used for pixel-level segmentation to accurately distinguish the tongue body and tongue coating areas, and color, texture and morphological features are extracted separately to generate tongue image feature vectors. For bioelectrical signal data, wavelet transform is performed to extract the energy distribution of specific frequency bands, and nonlinear indices such as sample entropy are calculated to generate bioelectrical feature vectors. Step 4: Intelligent Fusion Diagnosis The multimodal data fusion diagnostic module 23 normalizes the two feature vectors and then inputs them into the pre-trained TCM syndrome classification model. The model uses an attention mechanism to achieve weighted fusion. Attention weights are calculated using a single-layer neural network: α_t = softmax(W_a * V_tongue), α_b = softmax(W_a * V_bio). The feature vectors are weighted using attention weights: V_t' = α_t·V_tongue, V_b' = α_b·V_bio The weighted feature vectors are concatenated and then input into a fully connected layer to calculate the joint contribution probability distribution of each TCM syndrome. Step 5: Diagnostic Decision-Making and Personalized Optimization The final TCM syndrome diagnosis is determined based on the highest probability, and corresponding health recommendations are generated. Simultaneously, the model optimization module updates the user's historical records based on the diagnostic data and dynamically adjusts the attention weight matrix W_a using the gradient descent algorithm, enabling continuous personalized evolution of the diagnostic model.

[0036] Step 6: Results Feedback and Loop Completion The diagnostic results and health recommendations are returned to user terminal 1 via the communication network and displayed on the mirror component 11, completing the entire diagnostic process. Users can immediately obtain professional-grade TCM diagnostic results, while the system accumulates personalized data for the next diagnosis.

[0037] This invention realizes a complete closed loop from data acquisition to diagnostic feedback, transforming the traditional Chinese medicine process of combining the four diagnostic methods into a standardized and intelligent modern diagnostic process. It not only maintains the essence of TCM syndrome differentiation and treatment, but also possesses the accuracy and repeatability of modern medical technology.

[0038] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A TCM intelligent diagnostic system based on multimodal data fusion, characterized in that, include: The user terminal integrates: Mirror components are used to provide users with facial image feedback; An image acquisition module, located near the mirror component, is used to acquire images of the user's tongue. A bioelectric sensing module is disposed in the grip part of the user terminal and is used to collect bioelectric signals from the user's hand. The terminal processing and communication module is used to control the image acquisition module and the bioelectric sensing module to perform synchronous data acquisition, and to send the acquired tongue image data and bioelectric signal data to diagnostic data. A remote server, connected to the user terminal via a communication network, includes: The communication receiving module is used to receive diagnostic data from the user terminal; The feature extraction module is used to perform deep learning analysis on the tongue image data to extract tongue image feature vectors, and to perform time-frequency domain analysis on the bioelectric signal data to extract bioelectric feature vectors. The multimodal data fusion diagnostic module is used to normalize the tongue image feature vector and the bioelectric feature vector and then input them into a pre-trained TCM syndrome classification model. The TCM syndrome classification model calculates the joint contribution probability of the tongue image feature vector and the bioelectric feature vector to different TCM syndromes through a weighted fusion algorithm, and outputs the final TCM syndrome diagnosis result and health advice.

2. The TCM intelligent diagnostic system based on multimodal data fusion according to claim 1, characterized in that, The mirror component of the user terminal is a semi-transparent and semi-reflective mirror located in front of the tablet computer screen. The image acquisition module is the front-facing camera of the tablet computer. The terminal processing and communication module is implemented by the processor and network module of the tablet computer.

3. The TCM intelligent diagnostic system based on multimodal data fusion according to claim 1, characterized in that, The feature extraction module performs deep learning analysis on tongue image data, specifically by using a convolutional neural network to segment the tongue image, obtain the tongue body and tongue coating regions, and extract their color, texture, and morphological features respectively, which together constitute the tongue image feature vector.

4. The TCM intelligent diagnostic system based on multimodal data fusion according to claim 2 or 3, characterized in that, The feature extraction module performs time-frequency domain analysis of bioelectric signal data, specifically including: performing wavelet transform on the bioelectric signal, extracting the energy distribution of a specific frequency band as features, and calculating the sample entropy or complexity of the signal, which together constitute the bioelectric feature vector.

5. The TCM intelligent diagnostic system based on multimodal data fusion according to claim 1, characterized in that, The weighted fusion algorithm is implemented based on an attention mechanism, which specifically includes: Calculate attention weights for the tongue image feature vector and the bioelectric feature vector respectively; The attention weights are calculated using a single-layer neural network, and the calculation process is as follows: α_t = softmax(W_a * V_tongue), α_b = softmax(W_a * V_bio); Where α_t represents the attention weight vector of tongue image features, α_b represents the attention weight vector of bioelectrical features, W_a represents the attention weight matrix, V_tongue represents the normalized tongue image feature vector, and V_bio represents the normalized bioelectrical feature vector. softmax represents the normalized exponential function; The original feature vector is weighted using the calculated attention weights: V_t' = α_t·V_tongue; V_b' = α_b·V_bio; Where V_t' represents the weighted tongue image feature vector and V_b' represents the weighted bioelectric feature vector. The weighted feature vectors are concatenated and input into a fully connected layer for classification, and the probability distribution of each TCM syndrome is calculated.

6. The TCM intelligent diagnostic system based on multimodal data fusion according to claim 5, characterized in that, It also includes a feedback guidance module, which overlays the ideal tongue outline onto the user's image in the mirror during the data acquisition process at the user terminal, so as to guide the user to adjust the tongue position to the optimal acquisition area.

7. The TCM intelligent diagnostic system based on multimodal data fusion according to claim 5, characterized in that, The remote server also includes a model optimization module, which is used to dynamically adjust the weight matrix W_a in the attention mechanism based on the historical data and feedback continuously uploaded by the user, thereby realizing the personalized evolution of the diagnostic model.

8. A traditional Chinese medicine diagnostic method based on the system described in any one of claims 1 to 7, characterized in that, Including the following steps: S1. Synchronously collect tongue images and hand bioelectrical signals through the user terminal; S2. Send the data to the remote server; S3. The server extracts the tongue image feature vector and the bioelectric feature vector respectively; S4. Calculate the joint contribution probability of tongue appearance and bioelectrical features to different TCM syndromes using a weighted fusion algorithm based on attention mechanism. S5. Determine the final TCM syndrome diagnosis result based on the highest probability and generate a suggestion; S6. Feedback the diagnostic results to the user terminal.

Images