Multi-modal data fusion chronic disease risk prediction and dynamic management system

Abstract

The invention discloses a chronic disease risk prediction and dynamic management system based on multi-modal data fusion, and relates to the technical field of chronic disease management, the chronic disease risk prediction and dynamic management system comprises a data acquisition layer, a data fusion processing layer, a risk prediction management layer and an application service layer, the data fusion processing layer performs preprocessing, feature extraction and fusion analysis, the risk prediction management layer constructs a risk prediction model according to result analysis, and the application service layer provides a risk prediction result, a personalized management scheme and an interactive interface service for a user. Through the arrangement of the data acquisition layer, the data fusion processing layer and the risk prediction management layer, 'symptom-physiology-image 'full-dimensional data is covered, more comprehensive health state evaluation is supported, dynamic intelligent prediction can be carried out, diet, exercise and medication suggestions are automatically adjusted according to patient execution feedback and the latest prediction result, and the health state evaluation efficiency is improved. And a self-adaptive management cycle is formed.

Description

Technical Field

[0001] The present invention relates to the technical field of chronic disease management, and in particular to a chronic disease risk prediction and dynamic management system based on multimodal data fusion. Background Art

[0002] Chronic diseases (such as hypertension, diabetes, coronary heart disease, etc.) have become one of the major challenges to global public health. Their long-term nature, complexity and high incidence rate have placed higher demands on the allocation of medical resources and health management.

[0003] The traditional chronic disease management model mainly relies on single-dimensional clinical data (such as electronic medical records, laboratory tests, etc.), which is difficult to fully reflect the patient's health status and lacks dynamic risk assessment and personalized intervention capabilities. Although wearable devices can collect physiological data, they lack effective integration with clinical symptoms. In addition, existing risk prediction models are mostly based on static data and cannot be combined with real-time physiological indicators (such as blood sugar and blood pressure fluctuations) to dynamically adjust the prediction results, resulting in delayed management plans. Summary of the Invention

[0004] The purpose of the present invention is to provide a chronic disease risk prediction and dynamic management system based on multimodal data fusion to solve the problems raised in the above background technology.

[0005] To achieve the above objectives, the present invention provides the following technical solutions: a multimodal data fusion chronic disease risk prediction and dynamic management system, comprising:

[0006] A data acquisition layer, which is used to acquire multi-source heterogeneous data, including clinical symptom data, physiological data collected by wearable devices, and image recognition data;

[0007] A data fusion processing layer, which performs preprocessing, feature extraction and fusion analysis on the collected multimodal data;

[0008] The risk prediction management layer constructs a risk prediction model based on the analysis of the results of the data fusion processing layer;

[0009] The application service layer provides users with risk prediction results, personalized management solutions and interactive interface services.

[0010] Preferably, the data acquisition layer includes:

[0011] A clinical symptom data collection unit, which designs a structured electronic questionnaire to collect symptom data and extracts the patient's clinical data through natural language processing technology in combination with an electronic medical record system;

[0012] A wearable device data acquisition unit, which collects physiological data through multiple types of wearable devices;

[0013] An image recognition data acquisition unit integrates a medical imaging device interface to obtain the patient's imaging examination data, is equipped with a portable image acquisition device to collect the patient's surface features in real time, and uses edge computing technology for preprocessing and feature extraction.

[0014] Preferably, the data fusion processing layer includes:

[0015] A data preprocessing module that standardizes clinical symptom data, performs time synchronization and normalization on wearable device data, and performs preprocessing on image data such as noise reduction, enhancement, and segmentation;

[0016] A feature extraction module, which extracts symptom feature vectors from clinical symptom data, extracts time domain and frequency domain features from wearable device data, and uses a convolutional neural network to extract features from image data;

[0017] The data fusion module uses decision-level fusion to establish a multimodal data fusion model, designs a fusion strategy based on weighted voting to predict different diseases, dynamically adjusts weights to obtain prediction results, and automatically focuses on data modalities and features according to attention weights.

[0018] Preferably, the attention weight calculation formula between the modal features in the data fusion module is:

[0019] Among them, the clinical symptom feature vector S, the wearable device feature vector W and the image feature vector I are mapped to the same feature space through the fully connected layer to obtain S′, W′, I′, and similarly, a w and a i .

[0020] Preferably, the fusion feature vector calculation formula is:

[0021] F=a s ·S′+a w ·W′+a i ·I′.

[0022] Preferably, the risk prediction management layer includes:

[0023] A risk prediction module, which constructs a chronic disease risk prediction model based on multimodal fusion features and performs real-time risk assessment;

[0024] A dynamic management module generates a personalized dynamic management scheme based on the risk prediction result and the individual characteristics of the patient, and optimizes the scheme according to the execution of the patient on the management scheme and the change of the health state.

[0025] Preferably, the risk prediction module adopts algorithms including but not limited to random forest, gradient boosting tree and deep learning, establishes corresponding prediction models for different types of chronic diseases, dynamically updates the risk prediction result according to the latest collected multi-modal data, and evaluates the risk level and gives an early warning according to the result.

[0026] Preferably, the dynamic risk prediction model in the risk prediction module adopts a long short-term memory network combined with an attention mechanism based on time series, and the model structure is as follows:

[0027] The multi-modal fusion features are input into the LSTM network in time sequence to capture the time series characteristics of the data;

[0028] Through the attention mechanism, different weights are given to the features at different time points, and the influence of the features of the key time points on the risk prediction is highlighted;

[0029] The output layer adopts a softmax function to obtain probability scores of different risk levels.

[0030] Preferably, the application service layer comprises:

[0031] An interaction module, the interaction and feedback module comprises a patient end APP and a doctor end management platform;

[0032] A feedback module, the feedback module is used for collecting the use feedback of the patient and the doctor on the system.

[0033] Preferably, the patient end APP in the interaction module is used for viewing the risk prediction result and the management scheme, recording daily symptoms and living habits, and online communicating with the doctor, and the doctor end management platform is used for viewing the multi-modal data of the patient, adjusting the risk prediction result and the management scheme, and performing remote consultation.

[0034] Technical effects and advantages of the present application:

[0035] (1) The present application is set by the data acquisition layer, the data fusion processing layer and the risk prediction management layer, through the three acquisition units of structured electronic questionnaire, wearable device and image recognition, covers the full-dimensional data of "symptom-physiology-image", supports more comprehensive health state evaluation, can dynamically and intelligently predict, automatically adjusts the diet, exercise and drug suggestion according to the patient's execution feedback and the latest prediction result, and forms a self-adaptive management cycle.

[0036] (2) The application realizes doctor-patient collaborative interaction through the application service layer, links the patient end APP and the doctor end platform, supports remote consultation and scheme adjustment, improves management convenience, and facilitates personalized dynamic management according to patients. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 It is a system overall framework of the application.

[0038] Figure 2 It is a framework at the data acquisition layer of the application.

[0039] Figure 3 It is a framework at the data fusion processing layer of the application.

[0040] Figure 4 It is a framework at the risk prediction management layer of the application.

[0041] Figure 5 It is a framework at the application service layer of the application. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the application.

[0043] The application provides a multi-modal data fusion chronic disease risk prediction and dynamic management system as shown in Figure 1-5 The application provides a multi-modal data fusion chronic disease risk prediction and dynamic management system as shown in

[0044] The data collection layer includes a clinical symptom data collection unit, a wearable device data collection unit, and an image recognition data collection unit. The clinical symptom data collection unit designs a structured electronic questionnaire to collect symptom data, covering typical symptoms of common chronic diseases (such as hypertension, diabetes, and coronary heart disease), such as headache, dizziness, fatigue, polydipsia, and polyuria. It extracts patients' clinical data through natural language processing technology combined with an electronic medical record system, facilitating semantic analysis and standardized conversion of patients' unstructured symptom descriptions. It can also automatically extract clinical data such as patients' medical history, medication records, and laboratory test results.

[0045] The wearable device data acquisition unit collects physiological data from various wearable devices, such as smart watches, smart bracelets, ambulatory blood pressure monitors, and blood glucose meters. It collects physiological data in real time through communication methods such as Bluetooth and Wi-Fi, including heart rate, blood pressure, blood oxygen saturation, blood glucose, and body movement frequency. Data interface specifications are designed to ensure that data from wearable devices of different brands and models can be seamlessly integrated into the system. The collected physiological data is filtered and detected for outliers in real time to remove noise interference.

[0046] The image recognition data acquisition unit integrates medical imaging equipment interfaces such as CT, MRI, and ultrasound to obtain the patient's imaging examination data, such as dermatoscopes and fundus cameras, which are used to collect body surface features, fundus images, etc. It is equipped with portable image acquisition equipment to collect patient surface features in real time, and uses edge computing technology for preprocessing and feature extraction.

[0047] In addition, the data fusion processing layer includes a data preprocessing module, a feature extraction module, and a data fusion module. The data preprocessing module standardizes the clinical symptom data, establishes a symptom terminology dictionary, unifies the symptom description method, synchronizes and normalizes the wearable device data to ensure the consistency of the timestamps of data from different sources, and preprocesses the image data by denoising, enhancing, and segmenting to improve the accuracy of subsequent feature extraction;

[0048] The feature extraction module extracts symptom feature vectors from clinical symptom data, and extracts time domain, frequency domain features, and time-frequency domain features from wearable device data. Methods such as the bag-of-words model and TF-IDF are used to extract symptom feature vectors. For wearable device data, time domain features (such as mean, variance, standard deviation, etc.), frequency domain features (such as power spectral density, etc.), and time-frequency domain features (such as wavelet transform coefficients, etc.) are extracted. Convolutional neural networks (CNNs) are used to extract features from image data, such as texture features, shape features, and grayscale features.

[0049] The data fusion module uses decision-level fusion to establish a multimodal data fusion model, designs a fusion strategy based on weighted voting to predict different diseases, dynamically adjusts the weights to obtain prediction results, and automatically focuses on data modalities and features according to the attention weights; first, a single-modal risk prediction model is constructed using clinical symptom data, wearable device data, and image recognition data respectively. Then, a fusion strategy based on weighted voting is designed. According to the prediction accuracy of each single-modal model in different disease scenarios, the weights are dynamically adjusted to obtain the final multimodal fusion prediction results. The attention mechanism is introduced to enable the model to automatically focus on the data modalities and features that contribute more to disease risk prediction.

[0050] Specifically, the attention weight calculation formula between each modality feature in the data fusion module is:

[0051] Among them, the clinical symptom feature vector S, the wearable device feature vector W and the image feature vector I are mapped to the same feature space through the fully connected layer to obtain S′, W′, I′, and similarly, a w and a i .

[0052] The calculation formula of fusion feature vector is:

[0053] F=a s ·S′+a w ·W′+a i ·I′.

[0054] Furthermore, the risk prediction management layer includes a risk prediction module and a dynamic management module. The risk prediction module constructs a chronic disease risk prediction model based on multimodal fusion features and performs real-time risk assessment. The dynamic management module generates a personalized dynamic management plan based on the risk prediction results and combined with the patient's individual characteristics (such as age, gender, living habits, etc.). The management plan includes dietary recommendations, exercise plans, medication reminders, regular check-up arrangements, etc. According to the patient's implementation of the management plan and changes in health status, the plan is optimized to improve the management effect.

[0055] The risk prediction module uses algorithms including but not limited to random forests, gradient boosting trees, and deep learning to establish corresponding prediction models for different types of chronic diseases, such as hypertension risk prediction models and diabetes complications prediction models. It dynamically updates risk prediction results based on the latest collected multimodal data, evaluates risk levels based on the results, and issues warnings, assessing risk levels such as low, medium, and high risk.

[0056] The dynamic risk prediction model in the risk prediction module uses a long short-term memory network combined with an attention mechanism based on time series. Its model structure is as follows:

[0057] Input the multimodal fusion features into the LSTM network in chronological order to capture the time series characteristics of the data;

[0058] Through the attention mechanism, different weights are given to features at different time points, highlighting the impact of features at key time points on risk prediction;

[0059] The output layer uses the softmax function to obtain the probability scores of different risk levels.

[0060] Furthermore, the application service layer includes an interaction module and a feedback module. The interaction and feedback module includes a patient-side APP and a doctor-side management platform. In the interaction module, the patient-side APP is used to view risk prediction results and management plans, record daily symptoms and living habits, and communicate with doctors online. The doctor-side management platform is used to view the patient's multimodal data, adjust the management plan based on risk prediction results, and conduct remote consultations. The feedback module is used to collect feedback from patients and doctors on the use of the system for system optimization and improvement.

[0061] Finally, taking diabetes management as an example, the specific workflow of the system is as follows:

[0062] Data collection: Patients wear smart blood glucose meters and smart bracelets to collect real-time data such as blood glucose, heart rate, and body movement; they use the mobile phone app to fill in whether they have symptoms such as polydipsia and polyuria; and they undergo regular fundus examinations to obtain fundus images.

[0063] Data fusion processing: Perform time series analysis on blood glucose data to extract blood glucose fluctuation characteristics; perform semantic analysis on symptom data to extract symptom characteristics; perform feature extraction on fundus images to detect whether there are signs of diabetic retinopathy.

[0064] Risk prediction: Use multimodal fusion features to predict the patient's risk level for developing diabetic complications (such as retinopathy, nephropathy, etc.).

[0065] Dynamic Management: Based on risk prediction results, a personalized blood sugar control plan is generated, including dietary recommendations (such as carbohydrate intake), exercise plans (such as daily steps), and medication reminders. If a patient's blood sugar control is poor, the system automatically adjusts the management plan and issues an alert to the doctor.

[0066] Application services: Patients can view risk prediction results and management plans through the APP and record daily symptoms and living habits. Doctors can view patients' multimodal data and risk prediction results through the management platform to adjust management plans.

[0067] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multimodal data fusion chronic disease risk prediction and dynamic management system, characterized by: include: A data acquisition layer, which is used to acquire multi-source heterogeneous data, including clinical symptom data, physiological data collected by wearable devices, and image recognition data; A data fusion processing layer, which performs preprocessing, feature extraction and fusion analysis on the collected multimodal data; The risk prediction management layer constructs a risk prediction model based on the analysis of the results of the data fusion processing layer; The application service layer provides users with risk prediction results, personalized management solutions and interactive interface services.

2. The multimodal data fusion chronic disease risk prediction and dynamic management system according to claim 1, characterized in that: The data acquisition layer includes: A clinical symptom data collection unit, which designs a structured electronic questionnaire to collect symptom data and extracts the patient's clinical data through natural language processing technology in combination with an electronic medical record system; A wearable device data acquisition unit, which collects physiological data through multiple types of wearable devices; An image recognition data acquisition unit integrates a medical imaging device interface to obtain the patient's imaging examination data, is equipped with a portable image acquisition device to collect the patient's surface features in real time, and uses edge computing technology for preprocessing and feature extraction.

3. The multimodal data fusion chronic disease risk prediction and dynamic management system according to claim 1, characterized in that: The data fusion processing layer includes: A data preprocessing module that standardizes clinical symptom data, performs time synchronization and normalization on wearable device data, and performs preprocessing on image data such as noise reduction, enhancement, and segmentation; A feature extraction module, which extracts symptom feature vectors from clinical symptom data, extracts time domain and frequency domain features from wearable device data, and uses a convolutional neural network to extract features from image data; The data fusion module uses decision-level fusion to establish a multimodal data fusion model, designs a fusion strategy based on weighted voting to predict different diseases, dynamically adjusts weights to obtain prediction results, and automatically focuses on data modalities and features according to attention weights.

4. The multimodal data fusion chronic disease risk prediction and dynamic management system according to claim 3 is characterized in that: The attention weight calculation formula between each modality feature in the data fusion module is: Among them, the clinical symptom feature vector S, the wearable device feature vector W and the image feature vector I are mapped to the same feature space through the fully connected layer to obtain S′, W′, I′, and similarly, a w and a i .

5. The multimodal data fusion chronic disease risk prediction and dynamic management system according to claim 4 is characterized in that: The calculation formula of the fused feature vector is: F=a s ·S′+a w ·W′+a i ·I′。 6. The multimodal data fusion chronic disease risk prediction and dynamic management system according to claim 1, characterized in that: The risk profile management layer includes: A risk prediction module, which constructs a chronic disease risk prediction model based on multimodal fusion features and performs real-time risk assessment; A dynamic management module generates a personalized dynamic management plan based on risk prediction results and the patient's individual characteristics, and optimizes the plan according to the patient's implementation of the management plan and changes in health status.

7. The multimodal data fusion chronic disease risk prediction and dynamic management system according to claim 5, characterized in that: The risk prediction module uses algorithms including but not limited to random forests, gradient boosting trees, and deep learning to establish corresponding prediction models for different types of chronic diseases. It dynamically updates risk prediction results based on the latest collected multimodal data, evaluates risk levels based on the results, and issues warnings.

8. The multimodal data fusion chronic disease risk prediction and dynamic management system according to claim 5, characterized in that: The dynamic risk prediction model in the risk prediction module adopts a long short-term memory network combined with an attention mechanism based on time series, and its model structure is as follows: Input the multimodal fusion features into the LSTM network in chronological order to capture the time series characteristics of the data; Through the attention mechanism, different weights are given to features at different time points, highlighting the impact of features at key time points on risk prediction; The output layer uses the softmax function to obtain the probability scores of different risk levels.

9. The multimodal data fusion chronic disease risk prediction and dynamic management system according to claim 1, characterized in that: The application service layer includes: Interaction module, the interaction and feedback module includes the patient-side APP and the doctor-side management platform; The feedback module is used to collect feedback from patients and doctors on the use of the system.

10. The multimodal data fusion chronic disease risk prediction and dynamic management system according to claim 9, characterized in that: In the interactive module, the patient-side APP is used to view risk prediction results and management plans, record daily symptoms and living habits, and communicate with doctors online. The doctor-side management platform is used to view the patient's multimodal data, adjust the management plan based on risk prediction results, and conduct remote consultations.

Images