Medical care data intelligent evaluation system based on multi-modal data fusion

Abstract

The application discloses a medical care data intelligent evaluation system based on multi-modal data fusion and belongs to the technical field of medical care. The application solves the problems that the existing technology has subjective bias, the evaluation results of different medical staffs for the same patient can be different, and the medical data source is single and cannot comprehensively reflect the patient information. Through the real-time feedback of nursing staff, the frequency of model update is dynamically adjusted by combining the frequency of abnormal data in the feedback results in a period of time, so as to reduce the misjudgment caused by model bias; the abnormal data in the multi-modal data is identified by calculating Z-score, the time series data is set as high priority, and the abnormal data in the high priority is preferentially fed back to the nursing staff, which not only improves the response speed of the nursing staff in the face of abnormal conditions, but also optimizes the allocation of nursing resources, so that the nursing staff can more timely handle the emergency, thereby improving the nursing quality and patient safety.

Description

Technical Field

[0001] This invention relates to the field of medical and nursing technology, specifically to an intelligent assessment system for medical and nursing data based on multimodal data fusion. Background Technology

[0002] In the field of healthcare, patient assessment is a crucial basis for developing care plans and treatment protocols. Furthermore, with the advancement of medical technology, the amount of medical data is constantly increasing. Traditional healthcare assessments primarily rely on the subjective judgment of medical staff and limited medical data, such as medical records and examination reports. However, this method has several limitations. For example, the subjective judgment of medical staff may be biased, and different medical staff may arrive at different assessments of the same patient; traditional medical data sources are singular and cannot comprehensively reflect the patient's health status, relying solely on medical records and examination reports to fail to obtain information such as the patient's lifestyle habits and psychological state.

[0003] Therefore, it does not meet the existing needs, so we propose a medical care data intelligent assessment system based on multimodal data fusion. Summary of the Invention

[0004] The purpose of this invention is to provide an intelligent assessment system for medical and nursing data based on multimodal data fusion. By analyzing the frequency of abnormal data in the feedback results over a period of time through real-time feedback from nursing staff, the system dynamically adjusts the frequency of model updates, reducing misjudgments caused by model bias. It identifies abnormal data in the multimodal data by calculating Z-scores and prioritizes time-series data, feeding back high-priority abnormal data to nursing staff first. This not only improves the response speed of nursing staff to abnormal situations but also optimizes the allocation of nursing resources, enabling nursing staff to handle emergencies more promptly, thereby improving the quality of care and patient safety, and solving the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A medical care data intelligent assessment system based on multimodal data fusion includes:

[0007] The multimodal data acquisition unit is configured to acquire patients' multimodal data in real time. The multimodal data includes: time-series physiological data, text data, image data, and environmental data.

[0008] The multimodal data fusion unit is configured to extract multi-dimensional feature information from multimodal data based on the Transformer fusion model, and perform deep fusion through a progressive fusion framework constructed by timestamp alignment, anchor learning and mapping, and context association to generate joint features with context information.

[0009] The intelligent assessment unit is configured to analyze joint features based on an LSTM time-series prediction model to obtain the comprehensive symptoms of clinical patients.

[0010] The intelligent evaluation unit includes:

[0011] The data analysis module is configured to identify specific symptoms through a multi-label classification model, analyze multimodal data using an LSTM time-series prediction model, predict the probability of adverse events based on dynamic accuracy assessment and self-correction mechanisms based on time steps, and analyze multimodal data using knowledge graphs and rule engines to derive the comprehensive symptoms of current clinical patients.

[0012] Furthermore, the multimodal data fusion unit performs deep fusion of multi-dimensional feature information to generate joint features with contextual information, including:

[0013] The core feature key-value pair generation module is used to collect the time information corresponding to each modality data, add timestamps to each modality data according to the time information, perform time alignment of each modality data according to the timestamps, generate time-feature key-value pairs according to the alignment results, extract modality core features from the modality data according to the time-feature key-value pairs, and fuse the modality core features with the time-feature key-value pairs to generate core feature key-value pairs.

[0014] The anchor point determination module is used to collect multimodal sample data and divide the multimodal sample data according to the core feature key-value pairs to obtain positive sample pairs and negative sample pairs. At the same time, a loss function is constructed based on the positive sample pairs, and the anchor points between each modality data are determined based on the loss function, positive sample pairs, and negative sample pairs.

[0015] The cross-modal anchor alignment module is used to extract the initial position features of anchor points in each modal sample data, standardize the initial position features, construct a cross-modal mapping based on the standardization results and a preset calibration algorithm, and align different anchor points in the same coordinate system based on the cross-modal mapping results.

[0016] The multimodal association feature generation module is used to collect feature information of different dimensions in each modality based on the cross-modal anchor point alignment results, and perform modal adaptation processing on the collected feature information of different dimensions. At the same time, a context association network between features of different dimensions is constructed based on the anchor points, and the feature information of different dimensions after modal adaptation processing is input into the context association network for processing to generate multimodal association features.

[0017] The joint feature generation module is used to input multimodal associated features into the fusion network for deep fusion and generate joint features with contextual information.

[0018] Furthermore, the data analysis module uses an LSTM time-series prediction model to analyze multimodal data and predict the probability of adverse events occurring within a specific future time period, and also includes:

[0019] When using the LSTM time series prediction model to analyze multimodal data and predict the probability of adverse events occurring within a specific future time period, the prediction time step is collected, and the prediction accuracy of the LSTM time series prediction model is calculated based on the prediction time step.

[0020] Obtain a preset accuracy threshold and compare it with the prediction accuracy to determine whether the LSTM time series prediction model needs to be modified.

[0021] When the prediction accuracy is equal to or greater than the preset accuracy threshold, it is determined that no correction is needed to the LSTM time series prediction model.

[0022] Otherwise, it is determined that the LSTM time series prediction model needs to be modified.

[0023] Furthermore, the intelligent evaluation unit includes:

[0024] The nursing configuration module is configured to list the most urgent nursing problems based on symptom identification and risk prediction results; list the patient's specific nursing needs based on the output of the context understanding engine; formulate specific nursing measures and execution frequencies based on the nursing needs; and provide real-time risk warning information based on the output of the risk prediction engine.

[0025] The secondary confirmation module is configured to obtain feedback results from nursing staff on the intelligent assessment of multimodal data based on the interactive interface, and determine whether there is any bias in the model;

[0026] The module is configured to periodically update and retrieve feedback results indicating deviations over a certain period, marking them as abnormal results; analyze the frequency of abnormal results during that period, and mark them as abnormal frequencies.

[0027] The abnormal frequency is compared with a preset threshold to determine whether the abnormal frequency exceeds the threshold range. If it does, the frequency of model updates is adjusted.

[0028] Furthermore, the multimodal data acquisition unit includes:

[0029] The data acquisition interval acquisition module is configured to acquire the data acquisition interval of multiple sensors deployed in the intelligent assessment system for medical and nursing data, and to set Class A time thresholds and Class B time thresholds.

[0030] The multimodal data partitioning module is configured to partition the multimodal data collected by multiple sensors according to the A-level time threshold and the B-level time threshold to obtain the first dataset, the second dataset, and the third dataset.

[0031] Furthermore, the multimodal data acquisition unit also includes:

[0032] The transmission interval setting module is configured to extract the acquisition time interval of the first dataset, the second dataset, and the third dataset to obtain the first information, the second information, and the third information; set the corresponding transmission time interval based on the first information, the second information, and the third information, and transmit the first dataset, the second dataset, and the third dataset to the multimodal data fusion unit in sequence according to the specified transmission time interval.

[0033] Furthermore, the multimodal data fusion unit includes:

[0034] The abnormal data extraction module is configured to receive the first dataset, the second dataset, and the third dataset sequentially according to the transmission time interval, and to calculate the absolute Z-score of each data point in the first dataset, the second dataset, and the third dataset in sequence. If the absolute Z-score of any data point is greater than a predetermined threshold, the dataset containing the abnormal data is marked as an abnormal subset.

[0035] The priority processing module is configured to extract anomalous data from the anomalous subsets in the first, second, and third datasets, perform deep fusion processing on the anomalous data based on the Transformer fusion model, and prioritize the transmission of the fused joint features and anomalous subsets to the intelligent evaluation unit for analysis.

[0036] The anomaly analysis module is configured to set time-series data in multimodal data as high priority, obtain the data source category of the anomaly data, determine whether the anomaly data belongs to high priority, and if so, provide feedback to nursing staff based on the interactive interface; otherwise, no action is taken.

[0037] Furthermore, the intelligent evaluation unit also includes:

[0038] The model optimization module is configured to update the weights of the multi-label classification model in real time using the online gradient descent method, update the weights of the LSTM time series prediction model in real time using the incremental learning method of LSTM, and adjust the relationships and rules in the knowledge graph in real time using the incremental update method of knowledge graph, based on the results of the feedback analysis, and then feed back the adjusted model results to the nursing staff in real time.

[0039] Furthermore, based on symptom identification and risk prediction results, the nursing configuration module lists the most urgent nursing problems, including:

[0040] The severity and risk probability of symptoms are divided into three levels: high, medium, and low. Each nursing problem is comprehensively assessed to determine its priority.

[0041] Based on the priority assessment results, a list of nursing issues sorted by priority is generated, and the highest priority nursing issues are extracted from it;

[0042] Based on the new symptom identification and risk prediction results, the list of the most urgent nursing problems is updated in real time and displayed to nursing staff through an interactive interface so that timely measures can be taken.

[0043] Furthermore, the nursing configuration module sets the execution frequency of nursing measures based on the patient's condition and nursing needs, including:

[0044] Obtain information on the severity of the patient's symptoms, the probability of risk, the priority of the response, and the highest priority nursing problem;

[0045] Set monitoring and intervention frequencies according to priorities, and formulate corresponding nursing measures;

[0046] Generate a nursing execution form and provide feedback to nursing staff via an interactive interface.

[0047] Compared with the prior art, the beneficial effects of the present invention are:

[0048] 1. In this invention, real-time feedback from nursing staff can promptly determine whether there are any deviations in the model; and by analyzing the frequency of abnormal data in the feedback results over a period of time, the frequency of model updates can be dynamically adjusted to reduce misjudgments caused by model deviations, enhance the stability and practicality of the system, significantly improve the accuracy and adaptability of the intelligent assessment system for medical and nursing data, better meet clinical needs, and improve nursing quality and patient safety.

[0049] 2. In this invention, abnormal data in multimodal data is identified by calculating Z-score, and the abnormal data is preferentially fused to ensure rapid capture and accurate analysis of key data, thereby quickly determining the patient's nursing needs and improving the quality of nursing care. Furthermore, time-series data is set as high priority, and abnormal data with high priority is fed back to nursing staff first. This not only improves the response speed of nursing staff to abnormal situations, but also optimizes the allocation of nursing resources, enabling nursing staff to handle emergencies more promptly, thereby improving the quality of nursing care and patient safety.

[0050] 3. By acquiring time information from various modalities, adding timestamps, and aligning them, time-feature key-value pairs are effectively obtained. After extracting core features, these are fused into core feature key-value pairs. Positive and negative sample pairs are then divided, a loss function is constructed to learn and determine anchor points, anchor point location features are processed, and a mapping is built to achieve cross-modal alignment. This effectively ensures that anchor points from different modalities can directly interact. Finally, features are collected and adapted, and an association network is constructed to generate associated features. These features are then input into a fusion network to obtain joint features containing context. Through time alignment and feature fusion, multimodal data are accurately associated, key anchor points are mined, and cross-modal alignment is achieved. The generated joint features take into account information from each modality and contextual logic, improving the accuracy and comprehensiveness of medical and nursing data assessment and providing a reliable basis for intelligent assessment.

[0051] 4. By collecting the prediction time step and calculating the prediction accuracy of the LSTM time series prediction model based on the prediction time step, the prediction accuracy can be used to effectively judge the LSTM time series prediction model. When correction is needed, the LSTM time series prediction model can be corrected in a timely and accurate manner, which helps to ensure the accurate operation of the LSTM time series prediction model and thus provides effective guarantee for the output results. Attached Figure Description

[0052] Figure 1 This is a flowchart of the intelligent assessment system for medical and nursing data based on multimodal data fusion according to the present invention. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] To address the limitations of existing technologies, such as the potential for differing assessments from different healthcare professionals for the same patient, leading to subjective bias, and the reliance on limited data sources (medical records and examination reports alone) which fail to comprehensively reflect a patient's health status, lifestyle habits, and psychological state, please refer to [the relevant documentation / reference]. Figure 1 This embodiment provides the following technical solution:

[0055] A medical care data intelligent assessment system based on multimodal data fusion includes:

[0056] The multimodal data acquisition unit is configured to collect multimodal data from multiple heterogeneous data sources in real time. The multimodal data includes: time-series physiological data: real-time acquisition of waveform and numerical time-series signals of patients through ECG monitors, blood glucose meters, thermometers, etc.; text data: extraction of text information from electronic health records, nursing records, doctor's orders, test reports, etc.; image data: acquisition of image information from medical images, ward monitoring videos, and patient body surface photos; and environmental data: acquisition of environmental information through temperature and humidity sensors, light sensors, and activity monitoring sensors (such as radar and infrared) in the ward.

[0057] The multimodal data acquisition unit includes:

[0058] The data acquisition interval acquisition module is configured to acquire the data acquisition interval of multiple sensors deployed in the intelligent assessment system for medical and nursing data, and to set Class A time thresholds and Class B time thresholds.

[0059] The multimodal data partitioning module is configured to partition multimodal data collected by multiple sensors based on a Level A time threshold and a Level B time threshold, resulting in a first dataset, a second dataset, and a third dataset. The first dataset consists of multimodal data collected by sensors whose sensor acquisition time interval is less than the Level A time threshold. The second dataset consists of multimodal data collected by sensors whose sensor acquisition time interval is equal to or greater than the Level A time threshold but less than the Level B time threshold. The third dataset consists of multimodal data collected by sensors whose sensor acquisition time interval is equal to or greater than the Level B time threshold.

[0060] The transmission interval setting module is configured to extract the acquisition time interval of the first dataset, the second dataset, and the third dataset to obtain the first information, the second information, and the third information; set the corresponding transmission time interval based on the first information, the second information, and the third information, and transmit the first dataset, the second dataset, and the third dataset to the multimodal data fusion unit in sequence according to the specified transmission time interval.

[0061] The beneficial effects achieved by the above are as follows: By introducing time thresholds to classify multimodal data and prioritizing processing based on the degree of data anomalies, the efficiency and response speed of the intelligent assessment system for medical and nursing data are significantly improved; by calculating Z-scores to identify anomalous data in multimodal data and prioritizing the fusion processing of anomalous data, the rapid capture and accurate analysis of key data are ensured, thereby quickly determining patients' nursing needs and improving the quality of care; and by setting time-series data as high priority and feeding back high-priority anomalous data to nursing staff first, not only is the response speed of nursing staff to abnormal situations improved, but the allocation of nursing resources is also optimized, enabling nursing staff to handle emergencies more promptly, thereby improving the quality of care and patient safety.

[0062] The multimodal data fusion unit is configured to preprocess multimodal data, extract multi-dimensional feature information from the multimodal data based on the Transformer fusion model, and perform deep fusion of the multi-dimensional feature information to generate joint features with contextual information.

[0063] The multimodal data fusion unit includes:

[0064] The abnormal data extraction module is configured to receive the first, second, and third datasets sequentially according to the transmission time interval, and calculate the absolute Z-score of each data point in the first, second, and third datasets in sequence. If the absolute Z-score of any data point is greater than a predetermined threshold (e.g., 3), the dataset containing the abnormal data is marked as an abnormal subset. For example, the first dataset is transmitted every 5 minutes, the second dataset every 15 minutes, and the third dataset every 30 minutes. In the first dataset, data point X=120 belongs to electrocardiogram data, and the data source category is time-series physiological data. Assuming that the Z-score of data point X=120 is 3.5, it is marked as abnormal data, and the subset containing this data point is marked as an abnormal subset.

[0065] The priority processing module is configured to extract anomalous data from the first, second, and third datasets, perform deep fusion processing on the anomalous data based on the Transformer fusion model, and prioritize the transmission of the fused joint features and anomalous subsets to the intelligent evaluation unit for analysis. For example, data labeled as anomalous subsets are extracted from the first dataset, the extracted anomalous subset data is prioritized into the Transformer model to generate joint features, and then transmitted to the intelligent evaluation unit.

[0066] The anomaly analysis module is configured to prioritize time-series data within multimodal data. It retrieves the data source category of abnormal data, determines whether the abnormal data belongs to high priority, and if so, prioritizes feedback to nursing staff via the interactive interface, allowing them to promptly monitor the analysis results and patient status. If not, no action is taken. For example, when an extracted subset of abnormal data is prioritized and input into the Transformer model, if the abnormal data X=120 is identified as high-priority time-series physiological data, the interactive interface will simultaneously display: Patient ID 12345, Abnormal data: ECG data point X=120, immediate investigation recommended. This improves the response speed to abnormal situations, enabling nursing staff to handle emergencies more promptly, thereby improving nursing quality and patient safety.

[0067] The intelligent assessment unit includes: a data analysis module, a nursing configuration module, a secondary confirmation module, and a periodic update module.

[0068] The data analysis module is configured to use a multi-label classification model to analyze joint features and identify specific symptoms, such as "dyspnea," "painful expression," "anxiety," and "pressure sore erythema." For example, if the joint features simultaneously include decreased blood oxygen saturation, a report description of "difficulty breathing," and intercostal muscle retraction in the video, the engine is highly certain that the symptom of "dyspnea" exists. It uses an LSTM time-series prediction model to analyze multimodal data and predict the probability of adverse events occurring within a specific future time period, such as "85% probability of falling in the next 2 hours" and "60% probability of developing pressure sores in the next 24 hours." It also uses a knowledge graph and rule engine to analyze multimodal data and derive the current comprehensive symptoms of clinical patients, such as "postoperative pain in patients leads to limited mobility, resulting in ineffective drainage of respiratory secretions and thus the risk of atelectasis."

[0069] The nursing configuration module is configured to: list the most urgent nursing problems based on symptom identification and risk prediction results; list the patient's specific nursing needs based on the output of the context understanding engine; formulate specific nursing interventions and their frequency of execution based on these needs; and provide real-time risk warning information based on the output of the risk prediction engine, specifically including:

[0070] The severity and risk probability of symptoms were categorized into high, medium, and low levels. Each nursing problem was comprehensively assessed to determine its priority. The priority rule was as follows: nursing problems with high severity or high risk probability had the highest priority; those with medium severity or medium risk probability had the second highest priority; and those with low severity or low risk probability had the lowest priority. For example, symptom identification results: the patient exhibited "dyspnea" (severity: high) and "painful expression" (severity: medium); risk prediction results: 85% risk of fall in the next 2 hours (high risk), and 60% risk of pressure sores in the next 24 hours (medium risk). Therefore, "dyspnea" (high severity) and "85% risk of fall in the next 2 hours" (high risk) had the highest priority, followed by "painful expression" (medium severity) and "60% risk of pressure sores in the next 24 hours" (medium risk). This allows for the following priority classification: "Dyspnea" (high severity) and "85% probability of falling in the next 2 hours" (high risk) are assessed as the highest priority, while "painful expression" (moderate severity) and "60% probability of pressure sores in the next 24 hours" (medium risk) are assessed as the second highest priority. Based on the priority assessment results, a list of nursing problems is generated in order of priority, and the highest priority nursing problems are extracted from it. For example, the list of most urgent nursing problems includes: dyspnea (high severity), risk of falling in the next 2 hours (high risk), painful expression (moderate severity), and risk of pressure sores in the next 24 hours (medium risk). The most urgent nursing problems are then determined to be: dyspnea (high severity) and risk of falling in the next 2 hours (high risk). Based on the new symptom identification and risk prediction results, the list of most urgent nursing problems is updated in real time and displayed to nursing staff in real time through an interactive interface so that timely measures can be taken.

[0071] Obtain the severity, risk probability, and response priority of the patient's symptoms, as well as the highest priority nursing problem; for example: the patient has "dyspnea" (high severity) and "painful expression" (moderate severity); the probability of falling in the next 2 hours is 85% (high risk), and the probability of developing pressure ulcers in the next 24 hours is 60% (medium risk); the nursing needs list includes: oxygen therapy, pain management, fall prevention, and pressure ulcer prevention; set the monitoring and intervention frequency according to priority, and formulate corresponding nursing measures; for "dyspnea" (high severity): monitor blood oxygen saturation every 1 hour, assess respiratory status every 2 hours, and formulate oxygen therapy measures; for "painful expression" (moderate severity): conduct pain assessment every 4 hours, and administer analgesics every 6 hours; for high fall risk (high risk): make rounds every 1 hour to ensure bed rails are raised; for pressure ulcer risk (medium risk): turn the patient every 4 hours and check skin condition every 6 hours; generate a nursing execution table from this, and provide feedback to nursing staff through an interactive interface.

[0072] The secondary confirmation module is configured to obtain feedback results from nursing staff on the intelligent assessment of multimodal data based on the interactive interface, and determine whether there is any bias in the model.

[0073] The model optimization module is configured to update the weights of the multi-label classification model in real time using online gradient descent, update the weights of the LSTM time-series prediction model in real time using incremental learning of LSTM, and adjust the relationships and rules in the knowledge graph in real time using incremental update of the knowledge graph, based on the feedback analysis results. The adjusted model results are then fed back to nursing staff in real time. For example, nursing staff can confirm the accuracy of the model's prediction of the symptom "dyspnea" through the interface. The nursing staff inputs feedback: "The model's prediction of the symptom 'dyspnea' is inaccurate; the patient does not actually have 'dyspnea'." The resulting timestamp is 2023-10-01 10:00, the patient ID is 12345, and the feedback content is: "The model's prediction of the symptom 'dyspnea' is inaccurate." If the model is found to frequently make errors in predicting "dyspnea," the weights of relevant features can be increased or the model structure adjusted. If the model adjusts its prediction of the symptom "dyspnea" based on new feedback, the new prediction results are immediately pushed to the nursing staff so that they can understand the model's improvement in a timely manner.

[0074] The periodic update module is configured to retrieve feedback results indicating deviations over a certain period and mark them as abnormal results; analyze the frequency of abnormal results during that period and mark them as abnormal frequencies; compare the abnormal frequencies with a preset threshold to determine if the abnormal frequency exceeds the threshold range. If it does, the model update frequency is adjusted. For example, it collects feedback data from the past 24 hours, marks all feedback results with incorrect model predictions, calculates the frequency of model prediction errors in the past 24 hours (assumed to be 15%), sets a threshold of 10%, and if the abnormal frequency exceeds 10%, the model is considered to need more frequent updates; if the abnormal frequency is 15%, exceeding the threshold by 50%, the model update frequency is adjusted from once a day to once every 12 hours. This improves the accuracy and adaptability of the model, ensuring the efficient operation of the intelligent assessment system for medical and nursing data.

[0075] The beneficial effects achieved by the above are as follows: real-time feedback from nursing staff enables timely judgment of whether there are any biases in the model; and by combining the analysis of the frequency of abnormal data in the feedback results over a period of time, the frequency of model updates can be dynamically adjusted, reducing misjudgments caused by model bias, enhancing the stability and practicality of the system, significantly improving the accuracy and adaptability of the intelligent assessment system for medical and nursing data, better meeting clinical needs, and improving the quality of nursing care and patient safety.

[0076] Working Principle: This system collects multimodal patient data in real-time from multiple heterogeneous data sources. A Transformer fusion model extracts multi-dimensional features and performs deep fusion to generate joint features with contextual information. A multi-label classification model analyzes these joint features to identify specific symptoms. An LSTM time-series prediction model predicts the probability of adverse events occurring within a specific future timeframe. A knowledge graph and rule engine are used to derive the current comprehensive symptoms of the patient. Based on the analysis results, the most urgent nursing problems are identified. Specific nursing interventions and their frequency are determined according to the patient's specific needs, and real-time risk warnings are provided. This ensures the accuracy and adaptability of multimodal data fusion and model analysis, providing intelligent decision support for healthcare.

[0077] In one embodiment, the intelligent assessment system for medical care data based on multimodal data fusion provided in this embodiment, wherein the multimodal data fusion unit performs deep fusion of multi-dimensional feature information to generate joint features with contextual information, including:

[0078] The core feature key-value pair generation module is used to collect the time information corresponding to each modality data, add timestamps to each modality data according to the time information, perform time alignment of each modality data according to the timestamps, generate time-feature key-value pairs according to the alignment results, extract modality core features from the modality data according to the time-feature key-value pairs, and fuse the modality core features with the time-feature key-value pairs to generate core feature key-value pairs.

[0079] The anchor point determination module is used to collect multimodal sample data and divide the multimodal sample data according to the core feature key-value pairs to obtain positive sample pairs and negative sample pairs. At the same time, a loss function is constructed based on the positive sample pairs, and the anchor points between each modality data are determined based on the loss function, positive sample pairs, and negative sample pairs.

[0080] The cross-modal anchor alignment module is used to extract the initial position features of anchor points in each modal sample data, standardize the initial position features, construct a cross-modal mapping based on the standardization results and a preset calibration algorithm, and align different anchor points in the same coordinate system based on the cross-modal mapping results.

[0081] The multimodal association feature generation module is used to collect feature information of different dimensions in each modality based on the cross-modal anchor point alignment results, and perform modal adaptation processing on the collected feature information of different dimensions. At the same time, a context association network between features of different dimensions is constructed based on the anchor points, and the feature information of different dimensions after modal adaptation processing is input into the context association network for processing to generate multimodal association features.

[0082] The joint feature generation module is used to input multimodal associated features into the fusion network for deep fusion and generate joint features with contextual information.

[0083] In this embodiment, adding timestamps to each modal data based on time information can be the "visit time" of the text record or the "collection time in seconds" of the physiological signal, in order to solve the problem of data asynchrony.

[0084] In this embodiment, time alignment of each modal data based on timestamps can be achieved, for example, by binding the text "9:00 cough" with audio coughing segments and body temperature data from the same period to generate "time-feature key-value pairs", such as: 9:00, text: cough; audio: coughing sound; body temperature: 37.5℃.

[0085] In this embodiment, modal core features (such as "cough frequency" in text, "cough intensity" in audio, and "temperature rise trend") are extracted from modal data based on time-feature key-value pairs. The modal core features are then fused with time-feature key-value pairs to generate core feature key-value pairs (e.g., 9:00, core features: frequent cough + high sound intensity + slight temperature rise). By outputting core feature key-value pairs, the accurate correlation of the time dimension is preserved, and key clinical information is focused, thereby improving the quality of data feature extraction.

[0086] In this embodiment, positive sample pairs are divided from core feature key-value pairs, for example: time t, text: cough; audio: high-frequency cough sound, which is a clinically logical association; and negative sample pairs, for example: time t, text: no cough; audio: high-frequency cough sound, which is a logically conflicting association. A loss function is constructed based on the "true association pattern" of positive sample pairs (e.g., maximizing the feature similarity of positive sample pairs and minimizing the feature similarity of negative sample pairs). Through model learning (e.g., contrastive learning), the model gradually identifies those "strongly associated feature combinations" that appear stably in positive and negative sample pairs. These combinations are the "anchor points" (e.g., the cross-modal association of "text cough description + concurrent high-frequency audio + slight increase in body temperature"). The anchor points output are the "core coordinate points" for subsequent cross-modal alignment.

[0087] In this embodiment, the initial position features of anchor points in each modality sample data are extracted, such as the "timestamp 9:00" of text anchor points and the "pixel coordinates (200, 300)" of image anchor points. The initial position features are standardized (e.g., text timestamps are converted to Unix time, and image pixel coordinates are normalized to physical dimensions "millimeters") to eliminate modal format differences. Based on the standardization results and preset calibration algorithms (e.g., time synchronization to correct device time difference, spatial registration to eliminate image distortion), cross-modal mapping (e.g., conversion rules of "text timestamp → audio time axis → image spatial coordinates") is constructed. Finally, anchor points of different modalities are mapped to the same coordinate system (e.g., "Unix timestamp + three-dimensional spatial coordinates") to achieve accurate alignment of cross-modal anchor points (e.g., text anchor point "9:00 cough" and image anchor point "9:00 abnormal lung image area" are associated in the same spatiotemporal coordinates).

[0088] In this embodiment, feature information of different dimensions in each modality is collected based on the cross-modal anchor point alignment results. Multi-dimensional features can be collected in a targeted manner based on the anchor point alignment results. According to the aligned spatiotemporal coordinates, such as the text "cough", the image "lung image", and the physiological signal "heart rate fluctuation" corresponding to a certain timestamp, feature fragments associated with the anchor points are extracted from each modality. For the text modality, semantic entities such as "cough frequency" and "duration" of the event corresponding to the anchor point are extracted. For the image modality, visual features of the spatial region of the anchor point are focused, such as "shadow density" and "texture changes" of the lung image. For time-series physiological data, the trend of indicators within the time window of the anchor point is extracted, such as "amount of increase in heart rate" and "fluctuation of blood oxygen saturation". This step ensures that the collected features all point to the same event through the anchor points, avoiding the mixing of irrelevant information.

[0089] In this embodiment, modal adaptation processing of the collected feature information of different dimensions includes: the dimensions and distribution of features of different modalities are different. For example, the semantic vector is 128-dimensional and the image feature is a 256-dimensional tensor. It is necessary to unify the dimensions through feature mapping, such as a fully connected layer, and normalize, such as standardizing to the same mean and variance, to project various features into the same semantic space, laying the foundation for cross-modal interaction. At this time, the spatiotemporal coordinates of the anchor point serve as "implicit labels" to ensure that the adapted features still retain the original association, such as the text and image features under a certain timestamp remaining spatially adjacent.

[0090] In this embodiment, constructing a contextual association network between features of different dimensions based on anchor points can utilize the temporal continuity of anchor points, such as a sequence of anchor points with consecutive timestamps, and the semantic association, such as the causal relationship between the "cough" anchor point and the "fever" anchor point. The association between features can be modeled through graph neural networks or temporal attention mechanisms. For example, anchor points can be used as graph nodes, and edge weights can be used to represent the temporal order (e.g., "cough" occurs after "elevated body temperature") or semantic dependence (e.g., "lung shadow" supports the diagnosis of "pneumonia") between features, so that scattered features can form a logically related network.

[0091] In this embodiment, multimodal correlation features are input into a fusion network for deep fusion to generate joint features with contextual information. This can be achieved by inputting multimodal correlation features into a fusion network, where the fusion network can be a cross-attention layer of a Transformer. Based on the input results, the features of each modality can effectively complement each other in the interaction: the semantic logic of the text (e.g., "persistent cough for 3 days"), correcting ambiguities in image features (e.g., the benign / malignant judgment of lung shadows), and the quantitative data of physiological signals (e.g., "sudden increase in heart rate") to enhance the objectivity of the text description (e.g., the degree of "severe cough"). At the same time, the spatiotemporal coordinates of the anchor points are incorporated into the fusion process as positional encoding, ensuring that the output joint features contain both the key details of each modality (e.g., shadow size, heart rate value) and fully retain the contextual logic of the anchor points, such as "when the cough occurs → what physiological changes accompany it → how the corresponding image features evolve".

[0092] The working principle and beneficial effects of the above technical solution are as follows: By acquiring the time information of data from each modality, adding timestamps and aligning them, time-feature key-value pairs are effectively obtained. After extracting core features, they are fused into core feature key-value pairs. Then, positive and negative sample pairs are divided, a loss function is constructed to learn and determine anchor points, the anchor point position features are processed, and a mapping is constructed to achieve cross-modal alignment, effectively ensuring that anchor points of different modalities can directly interact. Finally, features are collected and adapted, an association network is constructed to generate associated features, and the features are input into the fusion network to obtain joint features containing context. Through time alignment and feature fusion, multimodal data are accurately associated, key anchor points are mined and aligned across modalities, and the generated joint features take into account the information of each modality and the contextual logic, improving the accuracy and comprehensiveness of medical and nursing data evaluation and providing a reliable basis for intelligent evaluation.

[0093] In one embodiment, the intelligent assessment system for medical care data based on multimodal data fusion provided in this embodiment includes a data analysis module that uses an LSTM time-series prediction model to analyze multimodal data and predict the probability of adverse events occurring within a specific future time period. The system further includes:

[0094] When using the LSTM time series prediction model to analyze multimodal data and predict the probability of adverse events occurring within a specific future time period, the prediction time step is collected, and the prediction accuracy of the LSTM time series prediction model is calculated based on the prediction time step.

[0095] ;

[0096] in, This represents the prediction accuracy of the LSTM time series prediction model. Indicates the number of time steps in the prediction; This represents the time sequence number within the predicted time period; Indicates the first The predicted risk probability at time t, and the predicted risk probability takes the value (0, 1). Indicates the first The actual result at that moment; Indicates the first The time weighting factor at time, where, ;

[0097] Obtain a preset accuracy threshold and compare it with the prediction accuracy to determine whether the LSTM time series prediction model needs to be modified.

[0098] When the prediction accuracy is equal to or greater than the preset accuracy threshold, it is determined that no correction is needed to the LSTM time series prediction model.

[0099] Otherwise, it is determined that the LSTM time series prediction model needs to be modified.

[0100] When it is necessary to correct the LSTM time series prediction model, the correction shall be performed according to the following formula until the prediction accuracy is equal to or greater than the preset accuracy threshold, and the correction of the LSTM time series prediction model shall be completed.

[0101] ;

[0102] in, Indicates the corrected number The probability of predicted risk at any given moment; This represents the correction factor, and its value is (0, 1). , This indicates the preset accuracy threshold; The time step representing the historical error; Indicates the size of the historical error window; express The prediction error at any given time; Indicates the time step from the current time. The previous The actual result corresponding to each time step; Indicates the time step from the current time. The previous The predicted risk probability corresponding to each time step.

[0103] In this embodiment, Indicates the first The actual results at any given time include 0 and 1, where 0 indicates that no adverse event occurred and 1 indicates that an adverse event occurred.

[0104] In this embodiment, the time weighting factor, The purpose is to force a higher weighting for recent predictions. In clinical settings, the accuracy of predictions of recent adverse events (such as those within the next 24 hours) is more critical for intervention decisions and requires timely handling, while long-term predictions (such as those within the next 7 days) have a lower urgency. Therefore, by using a time weighting factor, the error in recent predictions has a greater impact on the overall accuracy, which is more in line with actual needs.

[0105] In this embodiment, The time step representing the historical error, for example, when Time: Represents the error of the previous time step (the most recent historical error); when Time: Represents the error from the time step two time steps ago (slightly older historical error); and so on. The larger the value, the further away (older) the historical error is from the current time.

[0106] In this embodiment, Indicates the time step from the current time. Between the first A time step, that is, a historical moment, for example: if It means today, but It refers to yesterday.

[0107] In this embodiment, the predicted time step is the length of a specific time period that is set in advance.

[0108] The working principle and beneficial effects of the above technical solution are as follows: by collecting the prediction time step, and calculating the prediction accuracy of the LSTM time series prediction model based on the prediction time step, the prediction accuracy can be effectively used to determine the LSTM time series prediction model. When correction is needed, the LSTM time series prediction model can be corrected in a timely and accurate manner, which helps to ensure the accurate operation of the LSTM time series prediction model and thus provides effective protection for the output results.

[0109] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "possessing," or any other variations thereof are intended to cover non-exclusive possession, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0110] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that variations, modifications, substitutions and alterations may be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A medical care data intelligent assessment system based on multimodal data fusion, characterized in that, include: The multimodal data acquisition unit is configured to acquire patients' multimodal data in real time, including: time-series physiological data, text data, image data, and environmental data. The multimodal data fusion unit is configured to extract multi-dimensional feature information from multimodal data based on the Transformer fusion model, and perform deep fusion through a progressive fusion framework constructed by timestamp alignment, anchor learning and mapping, and context association to generate joint features with context information. The intelligent evaluation unit includes: The data analysis module is configured to use an LSTM time-series prediction model to analyze multimodal data, and predict the probability of adverse events based on dynamic accuracy assessment and self-correction mechanisms based on time steps; it identifies specific symptoms through a multi-label classification model, and uses knowledge graphs and rule engines to analyze multimodal data to derive the comprehensive symptoms of current clinical patients; The multimodal data fusion unit performs deep fusion of multi-dimensional feature information to generate joint features with contextual information, including: The core feature key-value pair generation module is used to collect the time information corresponding to each modality data, add timestamps to each modality data according to the time information, perform time alignment of each modality data according to the timestamps, generate time-feature key-value pairs according to the alignment results, extract modality core features from the modality data according to the time-feature key-value pairs, and fuse the modality core features with the time-feature key-value pairs to generate core feature key-value pairs. The anchor point determination module is used to collect multimodal sample data and divide the multimodal sample data according to the core feature key-value pairs to obtain positive sample pairs and negative sample pairs. At the same time, a loss function is constructed based on the positive sample pairs, and the anchor points between each modality data are determined based on the loss function, positive sample pairs, and negative sample pairs. The cross-modal anchor alignment module is used to extract the initial position features of anchor points in each modal sample data, and to standardize the initial position features. Based on the standardization results, a preset calibration algorithm is used to construct a cross-modal mapping, and different anchor points are aligned in the same coordinate system according to the cross-modal mapping results. The multimodal association feature generation module is used to collect feature information of different dimensions in each modality based on the cross-modal anchor point alignment results, and perform modal adaptation processing on the collected feature information of different dimensions. At the same time, a context association network between features of different dimensions is constructed based on the anchor points, and the feature information of different dimensions after modal adaptation processing is input into the context association network for processing to generate multimodal association features. The joint feature generation module is used to input multimodal associated features into the fusion network for deep fusion and generate joint features with contextual information; The multimodal data acquisition unit includes: The data acquisition interval acquisition module is configured to acquire the data acquisition interval of multiple sensors deployed in the intelligent assessment system for medical and nursing data, and to set Class A time thresholds and Class B time thresholds. The multimodal data partitioning module is configured to partition multimodal data collected by multiple sensors based on a Level A time threshold and a Level B time threshold, resulting in a first dataset, a second dataset, and a third dataset. The first dataset consists of multimodal data collected by sensors whose acquisition time intervals are less than the Level A time threshold; the second dataset consists of multimodal data collected by sensors whose acquisition time intervals are equal to or greater than the Level A time threshold but less than the Level B time threshold; and the third dataset consists of multimodal data collected by sensors whose acquisition time intervals are equal to or greater than the Level B time threshold. The transmission interval setting module is configured to extract the acquisition time interval of the first dataset, the second dataset, and the third dataset to obtain first information, second information, and third information; set the corresponding transmission time interval based on the first information, the second information, and the third information, and transmit the first dataset, the second dataset, and the third dataset to the multimodal data fusion unit in sequence according to the specified transmission time interval; The multimodal data fusion unit includes: The abnormal data extraction module is configured to receive the first dataset, the second dataset, and the third dataset sequentially according to the transmission time interval, and to calculate the absolute Z-score of each data point in the first dataset, the second dataset, and the third dataset in sequence. If the absolute Z-score of any data point is greater than a predetermined threshold, the dataset containing the abnormal data is marked as an abnormal subset. The priority processing module is configured to extract anomalous data from the anomalous subsets in the first, second, and third datasets, perform deep fusion processing on the anomalous data based on the Transformer fusion model, and prioritize the transmission of the fused joint features and anomalous subsets to the intelligent evaluation unit for analysis. The anomaly analysis module is configured to set time-series data in multimodal data as high priority, obtain the data source category of the anomaly data, determine whether the anomaly data belongs to high priority, and if so, provide feedback to nursing staff based on the interactive interface; otherwise, no action is taken.

2. The intelligent assessment system for medical and nursing data based on multimodal data fusion according to claim 1, characterized in that, The data analysis module uses an LSTM time-series prediction model to analyze multimodal data and predict the probability of adverse events occurring within a specific future time period. It also includes: When using the LSTM time series prediction model to analyze multimodal data and predict the probability of adverse events occurring within a specific future time period, the prediction time step is collected, and the prediction accuracy of the LSTM time series prediction model is calculated based on the prediction time step. Obtain a preset accuracy threshold and compare it with the prediction accuracy to determine whether the LSTM time series prediction model needs to be modified. When the prediction accuracy is equal to or greater than the preset accuracy threshold, it is determined that no correction is needed to the LSTM time series prediction model. Otherwise, it is determined that the LSTM time series prediction model needs to be modified.

3. The intelligent assessment system for medical and nursing data based on multimodal data fusion according to claim 1, characterized in that, The intelligent evaluation unit includes: The nursing configuration module is configured to list the most urgent nursing problems and specific nursing needs based on symptom identification results and risk probabilities; formulate specific nursing measures and execution frequencies based on nursing needs; and provide real-time risk warning information based on the risk probability of adverse events output by the LSTM time series prediction model. The secondary confirmation module is configured to obtain feedback results from nursing staff on the intelligent assessment of multimodal data based on the interactive interface, and determine whether there is any bias in the model; The module is configured to periodically update and retrieve feedback results indicating deviations over a certain period, marking them as abnormal results; analyze the frequency of abnormal results during that period, and mark them as abnormal frequencies. The abnormal frequency is compared with a preset threshold to determine whether the abnormal frequency exceeds the threshold range. If it does, the frequency of model updates is adjusted.

4. The intelligent assessment system for medical and nursing data based on multimodal data fusion according to claim 3, characterized in that, The intelligent evaluation unit also includes: The model optimization module is configured to update the weights of the multi-label classification model in real time using the online gradient descent method, update the weights of the LSTM time series prediction model in real time using the incremental learning method of LSTM, and adjust the relationships and rules in the knowledge graph in real time using the incremental update method of knowledge graph, based on the results of the feedback analysis, and then feed back the adjusted model results to the nursing staff in real time.

5. The intelligent assessment system for medical and nursing data based on multimodal data fusion according to claim 3, characterized in that, The nursing configuration module lists the most urgent nursing problems and specific nursing needs based on symptom identification results and risk probabilities, including: The severity and risk probability of symptoms are divided into three levels: high, medium, and low. Each nursing problem is comprehensively assessed and prioritized. Based on the priority assessment results, a list of nursing issues sorted by priority is generated, and the highest priority nursing issues are extracted from it; Based on the new symptom identification and risk prediction results, the list of the most urgent nursing problems is updated in real time and displayed to nursing staff through an interactive interface so that timely measures can be taken.

6. The intelligent assessment system for medical and nursing data based on multimodal data fusion according to claim 3, characterized in that, The nursing configuration module sets the execution frequency of nursing measures based on the patient's condition and nursing needs, including: Obtain the severity of the patient's symptoms, the probability of risk, and the corresponding priority, as well as the highest priority nursing problem; Set monitoring and intervention frequencies according to priorities, and formulate corresponding nursing measures; Generate a nursing execution form and provide feedback to nursing staff via an interactive interface.

Images