Intelligent early warning method and system for emergency patients based on multi-modal data fusion

Abstract

This application relates to the interdisciplinary field of emergency medicine and artificial intelligence, and particularly to an intelligent early warning method and system for emergency patients based on multimodal data fusion. The system collects multimodal data information from patients in real time; uses a pre-set edge computing device to extract and fuse features from the multimodal data information to obtain a comprehensive feature vector; uses the comprehensive feature vector to generate a real-time risk level based on a pre-set dynamic risk assessment algorithm; and automatically triggers the pre-preparation of resuscitation equipment based on the real-time risk level, and pushes early warning information to medical staff. The technical solution in this application overcomes the bottlenecks of traditional emergency early warning systems in data integration, response speed, model adaptability, and equipment coordination, constructing an intelligent early warning system with self-learning capabilities. This provides a new technical path for the early identification and efficient treatment of critically ill patients, significantly improving the safety and efficiency of emergency medical services.

Description

Technical Field

[0001] This application relates to the interdisciplinary field of emergency medicine and artificial intelligence, and in particular to an intelligent early warning method and system for emergency patients based on multimodal data fusion. Background Technology

[0002] As the frontline of hospital care for critically ill patients, the emergency department faces challenges such as high patient volume, complex conditions, and tight treatment timelines. Traditional emergency triage relies heavily on manual assessment by medical staff, such as the Modified Early Warning Score (MEWS) system. This system calculates risk levels by manually inputting single vital signs like heart rate and blood pressure, which suffers from high subjectivity, low efficiency, and poor real-time performance, easily leading to missed diagnoses or delayed treatment of critically ill patients. While existing technologies, such as IoT-based bedside monitoring devices, can automatically collect physiological parameters and upload them to the cloud for analysis, they only focus on single-modal data (e.g., blood oxygen, heart rate) and cannot integrate multi-dimensional information such as medical images (e.g., CT, MRI) and textual medical records (e.g., medical history, medication records). Furthermore, cloud processing suffers from high data transmission latency (average response time > 20 seconds) and static models (fixed and unadjustable thresholds). In addition, traditional systems lack equipment linkage mechanisms, preventing the pre-preparation of resuscitation equipment (e.g., ventilators, defibrillators) based on risk levels, resulting in inefficient resource allocation and further impacting treatment success rates.

[0003] To address the aforementioned issues, existing technologies urgently require an intelligent early warning system capable of real-time fusion of multimodal data, dynamic assessment of risk levels, and linkage with rescue equipment, in order to resolve the core pain points of single data dimensions, delayed early warning, and insufficient resource coordination. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide an intelligent early warning method and system for emergency patients based on multimodal data fusion, so as to solve the related problems existing in the prior art.

[0005] According to a first aspect of the present invention, an intelligent early warning method for emergency patients based on multimodal data fusion is provided, characterized in that the method includes:

[0006] Real-time acquisition of multimodal data information of patients; the multimodal data information includes: patient physiological parameters, medical images and text medical record data information;

[0007] The multimodal data information is subjected to feature extraction and fusion analysis using a pre-set edge computing device to obtain a comprehensive feature vector;

[0008] Using the comprehensive feature vector, a real-time risk level is generated based on a preset dynamic risk assessment algorithm;

[0009] The system automatically triggers the pre-preparation of rescue equipment based on the real-time risk level and sends early warning information to medical staff.

[0010] Furthermore, the step of using a preset edge computing device to perform feature extraction and fusion analysis on the multimodal data information to obtain a comprehensive feature vector includes:

[0011] The temporal features of patient physiological parameters in the multimodal data are extracted using a pre-defined edge computing device and a pre-defined LSTM network.

[0012] Convolutional feature information of medical images in the multimodal data is extracted using a pre-set edge computing device and a pre-trained ResNet50 model.

[0013] The text medical records in the multimodal data information are combined with preset BERT word vectors and structured data, and 256-dimensional fused feature data information is generated through a preset attention mechanism.

[0014] By utilizing the temporal features of patient physiological parameters, the convolutional features of medical images, and the 256-dimensional fusion features in the multimodal data, a comprehensive feature vector is obtained through dynamic weighted fusion.

[0015] Furthermore, it also includes:

[0016] When extracting temporal feature information of patient physiological parameters using a preset edge computing device and a preset LSTM network, the input physiological parameters are normalized to eliminate the dimensional differences between different parameters, and the input weights of the LSTM network are dynamically adjusted according to the preset values ​​of different physiological parameters.

[0017] Furthermore, it also includes:

[0018] When extracting convolutional feature information from medical images using a pre-trained ResNet50 model, the medical images are enhanced, including but not limited to random rotation, flipping, and scaling operations, to improve the model's generalization ability.

[0019] Furthermore, it also includes:

[0020] When generating 256-dimensional fused feature data information through a preset attention mechanism, the parameters of the attention mechanism are dynamically adjusted according to the length and content complexity of the text medical record.

[0021] Furthermore, the step of generating a real-time risk level based on a preset dynamic risk assessment algorithm using the comprehensive feature vector includes:

[0022] The comprehensive feature vector is input into a preset fully connected neural network structure, and the probability value of each risk level is output.

[0023] Further, the step of inputting the comprehensive feature vector into a preset fully connected neural network structure and outputting the probability value of each risk level includes:

[0024] The integrated feature vector is received using a preset input layer, and nonlinear feature transformation is performed through a preset number of decreasing fully connected layers. A preset ReLU activation function and a preset batch normalization algorithm are used to prevent overfitting.

[0025] The preset output layer, combined with the Softmax activation function, outputs the probability values ​​for each risk level.

[0026] Furthermore, the automatic triggering of pre-preparation of rescue equipment based on real-time risk levels and the push of early warning information to medical personnel include:

[0027] If the risk level is greater than or equal to the first threshold and the blood oxygen saturation is less than the second threshold, an instruction is automatically sent to the ventilator, the instruction including: an oxygen concentration adjustment instruction;

[0028] If the risk level is greater than the third threshold, the location of the nearest defibrillator will be obtained through a high-precision positioning system based on ultra-wideband wireless technology, and a navigation path will be pushed to the preset terminal.

[0029] Furthermore, it also includes:

[0030] If the risk level is greater than or equal to the first threshold, the CT or MRI examination will be automatically scheduled in the queue to shorten the waiting time.

[0031] According to a second aspect of the present invention, a method for intelligent early warning of emergency patients based on multimodal data fusion is provided, applied to any one of the above-described methods for intelligent early warning of emergency patients based on multimodal data fusion, characterized in that the system comprises:

[0032] The acquisition module is used to collect multimodal data information of patients in real time; the multimodal data information includes: patient physiological parameters, medical images and text medical record data information;

[0033] The first processing module is used to extract and fuse features from the multimodal data information using a preset edge computing device to obtain a comprehensive feature vector;

[0034] The second processing module is used to generate a real-time risk level based on a preset dynamic risk assessment algorithm using the comprehensive feature vector.

[0035] The third processing module is used to automatically trigger the pre-preparation of rescue equipment based on the real-time risk level and push early warning information to medical staff.

[0036] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects:

[0037] 1. A precise evaluation mechanism for multimodal data fusion

[0038] This approach creatively integrates three heterogeneous data types: physiological parameters, medical images, and text medical records, overcoming the limitations of traditional single-vital sign assessments. By capturing dynamic changes in physiological parameters using LSTM, analyzing image structural features using ResNet50, and mining semantics from medical record text using BERT, it forms a multi-dimensional feature complementarity, fundamentally improving the comprehensiveness and accuracy of risk assessment. This effectively reduces misjudgments caused by insufficient data dimensions and provides richer information support for clinical decision-making.

[0039] 2. Real-time response capability driven by edge computing

[0040] For the first time, complex models such as LSTM, ResNet50, and BERT are deployed locally on edge computing devices. Combined with model quantization and parallel optimization techniques, this enables end-to-end localized processing from data acquisition to risk output. Compared to traditional cloud architectures, this completely solves the problem of data transmission latency, ensuring the real-time nature of risk assessment results in emergency scenarios, securing a precious window of opportunity for critically ill patients, and meeting the clinical need for rapid response within the "golden five minutes."

[0041] 3. Intelligent learning system with dynamic threshold calibration

[0042] We designed a reinforcement learning mechanism based on historical misdiagnosis data to overcome the inherent defects of static thresholds in traditional early warning models. By continuously iterating and optimizing the risk level classification strategy, the model can adapt to the characteristic differences of different patient groups (such as elderly patients and patients with chronic diseases), significantly improving the universality and accuracy of the early warning system. This avoids the waste of resources caused by "over-warning" or the delay in treatment caused by "warning delay," and realizes the dynamic and personalized nature of risk assessment standards.

[0043] 4. Risk-level driven device collaboration network

[0044] An integrated equipment linkage system of "assessment-decision-execution" is constructed, which automatically triggers the pre-preparation of rescue equipment and resource allocation based on risk level. It innovatively realizes functions such as ventilator pre-oxygenation, intelligent defibrillator positioning, and priority allocation of examination resources, transforming passive response into proactive intervention, significantly shortening equipment preparation and resource allocation time, optimizing the work efficiency of the emergency department, forming an intelligent rescue chain of "early warning equals action", and improving the overall efficiency of medical resource utilization.

[0045] The organic combination of the above technical solutions breaks through the bottlenecks of traditional emergency early warning systems in terms of data integration, response speed, model adaptability, and equipment collaboration. It constructs an intelligent early warning system with self-learning capabilities, providing a new technical path for the early identification and efficient treatment of critically ill patients, and significantly improving the safety and efficiency of emergency medical services.

[0046] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0047] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0048] Figure 1 This is a schematic diagram of a multimodal data fusion-based intelligent early warning method for emergency patients, according to an exemplary embodiment.

[0049] Figure 2 This is a schematic diagram illustrating the components of an intelligent early warning system for emergency patients based on multimodal data fusion, according to an exemplary embodiment. Detailed Implementation

[0050] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0051] Example 1

[0052] Please see Figure 1 , Figure 1 This is a schematic flowchart of an intelligent early warning method for emergency patients based on multimodal data fusion, according to an exemplary embodiment. The method includes:

[0053] S1. Real-time acquisition of multimodal data information of patients; the multimodal data information includes: patient physiological parameters, medical images and text medical record data information;

[0054] S2. Using a preset edge computing device, feature extraction and fusion analysis are performed on the multimodal data information to obtain a comprehensive feature vector;

[0055] S3. Using the comprehensive feature vector, generate a real-time risk level based on a preset dynamic risk assessment algorithm;

[0056] S4. Automatically trigger the pre-preparation of rescue equipment based on the real-time risk level and push early warning information to medical staff.

[0057] In specific implementation, as described in step S1, the patient's multimodal data information is collected in real time, including:

[0058] Physiological parameter acquisition: The wearable smart bracelet (sampling frequency 50Hz) with integrated MAX30102 blood oxygen module collects physiological parameters such as blood oxygen saturation (SpO), heart rate (HR), and body temperature (Temp) in real time. The data is transmitted to the bedside edge computing node (NVIDIA Jetson AGX Orin) via BLE5.0 protocol with a transmission latency of ≤200ms.

[0059] It should be noted that the edge computing node (NVIDIA Jetson AGX Orin) is a high-performance edge computing device built on the NVIDIA Jetson AGX Orin platform, designed specifically for real-time data processing and running complex AI tasks at the "edge" (such as terminal devices or field environments) close to the data source.

[0060] NVIDIA Jetson AGX Orin: This is NVIDIA's third-generation edge AI computing platform, the flagship product in the Jetson series. It integrates powerful computing capabilities, low-power design, and rich I / O interfaces, and is designed to support complex AI tasks at the edge (such as image processing, video analysis, robot control, and multimodal data fusion).

[0061] Medical image acquisition: Through the DICOM3.0 interface, it connects with the hospital's CT / MRI equipment in real time to automatically acquire chest CT images (DICOM format) and uses the SimpleITK library for preprocessing (noise reduction, window width and window level adjustment to 500HU / 1500HU) to generate a standardized image of 512×512 pixels.

[0062] Synchronization of medical records: The hospital's HIS system extracts structured patient data (age, gender, allergy history, medication records) and unstructured text (chief complaint, present illness history) through its RESTful API. The text is then processed using the BERT segmenter, truncated to a preset length, and missing values ​​are handled using padding.

[0063] It should be noted that the multimodal data in the embodiments of this application mainly includes:

[0064] 1. Patient physiological parameters

[0065] This is quantitative data reflecting the patient's physical condition, typically collected in real time using various medical devices. For example:

[0066] Vital signs include: heart rate, which reflects how fast the heart beats; a normal adult's resting heart rate is generally 60-100 beats per minute; blood pressure, which reflects the pressure exerted by blood on the walls of blood vessels as it flows through them, including systolic and diastolic pressure; respiratory rate, which refers to the number of breaths per minute; a normal adult's resting respiratory rate is 12-20 breaths per minute; and body temperature, which reflects the balance between heat production and heat dissipation in the body.

[0067] Blood indicators include: blood oxygen saturation, which indicates the oxygen content in the blood; generally, the blood oxygen saturation of human arteries should not be lower than 95%; blood glucose level reflects the concentration of glucose in the blood and is of great significance in diagnosing diseases such as diabetes; and various indicators in a complete blood count, such as white blood cell count, red blood cell count, and platelet count, can be used to assess whether the body has infections, anemia, or other conditions.

[0068] 2. Medical Imaging

[0069] Images obtained using various medical imaging techniques can visually display the internal structures and pathological conditions of the human body. Common medical images include:

[0070] X-ray imaging: commonly used for the diagnosis of diseases of the skeletal system, such as clearly showing the location, type and displacement of fractures; it can also be used for chest examinations to check for lesions such as inflammation and tumors in the lungs.

[0071] CT (Computed Tomography) images: have high density resolution and can provide cross-sectional images of a part of the human body. They are more accurate in diagnosing diseases in the brain, chest, abdomen, and other parts of the body, and can detect some tiny lesions that are difficult to detect with X-rays.

[0072] MRI (Magnetic Resonance Imaging) images: have a strong ability to resolve soft tissues and are often used to diagnose diseases of the nervous system, musculoskeletal system, etc., such as detecting brain tumors and spinal cord injuries.

[0073] Ultrasound imaging: Images are formed by the reflection of ultrasound waves. It is commonly used in obstetrics and gynecology to examine the development of the fetus, and can also be used to examine the shape and structure of abdominal organs such as the liver, gallbladder, and pancreas.

[0074] 3. Text medical record data

[0075] Patient medical information, recorded in written form, includes various descriptions and diagnostic results from the patient's initial consultation to treatment. The main contents include:

[0076] Patient basic information: such as name, age, gender, occupation, etc. This information helps doctors understand the patient's basic situation. Some occupations may be associated with the occurrence of specific diseases.

[0077] Chief complaint: The patient's most prominent symptoms or discomfort when seeking medical attention, such as "cough, sputum production, and fever for 3 days." This is an important clue for doctors to understand the patient's condition.

[0078] Present Illness: Describe in detail the onset and development of the patient's current illness, including the onset time of symptoms, changes in symptoms, treatment process, etc., to provide important basis for diagnosis.

[0079] Past medical history: Record the patient's past health status, whether they have had other diseases, such as a history of chronic diseases such as hypertension and diabetes. This is of great reference value for determining the cause of the current disease and formulating a treatment plan.

[0080] Family history: To understand whether there are similar diseases in the patient's family, as some hereditary diseases may cluster in families.

[0081] Diagnosis: The diagnosis given by the doctor based on various examinations and assessments is an important basis for subsequent treatment.

[0082] In one embodiment, as described in step S2, a preset edge computing device is used to perform feature extraction and fusion analysis on the multimodal data information to obtain a comprehensive feature vector, including:

[0083] The temporal features of patient physiological parameters in the multimodal data are extracted using a pre-defined edge computing device and a pre-defined LSTM network.

[0084] Convolutional feature information of medical images in the multimodal data is extracted using a pre-set edge computing device and a pre-trained ResNet50 model.

[0085] The text medical records in the multimodal data information are combined with preset BERT word vectors and structured data, and 256-dimensional fused feature data information is generated through a preset attention mechanism.

[0086] The multimodal data information utilizes the temporal features of patient physiological parameters, the convolutional features of medical images, and 256-dimensional fusion features, which are then weighted and integrated through dynamic weight fusion to obtain a comprehensive feature vector. In specific implementation, multimodal data is processed in parallel using edge computing devices: First, physiological parameters (blood oxygen, heart rate, etc.) are framed through a sliding window and input into a bidirectional LSTM network to extract 128-dimensional temporal features; medical images are processed using a ResNet50 model to extract 2048-dimensional convolutional features; and textual medical records, combined with BERT word vectors and structured data, generate 256-dimensional fusion features through an attention mechanism. These three types of features are then weighted and integrated through dynamic weight fusion, ultimately outputting a 256-dimensional comprehensive feature vector. Edge computing nodes employ TensorRT to accelerate inference. Through model quantization and pipelined parallel processing, the single-sample processing latency is controlled within 45ms, achieving a 4.45-fold improvement in response speed compared to traditional cloud processing, while simultaneously reducing power consumption by 40%, ensuring a balance between real-time performance and energy efficiency.

[0087] In practical implementation, dynamic weighted fusion is to weight and integrate different features extracted from multimodal data (temporal features of patient physiological parameters, convolutional features of medical images, and fusion features of text medical records) to obtain a comprehensive feature vector. In practical applications, the following formula can be used for calculation.

[0088] In practical implementation, the temporal feature vector extracted from the patient's physiological parameters is X1 = (x 11 ,x 12 ,...,x 1n The dimension is n; the convolutional feature vector extracted from medical images is X2 = (x 21 ,x 22 ,...,x 2m The dimension is m; the 256-dimensional fusion feature vector generated from the text medical record is X3 = (x 31 ,x 32 ,...,x 3p The dimension is p (here p = 256). The comprehensive feature vector Y = (y1, y2, ..., yq), where q is the dimension of the comprehensive feature vector, which is usually the same as the dimension of a key feature vector involved in the fusion. Here, it is assumed to be consistent with the dimension of the feature vector fused from the text medical records, i.e., q = 256.

[0089] Each feature vector is assigned a dynamic weight, w1, w2, w3, satisfying w1 + w2 + w3 = 1, w1 ≥ 0, w2 ≥ 0, and w3 ≥ 0. The dynamic weights can be adjusted in real time based on factors such as the importance of the features and the reliability of the data.

[0090] The formula for calculating the comprehensive feature vector Y is:

[0091] yi =w1x 1i +w2x 2i +w3x 3i (i = 1, 2, ..., 256);

[0092] This formula enables the weighted integration of different features from multimodal data, generating a comprehensive feature vector for subsequent risk assessment. In practice, the determination of dynamic weights may be based on more complex algorithms, such as automatically learning and adjusting weights using machine learning algorithms based on the performance of different features in different cases, to achieve a more accurate fusion effect.

[0093] It should be noted that when extracting temporal feature information of patient physiological parameters using a preset edge computing device and a preset LSTM network, the input physiological parameters are normalized to eliminate the dimensional differences between different parameters, and the input weights of the LSTM network are dynamically adjusted according to the importance of different physiological parameters. When extracting convolutional feature information of medical images using a pre-trained ResNet50 model, the medical images are enhanced, including but not limited to random rotation, flipping, scaling, etc., to improve the model's generalization ability. When generating 256-dimensional fused feature data information through a preset attention mechanism, the parameters of the attention mechanism are dynamically adjusted according to the length and content complexity of the text medical record to more accurately focus on key information.

[0094] In specific implementation, as described in step S3, the real-time risk level is generated using the comprehensive feature vector based on a preset dynamic risk assessment algorithm, including:

[0095] First, a comprehensive feature vector, obtained through feature extraction and fusion of multimodal data (physiological parameters, medical images, and text medical records), is used as input. This model typically employs a fully connected neural network structure. The input layer receives the comprehensive feature vector, the hidden layer uses the ReLU activation function for nonlinear transformation, and the output layer uses the Softmax function to convert the output into a probability distribution corresponding to the five risk levels. After inputting the comprehensive feature vector, the model performs forward propagation calculations and finally outputs the probability value for each risk level.

[0096] Next, based on the output probability distribution, the risk level with the highest probability is determined as the patient's real-time risk level. For example, if the output probability distribution is [0.1, 0.2, 0.3, 0.25, 0.15], then the level 3 represented by the highest probability of 0.3 is the current risk level.

[0097] In addition, to ensure the accuracy of risk assessment, the system collects historical misdiagnosis data monthly, optimizes model parameters using reinforcement learning algorithms, and dynamically adjusts risk level thresholds. Each adjustment step is ±0.5%, with an interval of one training cycle (e.g., 1000 iterations). Through this continuous calibration, the model can more accurately reflect the actual risk of patients, reducing the probability of false positives and false negatives.

[0098] Specifically, the system automatically triggers the pre-preparation of rescue equipment based on the real-time risk level and sends early warning information to medical staff, including:

[0099] If the risk level is greater than or equal to the first threshold and the blood oxygen saturation is less than the second threshold, an instruction is automatically sent to the ventilator, the instruction including: an oxygen concentration adjustment instruction;

[0100] If the risk level is greater than the third threshold, the location of the nearest defibrillator will be obtained through a high-precision positioning system based on ultra-wideband wireless technology, and a navigation path will be pushed to the preset terminal.

[0101] Furthermore, it also includes:

[0102] If the risk level is greater than or equal to the first threshold, the patient will be automatically scheduled for a CT or MRI examination, shortening the waiting time. When the risk level is ≥3 (first threshold) and SpO2 < 90% (second threshold), the system will send a command to the preset ventilator via the MODBUS RTU protocol, automatically adjust the oxygen concentration to 100% and pre-oxygenate for 3 minutes, and mark the patient as a red priority on the nurse station screen.

[0103] In practice, the system sends commands to the preset ventilator via the MODBUS RTU protocol. This protocol is widely used in medical device communication, and its data transmission format, verification mechanism, and other characteristics meet the basic requirements for medical data transmission. It ensures the accuracy and stability of commands during transmission, providing a guarantee for the precise control of the ventilator.

[0104] It should be noted that during the system development and testing phase, interoperability tests were conducted on the MODBUS RTU protocol with various mainstream ventilator devices. The tests covered ventilators of different brands and models, simulating various clinical application scenarios, such as the transmission of oxygen concentration adjustment commands at different risk levels and continuous, long-term command interactions. After rigorous testing, the protocol demonstrated excellent performance in terms of data transmission accuracy, response time, and device compatibility, fully meeting the requirements of the emergency patient intelligent early warning system for ventilator control.

[0105] When the risk level is ≥4 (third threshold), the UWB positioning system is activated (positioning accuracy ±30cm) to obtain the location of the defibrillator closest to the patient, and the optimal navigation path is generated by the algorithm (avoiding congested channels) and pushed to the nurse's PDA (response time ≤2 seconds).

[0106] When the risk level is ≥3 (first threshold), the system automatically sends a queue-jumping request to the radiology management system, reducing the CT / MRI examination queue time from an average of 31 minutes to less than 22 minutes, and prioritizing the allocation of examination resources to high-risk patients.

[0107] It should be noted that after the system automatically triggers the pre-preparation of the resuscitation equipment and sends an early warning message to medical staff, it will monitor the preparation status of the resuscitation equipment and the response of medical staff in real time. If the resuscitation equipment fails to complete pre-preparation or the medical staff fails to respond to the early warning message within the preset time, the system will automatically escalate the early warning level, send an emergency warning to higher-level medical staff or management personnel, and simultaneously increase the frequency and intensity of the early warning messages to ensure that patients can receive timely treatment.

[0108] This invention deeply integrates multimodal data fusion, edge computing, dynamic model optimization, and device collaboration technologies to construct a closed-loop system of "data acquisition - real-time analysis - intelligent decision-making - device linkage," fundamentally solving the core defects of traditional emergency early warning systems and providing an efficient and reliable technical solution for improving the success rate of critical care.

[0109] Furthermore, in intelligent early warning systems for emergency patients, the protection of patient data privacy is of paramount importance. Specifically, measures to protect patient data privacy should be added to the documentation across four key stages: data collection, storage, transmission, and use, in order to enhance the system's objectivity and usability.

[0110] In practical implementation, during the data collection phase: anonymization techniques are employed when collecting multimodal patient data. Wearable smart bracelets, when collecting physiological parameters, only record the device ID, rather than directly linking it to patient identity information. The mapping between the device ID and the patient's real identity is stored in a separate, secure database accessible only to authorized personnel. When obtaining text medical record data through the hospital's HIS system, sensitive information such as patient names and ID numbers is encrypted, and the encrypted data is used for model training and analysis.

[0111] Data storage: A secure data storage environment is established, using encryption technologies that comply with medical industry standards, such as the AES encryption algorithm, to encrypt patient physiological parameters, medical images, and text medical records stored in the database. A strict access control mechanism is implemented, assigning different access levels based on the responsibilities and work needs of medical staff. For example, ordinary nurses can only access basic patient physiological parameters and some medical record information, while attending physicians can access the complete data after authorization. Simultaneously, storage devices are regularly subjected to security testing and vulnerability patching to prevent data leakage risks.

[0112] Data transmission: During data transmission, secure communication protocols, such as HTTPS, are used to encrypt the transmitted patient data, ensuring its security during network transmission. For physiological parameter data transmitted via the BLE 5.0 protocol, the device's encryption transmission function is enabled to prevent data theft or tampering. When data is transmitted to edge computing devices and other systems, authentication is performed; only authorized devices and systems can receive the data.

[0113] Data Usage: When using patient data for model training and risk assessment, adhere to the principle of minimal use, using only data relevant to the early warning task and avoiding excessive use of patient privacy data. Conduct rigorous background checks and privacy training for personnel involved in model training and development, sign confidentiality agreements, and clearly define the boundaries and responsibilities for data use. Establish a data usage audit mechanism to record all access to and use of patient data, enabling traceability and accountability in case of problems.

[0114] In practical implementation, this model is applicable to different populations. It integrates three heterogeneous data types: patient physiological parameters, medical images, and text medical records, providing information from multiple dimensions. For different populations, whether elderly patients, patients with chronic diseases, or other special groups, this multimodal data fusion approach can more comprehensively reflect the patient's health status. Taking elderly patients as an example, they often have multiple underlying diseases, and changes in their physiological parameters may be atypical, making it easy to miss diagnoses when relying solely on a single vital sign. This model, by comprehensively analyzing medical images and text medical records—such as identifying occult lesions from images and combining this with medical history and medication records—can more accurately assess risk.

[0115] Furthermore, a reinforcement learning mechanism based on historical misdiagnosis data is designed to enable the model to dynamically adjust its risk level classification strategy according to the characteristic differences of different patient groups. Different populations have different responses and tolerances to diseases. For example, patients with chronic diseases have long-term illnesses, and their physical indicators may differ significantly from those of healthy individuals. Through continuous iterative optimization, the model can adapt to these differences, avoiding "over-warning" or "delayed warning" for patients with chronic diseases, thus improving the universality and accuracy of the warning system across different populations.

[0116] Please see Figure 2 , Figure 2 This is a schematic diagram illustrating the components of an intelligent early warning system for emergency patients based on multimodal data fusion, according to an exemplary embodiment. The system includes:

[0117] The acquisition module 20 is used to collect multimodal data information of patients in real time; the multimodal data information includes: patient physiological parameters, medical images and text medical record data information.

[0118] In practice, this system is responsible for collecting multimodal patient data in real time and comprehensively. This is achieved through various sensors and medical information systems: using wearable medical devices (such as smart bracelets and dynamic electrocardiogram monitors) and hospital vital sign monitoring equipment (such as monitors) to collect real-time physiological parameters, including heart rate, blood pressure, blood oxygen saturation, respiratory rate, and body temperature; connecting to the hospital's Picture Archiving and Communication System (PACS) to obtain the patient's medical images, including X-rays, CT scans, and MRIs; and accessing the Hospital Information System (HIS) to extract the patient's textual medical record data, such as past medical history, diagnostic records, medication information, and laboratory reports.

[0119] The first processing module 21 is used to extract and fuse features from the multimodal data information using a preset edge computing device to obtain a comprehensive feature vector.

[0120] Specifically, relying on pre-set edge computing devices, feature extraction and fusion analysis are performed on the multimodal data collected by the acquisition module. For patient physiological parameters, signal processing algorithms are used to extract dynamic change features, such as heart rate variability and blood pressure fluctuation trends. For medical images, image recognition technology is used to identify lesion features and tissue morphology in the images. For textual medical record data, natural language processing technology is used to extract key semantic features such as disease diagnosis, symptom description, and treatment measures. Then, the features extracted from different modalities are fused according to a specific fusion strategy to obtain a comprehensive feature vector, which comprehensively represents the patient's condition. The second processing module 22 is used to generate a real-time risk level based on a pre-set dynamic risk assessment algorithm using the comprehensive feature vector.

[0121] Specifically, based on the comprehensive feature vector generated by the first processing module, a preset dynamic risk assessment algorithm (such as a risk assessment model built based on machine learning and deep learning, considering the dynamic changes in disease development) is used. By calculating and analyzing the comprehensive feature vector, combined with disease knowledge base and clinical experience data, the severity of the patient's condition is assessed in real time, generating a corresponding real-time risk level, such as low risk, medium risk, high risk, and extremely high risk. The third processing module 23 is used to automatically trigger the pre-preparation of rescue equipment based on the real-time risk level and push early warning information to medical staff.

[0122] Specifically, actions are taken based on the real-time risk level generated by the second processing module. When the risk level reaches the trigger threshold, the system automatically interacts with the hospital's medical equipment management system to pre-prepare resuscitation equipment (such as ventilators, defibrillators, and emergency medication reserves) to ensure they are readily available. Simultaneously, warning information is pushed to relevant medical staff through the hospital's internal communication system (such as SMS platforms, medical workstation terminals, and mobile nursing apps). This information includes basic patient information, real-time risk level, and a brief summary of the patient's condition, enabling medical staff to promptly understand changes in the patient's condition and take appropriate treatment measures.

[0123] This invention deeply integrates multimodal data fusion, edge computing, dynamic model optimization, and device collaboration technologies to construct a closed-loop system of "data acquisition - real-time analysis - intelligent decision-making - device linkage," fundamentally solving the core defects of traditional emergency early warning systems and providing an efficient and reliable technical solution for improving the success rate of critical care.

[0124] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0125] It should be noted that in the description of this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this invention, unless otherwise stated, "a plurality of" means at least two.

[0126] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0127] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0128] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0129] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0130] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0131] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0132] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. An intelligent early warning method for emergency patients based on multimodal data fusion, characterized in that... The method includes: Real-time acquisition of multimodal data information of patients; the multimodal data information includes: patient physiological parameters, medical images and text medical record data information; The multimodal data information is subjected to feature extraction and fusion analysis using a pre-set edge computing device to obtain a comprehensive feature vector; Using the comprehensive feature vector, a real-time risk level is generated based on a preset dynamic risk assessment algorithm; The system automatically triggers the pre-preparation of rescue equipment based on the real-time risk level and sends early warning information to medical staff.

2. The method according to claim 1, characterized in that, The step of using a preset edge computing device to extract and fuse features from the multimodal data information to obtain a comprehensive feature vector includes: The temporal features of patient physiological parameters in the multimodal data are extracted using a pre-defined edge computing device and a pre-defined LSTM network. Convolutional feature information of medical images in the multimodal data is extracted using a pre-set edge computing device and a pre-trained ResNet50 model. The text medical records in the multimodal data information are combined with preset BERT word vectors and structured data, and 256-dimensional fused feature data information is generated through a preset attention mechanism. By utilizing the temporal features of patient physiological parameters, the convolutional features of medical images, and the 256-dimensional fusion features in the multimodal data, a comprehensive feature vector is obtained through dynamic weighted fusion.

3. The method according to claim 1, characterized in that, Also includes: When extracting temporal feature information of patient physiological parameters using a preset edge computing device and a preset LSTM network, the input physiological parameters are normalized to eliminate the dimensional differences between different parameters, and the input weights of the LSTM network are dynamically adjusted according to the preset values ​​of different physiological parameters.

4. The method according to claim 2, characterized in that, Also includes: When extracting convolutional feature information from medical images using a pre-trained ResNet50 model, the medical images are enhanced, including but not limited to random rotation, flipping, and scaling operations, to improve the model's generalization ability.

5. The method according to claim 2, characterized in that, Also includes: When generating 256-dimensional fused feature data information through a preset attention mechanism, the parameters of the attention mechanism are dynamically adjusted according to the length and content complexity of the text medical record.

6. The method according to claim 1, characterized in that, The step of generating a real-time risk level based on the comprehensive feature vector and a preset dynamic risk assessment algorithm includes: The comprehensive feature vector is input into a preset fully connected neural network structure, and the probability value of each risk level is output.

7. The method according to claim 6, characterized in that, The step of inputting the comprehensive feature vector into a preset fully connected neural network structure and outputting the probability values ​​of each risk level includes: The integrated feature vector is received using a preset input layer, and nonlinear feature transformation is performed through a preset number of decreasing fully connected layers. A preset ReLU activation function and a preset batch normalization algorithm are used to prevent overfitting. The preset output layer, combined with the Softmax activation function, outputs the probability values ​​for each risk level.

8. The method according to claim 1, characterized in that, The automatic triggering of emergency equipment preparation based on real-time risk levels and the sending of early warning information to medical staff include: If the risk level is greater than or equal to the first threshold and the blood oxygen saturation is less than the second threshold, an instruction is automatically sent to the ventilator, the instruction including: an oxygen concentration adjustment instruction; If the risk level is greater than the third threshold, the location of the nearest defibrillator will be obtained through a high-precision positioning system based on ultra-wideband wireless technology, and a navigation path will be pushed to the preset terminal.

9. The method according to claim 8, characterized in that, Also includes: If the risk level is greater than or equal to the first threshold, the CT or MRI examination will be automatically scheduled in the queue to shorten the waiting time.

10. An intelligent early warning system for emergency patients based on multimodal data fusion, applied to the intelligent early warning method for emergency patients based on multimodal data fusion as described in any one of claims 1-9, characterized in that... The system includes: The acquisition module is used to collect multimodal data information of patients in real time; the multimodal data information includes: patient physiological parameters, medical images and text medical record data information; The first processing module is used to extract and fuse features from the multimodal data information using a preset edge computing device to obtain a comprehensive feature vector; The second processing module is used to generate a real-time risk level based on a preset dynamic risk assessment algorithm using the comprehensive feature vector. The third processing module is used to automatically trigger the pre-preparation of rescue equipment based on the real-time risk level and push early warning information to medical staff.

Images