An intelligent emergency decision-making system based on multimodal fusion

Abstract

The present invention seeks to protect an intelligent first aid decision-making system based on multimodal fusion, which includes a main control unit, the main control unit is connected to a display unit, a connection unit, an interaction unit and a cross-linking unit; when the main control unit determines whether the patient has health risks through the patient's life monitoring data, if it is determined that there are health risks, the main control unit predicts possible symptoms based on the patient's life monitoring data, and generates predicted first aid content according to all predicted symptoms; when the doctor determines that the patient has symptoms and needs first aid, the predicted first aid content is called by the main control unit for the doctor's reference. This application can save the time of the main control module to generate content when an emergency situation occurs by generating predicted first aid content in advance. In addition, the main control module records the first aid content, which plays the role of legal risk prevention and control. The timeline is highly accurate, and the digital signature and CA certification meet the requirements of judicial appraisal.

Description

Technical Field

[0001] The present invention relates to the field of medical technology, and more specifically, to an intelligent first aid decision-making system based on multimodal fusion. Background Art

[0002] Emergency care often involves dealing with critically ill patients. Given the limited time, urgency, and heavy workload, emergency care must be conducted in an orderly manner according to its specific characteristics and work standards to ensure timely and accurate diagnosis and treatment, ultimately saving patients' lives. The rescue process requires rapid development of emergency plans, recording of key time points, and multidisciplinary collaboration.

[0003] During clinical emergency treatment, traditional paper records and multi-platform, decentralized data management lead to inefficient treatment, delayed decision-making, and difficulties in quality control. Critical patient rescue milestones (such as medication, defibrillation, and consultation) are easily missed, data silos across multiple systems are prominent, and the lack of real-time intelligent support slows down the generation of emergency plans, impacting treatment success rates. To address these issues, it is necessary to build an intelligent emergency platform that integrates the Internet of Things, artificial intelligence, and multi-system collaboration to achieve digital management of the entire emergency process. Summary of the Invention

[0004] In response to the shortcomings of the existing technology, the present invention proposes an intelligent first aid decision-making system based on multimodal fusion, which can predict the patient's possible symptoms in advance through the patient's vital monitoring data, thereby generating predicted first aid content in advance. The first aid process will be recorded in an information-based manner to provide high-fidelity training data for subsequent quality improvement.

[0005] The present invention provides an intelligent first aid decision-making system based on multimodal fusion, and the technical solution is as follows:

[0006] An intelligent emergency decision-making system based on multimodal fusion includes a main control unit connected to a display unit, a connection unit, an interaction unit and a cross-linking unit;

[0007] The main control unit stores the patient's basic and detailed information. The connection unit obtains the patient's vital monitoring data through medical instruments. The display unit is used to display the patient's basic information and vital monitoring data. The interaction unit is used to receive and record the doctor's oral description of the patient's current condition. The cross-linking unit is used to connect to the hospital's Internet of Things to obtain drug information.

[0008] When the main control unit determines whether the patient has health risks based on the patient's vital monitoring data, if it is determined that there are health risks, the main control unit predicts possible symptoms based on the patient's vital monitoring data and generates predicted first aid content based on all predicted symptoms;

[0009] When the doctor determines that the patient has a disease and needs emergency treatment, the main control unit calls the predicted emergency treatment content for the doctor's reference. If the predicted emergency treatment content contains the patient's disease, the doctor calls the predicted emergency treatment content corresponding to the disease. If the predicted emergency treatment content does not contain the patient's disease, the doctor inputs the patient's current condition and disease through the interactive unit, and the main control unit generates the emergency treatment content online.

[0010] First aid content includes symptom analysis, first aid plan, medication recommendations, drug dosage recommendations, medication warnings and the nearest drug storage point.

[0011] In summary, the above technical solution has the following beneficial effects: The beneficial effects of this application include: the main control unit of this application determines whether the patient has health risks by collecting the patient's vital monitoring data, thereby predicting the patient's possible symptoms and generating predicted first aid content in advance. When the patient needs first aid, the doctor can observe the patient's symptoms and check whether the main control unit has predicted first aid content generated in advance for reference. If so, the doctor can refer to the predicted first aid content for first aid, and the main control unit records the first aid process. If not, the doctor inputs the symptoms online by voice input, or directly performs first aid according to the patient's specific symptoms, and the main control unit also records the first aid process. By generating predicted first aid content in advance, the time for the main control module to generate content can be saved when an emergency situation occurs. In addition, the main control module records the first aid content, which plays a role in legal risk prevention and control. The timeline accuracy is high, and the digital signature and CA certification meet the requirements of judicial appraisal. After each first aid data is recorded, it can be used as a reference in the next prediction, thereby improving the accuracy of the prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is a module connection diagram of an intelligent emergency decision-making system based on multimodal fusion;

[0013] Figure 2 A schematic diagram of life monitoring data for an intelligent emergency decision-making system based on multimodal fusion;

[0014] Figure 3 A schematic diagram of first aid content generation for an intelligent first aid decision-making system based on multimodal fusion.

[0015] Reference numerals: 10, main control unit; 20, display unit; 30, connection unit; 40, interaction unit; 50, cross-linking unit; 60, telephone dedicated line unit; 70, evaluation unit. DETAILED DESCRIPTION

[0016] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0017] like Figure 1 As shown, an intelligent emergency decision-making system based on multimodal fusion is characterized in that it includes a main control unit 10, which is connected to a display unit 20, a connection unit 30, an interaction unit 40 and a cross-linking unit 50; the main control unit 10 stores the patient's basic information and detailed information, the connection unit 30 obtains the patient's vital monitoring data through medical instruments, the display unit 20 is used to display the patient's basic information and the patient's vital monitoring data, the interaction unit 40 is used to receive and record the doctor's oral description of the patient's current condition, and the cross-linking unit 50 is used to connect to the hospital Internet of Things to obtain drug information; when the main control unit 10 determines whether the patient has a healthy state through the patient's vital monitoring data Hidden dangers. If it is determined that there are health hazards, the main control unit 10 predicts possible symptoms based on the patient's vital monitoring data, and generates predicted first aid content based on all predicted symptoms. When the doctor determines that the patient has symptoms and needs first aid, the main control unit 10 calls the predicted first aid content for the doctor's reference. If the predicted first aid content includes the patient's symptoms, the doctor calls the predicted first aid content corresponding to the symptoms. If the predicted first aid content does not include the patient's symptoms, the doctor inputs the patient's current condition and symptoms through the interactive unit 40, and lets the main control unit 10 generate first aid content online. The first aid content includes symptom analysis, first aid plan, medication recommendations, drug dosage recommendations, medication warnings, and the nearest drug reserve point.

[0018] The main control unit 10 of this application determines whether the patient has health risks by collecting the patient's vital monitoring data, thereby predicting the patient's possible symptoms and generating predicted first aid content in advance. When the patient needs first aid, the doctor can observe the patient's symptoms and check whether the main control unit 10 has predicted first aid content generated in advance for reference. If so, the doctor can refer to the predicted first aid content for first aid, and the main control unit 10 records the first aid process. If not, the doctor inputs the symptoms online by voice input, or directly performs first aid according to the patient's specific symptoms, and the main control unit 10 also records the first aid process. By generating predicted first aid content in advance, the time for the main control module to generate content can be saved when an emergency situation occurs. In addition, the main control module records the first aid content, which plays a role in legal risk prevention and control. The timeline accuracy is high, and the digital signature and CA certification meet the requirements of judicial appraisal. After each first aid data is recorded, it can be used as a reference in the next prediction, thereby improving the accuracy of the prediction.

[0019] like Figure 2and Figure 3 As shown, this system is installed in an all-in-one computer (touch screen) and a smart emergency vehicle. To use it, press the red start button on the emergency vehicle computer and select the emergency patient using the touch screen of the display unit 20. This quickly retrieves detailed information such as medical history, past medical history, major diagnosis and treatment history, and auxiliary examinations. The patient's current condition can then be quickly input via voice or text. The system quickly integrates the patient's current condition and vital monitoring data. Using a large-scale disease analysis model based on evidence-based medicine, a preliminary emergency plan is generated within 10-20 seconds. A dedicated emergency channel is then activated on the computing server, prioritizing computational speed.

[0020] Specifically, the main control unit 10 synchronizes the monitor's 12-lead waveform data, such as ECG, SpO2, and invasive blood pressure, in real time through the connection unit 30, with a sampling rate of ≥500Hz, and the connection is achieved through Bluetooth or WiFi.

[0021] Specifically, the interactive unit 40 is a medical-grade noise reduction microphone array that supports 10-meter far-field sound pickup and integrates a medical-specific speech recognition engine that can recognize professional terms such as "amiodarone 150mg intravenous injection".

[0022] Specifically, the cross-linking unit 50 can read data such as drug inventory, compatibility, etc. of the smart medicine cabinet in real time.

[0023] Specifically, the multimodal fusion algorithm framework of the main control module utilizes a "hierarchical feature fusion + dynamic weight allocation" model. This includes the bottom-level feature layer: vital data is extracted using an LSTM network for temporal features, voice commands are parsed into medical behavior intent vectors using a BERT model, and drug information is embedded into a knowledge graph to generate structured features. The middle-level decision layer: A cross-modal attention mechanism is constructed based on the Transformer architecture, incorporating clinical time window constraints (e.g., the 90-minute golden period for treatment of myocardial infarction patients after symptom onset) to dynamically adjust the weights of each modality.

[0024] The main control module includes a multimodal fusion algorithm. The multimodal fusion process can be formalized by the following mathematical expression:

[0025] Preset Input data of different modalities ,set up Indicates the data of different modalities, assuming Indicates the Modal data such as patient vital monitoring data, medication information prediction, and the patient's current condition and symptoms input by the doctor through the interactive unit 40.

[0026] First, each modal data is converted into feature representation through its own feature extraction network:

[0027]

[0028]

[0029] in For the The feature representation of the modality, For the The feature representation of the modality, and Represents the feature extraction network for the corresponding modality data.

[0030] Next, the cross-modal attention mechanism in the Transformer architecture is used to handle the interaction between different modalities. and modal The attention calculation between can be expressed as:

[0031]

[0032] in is the query matrix, which is used to extract query features from the current modality. Is the key matrix, used to store key information that can be matched, is the dimension of the key vector, is the modality-specific learnable parameter matrix obtained by learning the Q-map, is the modality-specific learnable parameter matrix obtained by learning the K mapping.

[0033] The attention output can be expressed as:

[0034]

[0035] Where V is a matrix of values, It is a modality-specific learnable parameter matrix responsible for mapping the original features to the value space; the value matrix = is the learnable parameter matrix Feature representation of mode j Obtained by performing a linear transformation.

[0036] The aggregation of cross-modal attention is expressed as:

[0037]

[0038] The value matrix plays the role of information content provider in the attention mechanism. When the similarity calculation between the query matrix Q and the key matrix K is completed and the attention weight is generated Finally, these weights are applied to the value matrix to obtain the actual information content that needs to be extracted. Aggregation of cross-modal attention Contains the interactive information between modalities, compared with the original feature representation Has richer semantics.

[0039] Then, a dynamic weight allocation mechanism constrained by clinical time window is introduced to perform weighted fusion of each modality feature:

[0040]

[0041] in Clinical time window The relevant scoring function, It is Dynamic weighting of modalities is used to dynamically adjust the importance of each modality in different clinical scenarios based on time window constraints. For example, during the golden 90-minute treatment period for a myocardial infarction patient, the weight of electrocardiogram data may be automatically increased by the system, while other vital sign data may receive higher weights at other time points.

[0042] The final multimodal fusion feature is expressed as:

[0043]

[0044] The fusion feature It will serve as the input of the subsequent decision-making model to obtain the first aid content.

[0045] Federated Learning Optimization: A federated learning platform was established within the hospital's internal data center. Each department's nodes upload encrypted model gradients (not raw data), and the global fusion model is regularly updated to meet the data privacy requirements of the "Medical and Health Data Management Measures." A real-time guarantee mechanism utilizes DDS (Data Distribution Service) technology to achieve millisecond-level data synchronization, with end-to-end latency under 200ms. A double-buffered queue design ensures that urgent data (such as ventricular fibrillation waveforms) trigger interrupt priority scheduling, ensuring critical information processing latency under 50ms.

[0046] The main control unit 10 generates predicted emergency care content based on the clinical decision-making engine generated by the intelligent emergency plan. This includes the construction of a dynamic knowledge graph. This involves integrating multi-source knowledge from multiple databases, including a basic rule base: integrating authoritative guidelines such as UpToDate clinical guidelines, the American Heart Association (AHA) cardiopulmonary resuscitation process, and the "Chinese Expert Consensus on the Diagnosis and Treatment of Acute Poisoning" to construct a decision tree containing over 8,000 medical rules. The hospital's experience base: using natural language processing technology to analyze over 20,000 emergency medical records over the past five years, extracting unique diagnosis and treatment pathways (such as the thrombolysis time window adjustment rules for patients with myocardial infarction in high altitude areas). Real-time evidence updates from multiple databases: integrating with PubMed Clinical Queries, automatically syncing the latest clinical research evidence (evidence level ≥ 2A) weekly.

[0047] A reinforcement learning decision model was then used, using state variables such as the rate of change in the patient's APACHE II score and lactate clearance rate to design a reward function for the sequence of emergency response measures. The decision model was trained using the Proximal Policy Optimization (PPO) algorithm to simulate the impact of different emergency response plans on 28-day mortality.

[0048] The main control unit 10 generates first aid content through the first aid decision model, which is mathematically expressed as follows:

[0049] The main control unit 10 defines the state space , including various physiological indicators and clinical parameters of patients, such as APACHE II score, lactate level, hemodynamic parameters and other multidimensional feature vectors. Action space , represents a discrete set of clinical interventions, including medication regimens, surgical interventions, life support measures, etc. Reward function , quantifies the medical value of performing action A in state S, usually mapped to patient prognosis indicators.

[0050] In the state space Based on the input feature Z, the action space The action is selected from the input features Z and the strategy is updated based on R(s,a). The multimodal fusion Z is used as the input feature of the emergency decision-making model. Meanwhile, S, composed of the patient's physiological indicators and clinical parameters, is used as the "state" in reinforcement learning. In state S, based on the input features Z, an action A is selected and the strategy is updated based on R(s,a).

[0051] The first aid decision is modeled as a Markov decision process (MDP) and the goal is to find the optimal policy , so that the expected cumulative reward is maximized:

[0052]

[0053] in is a discount factor that adjusts the trade-off between short-term and long-term benefits, reflecting the clinical time window constraint; t is a sequence position identifier: t starts at 0 and ends at T, and each t corresponds to a specific time point in the emergency treatment process; is the decision sequence length; It is a strategy for making decisions under different conditions; is the expected value under strategy π. This formula is mainly used to solve the optimal strategy. Represents the best course of action the system should take, equivalent to an "ideal first aid guide."

[0054] Use the PPO algorithm to update the emergency decision-making strategy by optimizing the objective function:

[0055]

[0056] in is the parameter of the emergency decision model, is the probability ratio of the new and old strategies, is the advantage function estimate, is the clipping function, is the cropping parameter, The empirical expectation at said time t.

[0057] The code for the visualization decision process is as follows:

[0058] # Pseudocode for decision making for patients with acute chest pain

[0059] def chest_pain_decision(patient_data):

[0060] if ecg_has_st_elevation(patient_data):

[0061] # Calculation of symptom onset time

[0062] if time_since_onset<120min:

[0063] # Catheterization lab status query

[0064] if can_perform_pci(patient_data):

[0065] # Start the PCI preoperative preparation process

[0066] return generate_pci_protocol()

[0067] else:

[0068] # Thrombolytic regimen in accordance with the STEMI Diagnosis and Treatment Guidelines

[0069] return generate_thrombolysis_protocol()

[0070] else:

[0071] return stable_angina_management()

[0072] else:

[0073] # Dynamic monitoring of myocardial injury markers

[0074] if troponin>99th_percentile:

[0075] return nstemi_treatment()

[0076] else:

[0077] # Start other cause investigation processes

[0078] return rule_out_acs()

[0079] Furthermore, the main control unit 10 is connected to a dedicated telephone line unit 60, which establishes dedicated telephone lines with consulting physicians, pharmacies, and blood banks, allowing for one-touch voice calls. The dedicated telephone line unit 60 selects a carrier from China Mobile, China Unicom, or China Telecom, opens a dedicated emergency voice line, and configures a one-touch call consultation system, "Yunlian," which embeds the hospital's emergency number and call templates, supporting one-touch voice calls to consulting physicians, pharmacies, and blood banks. The emergency plan lists the consultation matters that require urgent calls. Standardized help SMS messages and corporate WeChat account notifications are pushed simultaneously based on different needs. For example, calling the anesthesiology department for tracheal intubation: Hello, bed 12 in ward 222 requires urgent tracheal intubation, please go immediately; clinical department: Hello, bed 12 in ward 222 requires urgent consultation, please go immediately; blood bank: Hello, bed 12 in ward 222 requires urgent blood transfusion, the patient's blood type is A, blood cross will be sent immediately, please prepare blood immediately and deliver it as soon as possible; calling 100: Hello, bed 12 in ward 222 requires urgent blood specimen delivery, please go immediately; pharmacy: Hello, bed 12 in ward 222 requires 5 norepinephrines, please deliver immediately, etc. This allows for rapid initiation of multidisciplinary team rescue efforts.

[0080] Medication recommendations can provide the name of the medication the patient currently needs and display the medication's location information based on the cross-linking unit 50, making it easy to quickly access. They also provide dosage recommendations automatically calculated based on body weight. Medication warnings, such as "Warfarin currently in use; use heparin with caution," are displayed. When medication is low, the nearest stock point is displayed, such as "10 units in stock at the emergency pharmacy."

[0081] The display unit 20 with integrated touch is physically bound to the mobile rescue vehicle and can be adjusted at multiple angles.

[0082] The drug dosage recommendation is obtained based on the patient's weight, the medication warning is obtained based on the patient's detailed information, and the nearest drug storage point obtains the location information of the corresponding drug through the cross-linking unit 50.

[0083] The display unit 20 displays the standard first aid process step by step on the screen according to the first aid plan, supports touch-screen zooming, and highlights step-by-step instructions on the screen.

[0084] The main control unit 10 generates a first aid record based on the first aid steps and a timeline. It records medication administration and defibrillation as key points in the timeline, as well as outpatient examinations and department transfers. The following table shows the various stages of the timeline.

[0085]

[0086] Timeline granularity: Accurate to operation time (millisecond level) + operation subject (authenticated by swiping work badge) + equipment number (unique medical device code), for example: 2025-04-30T14:23:15.452+08:00|Nurse A|Bedside monitor #CN2024001|SpO2 increased from 88% to 94%.

[0087] Typical forensic forensic scenarios include verifying CPR compliance: using timeline data to calculate key metrics such as compression rate (100-120 times / minute) and interruption time (>10 seconds). Medication sequence dispute resolution: Accurately tracing the timing of the first epinephrine injection and the completion of defibrillator charging to resolve disputes regarding the rescue process.

[0088] After the interactive unit 40 records the doctor's oral description of the patient's current condition, the main control unit 10 parses the voice instructions of the patient's current condition and generates a standardized electronic medical order. The nurse confirms the execution of the electronic medical order through the touch screen of the display unit 20, and the main control unit 10 records the nurse's execution of the electronic medical order in the timeline.

[0089] The main control unit 10 records the voiceprints of each doctor and nurse, and the interaction unit 40 uses voiceprint recognition to distinguish between doctors and nurses. For example, if a nurse says, "09:25, 1mg epinephrine IV injection," "09:30, 200J biphasic defibrillation, one time," etc., the interaction unit 40 then converts the nurse's speech into text, and the main control unit 10 generates a rescue timeline with a timestamp in real time, making it easier to record structured data after the rescue is completed and to verify data integrity using time logic.

[0090] After the first aid is completed, the main control unit 10 generates a first aid file, which is used to automatically organize the first aid process according to the timeline, verify the integrity of the process data in combination with time logic; generate a first aid report that complies with medical standards; and trigger subsequent processes based on the first aid results. After the rescue is completed, click the rescue completion button, and then the main control unit 10 automatically organizes the rescue timeline, verifies the integrity of the data in combination with time logic, and generates a first aid report that complies with medical standards, which is reviewed and confirmed by medical staff. A consultation form is automatically generated based on the consultation call and consultation opinion, which is consistent with the consultation form in the current electronic medical record system, and allows doctors to import and modify it with one click. Finally, the entire first aid data is encrypted and archived. The subsequent rescue process includes automatically filling in content such as transfer to another department and death certificate. The main control unit 10 also traces the first aid process based on the first aid file and automatically marks the links to be improved, such as "defibrillation preparation was delayed by 50 seconds."

[0091] The main control unit 10 is further connected to an evaluation unit 70. The evaluation unit 70 is pre-set with a complete set of first aid items C = [C1, C2...Ci], where Ci represents the i-th first aid item, and each element of the complete set of first aid items C is respectively set with an importance score Si, where Si represents the importance score of the i-th first aid item;

[0092] The evaluation unit 70 obtains each first aid item on the timeline from the first aid record to form a content set P, where P = [P1, P2, ..., Pi], where Pi represents the i-th first aid item on the timeline, and a time set T, where T = [T1, T2, ..., Ti], where Ti represents the time spent on the i-th first aid item on the timeline. Each element of the content set P is mapped to the complete set of first aid items C, thereby obtaining an importance score Si for each element of the content set P. Each element of the content set P is assigned a weight B based on the importance score Si, where B = [B1, B2, ..., Bi], where Bi represents the weight of the i-th first aid item in the content set P. The evaluation unit calculates an evaluation score for the first aid item through the following process:

[0093]

[0094] Among them, Score represents the evaluation score, Bi represents the weight of the i-th first aid content in the content collection P, and Ti represents the time spent on the i-th first aid content on the timeline.

[0095] In order to improve the accuracy and comparability of the evaluation, the time set T can be normalized and the normalized time can be defined as for:

[0096]

[0097] in and are the minimum and maximum time spent on all first aid items respectively. The normalized evaluation score is calculated as:

[0098]

[0099] in is a small constant used to avoid the denominator being zero.

[0100] Each first aid record searches for the corresponding first aid item weight in the preset first aid item set based on the first aid items on the timeline, thereby calculating the weight of each first aid item in this first aid record. The weight can be obtained by comprehensively calculating the score of a certain first aid item in the first aid record over all first aid items. The longer the first aid item takes, the lower the score. The elements of the time set T can be normalized before participating in the calculation. Finally, the evaluation score of a first aid record is calculated, which is used to evaluate the score obtained for each first aid, including the score of each first aid item, so that each item can be reviewed later to find out the process that can be optimized. The evaluation unit calculates the average score of the first aid record through the following process;

[0101] in, It represents the average score of the first aid record, which is used to compare the scores of each first aid record and to judge the performance of medical staff in different first aid records.

[0102] In order to further improve the evaluation mechanism, a time sensitivity factor can be introduced , weighted according to the timeliness requirements of different emergency projects:

[0103]

[0104] in It is The time sensitivity factor of each emergency content is given a higher value for time-sensitive operations (such as defibrillation, endotracheal intubation, etc.) and a lower value for relatively less time-sensitive operations. An additional dimension was added to evaluate the time urgency of each emergency procedure. values, which may have completely different time sensitivity requirements: the original weight factor The timeliness score measures the general clinical importance of the procedure, while βi specifically addresses the speed with which the procedure must be performed. It can more reasonably evaluate the performance of medical staff in different first aid records.

[0105] The beneficial effects of this application include: 1. Significant improvement in rescue efficiency. 2. Enhanced management accuracy: Accurate control of the entire rescue process. 3. Guarantee of scientific decision-making: Reduce procedural errors through the clinical decision support system (CDSS). 4. Legal risk prevention and control: High timeline accuracy, digital signatures and CA certification meet forensic identification requirements. 5. Resource integration and optimization: Shortened rescue-related communication response time, improved multi-system data penetration rate. 6. Quality control closed loop: Automatically trigger quality control checkpoints, improve problem tracing efficiency, not only optimize the current rescue steps in real time, but also provide high-fidelity training data for subsequent quality improvements.

[0106] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions based on the principles of the present invention are within the scope of protection of the present invention. It should be noted that improvements and modifications that do not depart from the principles of the present invention are within the scope of protection of the present invention.

Claims

1. An intelligent emergency decision-making system based on multimodal fusion, characterized in that: The main control unit (10) is connected to a display unit (20), a connection unit (30), an interaction unit (40) and a cross-linking unit (50); The main control unit (10) stores the patient's basic information and detailed information, the connection unit (30) obtains the patient's vital monitoring data through medical instruments, the display unit (20) is used to display the patient's basic information and the patient's vital monitoring data, the interaction unit (40) is used to receive and record the patient's current condition orally stated by the doctor, and the cross-linking unit (50) is used to connect to the hospital Internet of Things to obtain drug information; The main control unit (10) determines whether the patient has health risks based on the patient's vital monitoring data. If it is determined that the patient has health risks, the main control unit (10) predicts possible symptoms based on the patient's vital monitoring data and medication information, and generates predicted first aid content based on all predicted symptoms. When the doctor determines that the patient has a symptom and needs emergency treatment, the main control unit (10) calls the predicted emergency treatment content for the doctor's reference. If the predicted emergency treatment content contains the patient's symptom, the doctor calls the predicted emergency treatment content corresponding to the symptom. If the predicted emergency treatment content does not contain the patient's symptom, the doctor inputs the patient's current condition and symptom through the interactive unit (40), and the main control unit (10) generates the emergency treatment content online. First aid content includes symptom analysis, first aid plan, medication suggestions, drug dosage recommendations, medication warnings, and the nearest drug storage point; The main control unit (10) generates first aid content through a multimodal fusion algorithm; Preset Input data of different modalities ,in Indicates the data of different modalities; Each modal data is converted into feature representation through its own feature extraction network: in For the The feature representation of the modality, For the The feature representation of the modality, and Represents the feature extraction network of the corresponding modality data; The cross-modal attention mechanism in the Transformer architecture is used to handle the interaction between different modalities. and modal The attention calculation between can be expressed as: in is the query matrix, which is used to extract query features from the current modality. Is the key matrix, used to store key information that can be matched, is the dimension of the key vector, is the modality-specific learnable parameter matrix obtained by learning the Q-map, is the modality-specific learnable parameter matrix obtained by learning the K mapping; The attention output can be expressed as: Where V is a matrix of values, It is a modality-specific learnable parameter matrix responsible for mapping the original features to the value space; The aggregation of cross-modal attention is expressed as: A dynamic weight allocation mechanism based on clinical time window constraints is introduced to perform weighted fusion of each modality feature: in Clinical time window The relevant scoring function, It is Dynamic weights of the modalities are used to dynamically adjust the importance of each modality according to the constraints of the time window in different clinical scenarios; The multimodal fusion feature is expressed as: The fusion feature As input, first aid content is obtained; The main control unit (10) is also connected to the evaluation unit (70), which introduces a time sensitivity factor to improve the evaluation mechanism. , weighted according to the timeliness requirements of different emergency projects: in is the time sensitivity factor of the i-th first aid content, Bi represents the weight of the i-th first aid content in the content collection P, and Ti represents the time spent on the i-th first aid content on the timeline.

2. The intelligent emergency decision-making system based on multimodal fusion according to claim 1, characterized in that: The main control unit (10) generates the first aid content mathematical expression as follows: The main control unit (10) defines the state space , including various physiological indicators and clinical parameters of patients; Action Space , represents a discrete set of clinical interventions; Reward Function , quantify the medical value of performing action A in state S; In the state space Based on the input feature Z, the action space Select an action and update the strategy based on R(s,a); The optimal strategy is found by the following formula , so that the expected cumulative reward is maximized: in is a discount factor that adjusts the trade-off between short-term and long-term benefits, reflecting the clinical time window constraint; t is a sequence position identifier: t starts at 0 and ends at T, and each t corresponds to a specific time point in the emergency treatment process; is the decision sequence length; It is a strategy for making decisions under different conditions; is the expected value under strategy π.

3. The intelligent emergency decision-making system based on multimodal fusion according to claim 2, characterized in that: The drug dosage recommendation is obtained based on the patient's weight, the medication warning is obtained based on the patient's detailed information, and the nearest drug storage point obtains the location information of the corresponding drug based on the cross-linking unit (50).

4. The intelligent emergency decision-making system based on multimodal fusion according to claim 2, characterized in that: The display unit (20) displays the standard first aid process step by step on the screen according to the first aid plan, supports touch-screen zooming, and provides step-by-step highlighting instructions on the screen.

5. The intelligent emergency decision-making system based on multimodal fusion according to claim 2 is characterized in that: The main control unit (10) is also connected to a dedicated telephone line unit (60), which establishes dedicated telephone lines with consulting physicians, pharmacies and blood banks respectively for one-touch voice calls.

6. The intelligent emergency decision-making system based on multimodal fusion according to claim 3, characterized in that: The main control unit (10) generates a first aid record according to the first aid steps and the time axis; Record medication time and defibrillation time as key nodes in the timeline, and also record outpatient examinations, transfers to other departments, etc. in the timeline.

7. The intelligent emergency decision-making system based on multimodal fusion according to claim 6, characterized in that: After the interactive unit (40) records the current condition of the patient as spoken by the doctor, the main control unit (10) parses the voice instructions of the patient's current condition and generates a standardized electronic medical order. The nurse confirms the execution of the electronic medical order through the touch screen of the display unit (20), and the main control unit (10) records the nurse's execution of the electronic medical order in a timeline.

8. The intelligent emergency decision-making system based on multimodal fusion according to claim 7, characterized in that: The main control unit (10) records the voiceprints of each doctor and nurse, and the interactive unit (40) distinguishes between doctors and nurses through voiceprint recognition.

9. The intelligent emergency decision-making system based on multimodal fusion according to claim 6, characterized in that: The evaluation unit (70) is preset with a first aid item set A=[A1, A2...Ai], Ai represents the i-th first aid item, and each element of the first aid item set A is respectively set with an importance score Si, Si represents the importance score of the i-th first aid item; The evaluation unit (70) obtains each first aid item on the time axis according to the first aid record to form a content collection P, P=[P1, P2...,Pi], Pi represents the first aid content of the i-th item on the time axis, and a time collection T, T=[T1, T2...,Ti], Ti represents the time spent on the first aid content of the i-th item on the time axis, and maps each element of the content collection P to the first aid item collection A, thereby obtaining an important score Si of each element of the content collection P. Each element of the content collection P obtains a weight Q of each element according to the important score Si, Q=[Q1, Q2...,Qi], Qi represents the weight of the first aid content of the i-th item in the content collection P. The evaluation unit (70) obtains an evaluation score for the first aid item through the following process: Among them, Score represents the evaluation score, Qi represents the weight of the i-th first aid content in the content collection P, and Ti represents the time spent on the i-th first aid content on the timeline.

Images