Intelligent first-aid decision-making system based on multi-modal fusion

Abstract

The invention relates to an intelligent first-aid decision system based on multi-modal fusion, which comprises a main control unit, and the main control unit is connected with a display unit, a connection unit, an interaction unit and a cross-linking unit. When the main control unit judges whether the patient has potential health hazards or not according to the patient life monitoring data, if yes, the main control unit predicts possible diseases according to the patient life monitoring data and generates predicted first-aid content according to all predicted diseases; when the doctor judges that the patient has diseases and needs emergency treatment, the predicted emergency treatment content is called through the main control unit for reference of the doctor. According to the application, the time for the main control module to generate the content can be saved when the first-aid situation occurs by generating the predicted first-aid content in advance, in addition, the main control module records the first-aid content, the legal risk prevention and control effect is achieved, the time axis precision is high, and the digital signature and CA authentication meet the judicial expertise requirement.

Description

Technical Field

[0001] The present invention relates to the field of medical technology, and more specifically, it relates to an intelligent first-aid decision-making system based on multimodal fusion. Background Art

[0002] First-aid scenarios often involve critically ill patients. Under the circumstances of short time, urgent tasks, and heavy workload, first-aid work must be carried out in an orderly manner according to its own characteristics and work specifications to ensure timely and correct diagnosis and treatment of patients, so as to save the lives of patients to the greatest extent. During the rescue process, it is necessary to quickly form a first-aid plan, record key time points, and require multi-disciplinary cooperation, etc.

[0003] During the clinical rescue process, traditional paper records and multi-platform, decentralized data management lead to low rescue efficiency, delayed decision-making, and difficult quality control. Key time nodes in patient rescue (such as medication, defibrillation, consultation) are prone to be missed, the problem of data islands in multiple systems is significant, and there is a lack of real-time intelligent auxiliary support. The generation speed of first-aid plans is slow, affecting the success rate of treatment. To solve these problems, it is necessary to build an intelligent first-aid platform integrating the Internet of Things, artificial intelligence, and multi-system collaboration to realize digital management of the entire rescue process. Summary of the Invention

[0004] Aiming at the deficiencies of the existing technology, the present invention proposes an intelligent first-aid decision-making system based on multimodal fusion, which can predict possible diseases of patients in advance through the patient's vital sign monitoring data, so as to generate predicted first-aid content in advance. The first-aid process will be recorded informatization, providing high-fidelity training data for subsequent quality improvement.

[0005] The present invention provides an intelligent first-aid decision-making system based on multimodal fusion, and the technical solution is as follows: An intelligent first-aid decision-making system based on multimodal fusion, which includes a main control unit, and the main control unit is connected with a display unit, a connection unit, an interaction unit, and a cross-linking unit; The main control unit stores the basic information and detailed information of the patient. The connection unit obtains the patient's vital sign monitoring data through medical instruments. The display unit is used to display the patient's basic information and the patient's vital sign monitoring data. The interaction unit is used to receive and record the current condition of the patient dictated by the doctor. The cross-linking unit is used to connect to the hospital Internet of Things to obtain drug information; When the main control unit determines whether the patient has health risks through the patient's vital sign monitoring data, if it is determined that there are health risks, the main control unit predicts possible diseases according to the patient's vital sign monitoring data, and generates predicted first-aid content according to all the predicted diseases; When a doctor determines that a patient has a medical condition and requires first aid, the main control unit is called to retrieve the predicted first aid content for the doctor's reference. If the predicted first aid content includes the patient's medical condition, the doctor can refer to the predicted first aid content corresponding to that condition. If the predicted first aid content does not include the patient's medical condition, the doctor inputs the patient's current situation and medical condition through the interaction unit, enabling the main control unit to generate the first aid content online. The first aid content includes medical condition analysis, first aid plan, medication advice, recommended drug dosage, medication warnings, and the nearest drug storage points.

[0006] In summary, the above technical solution has the following beneficial effects: The beneficial effects of this application include that the main control unit of this application determines whether there are health risks in the patient by collecting the patient's vital sign monitoring data, thereby predicting the possible medical conditions of the patient and generating the predicted first aid content in advance. When the patient needs first aid, the doctor can observe the patient's medical condition and check whether there is any predicted first aid content generated in advance by the main control unit for reference. If so, the doctor can refer to the predicted first aid content for first aid, and the main control unit records the first aid process. If not, the doctor can input the medical condition by voice to generate the plan online or directly conduct first aid according to the patient's specific medical condition, and the main control unit also records the first aid process. By generating the predicted first aid content in advance, it can save the time for the main control module to generate content in case of first aid. In addition, the main control module records the first aid content, which has the effect of preventing legal risks, with high time axis accuracy, and the digital signature and CA certification meet the requirements of forensic appraisal. Each first aid data, after being recorded, can be used as a reference for the next prediction, thereby improving the accuracy of the prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 It is a schematic diagram of the module connection of an intelligent first aid decision-making system based on multi-modal fusion; Figure 2 It is a schematic diagram of the vital sign monitoring data of an intelligent first aid decision-making system based on multi-modal fusion; Figure 3 It is a schematic diagram of the first aid content generation of an intelligent first aid decision-making system based on multi-modal fusion.

[0008] Reference numerals: 10, main control unit; 20, display unit; 30, connection unit; 40, interaction unit; 50, cross-linking unit; 60, dedicated telephone line unit; 70, evaluation unit. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0009] The following clearly and completely describes the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0010] As Figure 1 shown, an intelligent first-aid decision-making system based on multi-modal fusion, characterized in that it includes a main control unit 10, and the main control unit 10 is connected to a display unit 20, a connection unit 30, an interaction unit 40, and a cross-linking unit 50; the basic information and detailed information of the patient are stored in the main control unit 10, the connection unit 30 obtains the patient's vital sign monitoring data through medical instruments, the display unit 20 is used to display the patient's basic information and the patient's vital sign monitoring data, the interaction unit 40 is used to receive and record the current condition of the patient dictated by the doctor, and the cross-linking unit 50 is used to connect to the hospital Internet of Things to obtain drug information; when the main control unit 10 determines whether there are health hazards for the patient based on the patient's vital sign monitoring data, if it is determined that there are health hazards, the main control unit 10 predicts possible diseases according to the patient's vital sign monitoring data, and generates predicted first-aid content respectively according to all the predicted diseases; when the doctor determines that the patient has a disease and needs first aid, the predicted first-aid content is called through the main control unit 10 for the doctor to refer to. If the predicted first-aid content has the patient's disease, the doctor calls the predicted first-aid content corresponding to the disease. If the predicted first-aid content does not have the patient's disease, the doctor inputs the current condition and disease of the patient through the interaction unit 40, and the main control unit 10 generates the first-aid content online; the first-aid content includes disease analysis, first-aid plan, medication advice, drug dose recommendation, medication warning, and the nearest drug storage point.

[0011] In this application, the main control unit 10 determines whether there are health hazards for the patient by collecting the patient's vital sign monitoring data, thereby predicting the possible diseases of the patient and generating predicted first-aid content in advance. When the patient needs first aid, the doctor can observe the patient's disease and check whether the main control unit 10 has the predicted first-aid content generated in advance for reference. If so, the doctor can refer to the predicted first-aid content for first aid, and the main control unit 10 records the first-aid process. If not, the doctor inputs the disease by voice to generate a plan online, or directly performs first aid according to the specific disease of the patient, and the main control unit 10 also records the first-aid process. By generating the predicted first-aid content in advance, the time for the main control module to generate content can be saved in case of a first-aid situation. In addition, the main control module records the first-aid content, which has the effect of preventing legal risks, with high time-axis accuracy, and the digital signature and CA certification meet the requirements of forensic identification. After each first-aid data is recorded, it can be used as a reference for the next prediction, thereby improving the accuracy of the prediction.

[0012] As Figure 2and Figure 3 As shown, this system is installed in an all-in-one computer (touch screen) + intelligent rescue vehicle. When in use, press the red start button on the emergency vehicle computer, and use the touch screen of the display unit 20 to select the emergency patient, then detailed information such as the medical history, past history, main treatment process, and auxiliary examinations can be quickly retrieved. After that, quickly input the current situation through voice or text. This system can quickly integrate information such as the patient's current situation and vital sign monitoring data, and generate a preliminary first aid plan within 10 - 20 seconds based on evidence-based medicine through a disease analysis large model, and open a dedicated emergency channel for the computing power server, with the system giving priority to ensuring the computing speed.

[0013] Specifically, the main control unit 10 synchronizes the 12-lead waveform data such as ECG, SpO2, and invasive blood pressure of the monitor in real time through the connection unit 30, with a sampling rate ≥ 500Hz, and the connection method is achieved through Bluetooth or WiFi.

[0014] Specifically, the interaction unit 40 is a medical-grade noise reduction microphone array, supporting 10-meter far-field sound pickup, integrating a medical-specific voice recognition engine, and can recognize professional terms such as "150mg of amiodarone intravenous push".

[0015] Specifically, the cross-linking unit 50 can read the drug inventory, compatibility taboos, and other data of the intelligent medicine cabinet in real time.

[0016] Specifically, the multi-modal fusion algorithm framework of the main control module adopts a "hierarchical feature fusion + dynamic weight allocation" model. It includes a bottom feature layer: vital data extracts temporal features through an LSTM network, voice commands are parsed into medical behavior intention vectors by a BERT model, and drug information generates structured features through knowledge graph embedding. A middle decision layer: constructs a cross-modal attention mechanism based on the Transformer architecture, and introduces a clinical time window constraint (such as the 90-minute golden treatment period after the symptoms of a myocardial infarction patient appear) to dynamically adjust the weights of each modality.

[0017] The main control module includes a multi-modal fusion algorithm, and the multi-modal fusion process can be formalized by the following mathematical expression: There are preset types of input data of different modalities , let represent the data of the th modality, and let represent the data of the th modality. The data of the modality such as the patient's vital sign monitoring data, prediction of drug information, and the doctor's input of the patient's current situation and symptoms through the interaction unit 40.

[0018] First of all, the data of each modality is converted into a feature representation through its respective feature extraction network: where is the feature representation of the th modality, is the feature representation of the th modality, and represent the feature extraction network for the corresponding modality data.

[0019] Next, the cross-modal attention mechanism in the Transformer architecture is used to process the interaction between different modalities. For the attention calculation between modality and modality it can be expressed as: where is the query matrix, used to extract query features from the current modality, is the key matrix, used to store the key information that can be matched, is the dimension of the key vector, is the modality-specific learnable parameter matrix obtained by learning the Q mapping, is the modality-specific learnable parameter matrix obtained by learning the K mapping.

[0020] The attention output can be expressed as: where V represents the value matrix, where is the modality-specific learnable parameter matrix, responsible for mapping the original features to the value space; the value matrix = is obtained by linearly transforming the feature representation of modality j through the learnable parameter matrix .

[0021] The aggregation of cross-modal attention is expressed as: The value matrix plays the role of an information content provider in the attention mechanism. When the similarity calculation between the query matrix Q and the key matrix K is completed and the attention weights are generated, these weights are applied to the value matrix to obtain the actual information content that needs to be extracted finally. The aggregation of cross-modal attention contains the interaction information between modalities and has richer semantics compared to the original feature representation .

[0022] Then, a dynamic weight allocation mechanism with clinical time window constraints is introduced to perform weighted fusion on the features of each modality: Among them is a scoring function related to the clinical time window and is the dynamic weight of the th modality, which is used to dynamically adjust the importance of each modality according to the constraints of the time window in different clinical scenarios. For example, during the golden 90-minute treatment period for patients with myocardial infarction, the weight of electrocardiogram data may be automatically increased by the system, while at other time points, other vital sign data may obtain a higher weight.

[0023] The final multi-modal fusion feature is represented as: This fusion feature will be used as the input of the subsequent decision-making model to obtain the first-aid content.

[0024] Federated learning optimization: Build a federated learning platform in the hospital internal data center. Each department node uploads the encrypted model gradients (instead of the original data) to regularly update the global fusion model, meeting the data privacy requirements of the "Measures for the Management of Medical and Health Data". The real-time guarantee mechanism uses DDS (Data Distribution Service) technology to achieve millisecond-level data synchronization, with the end-to-end delay < 200 ms. Design a double-buffer queue: Emergency data (such as ventricular fibrillation waveforms) triggers interrupt priority scheduling to ensure that the processing delay of key information < 50 ms.

[0025] The main control unit 10 generates the predicted first-aid content based on the clinical decision-making engine generated by the intelligent first-aid plan, including the construction of a dynamic knowledge graph. First, multi-source knowledge fuses multiple databases, including a basic rule library: integrating authoritative guidelines such as UpToDate clinical guidelines, AHA cardiopulmonary resuscitation procedures, and "Expert Consensus on the Diagnosis and Treatment of Acute Poisoning in China" to construct a decision tree containing more than 8,000 medical rules; the hospital experience library: parsing more than 20,000 first-aid medical records in the past 5 years through natural language processing technology to extract the hospital's characteristic diagnosis and treatment paths (such as the thrombolysis time window adjustment rules for myocardial infarction patients in plateau areas). Real-time evidence update of multiple databases: Connect to PubMed Clinical Queries to automatically synchronize the latest clinical research evidence (evidence level ≥ 2A) every week.

[0026] Then, strengthen the learning decision model, using the change rate of the patient's APACHE II score, lactate clearance rate, etc. as state variables, and design a reward function for the first-aid measure sequence. Use the PPO (Proximal Policy Optimization) algorithm to train the decision model to simulate the impact prediction of different first-aid plans on the 28-day mortality rate.

[0027] The main control unit 10 generates the first-aid content through the first-aid decision model, and the mathematical expression is as follows: The main control unit 10 defines the state space , including various physiological indicators and clinical parameters of the patient, such as multi-dimensional feature vectors like APACHE II score, lactate level, hemodynamic parameters, etc. The action space , representing a discrete set of clinical intervention measures, including drug administration regimens, surgical interventions, life support measures, etc. The reward function , quantifies the medical value of performing action A in state S, usually mapped to patient prognosis indicators.

[0028] In the state space of physiological indicators and clinical parameters, based on the input feature Z, in the action space , select an action and update the policy according to R(s,a). The Z obtained by multi-modal fusion is used as the input feature of the first aid decision-making model; at the same time, the S composed of the patient's physiological indicators and clinical parameters is used as the "state" in reinforcement learning; in state S, based on the input feature Z, select action A and update the policy according to R(s,a).

[0029] The first aid decision-making is modeled as a Markov decision process (MDP), and the goal is to find the optimal policy , which maximizes the expected cumulative reward: where is the discount factor, a parameter that adjusts the trade-off between short-term and long-term rewards, reflecting the clinical time window constraint; t is the sequence position identifier: t starts from 0 and ends at T, and each t corresponds to a specific time point in the first aid process; is the decision sequence length; is the policy for making decisions in different states; is the expected value under policy π. This formula is mainly used to solve the optimal policy. The optimal policy represents the best action plan that the system should take, equivalent to an "ideal first aid guide".

[0030] Use the PPO algorithm to update the policy of the first aid decision-making by optimizing the objective function: where are the parameters of the first aid decision-making model, is the probability ratio of the new and old policies, is the advantage function estimate, is the clipping function, is the clipping parameter, is the empirical expectation at time t.

[0031] The code for visualizing the decision-making process is as follows: # Acute Chest Pain Patient Decision Pseudocode def chest_pain_decision(patient_data): if ecg_has_st_elevation(patient_data): # Symptom onset time calculation if time_since_onset < 120min: # Catheterization laboratory status query if can_perform_pci(patient_data): # Initiate pre-PCI preparation process return generate_pci_protocol() else: # Thrombolysis protocol compliant with the "STEMI Diagnosis and Treatment Guidelines" return generate_thrombolysis_protocol() else: return stable_angina_management() else: # Dynamic monitoring of myocardial injury markers if troponin > 99th_percentile: return nstemi_treatment() else: # Initiate other etiological investigation processes return rule_out_acs() Further, the main control unit 10 is also connected to a dedicated telephone line unit 60, which establishes dedicated telephone lines with the consulting physicians, pharmacy, and blood bank respectively for one-key voice calls. The dedicated telephone line unit 60 selects one of China Mobile / China Unicom / China Telecom as the operator to open a dedicated voice telephone line for first aid, sets up a one-key call consultation system - "Cloud Link", embeds the hospital emergency numbers and call templates, and supports one-key voice telephone calls to consult physicians / pharmacy / blood bank. The first aid plan lists the consultation matters that need to be urgently called. Standardized help-seeking text messages and enterprise WeChat notifications are pushed simultaneously according to different needs. For example, when calling the anesthesiology department for endotracheal intubation: Hello, the patient in bed 12 of ward 222 needs urgent endotracheal intubation, please go immediately; For the clinical department: Hello, the patient in bed 12 of ward 222 needs an emergency consultation, please go immediately; For the blood bank: Hello, the patient in bed 12 of ward 222 needs urgent blood transfusion. The patient's blood type is A. The blood cross-match should be sent immediately. Please prepare the blood and deliver it as soon as possible; When calling 100: Hello, the patient in bed 12 of ward 222 needs to send an emergency blood sample, please go immediately; For the pharmacy: Hello, the patient in bed 12 of ward 222 needs 5 norepinephrine, please deliver immediately, etc. Thus, it realizes the rapid activation of multi-disciplinary team cooperation for rescue.

[0032] The medication advice can give the name of the drug that the patient currently needs to use, as well as the location information of the drug shown according to the cross-linking unit 50, which is convenient for quick access. The dosage recommendation of the drug is automatically calculated according to the weight. Medication warnings such as "Taking warfarin, use heparin with caution" and other words are provided. When the drug is out of stock, the nearest storage point is displayed, such as "There are 10 tablets in the emergency pharmacy" and other words.

[0033] The integrated touch display unit 20 is physically bound to the mobile rescue vehicle and can be adjusted at multiple angles.

[0034] The drug dosage recommendation is obtained by calculating according to the patient's weight, the medication warning is obtained according to the patient's detailed information, and the nearest drug storage point is obtained according to the position information of the corresponding drug by the cross-linking unit 50.

[0035] The display unit 20 displays the standard first aid process step by step on the screen according to the first aid plan, supports touch zooming of the screen, and highlights the guidance step by step on the screen.

[0036] The main control unit 10 generates a first aid record according to the first aid steps along the time axis; records the medication time and defibrillation time as key nodes on the time axis, and records going out for examinations, transfer between departments, etc. on the time axis. Each stage of the time axis refers to the following table.

[0037] Time axis granularity of the time axis: Precise to the operation time (millisecond level) + operation subject (badge swiping authentication) + device number (unique medical device code), e.g., 2025-04-30T14:23:15.452+08:00|Nurse A|Bedside monitor#CN2024001|SpO2 increased from 88% to 94%.

[0038] Typical scenarios for forensic identification include verification of the compliance of cardiopulmonary resuscitation operations: calculating key indicators such as the compression frequency (whether it is between 100 - 120 times per minute) and interruption time (whether it is > 10 seconds) through time axis data. Determination of medication timing disputes: accurately tracing the time sequence of "the first intravenous injection of adrenaline" and "completion of defibrillator charging" to resolve disputes in the rescue process.

[0039] After the interaction unit 40 records the current condition of the patient dictated by the doctor, the main control unit 10 parses the voice command of the current condition of the patient and generates a standardized electronic medical order. The nurse confirms the execution of the electronic medical order through the touch screen of the display unit 20, and the main control unit 10 records the execution of the electronic medical order by the nurse in the time axis.

[0040] The main control unit 10 records the voiceprints of each doctor and nurse, and the interaction unit 40 distinguishes between doctors and nurses through voiceprint recognition. For example, the nurse says 09:25, intravenous injection of 1 mg of adrenaline, 09:30, 200 J biphasic defibrillation once, etc.; then the interaction unit 40 converts the content spoken by the corresponding nurse from voice to text, and the main control unit 10 generates a rescue time axis with timestamps in real time, which is convenient for supplementing structured data after the rescue, and can also verify the data integrity in combination with time logic.

[0041] After the first aid is over, the main control unit 10 generates a first aid file, which is used to automatically organize the first aid process according to the time axis, verify the integrity of the process data in combination with time logic; generate a first aid report that complies with medical norms; trigger subsequent processes according to the first aid results. After the rescue is completed, click the rescue completion button, and then the main control unit 10 automatically organizes the rescue timeline, verifies the data integrity in combination with time logic, generates a first aid report that complies with medical norms, and after being reviewed and confirmed by medical staff. An invitation for consultation form is automatically generated according to the consultation call and consultation opinions, which is consistent with the invitation for consultation form in the current electronic medical record system, and allows doctors to import and modify it with one click. Finally, the whole process data of the first aid is encrypted and archived. Subsequent rescue processes include automatically filling in transfer to another department, death certificate, etc. The main control unit 10 also traces the first aid process according to the first aid file and automatically marks the links to be improved, such as "the defibrillation preparation was delayed by 50 seconds".

[0042] The main control unit 10 is also connected to an evaluation unit 70. The evaluation unit 70 presets a complete set of first aid items C = [C1, C2...Ci], where Ci represents the i-th first aid item. Each element of the complete set of first aid items C is set with an importance score Si, and Si represents the importance score of the i-th first aid item. The evaluation unit 70 obtains each first aid item on the time axis according to the first aid record to form a content set P, P = [P1, P2..., Pi], where Pi represents the i-th first aid content on the time axis, and a time set T, T = [T1, T2..., Ti], where Ti represents the time spent on the i-th first aid content on the time axis. Map each element of the content set P to the complete set of first aid items C to obtain the importance score Si of each element of the content set P. Each element of the content set P obtains the weight B of each element according to the importance score Si, B = [B1, B2..., Bi], where Bi represents the weight of the i-th first aid content in the content set P. The evaluation unit obtains the evaluation score for the first aid items through the following process: Among them, Score represents the evaluation score, Bi represents the weight of the i-th first aid content in the content set P, and Ti represents the time spent on the i-th first aid content on the time axis.

[0043] To improve the accuracy and comparability of the evaluation, the time set T can be normalized, and the normalized time is defined as : Among them and are the minimum and maximum values of the time spent on all first aid items respectively. Then the normalized evaluation score is calculated as: Among them is a small constant used to avoid the denominator being zero.

[0044] Each first aid record finds the corresponding weight of the first aid item in the preset complete set of first aid items according to the first aid items on the time axis, so as to calculate the weights of each first aid item in this first aid record. The weight can be obtained by comprehensively calculating the scores of a certain first aid item in the first aid record accounting for all first aid items. The longer the time spent on a first aid item, the lower the score. The elements of the time set T can be normalized before participating in the calculation. Finally, the evaluation score of a certain first aid record is calculated, which is used to evaluate the score obtained from each first aid, including the scores of each first aid item, so that each item can be reviewed later to find the process that can be optimized. The evaluation unit calculates the average score of the first aid record through the following process; Among them, Represents the average score of first aid records, thus used to compare the scores of each first aid record and to judge the performance of medical staff in different first aid records.

[0045] To further improve the evaluation mechanism, a time sensitivity factor can be introduced , and weighted according to the timeliness requirements of different first aid items: Among them is the time sensitivity factor of the th first aid content. Higher values are given to time-sensitive operations (such as defibrillation, endotracheal intubation, etc.), and lower values are given to operations that are relatively less time-sensitive. Adds an evaluation dimension for evaluating and quantifying the time urgency dimension of each first aid procedure. For two equally important procedures with the same value, they may have completely different time sensitivity requirements: the original weight factor measures the general clinical importance of the procedure, while βi specifically targets the speed at which the procedure must be performed. The timeliness score can more reasonably evaluate the performance of medical staff in different first aid records.

[0046] The beneficial effects of this application include: 1. The rescue efficiency is significantly improved. 2. The management accuracy is enhanced: precise control of the entire rescue process is achieved. 3. The scientific nature of decision-making is guaranteed: process errors are reduced through the Clinical Decision Support System (CDSS). 4. Legal risk prevention and control: high time axis accuracy, digital signature and CA certification meet the requirements of forensic appraisal. 5. Resource integration and optimization: The communication response time related to rescue is shortened, and the data penetration rate of multiple systems is increased. 6. Quality control closed-loop: Quality control checkpoints are automatically triggered, and the problem tracing efficiency is improved. It can not only optimize the current rescue steps in real time, but also provide high-fidelity training data for subsequent quality improvement.

[0047] The above is only the preferred implementation mode of the present invention. The protection scope of the present invention is not limited to the above embodiments. All technical solutions within the idea of the present invention belong to the protection scope of the present invention. It should be noted that for those of ordinary skill in the art in this technical field, several improvements and refinements made without departing from the principle of the present invention should also be regarded as within the protection scope of the present invention.

Claims

1. An intelligent first-aid decision-making system based on multimodal fusion, characterized in that, It includes a main control unit (10), and the main control unit (10) is connected to a display unit (20), a connection unit (30), an interaction unit (40), and a cross-linking unit (50); The main control unit (10) stores the basic information and detailed information of the patient. The connection unit (30) obtains the patient's vital sign monitoring data through medical instruments. The display unit (20) is used to display the patient's basic information and the patient's vital sign monitoring data. The interaction unit (40) is used to receive and record the current condition of the patient dictated by the doctor. The cross-linking unit (50) is used to connect to the hospital Internet of Things to obtain drug information; When the main control unit (10) determines whether there are health hazards in the patient based on the patient's vital sign monitoring data, if it is determined that there are health hazards, the main control unit (10) predicts possible diseases based on the patient's vital sign monitoring data and medication information, and generates predicted first aid content for each predicted disease; When the doctor determines that the patient has a disease and needs first aid, the predicted first aid content is called through the main control unit (10) for the doctor's reference. If the predicted first aid content has the patient's disease, the doctor calls the predicted first aid content for the corresponding disease. If the predicted first aid content does not have the patient's disease, the doctor inputs the patient's current condition and disease through the interaction unit (40), and the main control unit (10) generates the first aid content online; The first aid content includes disease analysis, first aid plan, medication advice, drug dose recommendation, medication warning, and the nearest drug storage point.

2. The intelligent first aid decision-making system based on multimodal fusion according to claim 1, wherein The main control unit (10) generates the first aid content through a multi-modal fusion algorithm; There are preset input data of different modalities , where represents the data of the th modality; Each modal data is converted into a feature representation through its respective feature extraction network: wherein is the feature representation of the th modality, is the feature representation of the th modality, and represent the feature extraction network for the corresponding modality data; The cross-modal attention mechanism in the Transformer architecture is used to process the interaction between different modalities. For the attention calculation between modality and modality can be expressed as: Among them is a query matrix for extracting query features from the current modality, is a key matrix for storing key information that can be matched, is the dimension of the key vector, is a modality-specific learnable parameter matrix obtained by learning the Q mapping, is a modality-specific learnable parameter matrix obtained by learning the K mapping; The attention output can be expressed as: where V represents a value matrix, where is a modality-specific learnable parameter matrix responsible for mapping the original features to the value space; The aggregated representation of cross-modal attention is: Introduce a dynamic weight allocation mechanism with clinical time window constraints to perform weighted fusion on each modal feature: Among them is a scoring function related to the clinical time window , and is the dynamic weight of the th modality, which is used to dynamically adjust the importance of each modality according to the constraint conditions of the time window in different clinical scenarios; The multi-modal fusion feature representation is: The fused feature is used as input to obtain the first aid content.

3. The intelligent first aid decision-making system based on multi-modal fusion according to claim 2, characterized in that The mathematical expression for the main control unit (10) to generate the first aid content is as follows: The main control unit (10) defines a state space , including various physiological indexes and clinical parameters of the patient; Action space , representing a discrete set of clinical interventions; Reward function , quantifying the medical value of performing action A in state S; In the state space of physiological indicators and clinical parameters, based on the input feature Z, select an action in the action space and update the policy according to R(s,a); Find the optimal policy through the following formula such that the expected cumulative reward is maximized: where is the discount factor, a parameter that adjusts the trade-off between immediate and future rewards and reflects the clinical time window constraint; t is the sequence position identifier: t starts from 0 and ends at T, and each t corresponds to a specific time point in the first aid process; is the length of the decision sequence; is the policy for making decisions in different states; is the expected value under policy π.

4. An intelligent first aid decision-making system based on multimodal fusion according to claim 3, characterized in that The drug dose recommendation is obtained by calculating according to the patient's weight. The medication warning is obtained according to the patient's detailed information. The nearest drug storage point obtains the location information of the corresponding drug according to the cross-linking unit (50).

5. The intelligent first-aid decision-making system based on multimodal fusion according to claim 3, wherein The display unit (20) displays the standard first aid process step by step on the screen according to the first aid plan, supports touch zooming of the screen, and highlights the guidance step by step on the screen.

6. The intelligent first-aid decision-making system based on multi-modal fusion according to claim 3, wherein The main control unit (10) is also connected to a dedicated telephone line unit (60). The dedicated telephone line unit (60) establishes dedicated telephone lines with the consulting physician, pharmacy, and blood bank respectively for one-key voice calling.

7. An intelligent first aid decision-making system based on multimodal fusion according to claim 4, characterized in that, The main control unit (10) generates a first aid record according to the first aid steps along the time axis; The medication time and defibrillation time are recorded as key nodes on the time axis, and records such as going out for examination and transfer to another department are also recorded on the time axis.

8. An intelligent first aid decision-making system based on multimodal fusion according to claim 7, characterized in that, After the interaction unit (40) records the current condition of the patient dictated by the doctor, the main control unit (10) parses the voice command of the current condition of the patient and generates a standardized electronic medical order. The nurse confirms the execution of the electronic medical order through the touch screen of the display unit (20), and the main control unit (10) records the execution of the electronic medical order by the nurse in the timeline.

9. The intelligent first-aid decision-making system based on multimodal fusion according to claim 8, characterized in that The main control unit (10) records the voiceprints of each doctor and nurse, and the interaction unit (40) distinguishes doctors and nurses through voiceprint recognition.

10. An intelligent first aid decision-making system based on multimodal fusion according to claim 7, characterized in that, The main control unit (10) is also connected to an evaluation unit (70). The evaluation unit (70) presets a complete set of first aid items A = [A1, A2... Ai], where Ai represents the i-th first aid item. Each element of the complete set of first aid items A is set with an importance score Si, and Si represents the importance score of the i-th first aid item. The evaluation unit (70) obtains a content collection P on the timeline according to the first aid records, P = [P1, P2..., Pi], where Pi represents the i-th first aid content on the timeline, a time collection T, T = [T1, T2..., Ti], where Ti represents the time spent on the i-th first aid content on the timeline. Map each element of the content collection P to the complete set of first aid items A to obtain the importance score Si of each element of the content collection P. Each element of the content collection P obtains the weight Q of each element according to the importance score Si, Q = [Q1, Q2..., Qi], where Qi represents the weight of the i-th first aid content in the content collection P. The evaluation unit (70) obtains the evaluation score of the first aid item through the following process: Among them, Score represents the evaluation score, Qi represents the weight of the i-th first aid content in the content collection P, and Ti represents the time spent on the i-th first aid content on the timeline.

Images