Intensive care system based on multi-modal channel fusion

Abstract

The invention discloses an intensive care system based on multi-modal channel fusion in the technical field of intensive care, and the system comprises an information collection module which is used for collecting the vital sign information, image information and sound information of a patient and collecting the pressure sensing information of the patient in posture and motion changes; the information processing module is used for processing the information acquired by the information acquisition module; the multi-modal channel fusion module is used for performing multi-modal fusion according to the processed information; the decision module is used for performing symptom judgment and intention classification according to the fused multi-modal information; and the response report module is used for reporting the result obtained by the decision module to the medical staff and controlling equipment in the ward to perform emergency treatment on the patient. The pressure sensing unit is used for detecting subtle motion changes of an intensive care patient, the symptom condition and intention of the patient are accurately judged in combination with the multi-modal fusion technology, the symptom condition and intention are reported to medical staff for timely processing, and personalized and refined monitoring services are provided for the patient.

Description

Technical Field

[0001] The present invention belongs to the technical field of intensive care, and in particular is an intensive care system based on multimodal channel fusion. Background Art

[0002] Deep learning multimodal fusion technology refers to the technology that machines obtain information from multiple fields such as text, images, voice, video, etc., realize information conversion and fusion, and thus improve model performance. The main goal of multimodal fusion technology is to reduce the heterogeneity differences between modalities while maintaining the integrity of the specific semantics of each modality and achieving optimal performance in the deep learning model.

[0003] Critical care technology is a technology developed to deal with postoperative complications and critical conditions of patients. The core of critical care technology lies in real-time monitoring of patients' vital signs. Modern intensive care units (ICUs) are equipped with various advanced monitoring equipment, such as electrocardiographs, ventilators, blood oxygen saturation monitors, blood pressure monitors, etc. These devices can monitor patients' heart rate, breathing, blood oxygen saturation, blood pressure and other vital signs in real time. In addition to monitoring equipment, critical care technology also includes a series of complex treatment methods, such as ventilator-assisted breathing, drugs or mechanical means to maintain blood pressure and cardiac output, anti-infection treatment, etc. The implementation of these treatment methods requires doctors to have rich clinical experience and superb technical operation capabilities.

[0004] In recent years, with the aging of the population, the proportion of elderly people in ICU wards has gradually increased. Most of these patients have underlying diseases, which can easily lead to a series of complications after entering the ICU. Their conditions are complex and changeable, requiring more accurate symptom detection, more refined treatment and care. Some diseases often cause limb twitching and shaking in seriously ill patients, such as cardiovascular and cerebrovascular diseases and neurological diseases. These subtle limb movements are easily blocked by the quilt, resulting in untimely symptom monitoring and inaccurate judgment of symptom conditions.

[0005] To this end, it is necessary to propose an intensive care system based on multimodal channel fusion that can detect subtle changes in the body movements of intensive care patients, use multimodal fusion technology to accurately judge changes in patients' symptoms, and perform weighted learning of important symptom representation information. Summary of the invention

[0006] In order to solve the above problems, the purpose of the present invention is to provide an intensive care system based on multimodal channel fusion, which detects subtle movement changes of intensive care patients through a pressure sensing unit, accurately judges the patient's symptoms and intentions in combination with multimodal fusion technology, reports them to medical staff for timely processing, and performs weighted learning on important symptom representation information to further improve the system's judgment accuracy and provide patients with personalized and refined monitoring services.

[0007] In order to achieve the above object, the technical solution of the present invention is as follows: an intensive care system based on multimodal channel fusion, comprising an information acquisition module, an information processing module, a multimodal channel fusion module, a decision module and a reaction reporting module,

[0008] An information collection module is used to collect vital signs, image information and sound information of critical care patients, as well as pressure perception information of critical care patients during posture changes and limb movement changes;

[0009] An information processing module is used to process the information collected by the information collection module, including image information classification, action recognition and synthesis, speech recognition, and patient emotion recognition and synthesis;

[0010] A multimodal channel fusion module, which is used to perform multimodal fusion based on the processed information, including information mapping, information alignment and information fusion;

[0011] The decision module is used to make symptom judgment and intention classification based on the fused multimodal information;

[0012] The response reporting module is used to report the results obtained by the decision-making module to the attending physician and nurse, and control the equipment response in the intensive care unit to provide emergency treatment to the patient.

[0013] The principle of the basic solution is: the system collects multi-modal information of the patient in real time through the information acquisition module, including collecting subtle changes in body movements through pressure sensing units with high sensor density and resolution; the information processing module pre-processes the collected information and extracts key features; the multi-modal channel fusion module fuses the processed information to form a comprehensive understanding of the patient's condition and intention; the decision-making module makes symptom judgments and classifies intentions based on the fused information, and reports the results to medical staff through the response reporting module; medical staff take timely rescue measures based on the report results, such as adjusting treatment plans and using first aid equipment.

[0014] The beneficial effects of the basic solution are: 1. The system can simultaneously collect and process the patient's vital signs, image information, sound information, and pressure perception information, so as to have a more comprehensive understanding of the patient's health status. Through the fusion of multimodal information, the system can more accurately judge the patient's symptoms and intentions, and reduce misdiagnosis caused by insufficient information or misleading single modality information.

[0015] 2. The system can monitor the patient's movements and vital signs in real time, integrate and understand multimodal information, and immediately report any abnormalities to medical staff so that they can respond quickly and improve monitoring efficiency. On this basis, through automated and intelligent monitoring methods, the system can replace some manual monitoring work, reduce the burden on medical staff, and enable them to focus more on the treatment and care of patients.

[0016] 3. The system can accurately identify the patient's symptoms, such as limb twitching and shaking, through subtle movement monitoring and multimodal information fusion, providing medical staff with complete patient symptom information and personalized monitoring to improve monitoring effectiveness.

[0017] 4. By real-time monitoring of the patient's vital signs and subtle changes in movements, the system can promptly detect and prevent possible complications, and report them in a timely manner to reduce the patient's risks.

[0018] Furthermore, it also includes a feedback statistics module for counting the multimodal fusion information and the results of the decision module, allocating the weight of the processed information according to the attention learning mechanism, and applying the results to the next operation of the multimodal channel fusion module.

[0019] The beneficial effects of the basic solution are: 1. The feedback statistics module can dynamically adjust the weights of different modal information in the fusion process according to historical data and current conditions, making the system more adaptable to individual differences and changes in the condition of patients. Through continuous learning and adjustment, the system can find the optimal multimodal fusion strategy and improve the accuracy and efficiency of information fusion.

[0020] 2. The feedback statistics module can identify and filter out irrelevant or noise information, improving the system's ability to extract useful information. When a patient has an abnormal condition, the system can respond quickly based on historical data and current information to reduce the occurrence of misdiagnosis and missed diagnosis.

[0021] 3. The statistical data and weight distribution results provided by the feedback statistics module can provide more accurate and comprehensive information support for the decision-making module and improve the accuracy of decision-making. Through continuous learning and adjustment, the system can provide patients with more personalized monitoring plans and improve monitoring effects.

[0022] 4. The feedback statistics module, multimodal channel fusion module, decision-making module, etc. form a closed-loop feedback system, which can continuously promote the optimization and improvement of the system. Through continuous learning and adjustment, the system can gradually improve its performance and provide patients with better intensive care services.

[0023] Furthermore, the information collection module includes a vital sign monitoring unit, a pressure sensing unit, an interactive recorder and several cameras.

[0024] Vital Signs Monitoring Unit, which is used to monitor the vital signs of critical care patients, including temperature, pulse, respiration, blood pressure, oxygen saturation, central venous pressure, and urine output;

[0025] Cameras for recording image information of patients in intensive care;

[0026] An interactive voice recorder for recording voice messages from critical care patients and providing voice responses to critical care patients’ questions;

[0027] The pressure sensing unit is used to detect the movements of critical care patients on the bed. The pressure sensing unit resolution can reach 0.1% FS.

[0028] The beneficial effects of the basic solution are: 1. The vital signs monitoring unit, camera, interactive recorder and pressure sensing unit collect information from different angles and dimensions, forming a comprehensive description of the patient's health status. This information can complement and verify each other, improving the accuracy and reliability of the information. Through automated and intelligent information collection methods, the system can reduce errors caused by improper operation or negligence of medical staff. At the same time, the system can also monitor and record data in real time, avoiding data omissions or erroneous recording.

[0029] 2. The information collection module can monitor the patient's vital signs and movement changes in real time. After processing and integration, this information can more comprehensively reflect the patient's current physical condition and intentions. Once an abnormality or potential risk is found, it will immediately issue an early warning to medical staff so that they can respond quickly. This greatly improves the efficiency of monitoring and reduces the risk of danger to patients. Through automated and intelligent information collection methods, the system can reduce the monitoring burden of medical staff, allowing them to focus more on the treatment and care of patients. At the same time, the system can also provide rich reference information to help medical staff make more accurate judgments.

[0030] Furthermore, the pressure sensing unit includes a number of pressure sensors distributed on the limbs, trunk and head of the intensive care patient's bed, among which the number of pressure sensors under the hands is greater than that in other parts, and the density of hand sensors is greater than 50,000 / m 2 .

[0031] The beneficial effects of the basic solution are: 1. As one of the most active parts of the human body, the subtle movements of the hand can often reflect the patient's physical state and emotional changes. Increasing the number of pressure sensors under the hand can more carefully capture hand movements, such as bending, stretching, and grasping of fingers, providing the system with more accurate and richer movement information.

[0032] 2. The increase in the number of pressure sensors under the hand provides the system with more dimensional information input. This information can be integrated with the information of other components such as the vital signs monitoring unit, camera, interactive recorder, etc. to form a more comprehensive and accurate patient health portrait, providing strong support for the innovation and development of critical care technology.

[0033] Further, the information processing module includes an image information classification unit, an action recognition unit, a speech recognition unit, and an emotion recognition unit;

[0034] An image information classification unit, used for preliminarily classifying the patient's action type according to the image information;

[0035] An action recognition unit, used to recognize and describe the patient's accurate actions based on the image information classification results and pressure perception information;

[0036] A speech recognition unit, used to recognize and describe speech content based on speech information;

[0037] The emotion recognition unit is used to recognize and describe the patient's emotion information based on the action recognition and voice recognition results combined with the patient's facial image information.

[0038] The beneficial effects of the basic scheme are: 1. Through the preliminary classification of image information, the patient's action type can be quickly identified, providing a basis for subsequent action recognition and description. This step helps to reduce the amount of calculation for subsequent processing and improve the overall information processing efficiency.

[0039] 2. Combining the image information classification results and pressure perception information, it can accurately identify and describe the patient's movements. This multi-dimensional information fusion makes movement recognition more accurate and helps medical staff better understand the patient's physical condition and activity.

[0040] 3. By recognizing and describing the voice content, the system can capture the patient's voice information and provide medical staff with direct feedback and communication content from the patient. This helps medical staff to understand the patient's needs and feelings in a timely manner and improve patient satisfaction.

[0041] 4. Combining the results of motion recognition and voice recognition, as well as the patient's facial image information, the system can identify and describe the patient's emotional information. This function helps medical staff better understand the patient's mental state, provide psychological support and intervention in a timely manner, and promote the patient's recovery.

[0042] Furthermore, the multimodal channel fusion module includes an alignment unit and a fusion unit.

[0043] an alignment unit for mapping and aligning processing information according to timing;

[0044] The fusion unit is used to fuse information based on the processing information at the two levels of original data and features combined with the attention mechanism weights of the feedback statistics module.

[0045] The beneficial effects of the basic scheme are: 1. Through alignment processing, information of different modalities is consistent in time and semantics, which helps the system understand and process information more accurately. The alignment unit can deal with noise and interference in the data and improve the system's adaptability to complex environments. The aligned data is easier to be processed and used by the fusion unit, thereby improving the effect and accuracy of information fusion.

[0046] 2. The fusion unit can integrate data from different modalities and extract more comprehensive and accurate feature representations to provide strong support for downstream tasks. By combining the weights of the attention mechanism, the fusion unit can more efficiently process and utilize multimodal information, reducing redundancy and unnecessary calculations. The fusion unit can learn the correlation and complementarity between different modalities, thereby improving the system's generalization ability on unseen data.

[0047] 3. By integrating the advantages of the two units of alignment and fusion, the multimodal channel fusion module can significantly improve the performance and expression of the system in complex environments, and make more accurate judgments on the physical state and intention expression of intensive care patients.

[0048] Furthermore, the fusion unit uses an autoencoder to encode and decode all modal information, calculate the loss between the modal information, and find the implicit internal correlation structure of the information.

[0049] The benefits of the basic scheme are: 1. The autoencoder can compress the input data into a latent space representation, which is often more compact and informative than the original data. The decoder can restore the latent space representation to the original data or its approximation, which ensures the effective transmission and recovery of information.

[0050] 2. By calculating the loss of different modal information during encoding and decoding, the differences and similarities between them can be quantified. This loss calculation helps to discover the structure and association hidden in the information, thus providing a basis for more accurate information fusion.

[0051] 3. The autoencoder can learn a robust representation of the data during training, that is, it can resist a certain degree of noise and interference. This makes the system more stable and reliable when processing complex data in practical applications.

[0052] 4. Through the training of the autoencoder, the system can learn the common features and representations between different modal data. This cross-modal learning capability enables the system to better adapt to unseen data or new modalities, thereby enhancing generalization capabilities.

[0053] 5. Through the compression and representation learning of the autoencoder, the redundant information between different modal data can be reduced. This helps to extract more critical and useful information, thereby improving the fusion effect and reducing the computational cost.

[0054] Furthermore, the decision module includes a symptom judgment unit and an intention classification unit.

[0055] A symptom judgment unit, used to judge the type and danger of the current patient's symptoms based on the fusion information;

[0056] The intention classification unit is used to classify and judge the current patient's emotions and intentions based on the fused information.

[0057] The beneficial effects of the basic scheme are: 1. The symptom judgment unit can comprehensively consider multiple sources of information, reduce the subjectivity and uncertainty in manual diagnosis, and thus improve the accuracy of diagnosis. Through automated analysis, the symptom judgment unit can provide preliminary diagnosis results in a short time, buy precious treatment time for patients, and provide doctors with objective and comprehensive diagnostic basis to assist doctors in formulating more reasonable treatment plans.

[0058] 2. By accurately identifying the patient's emotions and intentions, the system can provide more intimate and personalized services and improve the patient's medical experience. The intention classification unit can help doctors better understand the patient's needs and feelings, thereby enhancing communication and trust between doctors and patients. For patients with abnormal emotions, the system can promptly detect and provide psychological intervention suggestions to help patients adjust their emotional state.

[0059] 3. The collaborative work of the symptom judgment unit and the intention classification unit enables the decision-making module to have comprehensive decision-making capabilities and to more comprehensively assess the patient's health status and needs. By introducing advanced algorithms and models, the decision-making module improves the intelligence level of the intelligent medical system and provides strong support for the intelligent transformation of medical services.

[0060] Further, the reaction reporting module includes a reporting unit and a reaction unit,

[0061] A reporting unit is used to report to the attending physician or nurse on duty based on the patient's symptom judgment and intention classification results;

[0062] The response unit is used to control the equipment in the intensive care unit to treat patients' emergency symptoms.

[0063] The beneficial effects of the basic solution are: 1. The reporting unit can automatically generate detailed reports, reducing the time for medical staff to manually record and analyze, and improving the efficiency of information transmission. Through intelligent symptom judgment and intent classification, the reporting unit can provide more accurate patient information and reduce the possibility of human error. The comprehensive information provided by the reporting unit helps medical staff to more accurately assess the health status of patients and formulate more reasonable treatment plans. The reporting unit makes information sharing between medical staff more convenient and helps to enhance communication and collaboration between teams.

[0064] 2. The response unit can quickly respond to the patient's emergency symptoms and start the treatment equipment, thereby shortening the treatment time and improving the success rate of treatment. Through automated equipment control, the response unit can reduce the operating burden of medical staff in emergency situations and improve the efficiency of treatment. The response unit can select appropriate treatment equipment and parameters according to the patient's specific symptoms and needs to ensure the accuracy of treatment. In an emergency, the response unit can quickly provide necessary treatment measures to reduce the risk of patients suffering from untimely or improper treatment.

[0065] 3. The collaborative work of the reporting unit and the response unit enables the response reporting module to have comprehensive treatment capabilities and to fully respond to the patient's emergency symptoms. Through intelligent reporting and treatment processes, the response reporting module improves the overall quality and efficiency of medical services.

[0066] Furthermore, the feedback statistics module includes a statistics unit and an attention learning unit.

[0067] Statistical unit, used for statistical processing information, fusion information and decision module results;

[0068] An attention learning unit is used to assign weights of processing information based on the attention learning mechanism, and weight the relevant processing information according to the frequency of symptom occurrence and the urgency of the symptoms.

[0069] The beneficial effects of the basic scheme are: 1. The statistical unit can ensure the integrity of all processed information, fused information and decision results, and avoid omission or loss of data. Through accurate statistics of data, the statistical unit can reduce human errors and improve data accuracy. The comprehensive data provided by the statistical unit helps medical staff to more accurately evaluate the treatment effect and provide a scientific basis for subsequent decision-making.

[0070] 2. The attention learning unit can intelligently allocate the weight of processing information according to the urgency and frequency of symptoms, ensuring that important information is processed first. Through intelligent weight allocation, the system can identify and process key information faster and improve the overall information processing efficiency. According to the different symptoms of different patients, the attention learning unit can provide personalized attention plans to ensure that each patient can receive the most suitable treatment and attention.

[0071] 3. The collaborative work of the statistical unit and the attention learning unit enables the feedback statistical module to show significant comprehensive performance improvements in data processing, decision support, and personalized attention. Through intelligent weight allocation and comprehensive data statistics, the system can more reasonably allocate computing resources, obtain accurate judgment results, and ensure that patients receive timely and effective treatment. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 Schematic diagram of an intensive care system based on multimodal channel fusion in an embodiment of the present invention.

[0073] Figure 2 Schematic diagram of an information acquisition module of an intensive care system based on multimodal channel fusion in an embodiment of the present invention.

[0074] Figure 3 Schematic diagram of an information processing module of an intensive care system based on multimodal channel fusion in an embodiment of the present invention.

[0075] Figure 4 Schematic diagram of a multimodal channel fusion module of an intensive care system based on multimodal channel fusion in an embodiment of the present invention.

[0076] Figure 5 Schematic diagram of a decision module of an intensive care system based on multimodal channel fusion in an embodiment of the present invention.

[0077] Figure 6 Schematic diagram of a response reporting module of an intensive care system based on multimodal channel fusion in an embodiment of the present invention.

[0078] Figure 7 Schematic diagram of a feedback statistics module of an intensive care system based on multimodal channel fusion in an embodiment of the present invention. DETAILED DESCRIPTION

[0079] The following is further described in detail through specific implementation methods:

[0080] Example 1

[0081] Basically as attached Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6 and Figure 7 As shown: A critical care system based on multimodal channel fusion, characterized in that it includes an information acquisition module for collecting vital signs information, image information and sound information of critical care patients, as well as pressure perception information of critical care patients during posture changes and limb movement changes. The information acquisition module includes a vital signs monitoring unit for monitoring the vital signs of critical care patients, including body temperature, pulse, respiration, blood pressure, blood oxygen saturation, central venous pressure and urine volume; a camera for recording image information of critical care patients; an interactive recorder for recording voice information of critical care patients and making voice answers to questions of critical care patients; a pressure sensing unit for detecting the movements of critical care patients on the bed, the pressure sensing unit resolution can reach 0.1% FS, and the pressure sensing unit includes a number of pressure sensors distributed on the bed under the limbs, trunk and head of the critical care patient, wherein the number of pressure sensors under the hand is greater than that of other parts, and the hand sensor density is greater than 50,000 / m 2 .

[0082] The information processing module is used to process the information collected by the information collection module, including image information classification, action recognition and synthesis, speech recognition, and patient emotion recognition and synthesis. The information processing module includes an image information classification unit, which is used to preliminarily classify the patient's action type based on the image information; a motion recognition unit, which is used to recognize and describe the patient's accurate actions based on the image information classification results and pressure perception information; a speech recognition unit, which is used to recognize and describe the speech content based on the speech information; and an emotion recognition unit, which is used to recognize and describe the patient's emotion information based on the action recognition and speech recognition results combined with the patient's facial image information.

[0083] The multimodal channel fusion module is used to perform multimodal fusion based on the processed information, including information mapping, information alignment and information fusion. The multimodal channel fusion module includes an alignment unit, which is used to process information according to time series mapping and alignment; a fusion unit, which is used to perform information fusion based on the processed information at the two levels of original data and features combined with the attention mechanism weights of the feedback statistics module. The fusion unit uses an autoencoder to encode and decode all modal information, calculates the loss between modal information, and finds the implicit internal correlation structure of the information.

[0084] The decision module is used to make symptom judgment and intention classification based on the fused multimodal information. The decision module includes a symptom judgment unit, which is used to judge the type and dangerousness of the current patient's symptoms based on the fused information; and an intention classification unit, which is used to classify and judge the current patient's emotions and intentions based on the fused information.

[0085] The reaction reporting module is used to report the results obtained by the decision-making module to the attending physician and nurse, and control the equipment in the intensive care unit to respond to treat the patient. The reaction reporting module includes a reporting unit, which is used to report to the attending physician or the nurse on duty based on the patient's symptom judgment and intention classification results; the reaction unit is used to control the equipment in the intensive care unit to provide emergency treatment for the patient's emergency symptoms.

[0086] The feedback statistics module is used to count the multimodal fusion information and the results of the decision module, allocate the weight of the processing information according to the attention learning mechanism, and apply the result to the next operation of the multimodal channel fusion module. The feedback statistics module includes a statistics unit, which is used to count the processing information, fusion information and the results of the decision module; an attention learning unit, which is used to allocate the weight of the processing information based on the attention learning mechanism, and weight the relevant processing information according to the frequency of symptom occurrence and the urgency of the symptoms.

[0087] The specific implementation process is as follows: Patients in the intensive care unit generally refer to patients with serious conditions, unstable vital signs, and who need close monitoring and advanced medical care. These patients may be seriously injured or seriously ill to the point of being in danger of death. In this case, serious injuries and illnesses are prone to cause a series of complications, causing the injury to explode and affect life. The other type is the elderly, most of whom suffer from underlying diseases and have complex and changeable conditions, so they need to receive 24-hour uninterrupted monitoring and treatment from a professional medical team.

[0088] This system uses the information collection module to comprehensively collect the vital signs, image information, voice information and pressure perception information of patients in the intensive care unit, such as the patient's lying posture on the bed, the movement of the limbs, whether there is convulsion, etc. These information may be blocked and difficult to collect through the camera. After the information processing module processes and identifies the information of different modes and reduces the noise, the multimodal channel fusion module aligns this information according to the time sequence, and performs multimodal fusion, cuts redundant information, complements the integrity of information, and finds the connection between the patient's vital signs, movement information and expression changes, so as to improve the monitoring performance of this system and obtain more accurate fusion information. Based on the fusion information, the decision module determines the patient's current possible symptoms, including type and severity. These judgment results are reported to the attending physician and the nurse on duty through the reporting unit, and the patient's symptoms are treated urgently through the response unit in time to avoid missing the critical rescue time and increase the patient's survival rate. After judging the patient's symptoms, the early processing information, the later fusion information and the results of the decision module are recorded by the statistical unit and used as the basis for establishing the patient's diagnosis plan. The attention learning unit weights the related fusion information according to the frequency and severity of different symptoms to improve the accuracy of the next multimodal fusion information.

[0089] For example, a patient who also suffers from cardiovascular and cerebrovascular diseases may have various neurological disorders that make it difficult to communicate and lose the ability to move. Under such circumstances, the chest pain, palpitations, dyspnea, nausea and left arm pain caused by cardiovascular diseases are difficult to be discovered by medical staff through monitoring and observation in time. At this time, in addition to the changes in their vital signs index, due to the presence of discomfort, they may repeatedly adjust their posture on the bed, their limbs tremble due to pain, their facial expressions shrink and their faces turn pale, and they make incomprehensible groans, etc. All this information will be collected by the information collection module, and after being processed by the information processing module, it will enter the decision module through multimodal fusion to determine that it may be a symptom of cardiovascular disease, and then report it to the medical staff in time, call the medical staff to come to the ward or strengthen remote attention, and control the equipment in the intensive care unit for emergency treatment, such as increasing the oxygen supply of the oxygen supply machine, etc., so as to reduce the patient's mortality rate. When making symptom judgments, the relevant symptom information will be weighted by the attention learning unit due to the causes that have occurred and invested in the next symptom judgment, thereby improving the response speed and accuracy of the judgment of prone symptoms.

[0090] Example 2: A mentally ill patient who suffers from mania due to shock may have a tendency to self-harm. When he has been seriously injured due to self-harm and entered the intensive care unit, he may not be able to be restrained due to the injury, and sedatives are injected to relieve manic symptoms. During the patient's coma, the cause of mania may be revealed by the patient through body posture and voice, which means that the patient is still immersed in related memories, which may cause a relapse of mania during the next awakening time. The information acquisition module of this system can collect the patient's pressure perception information and voice information, and perform motion recognition and voice recognition. These processed information are multimodally fused and the correlation between the information and the patient's manic attack is judged, and the medical staff is notified in time to come to the ward to stop the patient's self-harm behavior to avoid further damage. Then, after the symptom onset, the relevant action and voice processing information is weighted through the attention learning unit to improve the accuracy and response speed of multimodal fusion judgment, and also provide a basis for doctors to specify personalized solutions for treating patients' diseases.

[0091] Example 2

[0092] The difference from the above embodiment is that the patient who enters the intensive care unit may have lost the ability to express and move, or even has fallen into a coma. During the process of waking up and expressing the patient's intention, his vital signs may change slightly, but his body posture and facial expression generally have more obvious changes, which are mainly concentrated in the movements of fingers, arms, eye opening, and lips. The system can monitor these subtle movements at all times and remind medical staff of the patient's status, avoiding the need for medical staff to monitor the patient all the time and reducing the pressure on the consumption of medical human resources.

[0093] When language communication is difficult, the information collection module collects changes in the patient's facial expressions and pressure perception changes in hand movements, performs motion analysis and emotion recognition processing, and hand pressure sensors with a large number of sensors can even record the movement trajectory of the patient's fingers, integrate the motion analysis and emotion recognition processing information, and judge the patient's intentions, which are then reported to medical staff who enter the ward to ask the patient's exact intentions, obtain the patient's confirmation, and then help the patient complete it, such as needing to see someone, etc. This is conducive to promoting humanistic care in the medical care system and maintaining the dignity of intensive care patients.

[0094] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0095] The above is only an embodiment of the present invention. The common sense such as the known specific structure and characteristics in the scheme is not described in detail here. The ordinary technicians in the relevant field know all the common technical knowledge in the technical field of the invention before the application date or priority date, can obtain all the existing technologies in the field, and have the ability to apply the conventional experimental means before that date. The ordinary technicians in the relevant field can improve and implement this scheme in combination with their own abilities under the enlightenment given by this application. Some typical known structures or known methods should not become obstacles for ordinary technicians in the relevant field to implement this application. It should be pointed out that for those skilled in the art, without departing from the structure of the present invention, several deformations and improvements can be made, which should also be regarded as the protection scope of the present invention, which will not affect the effect of the implementation of the present invention and the practicality of the patent. The protection scope required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. An intensive care system based on multimodal channel fusion, characterized in that: It includes information collection module, information processing module, multimodal channel fusion module, decision module and reaction reporting module. An information collection module is used to collect vital signs, image information and sound information of critical care patients, as well as pressure perception information of critical care patients during posture changes and limb movement changes; An information processing module is used to process the information collected by the information collection module, including image information classification, action recognition and synthesis, speech recognition, and patient emotion recognition and synthesis; A multimodal channel fusion module, which is used to perform multimodal fusion based on the processed information, including information mapping, information alignment and information fusion; The decision module is used to make symptom judgment and intention classification based on the fused multimodal information; The response reporting module is used to report the results obtained by the decision-making module to the attending physician and nurse, and control the equipment response in the intensive care unit to provide emergency treatment to the patient.

2. The intensive care system based on multimodal channel fusion according to claim 1, characterized in that: It also includes a feedback statistics module, which is used to count the multimodal fusion information and decision module results, allocate weights for processing information according to the attention learning mechanism, and apply the results to the next run of the multimodal channel fusion module.

3. The intensive care system based on multimodal channel fusion according to claim 2, characterized in that: The information collection module includes a vital sign monitoring unit, a pressure sensing unit, an interactive recorder and several cameras. Vital Signs Monitoring Unit, which is used to monitor the vital signs of critical care patients, including temperature, pulse, respiration, blood pressure, oxygen saturation, central venous pressure, and urine output; Cameras for recording image information of patients in intensive care; An interactive voice recorder for recording voice messages from critical care patients and providing voice responses to critical care patients’ questions; The pressure sensing unit is used to detect the movements of critical care patients on the bed. The pressure sensing unit resolution can reach 0.1% FS.

4. The intensive care system based on multimodal channel fusion according to claim 3, characterized in that: The pressure sensing unit includes several pressure sensors distributed on the limbs, trunk and head of the intensive care patient's bed. The number of pressure sensors under the hands is greater than that in other parts, and the density of hand sensors is greater than 50,000 / m 2 .

5. The intensive care system based on multimodal channel fusion according to claim 4, characterized in that: The information processing module includes an image information classification unit, an action recognition unit, a speech recognition unit, and an emotion recognition unit; An image information classification unit, used for preliminarily classifying the patient's action type according to the image information; An action recognition unit, used to recognize and describe the patient's accurate actions based on the image information classification results and pressure perception information; A speech recognition unit, used to recognize and describe speech content based on speech information; The emotion recognition unit is used to recognize and describe the patient's emotion information based on the action recognition and voice recognition results combined with the patient's facial image information.

6. The intensive care system based on multimodal channel fusion according to claim 5, characterized in that: The multimodal channel fusion module includes an alignment unit and a fusion unit. an alignment unit for mapping and aligning processing information according to timing; The fusion unit is used to fuse information based on the processing information at the two levels of original data and features combined with the attention mechanism weights of the feedback statistics module.

7. The intensive care system based on multimodal channel fusion according to claim 6, characterized in that: The fusion unit uses an autoencoder to encode and decode all modal information, calculate the loss between modal information, and find the implicit internal correlation structure of the information.

8. The intensive care system based on multimodal channel fusion according to claim 7, characterized in that: The decision module includes a symptom judgment unit and an intention classification unit. A symptom judgment unit, used to judge the type and danger of the current patient's symptoms based on the fusion information; The intention classification unit is used to classify and judge the current patient's emotions and intentions based on the fused information.

9. The intensive care system based on multimodal channel fusion according to claim 8, characterized in that: The reaction reporting module includes a reporting unit and a reaction unit. A reporting unit is used to report to the attending physician or nurse on duty based on the patient's symptom judgment and intention classification results; The response unit is used to control the equipment in the intensive care unit to treat patients' emergency symptoms.

10. The intensive care system based on multimodal channel fusion according to claim 9, characterized in that: The feedback statistics module includes a statistics unit and an attention learning unit. Statistical unit, used for statistical processing information, fusion information and decision module results; An attention learning unit is used to assign weights of processing information based on the attention learning mechanism, and weight the relevant processing information according to the frequency of symptom occurrence and the urgency of the symptoms.

Images