Unmanned intelligent triage monitoring system and method based on multi-modal data fusion

Abstract

According to the unmanned intelligent triage monitoring system and method based on multi-modal data fusion provided by the invention, the accuracy and real-time performance of wounded evaluation are improved. The method comprises the steps that firstly, vital signs, head data, cardiopulmonary data, eye data and overall state data of a wounded person are collected in real time through a plurality of hardware devices, and data analysis is conducted through an advanced deep learning algorithm; then, on the basis of the collected multi-modal data, injury condition evaluation is automatically carried out, treatment priorities are divided, and unmanned intelligent injury detection and real-time monitoring are achieved; and finally, through an efficient intelligent decision-making module, an accurate treatment instruction set is provided, and medical personnel are helped to make an optimal decision in first aid. Through efficient fusion and artificial intelligence analysis of multi-modal data, the accuracy, the response speed and the efficiency of wounded rescue are remarkably improved, and the method can be widely applied to high-risk environments such as disaster rescue and battlefield first aid.

Description

Technical Field

[0001] The present invention relates to the field of unmanned intelligent triage and monitoring, and in particular to an unmanned intelligent triage and monitoring system and method based on multimodal data fusion. Background Art

[0002] In modern emergency medicine, triage systems are an essential component of disaster relief, battlefield first aid, and emergency medical services. Traditional triage systems typically rely on the experience of medical staff to determine the priority level of treatment for the injured and use a single vital sign collection device for monitoring. However, with the increasing complexity of emergency needs and the improvement in rescue efficiency requirements, traditional methods can no longer meet the needs for fast, accurate, and comprehensive casualty assessments.

[0003] Existing technologies are highly dependent on manual labor, easily affected by subjective factors, and inefficient. In addition, the data is single, and existing technologies only collect a single vital sign, lacking the ability to conduct multi-dimensional injury correlation analysis. In addition, due to insufficient real-time performance, the changes in the status of critically injured patients cannot be dynamically tracked, delaying the time for treatment. In addition, due to the limitations of hardware equipment, data collection equipment for special parts such as the eyes and intracranial blood vessels has poor portability and is difficult to adapt to extreme environments. Summary of the Invention

[0004] In order to solve the technical problems mentioned in the current background technology, the present invention proposes an unmanned intelligent injury detection and monitoring system and method based on multimodal data fusion.

[0005] To this end, the technical solution adopted in the present invention is as follows:

[0006] An unmanned intelligent triage monitoring system based on multimodal data fusion, characterized in that the system comprises:

[0007] The data acquisition module collects multimodal data sets of the injured and sick, including basic information, head data, cardiopulmonary data, vital signs data, and overall status data. During the head data acquisition process, the module integrates VR equipment to play customized soothing audio and video. The module also stores the multimodal data sets by building an independent disk.

[0008] A processing module performs preprocessing operations on the multimodal dataset, including data evaluation, screening, and quality control, to generate a usable dataset, and outputs fused global features through feature extraction and feature fusion; a database is constructed based on machine learning and deep learning algorithms, and a unique identifier containing the multimodal dataset and global features is intelligently generated;

[0009] The intelligent analysis and decision-making module constructs an unmanned intelligent triage and monitoring model based on the unique identifier and performs iterative training. The trained unmanned intelligent triage and monitoring model automatically analyzes the key information of the injured and patients, divides the treatment priorities, and simultaneously performs triage classification, injury association reasoning and ranking, and real-time monitoring, and outputs a treatment instruction set.

[0010] Furthermore, the basic information X includes the age, gender, height, and weight of the patient. The age and gender information of the patient is collected through automatic recognition technology, and the height and weight information of the patient is measured through smart wearable devices.

[0011] The collection of the head data A includes brain and eye data collection and virtual reality assisted psychological intervention. The brain data is collected through a non-invasive EEG signal collection device and a wearable ultrasound patch, and the eye data is collected through a high-sensitivity camera, a pressure sensor and a detachable eye-opening assist device.

[0012] The cardiopulmonary data B includes cardiac audio data and lung audio data, which are collected by a split audio collector, which includes a precordial collector and a lung collector.

[0013] The vital signs data C include body temperature, pulse, blood pressure and blood oxygen saturation data, which are collected synchronously by an integrated arm-type integrated sensor.

[0014] The overall status data D includes the appearance, expression, movement, limb movement, special behavior, body surface condition and wound condition of the injured person, which is collected by a high-resolution RGB camera.

[0015] Furthermore, the non-invasive EEG signal acquisition device is fixed on the patient's head, and collects EEG signal data through multi-channel electrodes to generate a non-invasive EEG image A1. The wearable ultrasound patch collects 3D intracranial vascular ultrasound data A2 of the patient in real time.

[0016] The highly sensitive camera collects pupil, eye movement trajectory, and fundus hemodynamic images to generate eye images A3. The pressure sensor calculates the intraocular pressure value by sensing the shape changes of the eyeball to obtain intraocular pressure data A4. The detachable eye-opening assistance device is activated by receiving instructions via Bluetooth, and uses a gear-driven paddle to perform a 30 to 45° progressive eyelid pressure and support operation on the injured person with closed eyes, assisting the injured person to open his eyes for eye data collection.

[0017] Furthermore, the precordial collector is attached to the precordial area of ​​the patient, and collects the patient's heart sounds and heart rhythm information in real time to generate the heart audio data B1; the lung collector is attached to the patient's bilateral lungs, and collects the patient's normal respiratory sounds, pathological lung rales, dry and wet rales, and murmurs in real time to generate the lung audio data B2;

[0018] The arm-type integrated sensor integrates body temperature, pulse, blood pressure and blood oxygen saturation sensors into a portable device to collect vital sign data C of the injured and sick in real time;

[0019] The basic information, head data, cardiopulmonary data, vital signs data and overall status data of the injured and sick are collected to output the multimodal dataset Z, which is expressed as:

[0020] Z={X,A,B,C,D}

[0021] Based on the category of the multimodal data, the multimodal data set is divided into text data, image data and video data; the text data H includes A2 and A4, the image data I includes A1, A3, C, and D, and the video data J includes B;

[0022] The independent disk stores the multimodal dataset using an AES-256 encryption method.

[0023] Furthermore, the data evaluation is to perform a quality evaluation on the data of each modality, and the formula is:

[0024]

[0025] Where SNR is the signal-to-noise ratio; P signal is the signal power; P noise is the noise power; when the signal-to-noise ratio is lower than the signal-to-noise ratio threshold, the data quality of this group of modalities is unqualified and needs to be re-collected;

[0026] The data screening removes outliers in each set of modal data by using the Z-score standardization method;

[0027] The data quality control is to correct the multimodal data through a bandpass filter.

[0028] Furthermore, the feature extraction includes text data feature extraction, image data feature extraction and video data feature extraction.

[0029] The text data features are extracted by word embedding technology, which maps each text data into a dense vector space to become a vector. In the vector space, the vector distance between the text data is the relationship between the text data. Through the mapping of the relationship, the text data is converted into text data features H h ;

[0030] The image data features are extracted through a convolutional neural network. A sliding convolution kernel is used to perform a convolution operation on the image data, and low-level, mid-level, and high-level features are extracted in sequence. The formula is:

[0031]

[0032] Among them, I i is the low-level image data feature; i and j are the row and column indexes of the low-level image data feature respectively; I is the input image data; O is the convolution kernel; m and n are the row and column indexes of the convolution kernel respectively; M and N are the sizes of the convolution kernel;

[0033] In the second convolutional layer, I becomes the output of the first convolutional layer, i.e., the low-level image data feature I i , I i Becomes intermediate image data feature I m , until we get the high-level image data feature I h ;

[0034] The video data features include spatial features and temporal features. Each frame in the video data is an independent image. The spatial feature J1 is extracted by the image data feature extraction method. The temporal feature is extracted by the optical flow method. The formula is:

[0035]

[0036] Among them, u and v are the optical flows in the x and y directions respectively; is the time feature; and are the brightness gradients in the x and y directions respectively;

[0037] The spatial and temporal features of the video data are combined to output the video data feature J h .

[0038] Furthermore, the feature fusion is performed through the self-attention mechanism of the Transformer algorithm, and the formula is:

[0039]

[0040] Among them, Attention (Q, K, V) is the fused global feature; Q is the query, which represents the data feature; K is the key, which represents the label of the data feature; V is the value, which represents the actual information of the data feature; d k is the dimension of the key; T is the transpose operation of the key K;

[0041] By calculating the relationship between the query and the key and using the value to perform weighted fusion, the fused global feature is finally obtained.

[0042] The unique identifier is obtained by performing hash calculation on the multimodal dataset and global features of the patient.

[0043] Furthermore, the iterative training process is training-evaluation-validation-retraining-reevaluation-revalidation.

[0044] The key information of the injured and sick patients includes vital sign data, trauma characteristics and psychological status;

[0045] The triage classification extracts global features and associates them with the multimodal dataset of the injured and sick. The unmanned intelligent triage monitoring model assesses the injury status and outputs four-level classification labels of the injured and sick based on the GCS scoring mechanism, including red, yellow, green and black labels.

[0046] The injury association reasoning and ranking is to rank the injury diagnosis based on the multimodal data set of the injured and sick, and give the probability of the disease suffered by the current injured and sick;

[0047] The real-time monitoring monitors the vital signs data and trauma conditions of the injured and sick, and predicts the vital signs trend of the injured and sick based on the LSTM algorithm. The classification labels of the injured and sick are updated based on the vital signs trend, and the priority of treatment is automatically adjusted according to the updated classification labels.

[0048] Based on the injury classification, injury association reasoning and ranking, and real-time monitoring, a treatment instruction set is generated.

[0049] An unmanned intelligent triage and monitoring method based on multimodal data fusion, characterized in that the method comprises:

[0050] Collect multimodal data sets of the injured and sick, including basic information, head data, cardiopulmonary data, vital signs data, and overall status data. During the head data collection process, integrate VR equipment to play customized soothing audio and video; and construct an independent disk to store the multimodal data sets.

[0051] Preprocessing the multimodal dataset to generate a usable dataset, and outputting fused global features through feature extraction and feature fusion; building a database based on machine learning and deep learning algorithms, and intelligently generating a unique identifier that includes the multimodal dataset and global features;

[0052] An unmanned intelligent triage and monitoring model is constructed based on the unique identifier and iterative training is performed. The trained unmanned intelligent triage and monitoring model analyzes key information of the injured and sick, divides treatment priorities, simultaneously performs triage classification, injury association reasoning and ranking, and real-time monitoring, and outputs a treatment instruction set.

[0053] Compared with the prior art, the advantages of the present invention are:

[0054] 1. Unmanned operation: This invention uses automated hardware equipment and artificial intelligence algorithms to achieve unmanned operation of the entire process from data collection to decision output, significantly reducing manual intervention and improving treatment efficiency.

[0055] 2. Multimodal data fusion: This invention not only collects conventional monitoring information such as cardiopulmonary data and vital signs data, but also integrates data collection from special parts such as EEG signals, intraocular pressure, and eye movement trajectories, and performs comprehensive analysis through deep learning algorithms. This multimodal data fusion significantly improves the accuracy and comprehensiveness of patient assessments.

[0056] 3. Hardware innovation: The hardware design of the present invention fully considers portability and adaptability. Through the detachable eye-opening auxiliary device, etc., it can efficiently complete data collection in harsh environments, solving the problem of inconvenient hardware use. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0058] Figure 1 This is a flow chart of the unmanned intelligent triage and monitoring system of the present invention;

[0059] Figure 2 Schematic diagram of the head data detection device of the present invention;

[0060] Figure 3 Schematic diagram of the wearable ultrasound patch of the present invention;

[0061] Figure 4 Schematic diagram of the inside view of the head data acquisition device of the present invention;

[0062] Figure 5 This is a schematic diagram of a split audio collector of the present invention;

[0063] Figure 6 This is a schematic diagram of the arm-type integrated sensor of the present invention;

[0064] Figure 7 Schematic diagram of a high-resolution RGB camera of the present invention. DETAILED DESCRIPTION

[0065] To achieve the above objectives, the present invention is implemented through the following technical solutions: the present invention provides an unmanned intelligent injury detection and monitoring system based on multimodal data fusion, please refer to Figures 1 to 7 , the system comprises:

[0066] M1, data acquisition module, collects multimodal data sets of the injured and sick, which include basic information, head data, cardiopulmonary data, vital signs data and overall status data. During the head data acquisition process, the VR device is integrated to play customized soothing audio and video, and the multimodal data sets are stored by building an independent disk.

[0067] Basic information X includes the patient's age, gender, height, and weight. The patient's age and gender information is collected through automatic recognition technology, and the patient's height and weight information is measured in real time through smart wearable devices.

[0068] The collection of head data A includes brain and eye data collection and virtual reality assisted psychological intervention.

[0069] Brain data is collected through a non-invasive EEG signal acquisition device and a wearable ultrasound patch. The non-invasive EEG signal acquisition device is fixed to the patient's head, and EEG signal data is collected through multi-channel electrodes to generate a non-invasive EEG image A1. EEG signals are generated by neuronal activity in the cerebral cortex, with a frequency range of 0.5-100Hz. EEG signals can reflect the patient's state of consciousness, brain damage, and severity. The wearable ultrasound patch collects 3D intracranial vascular ultrasound data A2 in real time to determine whether the brain's blood supply is normal and detect potential intracranial hemorrhage.

[0070] Eye data is collected using a highly sensitive camera, pressure sensor, and detachable eye-opening assistive device placed in the eye area. The highly sensitive camera tracks pupils, eye movement trajectories, and fundus hemodynamic changes to generate an eye image (A3). The pressure sensor calculates intraocular pressure (IOP) by sensing changes in the shape of the eyeball, generating IOP data (A4). The detachable eye-opening assistive device receives commands via Bluetooth and activates, using gear-driven paddles to perform 30 to 45° progressive eyelid compression and propulsion on patients with their eyes closed, assisting in opening the eye for eye data collection.

[0071] Virtual reality-assisted psychological intervention consists of virtual reality and audio playback equipment. During the head data collection process, video and audio are played in multi-sensory dimensions to intervene in the psychology, relieve the mental and psychological pressure of the injured and reduce their stress response.

[0072] The collection of cardiopulmonary data B includes cardiac audio data and lung audio data, which is completed through a split audio collector that fits the precordial area and bilateral lungs. The flexible silicone material adapts to the body shapes of different patients. The split audio collector includes a precordial collector and a lung collector.

[0073] Heart audio data B1 is collected by a precordial collector attached to the precordial area. The frequency range of heart sounds is 20Hz to 600Hz, including the first and second heart sounds, which are used to analyze the working status and rhythm of the heart.

[0074] Lung audio data B2 is collected using a lung collector attached to both lungs. The frequency range of lung sounds is 100Hz to 2000Hz, covering normal breath sounds, pathological lung rales, dry and wet rales, and murmurs. Changes in lung sounds can reflect the health of the respiratory tract.

[0075] The split audio collector adopts an acoustic cavity noise reduction design. By precisely adjusting the cavity shape, it enhances the reception of cardiopulmonary sound signals while reducing the impact of ambient noise, improving the signal-to-noise ratio (SNR), enhancing signal quality, and ensuring accurate data collection.

[0076] Vital sign data C includes body temperature, pulse, blood pressure, and blood oxygen saturation data, which are collected simultaneously through an all-in-one arm-mounted integrated sensor. The temperature, pulse, blood pressure, and blood oxygen saturation sensors are integrated into a lightweight device to ensure the patient's comfort and data collection accuracy.

[0077] The accuracy of vital signs data is improved through the fusion algorithm of bioelectrical impedance and optical sensing. Different weights are assigned according to the signal quality of each sensor to perform data fusion. The Kalman filter algorithm is used to dynamically adjust the fusion weight to improve data accuracy.

[0078] The overall status data D is collected through a set of high-resolution RGB cameras (resolution ≥ 1080p), including the appearance, expression, movements, limb movements, special behaviors, body surface condition and wound conditions of the injured person. When collecting limb movement and wound condition data, the YOLOv5 algorithm is used to identify abnormal limb movements and wound bleeding in real time.

[0079] Collect the basic information, head data, cardiopulmonary data, vital signs data and overall status data of the injured and sick, and output the multimodal dataset Z, which is expressed as:

[0080] Z={X,A,B,C,D}

[0081] Based on the category of multimodal data, the multimodal dataset is divided into text data, image data and video data; text data H includes A2 and A4, image data I includes A1, A3, C, D, and video data J includes B;

[0082] By building independent disks to store multimodal datasets, in this embodiment, the independent disk array uses a RAID1 configuration and stores data copies on multiple hard drives to prevent data loss caused by single point failures. The multimodal datasets are encrypted with AES-256 during disk storage to protect the patient's privacy and prevent unauthorized access.

[0083] The multimodal data sets of the injured and patients are displayed to the monitoring doctors in real time through a high-resolution 4K display, ensuring real-time and clear presentation of the data.

[0084] M2, the processing module, performs preprocessing operations on the multimodal dataset, including data evaluation, screening and quality control, generates a usable dataset, and outputs a feature dataset through feature extraction and feature fusion; builds a database based on machine learning and deep learning algorithms, and intelligently generates unique identifiers containing multimodal datasets and feature datasets.

[0085] After the multimodal data collection is completed, strict preprocessing operations are required first. Preprocessing operations include data evaluation, screening and quality control.

[0086] Data evaluation involves evaluating the quality of each modal data set by calculating the signal-to-noise ratio (SNR) of the multimodal data and comparing it with the SNR threshold to assess the validity and integrity of the data. The SNR calculation formula is:

[0087]

[0088] Among them, P signal is the signal power; P noise is the noise power; when the signal-to-noise ratio is lower than the signal-to-noise ratio threshold, the data quality of this group of modalities is unqualified and needs to be re-collected;

[0089] Data screening uses the Z-score normalization method to remove outliers in each set of modal data. In this embodiment, when the Z-score of a data point is greater than 3, the data point is an outlier and needs to be removed.

[0090] Data quality control involves correcting multimodal data through a bandpass filter to ensure that the data can operate normally within the predetermined frequency range.

[0091] Feature extraction includes text data feature extraction, image data feature extraction and video data feature extraction.

[0092] Text data features are extracted through word embedding technology. Word embedding technology maps each text data into a dense vector space, becoming a vector. In the vector space, the vector distance between text data is expressed as the relationship between text data. Through the mapping of relationship, the computer can understand the semantics and contextual relationship of text data and convert text data into text data features H. h .

[0093] The features of the image data are extracted through a convolutional neural network. The sliding convolution kernel is used to perform convolution operations on the image data, and low-level, medium-level, and high-level features are extracted in sequence. The formula is:

[0094]

[0095] Among them, I i is the low-level image data feature; i and j are the row and column indexes of the low-level image data feature respectively; I is the input image data; O is the convolution kernel; m and n are the indexes of the convolution kernel O, representing the row and column indexes in the convolution kernel respectively; M and N are the sizes of the convolution kernel;

[0096] In the second convolutional layer, I becomes the output of the first convolutional layer, that is, the low-level image data feature I i , I i Becomes intermediate image data feature I m , until we get the high-level image data feature I h .

[0097] Video data features include spatial features and temporal features. Each frame in the video data is regarded as an independent image. Therefore, the spatial features of the video data are extracted by the image data feature extraction method to obtain the spatial feature J1.

[0098] The temporal features of video data describe the dynamic changes between video frames. The optical flow method is used to extract temporal features. The optical flow method calculates the motion of objects by analyzing the pixel changes between consecutive video frames. The formula is:

[0099]

[0100] Among them, u and v are the optical flows in the x and y directions respectively; is the time feature; and are the brightness gradients in the x and y directions respectively;

[0101] Collect the spatial and temporal features of the video data and output the video data feature J h .

[0102] After feature extraction, the multimodal dataset obtains text data features, image data features, and video data features. The Transformer algorithm performs feature fusion operations. The Transformer algorithm uses the self-attention mechanism to process the long-distance dependencies between different data features. It is particularly suitable for processing data features with different dimensions and different time series characteristics. It weights different data features by calculating the relationship between different positions, and finally outputs the fused global features.

[0103] The calculation formula of the self-attention mechanism of the Transformer algorithm is:

[0104]

[0105] Among them, Attention (Q, K, V) is the fused global feature; Q is the query, which represents the data feature; K is the key, which represents the label of the data feature; V is the value, which represents the actual information of the data feature; d k is the dimension of the key; T is the transpose operation of the key K;

[0106] By calculating the relationship between the query and the key and using the value for weighted fusion, the fused global feature is finally obtained.

[0107] After feature extraction and fusion, the multimodal datasets are automatically associated with global features through machine learning and deep learning algorithms to build a structured database. The multimodal datasets and global features of the injured and sick are hashed to generate unique identifiers.

[0108] M3, intelligent analysis and decision-making module, builds an unmanned intelligent triage monitoring model based on the unique identifier and performs iterative training; the trained unmanned intelligent triage monitoring model automatically analyzes the key information of the injured and patients, divides the treatment priority, and simultaneously performs triage classification, injury association reasoning and ranking, and real-time monitoring, and outputs a treatment instruction set.

[0109] The unmanned intelligent triage model optimizes performance through iterative training to improve classification and prediction accuracy. It obtains multimodal datasets and global features from the database, and uses a standard training-evaluation-validation-retraining-reevaluation-revalidation process to conduct multiple rounds of iterative training to evaluate the model's performance and update the weight parameters in each round. During training, the model will gradually adjust to the relationship between different modal data and global features until the iteration conditions are met and the model training is completed.

[0110] After training, the unmanned intelligent triage monitoring model automatically analyzes the key information of the injured and sick, including vital signs data analysis, trauma characteristics analysis and psychological state analysis.

[0111] Vital sign data analysis involves analyzing the patient's body temperature, heart rate, blood oxygen, and other vital sign data to identify abnormal values, such as high body temperature or low blood oxygen, which could indicate a potential critical condition.

[0112] Trauma feature analysis automatically detects the type of trauma and bleeding of the injured through image and video data;

[0113] Psychological status analysis involves analyzing the psychological status of the injured (such as anxiety, panic, etc.) and adjusting priorities and decision support based on the relationship between the psychological characteristics of the injured and the injury.

[0114] Based on key information, the model prioritizes the treatment of the injured and sick and provides clear treatment recommendations.

[0115] Triage classification extracts global features and associates them with the multimodal datasets of the injured and sick. It then implements injury inspection and assessment based on the unmanned intelligent triage monitoring model. Based on the assessment results and the GCS scoring mechanism, it outputs four-level classification labels, including red, yellow, green, and black labels.

[0116] A red label indicates that the patient is in critical condition, a yellow label indicates that the patient's condition is more urgent, a green label indicates that the patient's condition is relatively stable, and a black label indicates that the patient cannot be treated.

[0117] Injury association reasoning and ranking is to rank the injury diagnosis based on the multimodal dataset of the injured and sick, and give the probability of the disease suffered by the current injured and sick. In this embodiment, specifically,

[0118] Patient 1 is suspected of having a brain injury (80% probability), and the recommended next step is surgery.

[0119] Patient 2 is suspected of having a penetrating chest injury (60% probability), and the recommended next step is surgery.

[0120] Patient 3 is presumed to be in shock (40% probability), and the recommended next step is resuscitation.

[0121] Patient 3 is presumed to be impaired consciousness (40% probability), and the recommended next step is resuscitation.

[0122] Real-time monitoring monitors the patient's vital signs and trauma status, predicts the patient's health trends, ensures continuous monitoring and timely response to the patient, uses the LSTM algorithm to predict the patient's vital signs trends, and dynamically adjusts priorities based on real-time data. When the patient's vital signs are abnormal (such as decreased blood oxygen and dilated pupils), an alarm (brain hypoxia alarm) is triggered to remind medical staff to deal with it immediately.

[0123] Through real-time monitoring, the classification labels of the injured and sick are updated, and the priority of treatment is automatically adjusted according to changes in the vital signs data of the injured and sick.

[0124] Based on the injury classification, injury association reasoning and ranking, and real-time monitoring results of the injured and sick, a treatment instruction set is generated to help the medical team formulate subsequent treatment strategies.

[0125] An unmanned intelligent triage and monitoring method based on multimodal data fusion, characterized in that the method comprises:

[0126] Collect multimodal data sets of the injured and sick, including basic information, head data, cardiopulmonary data, vital signs data, and overall status data. During the head data collection process, integrate VR equipment to play customized soothing audio and video; and construct an independent disk to store the multimodal data sets.

[0127] Preprocessing the multimodal dataset to generate a usable dataset, and outputting fused global features through feature extraction and feature fusion; building a database based on machine learning and deep learning algorithms, and intelligently generating a unique identifier that includes the multimodal dataset and global features;

[0128] An unmanned intelligent triage and monitoring model is constructed based on the unique identifier and iterative training is performed. The trained unmanned intelligent triage and monitoring model analyzes key information of the injured and sick, divides treatment priorities, simultaneously performs triage classification, injury association reasoning and ranking, and real-time monitoring, and outputs a treatment instruction set.

[0129] Experimental data:

[0130] In a simulated disaster scenario test, the classification accuracy of 200 injured and sick people reached 94.5%, the misjudgment rate was 32% lower than manual evaluation, and the evaluation time was 35.4s / case, which was less than the 69.1s / case of manual evaluation; the eye-opening assist device increased the pupil collection success rate from 70% to 95%.

[0131] Key performance indicators:

[0132] 1. The material of the data acquisition module is soft / the imaging clarity of the camera and lens is ≥1080P.

[0133] 2. High precision error of the sensor ≤ 2%.

[0134] 3. Data update frequency ≤ 2S, hardware can bear weight ≥ 60Kg, can withstand temperature -20℃ to +50℃, and anti-electromagnetic interference ≥ Class C.

[0135] The present invention proposes an unmanned intelligent triage and monitoring system and method based on multimodal data fusion. The invention integrates multiple data acquisition devices to collect multimodal data of the injured and sick in real time, including vital signs, head data, cardiopulmonary data, eye data and overall status data, etc., and uses high-precision sensors and deep learning algorithms to perform intelligent analysis and decision support, automatically analyze the injuries of the injured and sick, divide the treatment priorities, realize unmanned intelligent triage and real-time monitoring, and significantly improve the efficiency and accuracy of treatment.

[0136] In summary, the present invention overcomes the problems of manual dependence, single data, poor real-time performance, and poor hardware adaptability in traditional triage and monitoring technologies through multimodal data fusion technology, unmanned operation, and efficient intelligent analysis and decision-making mechanisms. It provides a new, accurate and efficient intelligent triage and monitoring solution. The present invention can quickly and accurately assess the health status of the wounded in scenarios such as disaster relief and battlefield first aid, improve the efficiency of the use of emergency resources, and provide effective decision-making support for medical teams.

[0137] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. An unmanned intelligent triage monitoring system based on multimodal data fusion, characterized in that: The system includes: The data acquisition module collects multimodal data sets of the injured and sick, including basic information, head data, cardiopulmonary data, vital signs data, and overall status data. During the head data acquisition process, the module integrates VR equipment to play customized soothing audio and video. The module also stores the multimodal data sets by building an independent disk. A processing module performs preprocessing operations on the multimodal dataset, including data evaluation, screening, and quality control, to generate a usable dataset, and outputs fused global features through feature extraction and feature fusion; a database is constructed based on machine learning and deep learning algorithms, and a unique identifier containing the multimodal dataset and global features is intelligently generated; The intelligent analysis and decision-making module constructs an unmanned intelligent triage and monitoring model based on the unique identifier and performs iterative training. The trained unmanned intelligent triage and monitoring model automatically analyzes the key information of the injured and patients, divides the treatment priorities, and simultaneously performs triage classification, injury association reasoning and ranking, and real-time monitoring, and outputs a treatment instruction set.

2. The unmanned intelligent triage monitoring system based on multimodal data fusion according to claim 1 is characterized in that: The basic information X includes the age, gender, height, and weight of the patient. The age and gender information of the patient are collected through automatic recognition technology, and the height and weight information of the patient are measured through smart wearable devices. The collection of the head data A includes brain and eye data collection and virtual reality assisted psychological intervention. The brain data is collected through a non-invasive EEG signal collection device and a wearable ultrasound patch, and the eye data is collected through a high-sensitivity camera, a pressure sensor and a detachable eye-opening assist device. The cardiopulmonary data B includes cardiac audio data and lung audio data, which are collected by a split audio collector, which includes a precordial collector and a lung collector. The vital signs data C include body temperature, pulse, blood pressure and blood oxygen saturation data, which are collected synchronously by an integrated arm-type integrated sensor. The overall status data D includes the appearance, expression, movement, limb movement, special behavior, body surface condition and wound condition of the injured person, which is collected by a high-resolution RGB camera.

3. The unmanned intelligent triage and monitoring system based on multimodal data fusion according to claim 2 is characterized in that: The non-invasive EEG signal acquisition device is fixed on the patient's head and collects EEG signal data through multi-channel electrodes to generate a non-invasive EEG image A1. The wearable ultrasound patch collects 3D intracranial vascular ultrasound data A2 of the patient in real time. The highly sensitive camera collects pupil, eye movement trajectory, and fundus hemodynamic images to generate eye images A3. The pressure sensor calculates the intraocular pressure value by sensing the shape changes of the eyeball to obtain intraocular pressure data A4. The detachable eye-opening assistance device is activated by receiving instructions via Bluetooth, and uses a gear-driven paddle to perform a 30 to 45° progressive eyelid pressure and support operation on the injured person with closed eyes, assisting the injured person to open his eyes for eye data collection.

4. The unmanned intelligent triage and monitoring system based on multimodal data fusion according to claim 2 is characterized in that: The precordial collector is attached to the precordial area of ​​the patient, and collects the patient's heart sounds and heart rhythm information in real time to generate the heart audio data B1; the lung collector is attached to the patient's bilateral lungs, and collects the patient's normal respiratory sounds, pathological lung rales, dry and wet rales, and murmurs in real time to generate the lung audio data B2; The arm-type integrated sensor integrates body temperature, pulse, blood pressure and blood oxygen saturation sensors into a portable device to collect vital sign data C of the injured and sick in real time; The basic information, head data, cardiopulmonary data, vital signs data and overall status data of the injured and sick are collected to output the multimodal dataset Z, which is expressed as: Z={X,A,B,C,D} Based on the category of the multimodal data, the multimodal data set is divided into text data, image data and video data; the text data H includes A2 and A4, the image data I includes A1, A3, C, and D, and the video data J includes B; The independent disk stores the multimodal dataset using an AES-256 encryption method.

5. The unmanned intelligent triage and monitoring system based on multimodal data fusion according to claim 3 or 4, characterized in that: The data evaluation is to evaluate the quality of each set of modal data using the following formula: Where SNR is the signal-to-noise ratio; P signal is the signal power; P noise is the noise power; when the signal-to-noise ratio is lower than the signal-to-noise ratio threshold, the data quality of this group of modalities is unqualified and needs to be re-collected; The data screening removes outliers in each set of modal data by using the Z-score standardization method; The data quality control is to correct the multimodal data through a bandpass filter.

6. The unmanned intelligent triage and monitoring system based on multimodal data fusion according to claim 5 is characterized in that: The feature extraction includes text data feature extraction, picture data feature extraction and video data feature extraction. The text data features are extracted by word embedding technology, which maps each text data into a dense vector space to become a vector. In the vector space, the vector distance between the text data is the relationship between the text data. Through the mapping of the relationship, the text data is converted into text data features H h ; The image data features are extracted through a convolutional neural network. A sliding convolution kernel is used to perform a convolution operation on the image data, and low-level, mid-level, and high-level features are extracted in sequence. The formula is: Among them, I i is the low-level image data feature; i and j are the row and column indexes of the low-level image data feature respectively; I is the input image data; O is the convolution kernel; m and n are the row and column indexes of the convolution kernel respectively; M and N are the sizes of the convolution kernel; In the second convolutional layer, I becomes the output of the first convolutional layer, i.e., the low-level image data feature I i , I i Becomes intermediate image data feature I m , until we get the high-level image data feature I h ; The video data features include spatial features and temporal features. Each frame in the video data is an independent image. The spatial features J1 are obtained by extracting the image data features using the image data feature extraction method. The temporal feature is extracted by the optical flow method, and the formula is: Among them, u and v are the optical flows in the x and y directions respectively; is the time feature; and are the brightness gradients in the x and y directions respectively; The spatial and temporal features of the video data are combined to output the video data feature J h .

7. The unmanned intelligent triage and monitoring system based on multimodal data fusion according to claim 6 is characterized in that: The feature fusion is performed through the self-attention mechanism of the Transformer algorithm, and the formula is: Among them, Attention (Q, K, V) is the fused global feature; Q is the query, which represents the data feature; K is the key, which represents the label of the data feature; V is the value, which represents the actual information of the data feature; d k is the dimension of the key; T is the transpose operation of the key K; By calculating the relationship between the query and the key and using the value to perform weighted fusion, the fused global feature is finally obtained. The unique identifier is obtained by performing hash calculation on the multimodal dataset and global features of the patient.

8. The unmanned intelligent triage and monitoring system based on multimodal data fusion according to claim 7 is characterized in that: The iterative training process is training-evaluation-validation-retraining-reevaluation-revalidation. The key information of the injured and sick patients includes vital sign data, trauma characteristics and psychological status; The triage classification extracts global features and associates them with the multimodal dataset of the injured and sick. The unmanned intelligent triage monitoring model assesses the injury status and outputs four-level classification labels of the injured and sick based on the GCS scoring mechanism, including red, yellow, green and black labels. The injury association reasoning and ranking is to rank the injury diagnosis based on the multimodal data set of the injured and sick, and give the probability of the disease suffered by the current injured and sick; The real-time monitoring monitors the vital signs data and trauma conditions of the injured and sick, and predicts the vital signs trend of the injured and sick based on the LSTM algorithm. The classification labels of the injured and sick are updated based on the vital signs trend, and the priority of treatment is automatically adjusted according to the updated classification labels. Based on the injury classification, injury association reasoning and ranking, and real-time monitoring, a treatment instruction set is generated.

9. An unmanned intelligent triage monitoring method based on multimodal data fusion, characterized in that: The method includes: Collect multimodal data sets of the injured and sick, including basic information, head data, cardiopulmonary data, vital signs data, and overall status data. During the head data collection process, integrate VR equipment to play customized soothing audio and video; and construct an independent disk to store the multimodal data sets. Preprocessing the multimodal dataset to generate a usable dataset, and outputting fused global features through feature extraction and feature fusion; building a database based on machine learning and deep learning algorithms, and intelligently generating a unique identifier that includes the multimodal dataset and global features; An unmanned intelligent triage and monitoring model is constructed based on the unique identifier and iterative training is performed. The trained unmanned intelligent triage and monitoring model analyzes key information of the injured and sick, divides treatment priorities, simultaneously performs triage classification, injury association reasoning and ranking, and real-time monitoring, and outputs a treatment instruction set.

Images