Emergency triage method and system based on multi-encoder and multi-modal information fusion

Abstract

The application discloses an emergency triage method and system based on multi-encoder and multi-modal information fusion, the system collects heterogeneous multi-modal medical data through a multi-modal data acquisition method, a multi-encoder strategy is used to firstly mix encode the results of image recognition, speech recognition, mouse keyboard input and the import of a life monitor by using a tokenizer, BERT and a One-Hot encoder, a multi-modal information fusion method is used to fuse the information of the encoded image, speech and text multi-modal data by using a low-level and high-level feature fusion method, finally, an algorithm model M1 and M2 based on machine learning are used to classify and identify the fused information, the severity grade of a patient and a doctor-seeing department are predicted, and medical triage is assisted. The emergency triage method and system can replace and assist artificial triage, so that the triage efficiency and accuracy are improved.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary fields of smart healthcare, artificial intelligence, robotics, instrument science, control science, computer science, sensor technology, and human-computer interaction technology, and relates to a medical triage method and system based on multi-encoder and multi-modal information fusion. Background Technology

[0002] Hospital triage refers to the process by which triage nurses determine the appropriate department for a patient upon arrival at the hospital, based on the severity of their condition and treatment priorities. The efficiency of this process is crucial for patients, especially critically ill ones, as it directly impacts their survival rate and even their chances of survival. Relying on subjective experience or insufficiently intelligent equipment for triage can lead to inefficiency, errors, and even secondary triage in complex situations due to differences in personnel and environmental interference. This can cause inconvenience for emergency patients and even medical accidents. Accurate and efficient triage ensures more timely and effective diagnosis and treatment. Upon arrival at the hospital, the triage nurse first communicates with the patient or their family to obtain their chief complaint. During this time, relevant physiological indicators are obtained using a thermometer, blood pressure monitor, heart rate sensor, and blood oxygen saturation sensor. The patient's level of consciousness is also assessed. Subsequently, a comprehensive judgment is made based on the chief complaint, physiological indicators, and level of consciousness to determine the triage level and thus whether the patient should be admitted to the emergency room or a relevant specialized department.

[0003] Chinese patent CN201710383287.4 proposes an intelligent emergency triage system, which uses a client-user relationship to link patient registration information with emergency triage information. This method shares and matches patient information, but it cannot solve the time-consuming problem of data collection and extraction of effective information. Chinese patent CN 201710955228.X proposes a method to match the optimal doctor based on the patient's chief complaint, thereby determining the triage department. However, the ranking of the optimal doctor's abilities has a certain degree of subjectivity, and it still cannot solve the problem of one doctor being responsible for multiple departments. Furthermore, most current mainstream triage methods are based on well-collected ideal data, neglecting the truly time-consuming steps: obtaining the chief complaint, physiological indicators, and making decisions. Summary of the Invention

[0004] To overcome the problems of lengthy, colloquial, unclear, and emotionally charged chief complaints, which can lead to subjective biases in the chief complaint, and to address the shortcomings of existing emergency triage systems in accurately collecting multimodal patient information in complex scenarios, efficiently predicting patient severity and target departments, and accurately and efficiently assisting nurses in triage decisions, this invention proposes an emergency triage method and system based on multi-encoder and multimodal information fusion. This system can acquire real-time image information, voice information, mouse and keyboard input information from patients, family members, and triage nurses through proactive human-computer interaction and communication interfaces, as well as physiological information such as blood pressure, temperature, blood oxygen saturation, and heart rate. Based on this proactively interacted heterogeneous information, the system updates the algorithm's weights to achieve intelligent and efficient prediction. It can simultaneously acquire patient severity and target departments to assist triage nurses in decision-making. This system combines proactive interaction with a grade-department prediction model, which not only improves triage efficiency but also possesses high intelligence and accuracy, providing strong support for smart healthcare.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows:

[0006] An emergency triage method based on multi-encoder and multi-modal information fusion includes: a multi-modal data acquisition step, an information fusion step of multi-modal heterogeneous medical data, and a patient severity level-department prediction step.

[0007] The multimodal data acquisition step includes the following sub-steps:

[0008] S1: Based on image sensors and voice sensors, acquire image and voice information of patients going to the triage station, and use image semantic parsing method and voice semantic parsing method to acquire corresponding semantic information respectively. At the same time, use electrocardiogram monitor, mouse and keyboard to monitor and record the patient's vital signs information.

[0009] S2: Based on the analyzed facial, trauma, and limb image information, voice information, vital sign information, and voice information from family members and doctors, match the patient's current symptoms.

[0010] S3: If symptom information cannot be obtained, and the patient's department and severity level cannot be determined, ask the patient about their current feelings through the microphone, return to step S1 to collect the patient's image and voice information, and proceed to S2 for reassessment. If the patient's current symptoms still cannot be matched, call the nurse for treatment through the display screen or speaker.

[0011] S4: The nurse interacts with the patient or family member via voice and expresses relevant information in a more standardized voice format, making it easier for the voice semantic analysis method to recognize and record semantics. Then, the nurse proceeds to steps S1 and S2 to match and identify the current symptoms.

[0012] S5: If one or more of the physiological indicators such as blood pressure, heart rate, blood oxygen saturation, respiratory rate, and body temperature reach a critical value, an alarm will be triggered and a nurse will be notified to intervene.

[0013] The information fusion step of the multimodal heterogeneous medical data involves data cleaning and information fusion of the multimodal heterogeneous medical data, which includes structured data and unstructured data obtained by the multimodal data acquisition method.

[0014] The data cleaning step takes multimodal data as input, analyzes the validity and completeness of the multimodal data, and uses filtering and correction methods to obtain effective triage data.

[0015] The information fusion step includes a low-level feature fusion step in the data encoding stage and a high-level feature fusion step in the model training stage.

[0016] The data encoding stage employs a multi-encoder combination strategy, using encoders including Tokenizer (E1 encoding), BERT (E2 encoding), and One-Hot (E3 encoding). The low-level feature fusion step in the data encoding stage includes the following sub-steps:

[0017] S6: Represent the feature space of multimodal fusion as in Let m represent the feature space of the k-th mode, and m k This represents the dimension of the space; L = (l1, l2, ..., l... q ) represents a label space with q categories, while S s and S u These represent the structured and unstructured feature spaces, respectively.

[0018] S7: The dataset can be represented as: D = {(x i k ,y i )|1≤k≤K,1≤i≤n}∈(S su ×L) n ,in Indicates a dimension m k The eigenvector of the k-th mode, and y i ∈L means with x i k The associated labels, where K represents the total number of modalities and n represents the total number of samples in the current modality;

[0019] S8: The three encoding strategies are represented as Φ={φ E1 ,φ E2 ,φ E3};

[0020] S9: Because there are many types of chief complaints and they are subjective, it can be represented as x u =(x1,x2,…,x i ,…,x n )∈S u , where x i =[ξ1,ξ2,…,ξ j ,…,ξ M ], ξ j M represents each word vector, where M represents x. i The maximum word vector length in the algorithm is used to ensure the integrity of each word vector feature.

[0021] S10: Two encoding methods, E1 and E2, are used for unstructured data: φ E1 (x u )=[α1,α2,…,α i ,…,α n ] and φ E2 (x u )=[β1,β2,…,β i ,…,β n Structured data uses E3 encoding: φ E3 (x n s )={e1 Ks e2 Ks ,…,e i Ks ,…,e n Ks}∈S s ;

[0022] S11: A hybrid coding strategy is used for structured and unstructured data to achieve two low-level feature fusion methods: F1 = Concate([φ E1 (x u ),φ E3 (x n s )]),F2=Concate([φ E2 (x u ),φ E3 (x n s )]);

[0023] The advanced feature fusion step during model training adds a concatenation layer to the network model to implement advanced feature fusion and attention mechanisms. The specific steps of the attention mechanism are as follows:

[0024] S12: Attention(Q,K,V)=softmax(QKT / d k 0.5 The linear transformation matrices Q, K, and V are respectively: Q = XW Q K = XW K V = XW V X is x i Let represent the input sequence of words, and W represent the learnable parameter matrix corresponding to Q, K, and V. Q and K are used to calculate the similarity between the current word vector and other word vectors, such that QK... T / d k 0.5 Used as attention weights; d k 0.5 QK represents the normalization factor; T / d k 0.5 The feature matrix V is weighted and summed to be used as the word vectors for the next layer;

[0025] S13: QK T Involving the interaction between two word vectors, extended to: QK T =(e i +p i W Q (e j +p j ) T (W K ) T , where e i and e j p represents the word vectors of the i-th and j-th tokens after E1 encoding. i and p j This represents the positional encoding of the i-th and j-th symbols;

[0026] S14: QK T Expanding the expression: QK T =e i W Q (W K ) T e j T +e i W Q (W K ) T p j T +p i W Q (W K ) T e j T +p i W Q (W K) T p j T The first item focuses on the interaction between two word vectors after word embedding, emphasizing explicit features; the fourth item involves positional encoding, which captures the positional relationship between word vectors.

[0027] S15: Where p t (i) p represents the positional encoding of the current word vector. i p j ∈p t (i) . d model The dimension of the word vector is represented by t, and the actual position of the word vector in the sequence is represented by t.

[0028] S16: Residual term p i p j T It contains relative positional information, which can be used without a trainable matrix W. Q (W K ) T Under the condition of obtaining: p i p j T =T(ij);

[0029] S17: Introducing Euler's formula: p i =[cosθ i sinθ i ] T p j T =[cosθ j sinθ j ] T p i and p j T θ represents the position matrix of word vectors in the feature space. i and θ j Let and represent the angles between the absolute positions of the current word vector in the feature space, assuming... θ μ = μθ0, where the scalar factor μ and θ μ Related to θ0;

[0030] S18: Introducing the trigonometric formula: p i p j T =cosθ i cosθ j +sinθ i sinθ j =cos(θ) i -θj )=>cos[(ij)θ0], the transformation function T is derived from this, and the training matrix is ​​updated; during model training, the first term in step S14 is entered to continuously extract the fused salient features; in addition, the attention mechanism is combined with the fourth term so that the fused feature information is continuously obtained through positional encoding during the encoding process;

[0031] The patient severity grade-department prediction step includes two strategies: based on Concate([φ E1 (x u ),φ E3 (x n s The mixed encoded data F1 of the data is based on Concate([φ)). E2 (x u ),φ E3 (x n s The difference between the mixed encoded data F2 and the structured data φ lies mainly in that the latter targets mixed feature data. E3 (x n s The feature extraction steps are the same; based on F1+φ E3 (x n s ) and F2+φ E3 (x n s The trained models are named Model M1 and Model M2, respectively.

[0032] The data processing steps for model M1 are as follows:

[0033] S19: Structured coded data and heterogeneous hybrid coded data are used as inputs to the parallel computing architecture and the serial architecture, respectively, with data dimensions of n×1×30 and n×43, where n represents the amount of data;

[0034] S20: The parallel computing architecture first consists of three serial branches. The first branch uses an LSTM module with a dropout of 0.2 and an output dimension of n×1×32. The second and third branches use ReLU activation functions, but with convolutional kernels of scales of 3×3 and 5×5, respectively, and the output dimension of each branch is n×1×32. Subsequently, the second and third branches use a Max Pooling module to gather high-level features, and the output dimension of each branch is n×1×32.

[0035] S21: The Concatenate layer is used to vertically merge each branch into n×3×32 features. Then, an LSTM module with dropout of 0.2 is used as the feature extractor and Max Pooling module, and the output dimensions are both n×1×36 high-level features.

[0036] S22: The serial architecture first performs position encoding on the data based on the TokenAnd Position Embedding layer, with an output dimension of n×43×36. Then, it uses an integrated Transformer layer with 6 heads to extract features from the position-encoded data, with an output dimension of n×43×36 as well.

[0037] S23: The Concatenate layer is used to merge the outputs of the serial and parallel architectures into a hybrid feature with a global receptive field, with an output dimension of n×46×36; then an attention mechanism with an output dimension of n×36 is used to capture more explicit and effective information, while also acting as a flattening layer to reduce the dimensionality of the data;

[0038] S24: The multi-output mode uses two parallel Dense linear layers to predict the patient's disease severity and target department, respectively. The output dimensions are n×L and n×D, where L and D represent the severity level and the number of types of departments to be visited, respectively.

[0039] The data processing steps for model M2 are as follows:

[0040] S25: Structured coded data and heterogeneous hybrid coded data are used as inputs to the parallel computing architecture and the serial architecture, respectively. The data dimensions are n×1×30 and n×31×312, respectively, where n represents the amount of data.

[0041] S26: The parallel computing architecture first consists of three serial branches. The first branch uses an LSTM module with a dropout of 0.2 and an output dimension of n×1×32. The second and third branches use ReLU activation functions, but with convolutional kernels of 3×3 and 5×5 scales respectively, and the output dimension of each branch is n×1×32. Then, each branch uses a Max Pooling module to gather high-level features, and the output dimension of each branch is n×1×32. This can enhance explicit features and reduce computational complexity.

[0042] S27: The Concatenate layer is used to horizontally merge the branches into n×1×96 features. Then, an LSTM module with dropout of 0.2 is used as the feature extractor and Max Pooling module, and the output dimensions are both n×1×32 high-level features.

[0043] S28: The serial architecture first uses a bidirectional LSTM module with a dropout of 0.2, with an output dimension of n×31×64. Then, it uses a Max Pooling module to obtain explicit features, with an output dimension of n×15×64. Next, it uses a convolution kernel with the activation function ReLU and a scale of 3×3, with an output dimension of n×15×32. Then, it uses the Max Pooling module again to obtain high-level features with a dimension of n×15×64, while suppressing information with low correlation.

[0044] S29: The Concatenate layer is used to fuse the outputs of the serial and parallel architectures into a hybrid feature with a global receptive field, with an output dimension of n×8×32; then an attention mechanism with an output dimension of n×32 is used to capture more explicit and effective information, while also acting as a flattening layer to reduce the dimensionality of the data;

[0045] S30: Employs a multi-output mode, using two parallel dense linear layers to predict the patient's disease severity and target department, with output dimensions of n×L and n×D, where L and D represent the severity level and the number of departments visited, respectively.

[0046] Furthermore, the image semantic parsing method includes the following steps:

[0047] (1): Image dataset construction: Construct an image dataset of emergency patients;

[0048] (2): Model training: Training of a classification model for emergency patients’ facial, trauma information and limb features based on deep neural networks;

[0049] (3): Model deployment and recognition: Actively track and collect images of emergency patients, and extract keyframes based on the patient's facial expressions, trauma information, admission method and gestures;

[0050] (4): Record and process key frame information of images in real time, and use the patient's pain level, trauma location, trauma severity, admission method, and combined speech response results as the output of the model to obtain a preliminary impression of the disease.

[0051] Furthermore, the speech semantic parsing method includes the following steps:

[0052] (1): Voice dataset construction: Construct a voice keyword dataset of emergency patients, their families and nurses;

[0053] (2): An adaptive sliding window method is used to extract effective components, and feature engineering is constructed based on the audio itself, the time domain, and the frequency domain to obtain feature information;

[0054] (3): Model training: Training of speech recognition models based on deep neural networks;

[0055] (4): Model deployment: Actively identify and track the voice recordings of patients, family members and nurses;

[0056] (5): Record and process voice information in real time, quickly and accurately capture keywords from the voice and establish association with medical terms to obtain preliminary chief complaint, past medical history, language function, patient vital signs information, etc.

[0057] Furthermore, the structured data includes vital signs, admission method, demographic characteristics, image semantic information, consciousness, and language function. The vital signs include the patient's heart rate, blood pressure, blood oxygen saturation, respiratory rate, and body temperature. The admission method includes ambulance, walking, wheelchair, and being carried in. The demographic characteristics include age and gender. The image semantic information includes pain level, presence or absence of facial paralysis symptoms, consciousness, location and severity of trauma, and presence or absence of limb movement disorders. The unstructured data includes the chief complaint, past medical history, and speech information. The results of the speech information parsing are linked with the structured or unstructured parameters and used for filling or supplementing.

[0058] Furthermore, the data cleaning step first sorts the triage data according to the patient's latest consultation time and fills in missing values ​​to facilitate further data processing by the pandas library. Then, it iterates through all data and records the index. Next, it comprehensively judges the validity of the triage data based on the patient's name and ID, whether keywords in the chief complaint are included in the set blacklist, or whether the chief complaint is the filled-in character from the previous step. If invalid, the entire triage information is deleted, and the data is iterated and the index is recorded again. If valid, it checks whether identical or incorrect triage information is due to keyboard and mouse input. If so, duplicates are deleted; if not, it continues to check whether the same patient has been triaged multiple times. If not, the data is directly considered valid; otherwise, it indicates an incorrect triage for the patient, and the validity of the triage needs to be judged based on the consultation time. If the same patient has multiple consultations at the same time, it indicates that the doctor temporarily corrected the incorrect triage result, and the latest consultation time represents valid data. Then, any formatting errors in the valid data are corrected, and the final triage data obtained is the valid triage data.

[0059] This invention also provides an emergency triage system based on multi-encoder and multimodal information fusion, used to implement the aforementioned emergency triage method based on multi-encoder and multimodal information fusion. The system includes a data acquisition and interaction module, an interface conversion device, a main controller module, and a power supply. The data acquisition and interaction module includes an image sensor, a sound sensor, an electrocardiogram monitor, a display, a speaker, a mouse, and a keyboard. The main controller module includes a CPU, a GPU, and memory. The data acquisition and interaction module is connected to the main controller module through the interface conversion device, and the power supply provides power to both the data acquisition and interaction module and the main controller module.

[0060] The beneficial effects of this invention are as follows:

[0061] (1) By adopting multiple coding strategies and multimodal fusion methods, structured and unstructured triage information can be fully utilized in clinical practice to improve the system’s fault tolerance and robustness, making the triage basis more comprehensive.

[0062] (2) The model has high prediction accuracy and is lightweight and easy to deploy, enabling efficient self-service triage and real-time assisted triage, reducing the waiting time of triage patients and saving medical resources.

[0063] (3) The prediction model can be configured with single or multiple outputs to meet different medical application needs.

[0064] (4) Two lightweight triage models with different computing resources are provided for selection. Model M1 has slightly higher accuracy than model M2, but it occupies more GPU memory. Attached Figure Description

[0065] Figure 1 This is a block diagram of the hardware system composition of the present invention.

[0066] Figure 2 This is a flowchart of the data cleaning process of the present invention.

[0067] Figure 3 This is a flowchart of the prediction network construction process of the present invention.

[0068] Figure 4 The multimodal data acquisition method and triage application scenario of the present invention are illustrated in the diagram. Detailed Implementation

[0069] The technical solutions provided by the present invention will be described in detail below with reference to specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0070] This invention provides an emergency triage system based on multi-encoder and multimodal information fusion, comprising a data acquisition and interaction module, an interface conversion device, a main controller module, and a power supply. The overall framework is as follows: Figure 1 As shown. The data acquisition and interaction module includes an image sensor, a sound sensor, an electrocardiogram monitor (including a heart rate sensor, blood oxygen saturation sensor, temperature sensor, blood pressure sensor, and respiratory rate sensor), a display, a speaker, a mouse, and a keyboard. The main controller module includes a CPU, GPU, and memory. The data acquisition and interaction module is connected to the main controller module via an interface conversion device. A power supply provides power to both the data acquisition and interaction module and the main controller module. The emergency triage system based on multi-encoder and multimodal information fusion is used to implement an emergency triage method based on multi-encoder and multimodal information fusion.

[0071] The emergency triage method based on multi-encoder and multi-modal information fusion provided by the present invention includes: a multi-modal data acquisition step, an information fusion step of multi-modal heterogeneous medical data, and a patient severity level-department prediction step.

[0072] The specific process of multimodal data acquisition includes the following sub-steps:

[0073] S1: Based on image sensors and voice sensors, acquire image and voice information of patients heading to the triage station, and use image semantic parsing method and voice semantic parsing method respectively to acquire corresponding semantic information. At the same time, use an electrocardiogram monitor (including heart rate sensor, blood oxygen saturation sensor, temperature sensor, blood pressure sensor, and respiratory rate sensor), mouse and keyboard to monitor and record the patient's vital signs information.

[0074] The image semantic parsing method includes the following steps:

[0075] (1): Image dataset construction: Construct an image dataset of emergency patients;

[0076] (2): Model training: Training of a classification model for emergency patients’ facial, trauma information and limb features based on deep neural networks;

[0077] (3): Model deployment and recognition: Actively track and collect images of emergency patients, and extract keyframes based on the patient's facial expressions, trauma information, admission method and gestures;

[0078] (4): Record and process key frame information of images in real time, and use the patient's pain level, trauma location, trauma severity, admission method, and combined voice response results as the output of the model to obtain a preliminary impression of the disease, including trauma, coma, limb movement disorder, etc.

[0079] The speech semantic parsing method includes the following steps:

[0080] (1): Voice dataset construction: Construct a voice keyword dataset of emergency patients, their families and nurses;

[0081] (2): An adaptive sliding window method is used to extract effective components, and feature engineering is constructed based on the audio itself, the time domain, and the frequency domain to obtain feature information;

[0082] (3): Model training: Training of speech recognition models based on deep neural networks;

[0083] (4): Model deployment: Actively identify and track the voice recordings of patients, family members and nurses;

[0084] (5): Record and process voice information in real time, quickly and accurately capture keywords from the voice and establish association with medical terms to obtain preliminary chief complaint, past medical history, language function, patient vital signs information, etc.

[0085] S2: Based on the analyzed facial, trauma, and limb image information, voice information, vital sign information, and voice information from family members and doctors, match the patient's current symptoms.

[0086] S3: If symptom information cannot be obtained, and the patient's department and severity level cannot be determined, ask the patient about their current feelings through the microphone, return to step S1 to collect the patient's image and voice information, and proceed to S2 for reassessment. If the patient's current symptoms still cannot be matched, call the nurse for treatment through the display screen or speaker.

[0087] S4: The nurse interacts with the patient or family member via voice and expresses relevant information in a more standardized voice format, making it easier for the voice semantic analysis method to recognize and record semantics. Then, the nurse proceeds to steps S1 and S2 to match and identify the current symptoms.

[0088] S5: If one or more of the following physiological indicators—blood pressure, heart rate, blood oxygen saturation, respiratory rate, or body temperature—reach a critical value, an alarm will be triggered and a nurse will be notified to intervene.

[0089] The information fusion steps for multimodal heterogeneous medical data involve data cleaning and information fusion. Heterogeneous medical data includes structured and unstructured data obtained through multimodal data acquisition methods. Structured data includes vital signs, admission methods, demographic characteristics, image semantic information, consciousness, and language function. Vital signs include heart rate, blood pressure, blood oxygen saturation, respiratory rate, and body temperature. Admission methods include ambulance, walking, wheelchair, and being carried in. Demographic characteristics include age and gender. Image semantic information includes pain level, presence of facial paralysis symptoms, consciousness, location and severity of trauma, and presence of limb movement impairment. Unstructured data includes chief complaint, past medical history, and speech information. The results of speech information parsing are linked with structured or unstructured parameters for use in filling or supplementing information.

[0090] The data cleaning process is as follows: Figure 2 As shown, this step uses multimodal data as input. Each module analyzes the validity and completeness of the multimodal data and uses filtering and correction methods to obtain valid triage data. Specifically, the method first sorts the triage data according to the patient's latest consultation time and fills in missing values ​​to facilitate further data processing by the pandas library. Then, it iterates through all data and records the index. Next, it comprehensively judges the validity of the triage data based on the patient's name and ID, whether keywords in the chief complaint are included in the set blacklist, or whether the chief complaint is the filled-in character from the previous step. If invalid, the entire triage information is deleted, and the data is iterated again and the index is recorded. If valid, it checks whether the same or incorrect triage information was entered due to keyboard or mouse input. If duplicates exist, they are deleted; if not, it continues to check whether the same patient has been triaged multiple times. If not, the data is directly considered valid; otherwise, it indicates an incorrect triage for the patient, and the validity of the triage needs to be determined based on the consultation time. If the same patient has multiple consultations at the same time, it means the doctor temporarily corrected the incorrect triage result, and the latest consultation time represents valid data. Then, any formatting errors in the valid data are corrected, and the final triage data obtained is the valid triage data. The keyword blacklist contains entries unrelated to triage; the prediction model cannot predict the patient's severity level and the department they should visit based on these entries.

[0091] The information fusion step includes a low-level feature fusion step in the data encoding phase and a high-level feature fusion step in the model training phase. The data encoding phase employs a multi-encoder combination strategy, using Tokenizer (E1), BERT (E2), and One-Hot (E3) encoders. The low-level feature fusion step in the data encoding phase includes the following sub-steps:

[0092] S6: Represent the feature space of multimodal fusion as in Let m represent the feature space of the k-th mode, and m k This represents the dimension of the space. L = (l1, l2, ..., l...) q ) represents a label space with q categories, while S s and S u These represent the structured and unstructured feature spaces, respectively.

[0093] S7: The dataset can be represented as: D = {(x i k ,y i )|1≤k≤K,1≤i≤n}∈(S su ×L) n,in Indicates a dimension m k The eigenvector of the k-th mode, and y i ∈L means with x i k Associated labels. K represents the total number of modalities, and n represents the total number of samples in the current modality;

[0094] S8: The three encoding strategies are represented as Φ={φ E1 ,φ E2 ,φ E3};

[0095] S9: Because there are many types of chief complaints and they are subjective, it can be represented as x u =(x1,x2,…,x i ,…,x n )∈S u , where x i =[ξ1,ξ2,…,ξ j ,…,ξ M ], ξ j M represents each word vector, where M represents x. i The maximum word vector length in the algorithm is used to ensure the integrity of each word vector feature.

[0096] S10: Two encoding methods, E1 and E2, are used for unstructured data: φ E1 (x u )=[α1,α2,…,α i ,…,α n ] and φ E2 (x u )=[β1,β2,…,β i ,…,β n Structured data uses E3 encoding: φ E3 (x n s )={e1 Ks e2 Ks ,…,e i Ks ,…,e n Ks}∈S s ;

[0097] S11: A hybrid coding strategy is used for structured and unstructured data to achieve two low-level feature fusion methods: F1 = Concate([φ E1 (x u ),φ E3 (x n s )]),F2=Concate([φ E2 (xu ),φ E3 (x n s )]);

[0098] The advanced feature fusion step during model training involves adding a concatenation layer to the network model to implement advanced feature fusion and an attention mechanism. The specific steps of the attention mechanism are as follows:

[0099] S12: Attention(Q,K,V)=softmax(QK T / d k 0.5 The linear transformation matrices Q, K, and V are respectively: Q = XW Q K = XW K V = XW V X is x i Let represent the input sequence of words, and W represent the learnable parameter matrix corresponding to Q, K, and V. Q and K are used to calculate the similarity between the current word vector and other word vectors, such that QK... T / d k 0.5 Used as attention weights. d k 0.5 This represents the normalization factor to ensure the stability of the SoftMax gradient. QK T / d k 0.5 The feature matrix V is weighted and summed to serve as the word vector for the next layer. This operation maintains the consistency of the input.

[0100] S13: QK T Involving the interaction between two word vectors, it can be extended to: QK T =(e i +p i W Q (e j +p j ) T (W K ) T , where e i and e j p represents the word vectors of the i-th and j-th tokens after E1 encoding. i and p j This represents the positional encoding of the i-th and j-th symbols;

[0101] S14: QK T Expanding the expression: QK T =e i W Q (W K )T e j T +e i W Q (W K ) T p j T +p i W Q (W K ) T e j T +p i W Q (W K ) T p j T The first item focuses on the interaction between two word vectors after word embedding, emphasizing explicit features; the fourth item involves positional encoding, which captures the positional relationship between word vectors.

[0102] S15: in

[0103] p t (i) p represents the positional encoding of the current word vector. i p j ∈p t (i) . d model The dimension of the word vector is represented by t, and the actual position of the word vector in the sequence is represented by t.

[0104] S16: Residual term p i p j T It contains relative position information, which can be used without a trainable matrix W. Q (W K ) T Under the condition of obtaining: p i p j T =T(ij);

[0105] S17:p i =[cosθ i sinθ i ] T p j T =[cosθ j sinθ j ] T p i and p j T θ represents the position matrix of word vectors in the feature space.i and θ j Let θ and θ represent the absolute position and angle of the current word vector in the feature space, respectively. μ =μθ0, where the scalar factor μ and θ μ Related to θ0;

[0106] S18: Introducing the trigonometric formula: p i p j T =cosθ i cosθ j +sinθ i sinθ j =cos(θ) i -θ j The transformation function T can be derived from cos[(ij)θ0], and the training matrix is ​​updated accordingly. During model training, the first term in step S14 is entered to continuously extract the fused salient features. Furthermore, the attention mechanism is combined with the fourth term to continuously acquire the fused feature information through positional encoding during the encoding process.

[0107] Neural network model methods for predicting patient severity and department of care include two strategies: one based on Concate([φ E1 (x u ),φ E3 (x n s The mixed encoded data F1 of the data is based on Concate([φ)). E2 (x u ),φ E3 (x n s The difference between the mixed encoded data F2 and the structured data φ lies mainly in that the latter targets mixed feature data. E3 (x n s The feature extraction steps are the same. Based on F1+φ E3 (x n s ) and F2+φ E3 (x n s The trained models are named Model M1 and Model M2, respectively. The workflow framework for this module is as follows: Figure 3 As shown.

[0108] The data processing steps for model M1 are as follows:

[0109] S19: Structured coded data and heterogeneous hybrid coded data are used as inputs to the parallel computing architecture and the serial architecture, respectively, with data dimensions of n×1×30 and n×43, where n represents the amount of data;

[0110] S20: The parallel computing architecture initially consists of three sequential branches. The first branch uses an LSTM module with a dropout of 0.2, resulting in an output dimension of n×1×32. The second and third branches use ReLU activation functions with convolutional kernels of 3×3 and 5×5 scales, respectively, both with an output dimension of n×1×32. Subsequently, the second and third branches use a Max Pooling module to aggregate high-level features, with an output dimension of n×1×32 for each branch. This enhances explicit features while reducing computational complexity.

[0111] S21: A concatenate layer is used to vertically fuse the branches into n×3×32 features. To avoid semantic redundancy due to multimodality, an LSTM module with dropout of 0.2 is then used as the feature extractor and Max Pooling module, both outputting high-level features with dimensions of n×1×36.

[0112] S22: The serial architecture first performs position encoding on the data based on the TokenAnd Position Embedding layer, with an output dimension of n×43×36. Then, it uses an integrated Transformer layer with 6 heads to extract features from the position-encoded data, with an output dimension of n×43×36 as well.

[0113] S23: A concatenate layer is used to fuse the outputs of the serial and parallel architectures into a hybrid feature with a global receptive field, with an output dimension of n×46×36. Subsequently, an attention mechanism with an output dimension of n×36 is used to capture more explicit and effective information, while also acting as a flattening layer to reduce the dimensionality of the data.

[0114] S24: The multi-output mode uses two parallel dense linear layers to predict the patient's disease severity and target department, respectively. The output dimensions are n×L and n×D, where L and D represent the severity level and the number of types of departments, respectively.

[0115] The data processing steps for model M2 are as follows:

[0116] S25: Structured coded data and heterogeneous hybrid coded data are used as inputs to the parallel computing architecture and the serial architecture, respectively. The data dimensions are n×1×30 and n×31×312, respectively, where n represents the amount of data.

[0117] S26: The parallel computing architecture initially consists of three sequential branches. The first branch uses an LSTM module with a dropout of 0.2, resulting in an output dimension of n×1×32. The second and third branches use ReLU activation functions with convolutional kernels of 3×3 and 5×5 scales, respectively, both with an output dimension of n×1×32. Subsequently, each branch uses a Max Pooling module to aggregate high-level features, with an output dimension of n×1×32. This enhances explicit features while reducing computational complexity.

[0118] S27: A concatenate layer is used to horizontally merge the branches into n×1×96 features. To avoid semantic redundancy due to multimodality, an LSTM module with dropout of 0.2 is then used as the feature extractor and Max Pooling module, both outputting high-level features with dimensions of n×1×32.

[0119] S28: The serial architecture first uses a bidirectional LSTM module with a dropout of 0.2, outputting n×31×64 dimensions. Then, it uses a Max Pooling module to obtain explicit features, outputting n×15×64 dimensions. Next, it uses a convolutional kernel with ReLU activation function and a scale of 3×3, outputting n×15×32 dimensions. Then, it uses a Max Pooling module again to obtain high-level features with dimensions of n×15×64 dimensions, while suppressing information with low correlation.

[0120] S29: A concatenate layer is used to fuse the outputs of the serial and parallel architectures into a hybrid feature with a global receptive field, with an output dimension of n×8×32. Subsequently, an attention mechanism with an output dimension of n×32 is used to capture more explicit and effective information, while also acting as a flattening layer to reduce the dimensionality of the data;

[0121] S30: Employs a multi-output mode, using two parallel dense linear layers to predict the patient's disease severity and target department, with output dimensions of n×L and n×D, where L and D represent the severity level and the number of departments visited, respectively.

[0122] Preferred embodiments of the present invention can be referred to the application scenario illustration. Figure 4 The specific steps are as follows:

[0123] 1. Audio information from the patient (4-2) and doctor (4-12) is collected via microphone 4-1. This audio information is then imported into the information processing module of the main controller (4-9) via interface conversion device 4-8 for timbre recognition and semantic analysis. Based on a "medical terminology keyword dictionary," natural language processing methods are used to analyze the doctor's judgment and the patient's relevant symptoms. The resulting keywords are then used to supplement or fill in information for other modalities and serve as the original unstructured data type. Table 1: Medical Terminology Keyword Dictionary.

[0124]

[0125] 2. Cameras 4-5 collect keyframe information of patients through target recognition and tracking. Then, the image recognition algorithm in the information processing module performs some pathological auxiliary judgments, such as facial paralysis symptom recognition, pain level recognition, trauma location and severity recognition, limb coordination degree recognition, etc. Subsequently, the semantics of the image are used to supplement or fill information in other modalities and serve as the original structured data.

[0126] 3. Use an electrocardiogram monitor to collect vital signs data and compare them with the reference range. If a critical value is found, immediately alert or call the doctor. Otherwise, make a more detailed judgment based on the reference range and use all judgment results as a supplement to the original structured data.

[0127] 4. For missing structured and unstructured data, use keyboard 4-11 and mouse 4-7 to fill or supplement them, such as: chief complaint, admission method, age, etc.

[0128] 5. Unstructured data is encoded using Tokenizer and BERT encoder respectively, and structured data is encoded using One-Hot encoder. Low-level feature fusion is used to fuse the encoded data. The resulting global mixed feature data and encoded structured data will be used as input to the prediction model.

[0129] 6. Predictive model M1 or M2 integrates the above-coded multimodal data to make real-time predictions of the patient's critical level and the department they should visit;

[0130] 7. Based on the data acquisition quality and model prediction results, the system can provide doctors with reference opinions or reminders through the earphone 4-4, and provide suggestions or guidance to patients or their families through the speaker 4-6 and the display screen 4-10.

[0131] 8. The proposed models M1 and M2 are compared with LSTM, LightGBM and Transformer on the same test set, respectively, with Accuracy, Sensitivity, F1-score, Size and detection time as performance indicators.

[0132] Table 2 Performance Comparison of Algorithms

[0133]

[0134] As shown in the table above, different models all exhibit good recognition performance on heterogeneous medical data after hybrid encoding. In particular, models M1 and M2, after advanced feature fusion, demonstrate higher recognition performance than other models. Therefore, the emergency triage method and system proposed in this invention, based on multi-encoder and multimodal information fusion, can effectively utilize various encoding strategies to characterize the features of heterogeneous medical data. Furthermore, the use of different levels of multimodal fusion methods enhances information complementarity between modalities. This approach has high practical value for emergency triage tasks in complex scenarios.

[0135] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. An emergency triage method based on multi-encoder and multi-modal information fusion, characterized in that, include: The steps include multimodal data acquisition, information fusion of multimodal heterogeneous medical data, and patient severity level-department prediction. The information fusion step of the multimodal heterogeneous medical data involves data cleaning and information fusion of the multimodal heterogeneous medical data, which includes structured data and unstructured data obtained by the multimodal data acquisition method. The data cleaning step takes multimodal data as input, analyzes the validity and completeness of the multimodal data, and uses filtering and correction methods to obtain effective triage data. The information fusion step includes a low-level feature fusion step in the data encoding stage and a high-level feature fusion step in the model training stage. The data encoding stage employs a multi-encoder combination strategy, using encoders including Tokenizer (E1 encoding), BERT (E2 encoding), and One-Hot (E3 encoding). The low-level feature fusion step in the data encoding stage includes the following sub-steps: S6: Represent the feature space of multimodal fusion as S su =ℝ m1 ×ℝ m2 ×⋯×ℝ mk ∈ (S s ×S u ), of which ℝ mk Let m represent the feature space of the k-th mode, and m k This represents the dimension of the space; L = (l1, l2, ..., ln) q ) represents a label space with q categories, while S s and S u These represent the structured and unstructured feature spaces, respectively. S7: The dataset can be represented as: D={(x i k , y i )|1≤k≤K, 1≤i≤n}∈(S su ×L) n ,in ϵ ℝ mk Indicates a dimension m k The eigenvector of the k-th mode, and y i ϵ L represents the relationship with x i k The associated labels, where K represents the total number of modalities and n represents the total number of samples in the current modality; S8: The three encoding strategies are represented as Φ = {ϕ E1 ϕ E2 ϕ E3 }; S9: Because there are many types of chief complaints and they are subjective, it can be represented as x u =(x1, x2, ⋯, x i , ⋯, x n ) ϵ S u , where x i =[ξ1, ξ2,⋯, ξ j ,⋯, ξ M ], ξ j M represents each word vector, where M represents x. i The maximum word vector length is used to ensure the integrity of each word vector feature; S10: Two encoding methods, E1 and E2, are used for unstructured data: ϕ E1 (x u )=[α1, α2,⋯, α i ,⋯, α n ] and ϕ E2 (x u )=[k1, k2,⋯, k i ,⋯, 𝛽 n Structured data uses E3 encoding: ϕ E3 (x n s )= {e1 Ks e2 Ks , ⋯,e i Ks , ⋯, e n Ks } ϵ S s ; S11: A hybrid coding strategy is used for structured and unstructured data to achieve two low-level feature fusion methods: F1 = Concate([ϕ E1 (x u ), ϕ E3 (x n s )]), F2 = Concate([ϕ E2 (x u ), ϕ E3 (x n s )]); The advanced feature fusion step during model training adds a concatenation layer to the network model to implement advanced feature fusion and attention mechanisms. The specific steps of the attention mechanism are as follows: S12: Attention(Q, K, V)=softmax(QK T / d k 0.5 )∙V, the linear transformation matrices Q, K and V are respectively: Q=XW Q K=XW K V=XW V X is x i Let represent the input sequence of words, and W represent the learnable parameter matrix corresponding to Q, K, and V. Q and K are used to calculate the similarity between the current word vector and other word vectors, such that QK T / d k 0.5 Used as attention weights; d k 0.5 QK represents the normalization factor. T / d k 0.5 The feature matrix V is weighted and summed to be used as the word vectors for the next layer; S13: QK T Involving the interaction between two word vectors, extended to: QK T =(e i +p i W Q (W K ) T (e j +p j ) T , where e i and e j p represents the word vectors of the i-th and j-th tokens after E1 encoding. i and p j This represents the positional encoding of the i-th and j-th symbols; S14: QK T Expanding the expression: QK T = e i W Q (W K ) T e j T +e i W Q (W K ) T p j T +p i W Q (W K ) T e j T +p i W Q (W K ) T p j T The first item focuses on the interaction between two word vectors after word embedding, emphasizing explicit features; the fourth item involves positional encoding, which captures the positional relationship between word vectors. S15: , , i = 0, 1, ..., (0.5 × d model -1), where p t (i) p represents the positional encoding of the current word vector. i p j ϵ p t (i) ; d model The dimension of the word vector is represented by the variable t, which represents the actual position of the word vector in the sequence. S16: Residual term p i p j T It contains relative positional information, which can be used without a trainable matrix W. Q (W K ) T Under the condition of obtaining: p i p j T =T(ij); S17: Introducing Euler's formula: p i =[cosθ i sinθ i ] T p j T =[cosθ j sinθ j ] T p i and p j T θ represents the position matrix of word vectors in the feature space. i and θ j Let represent the angles between the absolute positions of the current word vectors in the feature space, assuming 𝜃 𝜇 = 𝜇𝜃0, where the scalar factors 𝜇 and 𝜃 are equal to 0. 𝜇 Related to 𝜃0; S18: Introducing the trigonometric formula: p i p j T =cosθ i cosθ j +sinθ i sinθ j =cos(θ i −θ j )=>cos[(i−j)θ0], the transformation function T is derived from this, and the training matrix is ​​updated; during model training, the first term in step S14 is entered to continuously extract the fused salient features; in addition, the attention mechanism is combined with the fourth term to continuously obtain the fused feature information through positional encoding during the encoding process.

2. The emergency triage method based on multi-encoder and multi-modal information fusion according to claim 1, characterized in that, The multimodal data acquisition step includes the following sub-steps: S1: Based on image sensors and voice sensors, acquire image and voice information of patients going to the triage station, and use image semantic parsing method and voice semantic parsing method to acquire corresponding semantic information respectively. At the same time, use electrocardiogram monitor, mouse and keyboard to monitor and record the patient's vital signs information. S2: Based on the analyzed facial, trauma, and limb image information, voice information, vital sign information, and voice information from family members and doctors, match the patient's current symptoms. S3: If symptom information cannot be obtained, and the patient's department and severity level cannot be determined, ask the patient about their current feelings through the microphone, return to step S1 to collect the patient's image and voice information, and proceed to S2 for reassessment. If the patient's current symptoms still cannot be matched, call the nurse for treatment through the display screen or speaker. S4: The nurse interacts with the patient or family member via voice and expresses relevant information in a more standardized voice format, making it easier for the voice semantic analysis method to recognize and record semantics. Then, the nurse proceeds to steps S1 and S2 to match and identify the current symptoms. S5: If one or more of the following physiological indicators—blood pressure, heart rate, blood oxygen saturation, respiratory rate, or body temperature—reach a critical value, an alarm will be triggered and a nurse will be notified to intervene.

3. The emergency triage method based on multi-encoder and multi-modal information fusion according to claim 2, characterized in that, The image semantic parsing method includes the following steps: (1): Image dataset construction: Construct an image dataset of emergency patients; (2): Model training: Training of a classification model for emergency patients’ facial, trauma information and limb features based on deep neural networks; (3): Model deployment and recognition: Actively track and collect images of emergency patients, and extract keyframes based on the patient's facial expressions, trauma information, admission method and gestures; (4): Record and process key frame information of images in real time, and use the patient's pain level, trauma location, trauma severity, admission method, and combined speech response results as the output of the model to obtain a preliminary impression of the disease.

4. The emergency triage method based on multi-encoder and multi-modal information fusion according to claim 2, characterized in that, The speech semantic parsing method includes the following steps: (1): Construction of speech dataset: Construct speech keyword datasets for emergency patients, their families and nurses; (2): The effective components are extracted by using an adaptive sliding window method, and feature engineering is constructed based on the audio itself, the time domain and the frequency domain to obtain feature information; (3): Model training: Training of speech recognition models based on deep neural networks; (4): Model deployment: Actively identify and track the voice recordings of patients, family members and nurses; (5): Record and process voice information in real time, quickly and accurately capture keywords from the voice and establish association with medical terms to obtain preliminary chief complaint, past medical history, language function and patient vital signs information.

5. The emergency triage method based on multi-encoder and multi-modal information fusion according to claim 1, characterized in that, The patient severity-department prediction step includes two strategies: based on Concate([ϕ E1 (x u ), ϕ E3 (x n s The mixed encoded data F1 of the data is based on Concate([ϕ)). E2 (x u ), ϕ E3 (x n s The difference between the mixed encoded data F2 and the structured data F2 lies mainly in that they are for mixed feature data, while the structured data F2 is for mixed feature data. E3 (x n s The feature extraction steps are the same; the F1+ϕ-based method will be used. E3 (x n s ) and F2+ϕ E3 (x n s The trained models are named Model M1 and Model M2, respectively. The data processing steps for model M1 are as follows: S19: Structured coded data and heterogeneous hybrid coded data are used as inputs to the parallel computing architecture and the serial architecture, respectively, with data dimensions of n×1×30 and n×43, where n represents the amount of data; S20: The parallel computing architecture first consists of three serial branches. The first branch uses an LSTM module with a dropout of 0.2 and an output dimension of n×1×32. The second and third branches use ReLU activation functions, but with convolutional kernels of scales of 3×3 and 5×5, respectively, and the output dimension of each branch is n×1×32. Subsequently, the second and third branches use a Max Pooling module to gather high-level features, and the output dimension of each branch is n×1×32. S21: The Concatenate layer is used to vertically merge each branch into n×3×32 features. Then, an LSTM module with dropout of 0.2 is used as the feature extractor and Max Pooling module, and the output dimensions are both n×1×36 high-level features. S22: The serial architecture first performs position encoding on the data based on the Token And Position Embedding layer, with an output dimension of n×43×36. Then, it uses an integrated Transformer layer with 6 heads to extract features from the position-encoded data, with an output dimension of n×43×36 as well. S23: The Concatenate layer is used to merge the outputs of the serial and parallel architectures into a hybrid feature with a global receptive field, with an output dimension of n×46×36; then an attention mechanism with an output dimension of n×36 is used to capture more explicit and effective information, while also acting as a flattening layer to reduce the dimensionality of the data; S24: The multi-output mode uses two parallel Dense linear layers to predict the patient's disease severity and target department, respectively. The output dimensions are n×L and n×D, where L and D represent the severity level and the number of types of departments, respectively. The data processing steps for model M2 are as follows: S25: Structured coded data and heterogeneous hybrid coded data are used as inputs to the parallel computing architecture and the serial architecture, respectively. The data dimensions are n×1×30 and n×31×312, respectively, where n represents the amount of data. S26: The parallel computing architecture first consists of three serial branches. The first branch uses an LSTM module with a dropout of 0.2 and an output dimension of n×1×32. The second and third branches use ReLU activation functions, but with convolutional kernels of 3×3 and 5×5 scales respectively, and the output dimension of each branch is n×1×32. Then, each branch uses a Max Pooling module to gather high-level features, and the output dimension of each branch is n×1×32. This can enhance explicit features and reduce computational complexity. S27: The Concatenate layer is used to horizontally merge the branches into n×1×96 features. Then, an LSTM module with dropout of 0.2 is used as the feature extractor and Max Pooling module, and the output dimensions are both n×1×32 high-level features. S28: The serial architecture first uses a bidirectional LSTM module with a dropout of 0.2, with an output dimension of n×31×64. Then, it uses a Max Pooling module to obtain explicit features, with an output dimension of n×15×64. Next, it uses a convolution kernel with the activation function ReLU and a scale of 3×3, with an output dimension of n×15×32. Then, it uses the Max Pooling module again to obtain high-level features with a dimension of n×15×64, while suppressing information with low correlation. S29: The Concatenate layer is used to fuse the outputs of the serial and parallel architectures into a hybrid feature with a global receptive field, with an output dimension of n×8×32; then an attention mechanism with an output dimension of n×32 is used to capture more explicit and effective information, while also acting as a flattening layer to reduce the dimensionality of the data; S30: Employs a multi-output mode, using two parallel dense linear layers to predict the patient's disease severity and target department, with output dimensions of n×L and n×D, where L and D represent the severity level and the number of departments visited, respectively.

6. The emergency triage method based on multi-encoder and multi-modal information fusion according to claim 1, characterized in that, The structured data includes vital signs, admission method, demographic characteristics, image semantic information, consciousness, and language function. The vital signs include the patient's heart rate, blood pressure, blood oxygen saturation, respiratory rate, and body temperature. The admission method includes ambulance, walking, wheelchair, and being carried in. The demographic characteristics include age and gender. The image semantic information includes pain level, presence or absence of facial paralysis symptoms, consciousness, location and severity of trauma, and presence or absence of limb movement impairment. The unstructured data includes the chief complaint, past medical history, and speech information. The results of the speech information parsing are linked with structured or unstructured parameters and then used for filling or supplementing.

7. The emergency triage method based on multi-encoder and multi-modal information fusion according to claim 1, characterized in that, The data cleaning steps first sort the triage data according to the patient's latest visit time and fill in missing values ​​to facilitate further data processing by the pandas library. Then, all data is traversed and indexes are recorded. Next, the validity of the triage data is comprehensively judged based on the patient's name and ID, whether keywords in the chief complaint are included in the set blacklist, or whether the chief complaint is the filler character from the previous step. If invalid, the entire triage information is deleted, and the data is traversed and indexes are recorded again. If valid, it is determined whether identical or incorrect triage information is due to keyboard and mouse input. If so, duplicates are deleted; if not, it is further determined whether the same patient has been triaged multiple times. If not, the data is directly considered valid; otherwise, it indicates an incorrect triage for the patient, and the validity of the triage needs to be judged based on the visit time. If the same patient has multiple visits at the same time, it indicates that the doctor temporarily corrected the incorrect triage result, and the latest visit time represents valid data. Then, any formatting errors in the valid data are corrected, and the final triage data obtained is the valid triage data.

8. An emergency triage system based on multi-encoder and multimodal information fusion, used to implement the emergency triage method based on multi-encoder and multimodal information fusion as described in any one of claims 1-7, comprising a data acquisition and interaction module, an interface conversion device, a main controller module, and a power supply, wherein the data acquisition and interaction module includes an image sensor, a sound sensor, an electrocardiogram monitor, a display, a speaker, a mouse, and a keyboard; the main controller module includes a CPU, a GPU, and memory; the data acquisition and interaction module is connected to the main controller module through the interface conversion device; and the power supply provides power to both the data acquisition and interaction module and the main controller module.

Images