Ward rotation evaluation and model construction method based on multi-modal information

Abstract

The invention provides a ward rotation evaluation and model construction method based on multi-modal information, and belongs to the technical field of deep learning. The method comprises the following steps: firstly, constructing a multi-modal clinical data set, integrating five types of heterogeneous data including operation records, vital sign time sequence data, medical images, medication records and nursing logs, labeling postoperative state labels of patients, and simulating a clinical real scene to generate a training set and a test set containing random missing modals; secondly, constructing a ward rotation evaluation model, wherein the model is used for improving and designing a traditional teacher-student network architecture into a double-path heterogeneous distillation model; meanwhile, designing a multi-task loss joint optimization strategy for aligning the characteristics of the teacher-student network and constraining the evaluation result of the teacher network, and finally deploying a trained model to an edge computing device, receiving a multi-modal data stream in real time and outputting a transfer suggestion and a risk score; the invention provides a brand new intelligent solution under a ward rotation scene, and improves the evaluation robustness of a multi-modal missing scene.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning technology, and in particular relates to a ward rotation evaluation and model building method based on multimodal information. Background Art

[0002] Ward rotation assessment is the core link to optimize the allocation of medical resources and ensure patient safety. Its assessment results need to integrate multimodal information such as surgical information, vital signs, imaging data, medication records, and nursing logs. These modal data are not only used to judge the degree of recovery of the patient's physiological state and predict the risk of complications, but also an important basis for formulating personalized nursing plans and dynamically adjusting ward resource allocation. Studies have shown that there is a significant correlation between the patient's postoperative recovery process and multimodal data, but different disease types, surgical methods and individual differences will lead to highly complex correlation patterns between modalities. Traditional ward rotation assessment methods mainly rely on manual experience judgment and single-modal data analysis. Manual experience judgment is a highly subjective assessment method with the following limitations: 1) The assessment process depends on the clinical experience of medical staff and is easily affected by subjective cognitive biases, so the objectivity and consistency of the results are difficult to guarantee; 2) The real-time integration and analysis of multimodal data (such as dynamic vital signs and imaging changes) is inefficient and cannot quickly respond to sudden changes in the condition.

[0003] Single-modal data analysis is based on a single data source (such as blood oxygen or heart rate only) for risk assessment, which has the following limitations: 1) It ignores the synergy and complementarity between multiple modalities, for example, the correlation between early postoperative vital signs and nursing logs may be omitted; 2) When some modalities are missing (such as data loss due to equipment failure), the model performance is significantly reduced, which is prone to misjudgment. With the development of artificial intelligence technology, multimodal fusion methods are gradually applied in the field of medical assessment, mainly including traditional fusion methods and fusion methods based on deep learning.

[0004] Traditional fusion methods: usually use weighted average fusion or decision-level fusion to integrate multimodal features through rules or statistical methods. However, these methods have limitations: 1) they rely on complete modal input, and manual data supplementation is required in missing scenarios, resulting in evaluation delays; 2) they cannot dynamically adapt to nonlinear relationships between modalities, such as the complex temporal association between surgical information and image features.

[0005] Fusion method based on deep learning: Use neural network to automatically learn the interaction relationship between modalities. However, existing methods still have the following problems: 1) In scenarios with high missing rate (>30%), traditional zero filling or mean filling will introduce noise and destroy the semantic consistency of features; 2) Weak modal combinations (such as text records and discrete vital signs data) are not discriminative enough, and the model is prone to ignore potential risk features; 3) Data privacy restrictions in medical scenarios lead to desensitization of some modalities, and existing methods are difficult to adapt to dynamic missing patterns. Summary of the Invention

[0006] In view of the above problems, the first aspect of the present invention provides a method for constructing a ward rotation evaluation model based on multimodal information, including the following processes:

[0007] S1. Make a multimodal clinical data set, including the collection of five types of heterogeneous data: surgical records, vital sign time series data, medical images, medication records, and nursing logs. Corresponding postoperative status labels of patients are marked, namely "can be transferred to a general ward", "need to be continuously observed", and "high-risk complications". After combining the data samples with the labels, they are randomly divided into a training set and a test set;

[0008] S2. Construct a ward rotation evaluation model, and improve the traditional teacher-student network architecture into a dual-path heterogeneous distillation model, including a teacher network and a student network. The teacher network includes a patient cross-modal data fusion module and a Softmax activation function layer, and the student network includes a randomly missing modal data fusion module and a softmax activation function layer; when the multimodal data is complete, input it into the teacher network to obtain the predicted value of the patient's postoperative status, and when part of the multimodal data is missing, input it into the student network to obtain the predicted value of the patient's postoperative status;

[0009] S3. Train the dual-path heterogeneous distillation model, design a multi-task loss joint optimization strategy, including a boundary-aware distillation loss and a multimodal regularization loss, to align the features of the teacher-student network, constrain the prediction results of the teacher network, and design a risk assessment loss to optimize the prediction performance of the patient's postoperative status, and obtain the final ward rotation evaluation model.

[0010] Preferably, when making the multimodal clinical data set, five types of heterogeneous data are collected from the hospital information system, including surgical records, vital sign time series data, medical images, medication records, and nursing logs; the surgical record is structured text data, including information such as the type of surgery, the duration of the surgery, and the anesthesia method; the vital sign time series data is time series data, including heart rate, blood pressure, and blood oxygen saturation; the medical image is a postoperative CT image; the medication record is structured data, including the drug name, dosage, and administration time; the nursing log is unstructured text data, recording the patient's nursing situation and postoperative recovery status;

[0011] It is marked by clinical experts according to the postoperative patient status, and the marked labels include three categories: "can be transferred to a general ward", "need to be continuously observed", and "high-risk complications".

[0012] Preferably, the teacher network includes a patient cross-modal data fusion module and a softmax activation function layer. The patient cross-modal data fusion module extracts data features of different modalities from five types of modal data, namely surgical records, vital sign time-series data, medical images, medication records, and nursing logs, and performs cross-modal fusion to obtain a patient state representation. The patient cross-modal data fusion module includes a lightweight encoder module and a multi-modal co-attention fusion module based on semantic grouping. The lightweight encoder module takes the five types of modal data, namely surgical records, vital sign time-series data, medical images, medication records, and nursing logs, as inputs, obtains the features of the five modalities respectively, then divides the features of the five modalities into high-dimensional semantic groups and low-dimensional semantic groups, and uses the multi-modal co-attention fusion module based on semantic grouping to fuse the features of different dimensions to obtain a fused feature. The softmax activation function layer is used to map the fused feature to the patient's postoperative state.

[0013] Preferably, the lightweight encoder module includes a surgical record feature encoder, a vital sign time-series data feature encoder, a medical image feature encoder, a medication record feature encoder, and a nursing log feature encoder.

[0014] The surgical record feature encoder includes a BERT model, a bidirectional LSTM, and a global max pooling layer.

[0015] The vital sign time-series data feature encoder includes a Z-score normalization layer, an LSTM layer, and a multi-layer perceptron layer.

[0016] The medical image feature encoder includes a pre-trained ResNet50 and a fully connected layer.

[0017] The medication record feature encoder consists of an attention mechanism layer.

[0018] The nursing log feature encoder includes a rule matching layer, which extracts key events through rule matching and converts them into 32-dimensional binary vectors.

[0019] Preferably, the multi-modal co-attention fusion module based on semantic grouping fuses multi-modal features. The five types of modalities are divided into a high-dimensional semantic group based on feature dimension: surgical record features, vital sign time-series data features, medical image features, and a low-dimensional auxiliary group: medication record features, nursing log features. Intra-group and cross-group attention calculations are performed respectively, as follows:

[0020] S51, for surgical record feature f T and medical image feature f I , first generate a query matrix Q through a linear transformation T = W Q f T and key matrix K I= W K f I , calculate the cross-modal attention weight A TI , A TI reflects the semantic association strength between the surgical text description and the image features. Secondly, through the value matrix V I = W V f I perform weighted fusion to obtain the cross-modal feature f of the surgical record and the medical image TI = A TI V I , thirdly, generate the query matrix Q through linear transformation I = W Q f I and the key matrix K T = W K f T , calculate the cross-modal attention weight A TI , A TI reflects the semantic association strength between the medical image features and the surgical record features. Through the value matrix V T = W V f T perform weighted fusion to obtain the cross-modal feature f of the medical image and the surgical record IT = A IT V T ;

[0021] Repeat the above process to finally obtain the cross-modal feature f of the surgical record and the medical image TI , the cross-modal feature f of the medical image and the surgical record IT , the cross-modal feature f of the surgical record and the vital sign time series data TS , the cross-modal feature f of the vital sign time series data and the surgical record ST , the cross-modal feature f of the medical image and the vital sign time series data IS , the cross-modal feature f of the vital sign time series data and the medical image SI ;

[0022] S52, then perform attention weighted fusion on the medication record features and nursing log features of the low-dimensional semantic group in the way of S51 to obtain the cross-modal feature f of the medication record and the nursing log MP and the cross-modal feature f of the nursing log and the medication record PM ;

[0023] S53, combine f TI , f IT , f TS , f ST , f IS , f SI , f MP and f PMInput the global attention pooling layer to integrate the co-attention features of the high-dimensional semantic group and the co-attention features of the low-dimensional auxiliary group, and integrate and pool the features of different subspaces into a 256-dimensional fusion feature F final 。

[0024] Preferably, the student network includes a randomly missing modality data fusion module and a softmax activation function layer. The randomly missing modality data fusion module includes a lightweight encoder module, a dynamic modality missing layer, and a multiple co-attention fusion module based on semantic grouping. Among them, the lightweight encoder module and the multiple co-attention fusion module based on semantic grouping have the same module structure as those of the teacher network. The specific data processing process includes:

[0025] S61, Input the complete five types of modality data X = {T, S, I, M, P} into the lightweight encoder module to obtain the features f T 、f S 、f I 、f M and f P ;

[0026] S62, Input the features of the five modalities into the dynamic modality missing layer for modality missing processing to obtain the missing modality features. Specifically, use the Bernoulli indicator to set α% of the eigenvalue of each of f T 、f S 、f I 、f M and f P in the five modalities to 0, and obtain the missing modality features If T 、If S 、If I 、If M and If P ;

[0027] S63, Input the missing modality features If T 、If S 、IF I 、If M and If P into the multiple co-attention fusion module based on semantic grouping to obtain the missing modality fusion feature IF final ;

[0028] S64, Input the missing modality fusion feature IF final into the softmax activation function layer to obtain the postoperative status label y d 。

[0029] Preferably, the multi-task loss joint optimization strategy jointly optimizes three types of losses for end-to-end optimization. The multi-task loss is defined as:

[0030] L total = L CE + ∈L MAD +(1 - ∈)L MAR

[0031] L CE is the risk assessment loss, L MAD is the boundary-aware distillation loss, L MAR is the multi-modal perception regularization loss, where ∈ is the weight coefficient, and L MAD and L MAR have a weight sum of 1.

[0032] The second aspect of the present invention also provides a ward rotation evaluation method based on multi-modal information. The ward rotation evaluation model constructed by the construction method described in the first aspect is deployed in a medical edge computing device, and includes the following processes:

[0033] Real-time collect multi-modal clinical data streams through the hospital information system and Internet of Things devices, including surgical records, extract structured text data from the electronic medical record system EMR; vital sign time-series data, collect indicators including heart rate, blood pressure, and blood oxygen saturation every minute through bedside monitoring devices; medical images, receive DICOM-format images transmitted by postoperative X-ray machines or CT devices in real time; medication records and nursing logs, synchronize structured medication data and unstructured nursing texts from the pharmacy management system and mobile nursing terminals;

[0034] Input into the ward rotation evaluation model. When the multi-modal data is complete, input into the teacher network to obtain the patient's postoperative status. When the multi-modal data is missing, input into the student network to obtain the patient's postoperative status. The status includes three categories: "can be transferred to a general ward", "needs to be observed continuously", and "high-risk complications".

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] (1) The present invention has carried out a brand-new scenario data design, using five types of heterogeneous data, namely surgical records, vital sign time-series data, medical images, medication records, and nursing logs, as the input of the learning model, and the output is three categories of status: "can be transferred to a general ward", "needs to be observed continuously", and "high-risk complications", providing a brand-new intelligent solution for the ward rotation scenario.

[0037] (2) The present invention improves the evaluation robustness in multi-modal missing scenarios: The dual-path heterogeneous distillation model designed by the present invention models the correlation information among five modalities through mutual-guided learning, effectively modeling the interaction knowledge between different modalities, thereby robustly modeling the missing modal features in the student network and improving the stability of the model in the scenario of incomplete modalities; The distillation strategy optimized for the medical scenario aligns the decision boundaries of the teacher-student network, improves the recognition sensitivity for high-risk patients, and reduces the risk of clinical misjudgment.

[0038] (3) Enhance the discriminative ability of weak modal combinations: The multi-modal perception regularization (MAR) loss proposed by the present invention strengthens the representation learning of weak combinations through a dynamic weighting strategy, reducing the risk of misjudgment caused by the missing of key modalities.

[0039] (4) Support the needs of clinical real-time decision-making: The lightweight student network and edge computing framework deployed by the present invention support real-time inference of bedside devices, meeting the requirements of clinical scenarios for low-latency evaluation; Through the dynamic mask compensation mechanism and the training strategy simulating clinical missing patterns, it can adapt to complex data missing scenarios such as device failures and privacy desensitization, and improve the clinical practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is the overall method technical roadmap of the present invention.

[0041] Figure 2 It is a schematic diagram of the collection and processing process of multi-modal data.

[0042] Figure 3 It is a schematic diagram of the processing flow of the dual-path heterogeneous distillation model of the present invention.

[0043] Figure 4 It is a schematic diagram of the structure of the multi-modal co-attention fusion module based on semantic grouping of the present invention.

[0044] Figure 5 It is a schematic diagram of the architecture of the boundary-aware distillation loss MAD of the present invention.

[0045] Figure 6 It is a schematic diagram of the model evaluation results in the embodiments of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0046] The present invention mainly provides a ward rotation evaluation method based on multi-modal information, and the overall process is as Figure 1 shown:

[0047] Constructing a multimodal clinical dataset: Collecting and annotating five types of heterogeneous data, including surgical records, time-series vital sign data, medical images, medication records, and nursing logs, and annotating the postoperative status labels of patients, namely "transferable to general ward", "needs continuous observation", and "high-risk complications". After combining the data samples with the labels, randomly divide them into a training set and a test set.

[0048] Constructing a ward rotation evaluation model: Improving the traditional teacher-student network architecture into a dual-path heterogeneous distillation model, including a teacher network and a student network. The teacher network includes a patient cross-modal data fusion module and a Softmax activation function layer, and the student network includes a missing modality data fusion module, a dynamic modality missing layer, and a boundary-aware distillation module. When the multimodal data is complete, input it into the teacher network to obtain the patient's postoperative status. When the multimodal data is missing, input it into the student network to obtain the patient's postoperative status.

[0049] Model training based on a multi-task loss function: Designing a boundary-aware distillation loss and a multimodal regularization loss to align the features of the teacher-student network and constrain the evaluation results of the teacher network.

[0050] Model deployment and real-time inference: Deploying the trained dual-path heterogeneous distillation model (ward rotation evaluation model) to a medical edge computing device, receiving the multimodal data stream in real time, and outputting the predicted value of the patient's postoperative status as the basis for ward transfer.

[0051] The invention will be further described below in conjunction with specific embodiments.

[0052] I. Production of multimodal clinical dataset

[0053] The overall process is as Figure 2 shown:

[0054] Data acquisition: Collecting five types of heterogeneous data from the hospital information system, including surgical records, time-series vital sign data, medical images, medication records, and nursing logs; the surgical records are structured text data, containing information such as surgical type, surgical duration, and anesthesia method; the time-series vital sign data is time-series data, including heart rate, blood pressure, and blood oxygen saturation, recorded once per minute; the medical images are postoperative CT images with a resolution of 512×512 pixels; the medication records are structured data, containing drug name, dosage, and administration time; the nursing logs are unstructured text data, recording the patient's nursing situation and postoperative recovery status.

[0055] Data analysis and annotation: Clinical experts make annotations based on the postoperative status of patients. The annotation labels include three categories: "Can be transferred to the general ward", "Needs continuous observation", and "High-risk complications". During the annotation process, experts need to comprehensively judge by combining the changes in vital signs, imaging examination results, and nursing log content within 24 hours after the operation to ensure the accuracy and consistency of the annotations.

[0056] Data preprocessing: Standardize the collected raw data; perform word segmentation, stop word removal, and word vectorization on the operation records and nursing logs; normalize the time series data of vital signs to eliminate the influence of dimensions; perform grayscale conversion, noise reduction, and size unification on medical images; perform drug category coding and discretization of administration time on the medication records.

[0057] II. Construct a ward rotation evaluation model

[0058] In response to the challenge of random missing of multi-modal data in the clinical scenario. Specifically, in the evaluation of the postoperative status of patients, the key modality may be missing due to equipment failures or data collection limitations, resulting in misjudgments by the model relying on a single modality. The present invention proposes a dual-path heterogeneous distillation model, which transfers complete modality knowledge to the student network through the teacher network and aligns the decision boundaries by combining the Margin-Aware Distillation (MAD) strategy, thereby solving the problems of feature deviation and insufficient generalization ability caused by modality missing in traditional models. Its structure diagram can be seen Figure 3 , the model, through the dual-path heterogeneous distillation architecture, forces the deployed network to learn the complementary relationship between modalities (such as the compensatory effect of medication records on missing images) during the training phase and inherit the refined decision boundary refined by the teacher network in the complete modality, so that even in the inference phase when facing partial modality missing, it can still maintain high-precision risk assessment.

[0059] 1. Design of the teacher network:

[0060] The teacher network aims to learn comprehensive cross-modal knowledge. The teacher network includes a patient cross-modal data fusion module and a softmax activation function layer. The patient cross-modal data fusion module designed in the present invention extracts data features of different modalities from five modalities of operation records, time series data of vital signs, medical images, medication records, and nursing logs and performs cross-modal fusion to obtain a patient status representation. The patient cross-modal data fusion module includes a lightweight encoder module and a multiple co-attention fusion module based on semantic grouping. The lightweight encoder module takes five modalities of data including operation records, time series data of vital signs, medical images, medication records, and nursing logs as inputs and obtains features of the five modalities respectively. Then, the features of the five modalities are divided into high-dimensional semantic groups and low-dimensional semantic groups, and the multiple co-attention fusion module based on semantic grouping is used to fuse features of different dimensions to obtain fused features; the softmax activation function layer is used to map the fused features to the postoperative status of the patient.

[0061] First, input the complete five types of modal data X = {T, S, I, M, P} into the patient cross-modal data fusion module to obtain the fusion feature F final , and then input the fusion feature F final into the softmax activation function layer to obtain the patient's postoperative status label y t .

[0062] (1) Lightweight encoder module

[0063] Use the lightweight encoder module to encode the multi-modal data: the multi-modal data includes surgical records, vital sign time-series data, medical images, medication records, and nursing logs. The lightweight encoder group includes a surgical record feature encoder, a vital sign time-series data feature encoder, a medical image feature encoder, a medication record feature encoder, and a nursing log feature encoder, as follows:

[0064] Surgical record data (T) feature extraction: Input the surgical record data T into the surgical record feature encoder, which includes a BERT model, a bidirectional LSTM, and a global max pooling layer.

[0065] First, use the BERT model for word embedding (output dimension d = 768), and then extract sequence features through the bidirectional LSTM network to capture the context semantics of the text:

[0066] h t = LSTM(F BERT (T), h t-1 )

[0067] where h t-1 is the hidden state at time step t. The bidirectional LSTM can capture both the global description and local details of the surgical record through the concatenation of the forward and backward hidden states;

[0068] Secondly, obtain the 256-dimensional surgical record data feature f T through the global max pooling layer:

[0069] f T = MaxPool([h1, h2,..., h n )

[0070] This design avoids the defect of the traditional bag-of-words model ignoring context relationships and ensures the complete expression of key surgical information.

[0071] Vital sign time series data (S): The vital sign time series data (S) is input into the vital sign time series data feature encoder, which includes a Z-score normalization layer, an LSTM layer, and a multi-layer perceptron layer (MLP). First, due to the dimensionality differences of different vital signs interfering with model learning, the vital sign time series data needs to be input into the Z-score normalization layer for Z-score normalization to obtain the normalized vital sign time series data

[0072] Secondly, the is input into the LSTM network to model the temporal dependence relationship through the gating mechanism:

[0073]

[0074] The choice of LSTM stems from its memory ability for long sequence data, which can effectively capture the short-term fluctuations and long-term trends of postoperative vital signs.

[0075] Finally, the is input into the MLP, and the MLP is used to extract the hidden state as a 128-dimensional vital sign time series data feature vector f S :

[0076] f S = MLP(s n )

[0077] Medical image (I): The medical image (I) is input into the medical image feature encoder for extracting high-level semantic features to evaluate the recovery situation. The medical image feature encoder includes a pre-trained ResNet50 and a fully connected layer.

[0078] First, to avoid the high computational cost of training from scratch, a pre-trained ResNet50 model (freezing the weights of the first 20 layers) is used to extract the high-level semantic features f I :

[0079]

[0080] The residual structure of ResNet50 can alleviate the problem of gradient disappearance in deep networks, and its pre-trained weights on large-scale image datasets significantly improve the representation ability of medical image features.

[0081] Secondly, the is reduced to 512-dimensional features through the fully connected layer:

[0082]

[0083] where W l and b Iare the weights and biases of the fully connected layer.

[0084] Medication record (M): The medication record (M) is used as structured data and input into the medication record feature encoder, which consists of an attention mechanism layer, specifically as follows:

[0085] α m = Softmax(W m ·M)

[0086]

[0087] where W m is the weight matrix, α m is the attention coefficient of the medication record feature channel, and f M is the medication record feature, with a dimension of 64.

[0088] The attention mechanism makes f M more focused on the key therapeutic drug text information and ignores the background text information.

[0089] Nursing log (P): The nursing log data is input into the nursing log feature encoder, which includes a rule matching layer. Key events such as "wound exudation" and "abnormal body temperature" are extracted through rule matching and transformed into 32-dimensional binary vectors, specifically as follows:

[0090] f p = RuleMatch(P)

[0091] where f p is the nursing log feature.

[0092] This design balances the automatic learning of structured data and the domain knowledge guidance of unstructured text, avoiding overfitting of the nursing log by complex models.

[0093] (2) Multi-modal co-attention fusion module based on semantic grouping

[0094] The multi-modal features are fused using the multi-modal co-attention fusion module based on semantic grouping. The specific architecture diagram can be seen Figure 4 : To solve the semantic gap problem between heterogeneous modalities, this module introduces a cross-modal co-attention mechanism to enhance the correlation between the five modalities through interactive feature alignment. Traditional single-modal attention only focuses on its own features and is difficult to capture the potential cross-modal correlations, such as the correspondence between the surgical type and the postoperative imaging features. Therefore, this module divides the five types of modalities into a high-dimensional semantic group (surgical record features, vital sign time series data features, medical imaging features) and a low-dimensional auxiliary group (medication record features, nursing log features) based on the feature dimension, and calculates the intra-group and cross-group attention respectively, including the following processes:

[0095] 1) Perform attention-weighted fusion on the surgical record features, vital sign time-series data features, and medical image features of the high-dimensional semantic group, specifically as follows:

[0096] For the surgical record feature f T and the medical image feature f I , first generate the query matrix Q through a linear transformation T = W Q f T and the key matrix K I = W K f I , and calculate the cross-modal attention weights:

[0097]

[0098] This weight matrix A TI reflects the semantic association strength between the surgical text description and the image features. Secondly, perform weighted fusion through the value matrix V I = W V f I to obtain the cross-modal feature f TI of the surgical record and the medical image = A TI V I . Thirdly, generate the query matrix Q through a linear transformation I = W Q f I and the key matrix K T = W K f T , and calculate the cross-modal attention weights:

[0099]

[0100] This weight matrix A TI reflects the semantic association strength between the medical image features and the surgical record features. Secondly, perform weighted fusion through the value matrix V T = W V f T to obtain the cross-modal feature f IT of the medical image and the surgical record = A IT V T .

[0101] Repeat the above steps to finally obtain the cross-modal feature f TI of the surgical record and the medical image, the cross-modal feature f IT of the medical image and the surgical record, the cross-modal feature f TS of the surgical record and the vital sign time-series data, and the cross-modal feature f ST, Cross-modal features f of medical images and vital sign time-series data IS , Cross-modal features f of vital sign time-series data and medical images SI .

[0102] 2) Perform attention-weighted fusion on the medication record features and nursing log features of the low-dimensional semantic group. The method is the same as 1), and cross-modal features f of the medication record and nursing log are obtained MP and cross-modal features f of the nursing log and medication record PM .

[0103] 3) Input f TI , f IT , f TS , f ST , f IS , f SI , f MP and f PM into the global attention pooling layer to integrate the co-attention features of the high-dimensional semantic group and the co-attention features of the low-dimensional auxiliary group, and pool the features of different subspaces into a 256-dimensional fusion feature F final .

[0104] The multiple co-attention mechanism helps to adapt to the challenge of random modality loss in clinical scenarios and achieve adaptive fusion under missing modalities. In the case of complete modalities, the model needs to balance the attention values between modalities; while in the case of partial modality loss, it is necessary to dynamically adjust the attention contribution of the remaining modalities to the missing modalities to avoid feature bias caused by the loss. Specifically, first calculate the initial attention weights of the five types of modalities through the fully connected layer.

[0105] 2. Student network design

[0106] The student network is used to imitate the output or intermediate representation of the teacher network to achieve model compression or efficiency improvement, and is suitable for deployment on resource-constrained devices. At the same time, the student network is used to learn the feature representation under modality loss to adapt to the clinical real scenario. The student network includes a random missing modality data fusion module and a softmax activation function layer. The random missing modality data fusion module includes a lightweight encoder module, a dynamic modality missing layer, and a multiple co-attention fusion module based on semantic grouping. Among them, the lightweight encoder module and the multiple co-attention fusion module based on semantic grouping have the same module structure as those of the teacher network; its specific data processing process includes:

[0107] First, input the complete five types of modality data X = {T, S, I, M, P} into the random missing modality data fusion module to obtain the missing modality fusion feature IF final , specifically

[0108] 1) Input X = {T, S, I, M, P} into the lightweight encoder module to obtain the features f of five modalities T , f S , f I , f M and f P ;

[0109] 2) Next, input the features of the five modalities into the dynamic modality determination layer for modality missing processing to obtain the missing modality features. Specifically, use the Bernoulli indicator to set α% of the eigenvalue of each of the five modalities of f T , f S , f I , f M and f P to 0 (α% is a random value, with a value range of 10% - 80%) to obtain the missing modality features If T , If S , If I , If M and If P . This design forces the student network to explore the redundancy and complementarity between modalities through random modality dropout, thereby improving robustness and realizing the patient risk assessment inference in the case of missing modalities.

[0110] 3) Input the missing modality features If T , If S , If I , If M and If P into the multi - co - attention fusion module based on semantic grouping to obtain the missing modality fusion feature IF final .

[0111] Then, input the missing modality fusion feature IF final into the softmax activation function layer to obtain the patient's postoperative status label y d .

[0112] III. Model Training Based on Multi - task Loss Function

[0113] This embodiment designs a multi - task loss joint optimization strategy, including three types of losses: boundary - aware distillation loss, weak modality regularization loss, and risk assessment, to force the dual - path heterogeneous distillation module to balance the global decision - making ability and local feature robustness in the real - world clinical scenario of missing modalities.

[0114] 1. Boundary - aware distillation (MAD) loss:

[0115] MAD transfers the decision - boundary knowledge of the teacher network to the student network through relation consistency to solve the problem that it is difficult to distinguish patient samples near the classification boundary. To avoid the output features F of the teacher network finalDirectly align the output features IF of the student network final This causes overfitting in both the teacher network and the student network. Meanwhile, to maintain the consistency between the teacher network and the student network in the sample relationship space, the present invention designs the MAD loss to train the dual-path heterogeneous distillation module, which is specifically as follows:

[0116] First, flatten the output features F of the teacher network final and the output features IF of the student network final respectively to calculate the cosine similarity relationship matrix, which is specifically as follows:

[0117]

[0118] where r t (i, j) is the cosine similarity relationship matrix of the output features of the teacher network, and r d (i, j) is the cosine similarity relationship matrix of the output features of the student network.

[0119] Second, calculate the difference between r t (i, j) and r d (i, j) and sum them up to obtain the difference vector g td :

[0120]

[0121] where N and M are the number of rows and columns of the cosine similarity relationship matrix respectively.

[0122] Third, for patient samples near the decision boundary, such as patients with normal vital signs but abnormal imaging, these samples usually have higher classification uncertainty and need to be given greater weights to strengthen boundary learning. Therefore, based on the output yt of the teacher network, calculate the information entropy as a measure of uncertainty:

[0123]

[0124] where σ is the Softmax function, and the larger the value of π t (k), the closer the patient sample k is to the decision boundary.

[0125] Finally, use the uncertainty measure π t (k) as the weight of the difference vector g td to calculate the MAD loss function:

[0126]

[0127] where K is the number of patient samples.

[0128] The overall architecture is as Figure 5 shown.

[0129] 2. Multi-modal Perception Regularization Loss:

[0130] In view of the model bias problem caused by random missing of modalities in the clinical scenario, the present invention designs a multi-modal perception regularization (MAR) loss. This loss dynamically identifies weak modality combinations, such as combinations containing only medication records and nursing logs, and forces the deployed network to strengthen the learning of these combinations, thereby preventing the model from over-relying on strong modality combinations and improving the generalization ability in complex missing scenarios. Specifically as follows:

[0131] Quantification of Modality Combination Confidence: In the postoperative patient status assessment, the prediction reliabilities of different missing modality combinations vary significantly. For example, relying solely on nursing logs (weak combination) may lead to misjudgment due to vague text descriptions, while imaging + vital signs (strong combination) has high confidence. To quantify this difference, the present invention calculates the KL divergence difference g between the prediction results of the missing modality and the complete modality d :

[0132]

[0133] where σ is the Softmax function, and the larger g d is, the greater the difference between the prediction result distribution of the k-th patient sample under modality missing and the prediction result distribution under the complete modality, that is, the lower the confidence of the prediction result under modality missing. This design captures the global confidence difference (such as the prediction consistency of the "high-risk complication" category) through category-level distribution alignment, reducing the influence of training randomness.

[0134] 2) Identification of Weak Modality Combinations: To stably identify weak modality combinations that pose a threat to model generalization, this module introduces a memory bank and an average filtering mechanism to solve the volatility problem in neural network training. A memory bank is constructed to store the difference vector g d in each training cycle. After the training ends, the difference vectors in the memory bank are averaged over time to suppress single-cycle fluctuations, specifically as follows:

[0135]

[0136] where M d is an additional memory storage bank that records the difference vectors calculated for different modality combinations during the model training cycle, that is,

[0137] 3) Select the top R modality data fusion features with the largest value to form a set of weak modality combinations Ω, and generate a mask M r according to the set of weak modality combinations Ω:

[0138] M r= softmax(Ω(r))

[0139] 4) Construct the multi-modal perception regularization loss:

[0140]

[0141] where CE(·) represents the cross-entropy loss function, y d (k) represents the predicted label of the k-th patient sample, and y(k) represents the true label of the k-th patient sample.

[0142] 3. Risk assessment loss:

[0143] The main task loss function directly optimizes the patient state classification performance of the dual-path heterogeneous distillation model:

[0144]

[0145] This loss ensures that the model can still maintain the basic classification ability under modality absence.

[0146] Jointly optimize the three types of losses end-to-end, and the multi-task loss is defined as: The loss is defined as:

[0147] L total = L CE + ∈L MAD +(1 - ∈)L MAR

[0148] where ∈ is the weight coefficient, and the sum of the weights of L MAD and L MAR is 1.

[0149] The dual-path heterogeneous distillation model solves the problem of model degradation caused by modality absence in clinical scenarios through the knowledge transfer and boundary awareness strategy of the teacher network, providing high-robustness support for the real-time assessment of the postoperative patient state.

[0150] IV. Model Deployment and Experiments

[0151] Deploy the trained dual-path heterogeneous distillation model (ward rotation assessment model) to the medical edge computing device. The real-time processing pipeline first inputs the data stream through the dual-path heterogeneous distillation module to generate the postoperative state of the patient.

[0152] Real-time collection of multimodal clinical data streams through the hospital information system and Internet of Things devices, including surgical records, and extraction of structured text data (such as surgical type, anesthesia method, etc.) from the electronic medical record system (EMR); time-series vital sign data, collecting indicators such as heart rate, blood pressure, and blood oxygen saturation every minute through bedside monitoring devices; medical images, receiving DICOM format images (resolution 512×512) transmitted by postoperative X-ray machines or CT devices in real time; medication records and nursing logs, synchronizing structured medication data and unstructured nursing text from the pharmacy management system and mobile nursing terminals.

[0153] Input into the ward rotation assessment model. When the multimodal data is complete, input it into the teacher network to obtain the patient's postoperative status. When there are missing multimodal data, input it into the student network to obtain the patient's postoperative status. The status includes three categories: "can be transferred to the general ward", "needs to be continuously observed", and "high-risk complications".

[0154] Use the multimodal fusion model proposed by the present invention to perform real-time risk assessment on 328 postoperative patients. According to the verification results marked by clinical experts, the performance of the model in the scenario of missing modalities is shown in Table 1. The absolute error range of the risk score is between 0.03 - 0.18, and the mean absolute error is 0.09; the absolute error range of the confidence level of the transfer decision is between 0 - 0.15, and the mean absolute error is 0.05. The recall rate of high-risk complications reaches 97.2% in the complete modality and remains 90.3% when 80% of the modalities are missing, indicating that the model has high robustness and accuracy in complex clinical scenarios and meets the needs of medical decision-making. The specific results can be seen in Table 1 and Figure 6 。

[0155] Table 1 Multimodal Real-time Risk Assessment Performance

[0156]

[0157] The above are only the preferred embodiments of the present application and are not used to limit the present application. For those skilled in the art, various changes and modifications can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the protection scope of the present application.

[0158] Although the specific implementation manners of the present invention are described above, it is not a limitation to the protection scope of the present invention. Those skilled in the art should understand that based on the technical solutions of the present invention, various modifications or deformations that can be made without creative efforts by those skilled in the art are still within the protection scope of the present invention.

Claims

1. A method for constructing a ward rotation evaluation model based on multimodal information, characterized in that It includes the following processes: S1. Make a multi-modal clinical data set, including the collection of five types of heterogeneous data: surgical records, time-series vital sign data, medical images, medication records, and nursing logs. Correspondingly, label the postoperative status labels of patients, which are "can be transferred to a general ward", "need to be continuously observed", and "high-risk complications". After combining the data samples with the labels, randomly divide them into a training set and a test set; S2. Construct a ward rotation evaluation model, and improve the traditional teacher-student network architecture into a dual-path heterogeneous distillation model, including a teacher network and a student network. The teacher network includes a patient cross-modal data fusion module and a Softmax activation function layer, and the student network includes a randomly missing modal data fusion module and a softmax activation function layer; when the multi-modal data is complete, input it into the teacher network to obtain the predicted value of the patient's postoperative status, and when part of the multi-modal data is missing, input it into the student network to obtain the predicted value of the patient's postoperative status; S3. Train the dual-path heterogeneous distillation model, design a multi-task loss joint optimization strategy, including a boundary-aware distillation loss and a multi-modal regularization loss, to align the features of the teacher-student network, constrain the prediction results of the teacher network, and design a risk assessment loss to optimize the prediction performance of the patient's postoperative status, and obtain the final ward rotation evaluation model.

2. A method for constructing a ward rotation evaluation model based on multi-modal information according to claim 1, characterized in that: When making the multi-modal clinical data set, collect five types of heterogeneous data from the hospital information system, including surgical records, time-series vital sign data, medical images, medication records, and nursing logs; the surgical records are structured text data, including surgical type, surgical duration, and anesthesia method information; the time-series vital sign data is time-series data, including heart rate, blood pressure, and blood oxygen saturation; the medical images are postoperative CT images; the medication records are structured data, including drug name, dosage, and administration time; the nursing logs are unstructured text data, recording the patient's nursing situation and postoperative recovery status; It is labeled by clinical experts according to the postoperative patient status, and the labeled labels include three categories: "can be transferred to a general ward", "need to be continuously observed", and "high-risk complications".

3. A method for constructing a ward rotation evaluation model based on multi-modal information according to claim 1, characterized in that: The teacher network includes a patient cross-modal data fusion module and a softmax activation function layer. The patient cross-modal data fusion module extracts data features of different modalities from five types of modal data, namely surgical records, vital sign time-series data, medical images, medication records, and nursing logs, and performs cross-modal fusion to obtain a patient status representation. The patient cross-modal data fusion module includes a lightweight encoder module and a multi-modal co-attention fusion module based on semantic grouping. The lightweight encoder module takes the five types of modal data, namely surgical records, vital sign time-series data, medical images, medication records, and nursing logs, as inputs, obtains the features of the five modalities respectively, then divides the features of the five modalities into high-dimensional semantic groups and low-dimensional semantic groups, and uses the multi-modal co-attention fusion module based on semantic grouping to fuse the features of different dimensions to obtain a fused feature. The softmax activation function layer is used to map the fused feature to the postoperative status of the patient.

4. A method for constructing a ward rotation evaluation model based on multi-modal information according to claim 3, characterized in that: The lightweight encoder module includes a surgical record feature encoder, a vital sign time-series data feature encoder, a medical image feature encoder, a medication record feature encoder, and a nursing log feature encoder. The surgical record feature encoder includes a BERT model, a bidirectional LSTM, and a global max pooling layer. The vital sign time-series data feature encoder includes a Z-score normalization layer, an LSTM layer, and a multi-layer perceptron layer. The medical image feature encoder includes a pre-trained ResNet50 and a fully connected layer. The medication record feature encoder consists of an attention mechanism layer. The nursing log feature encoder includes a rule matching layer, which extracts key events through rule matching and converts them into 32-dimensional binary vectors.

5. The method for constructing a ward rotation evaluation model based on multi-modal information according to claim 3, wherein: The multi-modal co-attention fusion module based on semantic grouping fuses multi-modal features. The five types of modalities are divided into a high-dimensional semantic group based on feature dimensions: surgical record features, vital sign time-series data features, medical image features, and a low-dimensional auxiliary group: medication record features, nursing log features. Intra-group and cross-group attention calculations are performed respectively, as follows: S51, for the surgical record feature f T and the medical image feature f I , first, generate the query matrix Q through linear transformation T = W Q f T and the key matrix K I = W K f I , calculate the cross-modal attention weight A TI , A TI reflects the semantic association strength between the surgical text description and the image feature. Secondly, through the value matrix V I = W V f I for weighted fusion to obtain the cross-modal feature f of the surgical record and the medical image TI = A TI V I , again, generate the query matrix Q through linear transformation I = W Q f I and the key matrix K T = W K f T , calculate the cross-modal attention weight A TI , A TI reflects the semantic association strength between the medical image feature and the surgical record feature. Through the value matrix V T = W V f T for weighted fusion to obtain the cross-modal feature f of the medical image and the surgical record IT = A IT V T ; Repeat the above process to finally obtain the cross-modal features f of the operation record and medical image TI 、the cross-modal features f of the medical image and operation record IT 、the cross-modal features f of the operation record and vital sign time-series data TS 、the cross-modal features f of the vital sign time-series data and operation record ST 、the cross-modal features f of the medical image and vital sign time-series data IS 、the cross-modal features f of the vital sign time-series data and medical image SI ; S52, then the medication record features and nursing log features of the low-dimensional semantic group are fused with attention weighting in the manner of S51 to obtain the cross-modal features f of the medication record and the nursing log MP and the cross-modal features f of the nursing log and the medication record PM ; S53, input f TI , f IT , f TS , f ST , f IS , f SI , f MP and f PM into the global attention pooling layer to integrate the co-attention features of the high-dimensional semantic group and the co-attention features of the low-dimensional auxiliary group, and integrate and pool the features of different subspaces into a 256-dimensional fused feature F final .

6. A method for constructing a ward rotation evaluation model based on multimodal information as described in claim 1, characterized in that: The student network includes a randomly missing modal data fusion module and a softmax activation function layer. The randomly missing modal data fusion module includes a lightweight encoder module, a dynamic modal missing layer, and a multi-modal co-attention fusion module based on semantic grouping. Among them, the lightweight encoder module and the multi-modal co-attention fusion module based on semantic grouping have the same module structure as those of the teacher network. Its specific data processing process includes: S61, input the complete five - category modal data X = {T ∈ S, I, M, P} into the lightweight encoder module to obtain the features f T of five modalities, f S , f I , f M , and f P ; S62, input the features of the five modalities into the dynamic modality certainty layer for modality missing processing to obtain the missing modality features. Specifically, use Bernoulli indicators to set α% of the eigenvalue of each of the five modalities of f T , f S , f I , f M and f P to 0, and obtain the missing modality features If T , If S , If I , If M and If P ; S63, the missing modality feature If T , If S , If I , If M and If P are input into the multi - co - attention fusion module based on semantic grouping to obtain the missing modality fusion feature IF final ; S64, fuse the missing modality feature IF final Input it into the softmax activation function layer to obtain the postoperative status label y of the patient d .

7. The method for constructing a ward rotation evaluation model based on multi-modal information according to claim 1, characterized in that: The multi-task loss joint optimization strategy jointly optimizes three types of losses in an end-to-end manner. The multi-task loss is defined as the loss defined as: L total = L CE + ∈L MAD +(1 - ∈)L MAR L CE is the risk assessment loss, L MAD is the boundary-aware distillation loss, L MAR is the multi-modal perception regularization loss, where ∈ is the weight coefficient, and L MAD and L MAR have a weight sum of 1.

8. A ward rotation evaluation method based on multi-modal information, characterized in that: Deploy the ward rotation evaluation model constructed by the construction method according to any one of claims 1 to 7 in a medical edge computing device, and the process includes: Real-time collection of multimodal clinical data streams through hospital information systems and Internet of Things devices, including surgical records, and extraction of structured text data from the electronic medical record system EMR; time-series vital sign data, collecting indicators including heart rate, blood pressure, and blood oxygen saturation every minute through bedside monitoring devices; medical images, receiving DICOM-format images transmitted by postoperative X-ray machines or CT devices in real time; medication records and nursing logs, synchronizing structured medication data and unstructured nursing texts from the pharmacy management system and mobile nursing terminals. Input into the ward rotation assessment model. When the multimodal data is complete, input it into the teacher network to obtain the patient's postoperative status. When there are missing multimodal data, input it into the student network to obtain the patient's postoperative status. The status includes three categories: "can be transferred to the general ward", "needs continuous observation", and "high-risk complications".

Images