ICU (Intensive Care Unit) multi-task auxiliary diagnosis and treatment method and device based on multi-modal data fusion

Abstract

The invention discloses an ICU (Intensive Care Unit) multi-task auxiliary diagnosis and treatment method and device based on multi-modal data fusion, and the method comprises the steps: constructing a multi-modal data set comprising electronic medical record time sequence data and medical image data, and recording a multi-task prediction label of a patient in the multi-modal data set; using the multi-modal data set to train a constructed ICU multi-task prediction model comprising a multi-modal feature learning module, a multi-modal heterogeneous graph construction module, a heterogeneous graph learning module and a multi-task prediction module; and inputting newly collected patient data into the trained ICU multi-task prediction model for multi-task prediction. According to the method, effective integration of multi-modal information and knowledge migration among similar patients can be realized, death rate prediction, hospitalization duration prediction and disease risk prediction can be completed at the same time, the adaptability, the expandability and the prediction performance of the model in a real clinical environment are remarkably improved, and the method is suitable for popularization and application. The method can be widely applied to auxiliary intelligent diagnosis and treatment in actual clinical scenes.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of intelligent medical treatment, and particularly relates to an ICU multi-task auxiliary diagnosis and treatment method and device based on multi-modal data fusion. BACKGROUND

[0002] In modern intensive care medicine (Intensive Care Unit, ICU), clinicians often rely on multiple heterogeneous information sources for decision-making, including structured time series data (such as vital signs, laboratory test indicators, etc.) recorded in electronic medical records and medical image data (such as chest X-ray images). Taking pneumonia diagnosis as an example, clinical indicator data such as blood tests can provide information about a patient's infection status and immune response, while medical images such as chest X-rays can reveal the extent of lung inflammation. Integrating these data modalities can help gain a more comprehensive and accurate understanding of a patient's health status, which can lead to better clinical outcomes.

[0003] With the increasing popularity of digital clinical data, recent studies have focused on using multi-modal machine learning methods to improve the performance of clinical prediction tasks, including disease prediction and mortality prediction. The main goal of multi-modal learning is to use relevant information from different modalities to improve the performance of downstream tasks. Currently, multi-modal fusion strategies are mainly divided into early fusion, joint fusion and late fusion. Among them, the joint fusion paradigm is considered to be the most promising, and its core idea is to simulate the interaction between the representations of input modalities. However, existing multi-modal fusion strategies have many shortcomings in practical applications, especially when facing common multi-modal challenges in real ICU environments, the effect is obviously limited, which is specifically manifested as follows: (1) Serious modality missing: existing multi-modal fusion methods generally assume that all training and test samples contain all modal data, but this is not practical in actual clinical environments. Daily clinical practice produces heterogeneous data with varying degrees of sparsity. In the ICU, physiological data is collected more frequently than chest X-rays, and many hospitalized patients who need X-ray scans cannot receive the examination due to clinical or management reasons, resulting in a large number of patients missing picture modality information.

[0004] (2) Strong modality heterogeneity: different modal data differ significantly in dimension, structure, noise distribution, etc. Existing methods mostly use unified vector space projection or simple concatenation, which is difficult to retain the unique representation ability of each modality.

[0005] (3) Time misalignment problem is prominent: clinical indicator data is strictly time series, while images are often sparsely sampled and have no clear time alignment method, making it difficult to directly apply traditional time series modeling and alignment mechanisms.

[0006] (4) Ignoring the knowledge transfer relationship between similar patients: Most existing methods only focus on the modal data of a single patient itself, without considering the clinical similarity (such as similar disease course, similar etiology) between known patients in the training set, so it is difficult to transfer the diagnosis and treatment experience from the historical similar cases, which limits the generalization ability and clinical credibility of the model.

[0007] In addition, most researchers currently develop new methods by targeting only one clinical prediction task at a time, which can cause a disconnect from the realities of clinical decision-making. In clinical decision-making, all clinical tasks (mortality prediction, length of stay prediction, disease risk prediction) are usually performed simultaneously by clinical staff at an early stage. More importantly, there is increasing evidence that these prediction tasks are interrelated. For example, the patients with the highest risk and the highest cost are often those with complex conditions. Therefore, dealing with each prediction task in isolation cannot fully utilize the associated information between these tasks, which is not conducive to improving the accuracy and effectiveness of clinical decision-making. SUMMARY

[0008] In view of the above, the purpose of the present application is to provide an ICU multi-task auxiliary diagnosis and treatment method and device based on multi-modal data fusion, which constructs a patient-level multi-modal heterogeneous graph based on electronic medical record time series data and medical image data, and designs a heterogeneous graph neural network to model the similarity between patients and fuse multi-modal information, realizes the effective integration of multi-modal information and the knowledge transfer between similar patients, has the ability to support diagnosis and treatment prediction under real clinical conditions, and can simultaneously complete mortality prediction, length of stay prediction and disease risk prediction, significantly improving the adaptability, scalability and prediction performance of the model in real clinical environments.

[0009] To achieve the above invention purpose, the technical solutions provided by the present application are as follows: In a first aspect, the present application provides an ICU multi-task auxiliary diagnosis and treatment method based on multi-modal data fusion, comprising the following steps: Constructing a multi-modal data set including electronic medical record time series data and medical image data, and recording the multi-task prediction labels of the patients in the multi-modal data set; The constructed ICU multi-task prediction model including a multi-modal feature learning module, a multi-modal heterogeneous graph construction module, a heterogeneous graph learning module and a multi-task prediction module is trained by using a multi-modal data set, including: inputting the electronic medical record time series data and medical image data in the multi-modal data set into the multi-modal feature learning module for feature extraction to obtain multi-modal feature representation; constructing a multi-modal heterogeneous graph based on the multi-modal feature representation through the multi-modal heterogeneous graph construction module, the node set in the multi-modal heterogeneous graph including the electronic medical record time series node and the medical image node of each patient, and the edge set including the inter-modal connection of the same patient and the connection between similar patients; the information between different nodes is fused based on the multi-modal heterogeneous graph through the heterogeneous graph learning module, and the final fusion feature of each patient node is output; the training of multi-task prediction based on the final fusion feature is performed through the multi-task prediction module. The newly collected patient data is input into the trained ICU multi-task prediction model for multi-task prediction.

[0010] Preferably, the electronic medical record time series data and medical image data in the multi-modal data set are input into the multi-modal feature learning module for feature extraction to obtain multi-modal feature representation, including: The electronic medical record time series data and medical image data are encoded by separate encoders, wherein the electronic medical record time series data is encoded by a Transformer model, and the medical image data is encoded by a ResNet model to obtain encoded multi-modal feature representation.

[0011] Preferably, in the multi-modal heterogeneous graph, the connection between similar patients is constructed by KNN algorithm, including: Using KNN algorithm to find out the samples in the multi-modal data set whose clinical indicator distribution is closest to the patient, and constructing the connection edge between the electronic medical record time series data features of similar patients , wherein, represents a cosine similarity function, and represent the feature representation of the electronic medical record time series data of patient and patient , represents selecting the K samples with the highest similarity, and each pair of represents that patient and patient are similar in clinical characteristics, represents the total number of patients.

[0012] ​Preferably, the information between different nodes is fused by the heterogeneous graph learning module based on the multi-modal heterogeneous graph, and the final fusion features of each patient node are output, including: The heterogeneous graph learning module designs a multi-layer heterogeneous graph neural network based on a multi-head attention mechanism. First, the information transmitted by all neighbor nodes in the multi-modal heterogeneous graph to the target patient node is calculated using the multi-head attention mechanism. Then, the aggregation weights of the information transmitted by all neighbor nodes in the multi-modal heterogeneous graph to the target patient node are calculated using the multi-head attention mechanism. The information and the corresponding aggregation weights are multiplied and aggregated to obtain the representation of the target patient node aggregated with the information of all neighbor nodes. Finally, the final fusion features of each target patient node are obtained by linear mapping and residual connection of the aggregation of the self-features of the target patient node and the neighbor node information.

[0013] Preferably, the information transmitted by all neighbor nodes in the multi-modal heterogeneous graph to the target patient node is calculated using the multi-head attention mechanism, including: For the first layer heterogeneous graph attention mechanism neural network, the information transmitted by all neighbor nodes to the target patient node is calculated using the multi-head attention mechanism : , wherein, and represent the target patient node and the neighbor node, represents the edge between the target patient node and the neighbor node, represents aggregation, represents the number of heads of the multi-head attention mechanism, represents the output of the i-th attention head: , wherein, represents a linear projection specific to different node types to capture the information of different nodes, represents the information transmitted by all neighbor nodes of the output of the neural network of the previous layer heterogeneous graph attention mechanism to the target patient node , represents a learnable matrix specific to the edge type to capture the dependency relationship of different edges.

[0014] Preferably, the aggregation weights of the information transmitted by all neighbor nodes in the multi-modal heterogeneous graph to the target patient node are calculated using the multi-head attention mechanism, including: First, the i-th attention head in the multi-head attention mechanism is calculated ​information weights output by the attention heads : , wherein, represents a learnable, edge-type-specific matrix, represents a prior tensor representing the importance of different relations, and the superscript represents the transpose, represents the feature dimension, and are the key vector and the query vector, respectively: , , wherein, and represent linear projection representations specific to different node types and , respectively, and represent the information passed by all neighbor nodes of the previous layer to the target patient node, and the information received by the current neighbor node as the target patient node of the previous layer when passing all neighbor nodes of the previous layer to it; the final fusion of the information weights output by all attention heads obtains the aggregation weight of the information passed by all neighbor nodes to the target patient node : , wherein, represents a Softmax function, represents the set of all neighbor nodes of the node .

[0015] Preferably, the multiplication and aggregation of each information and its corresponding aggregation weight obtain the representation of the target patient node aggregated with all neighbor node information, and finally the final fusion feature of each target patient node is obtained by linear mapping and residual connection to aggregate the own feature of the target patient node and the neighbor node information, including: the representation of the target patient node aggregated with all neighbor node information is: , Finally, the own feature of the target patient node and the neighbor information are aggregated by linear mapping and residual connection to obtain the final representation: , ​wherein, represents the final fusion feature of each target patient node, represents a linear projection representation specific to different node types represents an activation function.

[0016] Preferably, the training of the multi-task prediction module based on the final fusion feature for multi-task prediction comprises: inputting the final fusion feature of each patient into the multi-task prediction module constructed by the hybrid expert model to perform multi-task prediction including mortality prediction, hospitalization duration prediction and disease risk prediction; In the hybrid expert model, the final fusion feature is first input into the shared layer to obtain the shared feature, and then the three expert networks are input in parallel, while the gate network calculates the loss function weight of each task to ensure the adaptive convergence of each task in the training process, and finally the trained ICU multi-task prediction model is obtained.

[0017] In a second aspect, the embodiments of the present application also provide an ICU multi-task auxiliary diagnosis and treatment device based on multi-modal data fusion, which is implemented by using the above-mentioned ICU multi-task auxiliary diagnosis and treatment method based on multi-modal data fusion, and comprises a multi-modal data set construction module, a multi-task prediction model training module and an ICU multi-task prediction module. The multi-modal data set construction module is configured to construct a multi-modal data set comprising electronic medical record time series data and medical image data, and record the multi-task prediction labels of patients in the multi-modal data set. The multi-task prediction model training module is configured to train the constructed ICU multi-task prediction model comprising a multi-modal feature learning module, a multi-modal heterogeneous graph construction module, a heterogeneous graph learning module and a multi-task prediction module by using the multi-modal data set, including: inputting the electronic medical record time series data and the medical image data in the multi-modal data set into the multi-modal feature learning module to extract features and obtain multi-modal feature representations; constructing a multi-modal heterogeneous graph based on the multi-modal feature representations by using the multi-modal heterogeneous graph construction module, wherein the node set in the multi-modal heterogeneous graph comprises the electronic medical record time series nodes and the medical image nodes of each patient, and the edge set comprises the inter-modal connection of the same patient and the connection between similar patients; fusing the information between different nodes based on the multi-modal heterogeneous graph by using the heterogeneous graph learning module and outputting the final fusion feature of each patient node; and training the multi-task prediction based on the final fusion feature by using the multi-task prediction module. The ICU multi-task prediction module is configured to input the newly collected patient data into the trained ICU multi-task prediction model to perform multi-task prediction.

[0018] ​In a third aspect, the embodiments of the present application also provide an electronic device comprising a memory for storing a computer program and one or more processors for implementing the above-mentioned ICU multi-task auxiliary diagnosis and treatment method based on multi-modal data fusion when executing the computer program.

[0019] Compared with the prior art, the present application has beneficial effects at least including: (1) Enhancing the adaptability and robustness to complex clinical data: In view of the problems of missing, heterogeneous, time misalignment, and incomplete patient information of multi-modal medical data in real clinical environment, the present application constructs a heterogeneous graph structure centered on patients, and fuses modal-specific encoders to realize flexible modeling and fusion of heterogeneous, misaligned, and partially missing modalities. This enables the model to maintain stable diagnosis and treatment prediction ability under inconsistent data collection frequency, significant structural difference, and missing image data, thereby improving the adaptability and robustness of the model to real ICU multi-modal data.

[0020] (2) Realizing knowledge transfer between similar patients and enhancing reasoning: Traditional methods often ignore the knowledge transfer mechanism between similar patients and only make predictions based on single patient features. The present application models similar patients based on KNN, propagates features through a heterogeneous graph neural network, enables the model to learn diagnosis and treatment experience and patterns from similar patients, realizes implicit knowledge transfer and enhanced reasoning, and further enhances the generalization ability of the model to complex cases and low-frequency samples.

[0021] (3) Mining the correlation between multi-tasks and realizing collaborative optimization: Considering the significant correlation between different clinical tasks, such as long hospitalization time and complex disease phenotypes for patients with high mortality risk, the present application introduces a hybrid expert model with a shared-specific structure for each task based on multi-modal graph neural fusion features. This design effectively mines the potential correlation between different diagnosis and treatment tasks, realizes collaborative optimization and information sharing between multi-tasks without significantly increasing the complexity of the model, improves the overall prediction performance and clinical adaptability, and effectively reduces the time cost of application.

[0022] (4) Having high scalability and clinical deployment value: The method proposed in the present application has good scalability and can flexibly adapt to new modalities or tasks, meets the needs of intelligent diagnosis and treatment auxiliary system construction in actual ICU environment, has high clinical deployment value, and can be widely applied to actual clinical scenarios for auxiliary intelligent diagnosis and treatment. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings required by the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0024] Figure 1 is a flowchart of the ICU multi-task auxiliary diagnosis and treatment method based on multi-modal data fusion provided by the embodiments of the present application. Figure 2 is a framework diagram of the ICU multi-task auxiliary diagnosis and treatment method based on multi-modal data fusion provided by the embodiments of the present application. Figure 3 is a structure and processing flow diagram of the hybrid expert model provided by the embodiments of the present application. Figure 4 is a structure diagram of the ICU multi-task auxiliary diagnosis and treatment device based on multi-modal data fusion provided by the embodiments of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of the present application more clear, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the protection scope of the present application.

[0026] The inventive concept of the present application is that: in view of the core problems of modal heterogeneity, information loss, task coupling, lack of knowledge transfer and the like existing in the current ICU scene, the embodiments of the present application provide an ICU multi-task auxiliary diagnosis and treatment method and device based on multi-modal data fusion, by constructing a heterogeneous graph structure centered on patients, fusing modal exclusive encoders, effectively dealing with the problems of data heterogeneity and loss, and realizing flexible modeling fusion. A similar patient graph modeling mechanism based on K-nearest neighbors is introduced, and features are propagated by means of a heterogeneous graph neural network to achieve knowledge transfer and enhanced reasoning. At the same time, a hybrid expert mechanism is used to model the sharing-specific structure of multi-task, to mine the potential correlation between tasks, to realize collaborative optimization and information sharing, and to comprehensively improve the diagnosis and treatment assistance capability and practicality of the model in the complex ICU environment.

[0027] As shown in Figure 1 and Figure 2 , the embodiments provide an ICU multi-task auxiliary diagnosis and treatment method based on multi-modal data fusion, comprising the following steps: S1, constructing a multi-modal data set including electronic medical record time series data and medical image data, and recording the multi-task prediction labels of the patient in the multi-modal data set.

[0028] In the embodiment, the clinical variable data of the patient within 48 hours before entering the ICU is periodically sampled every two hours as the electronic medical record time series data, the last chest X-ray film taken within 48 hours before the patient enters the ICU is collected as the medical image data, and the in-hospital death of the patient, the remaining hospitalization time after 48 hours and whether the patient has multiple diseases are recorded as labels, and the whole is constructed as a multi-modal data set.

[0029] Among them, 10 continuous clinical variables of ICU routine monitoring are extracted: diastolic blood pressure, inhaled oxygen fraction, blood glucose, heart rate, mean blood pressure, blood oxygen saturation, respiratory rate, systolic blood pressure, body temperature and pH value. In order to better train the model to face the generalization of missing modalities, the present application will randomly remove 20% of the X-ray films of the patients in the collected training set as additional data enhancement.

[0030] S2, using the multi-modal data set to train the ICU multi-task prediction model constructed including the multi-modal feature learning module, the multi-modal heterogeneous graph construction module, the heterogeneous graph learning module and the multi-task prediction module.

[0031] In the embodiment, the ICU multi-task prediction model including the multi-modal feature learning module, the multi-modal heterogeneous graph construction module, the heterogeneous graph learning module and the multi-task prediction module is constructed, and the ICU multi-task prediction model is trained using the multi-modal data set, which specifically includes the following steps.

[0032] S2.1, inputting the electronic medical record time series data and the medical image data in the multi-modal data set into the multi-modal feature learning module for feature extraction to obtain multi-modal feature representation.

[0033] Specifically, time series and pictures are two highly heterogeneous modalities, so the multi-modal feature learning module needs to use separate encoders to encode the original input data. For electronic medical record time series (EHR) data , a Transformer model is used for encoding, and for medical image (CXR) data , a convolutional network ResNet50 is used for encoding, and the formula is as follows: , , Among them, is the projection of the original input, is the position encoding, and are the extracted features.

[0034] S2.2, constructing a multi-modal heterogeneous graph based on the multi-modal feature representation by a multi-modal heterogeneous graph construction module, the node set in the multi-modal heterogeneous graph includes the electronic medical record time series node and the medical image node of each patient, and the edge set includes the inter-modal connection of the same patient and the connection between similar patients.

[0035] Specifically, the multi-modal heterogeneous graph construction module constructs a multi-modal heterogeneous graph for all modal data of all patients in the multi-modal data set , wherein the node set has two types of nodes: electronic medical record time series nodes , also representing patient nodes and chest X-ray image nodes , and only when the patient has an image, there is a node. The vector of the node is and respectively. The edge set is a series of relationships between node pairs, and there are two types , wherein is the set of all EHR-CXR, representing the multi-modal node connection of the same patient, is the set of all EHR-EHR, representing the connection between similar patients. is the node type combination, is the edge type combination, and for all nodes and edges, there is a mapping function , This design not only solves the actual problem of missing modalities of some patients, but also enables knowledge transfer and sharing across patients based on the graph structure, thereby improving the adaptability of the model to clinical complex situations.

[0036] The capture of similar patients is to find the samples closest to the patient in the multi-modal data set in terms of clinical indicators, and to construct the connection edge between the electronic medical record time series data features based on similar patients : , wherein represents the cosine similarity function, and represent the feature representation of the electronic medical record time series data of patient and patient respectively, represents selecting the K samples with the highest similarity, and each pair represents that patient and patient are similar in clinical features, represents the total number of patients.

[0037] S2.3, through the heterogeneous graph learning module, information between different nodes is fused based on multimodal heterogeneous graphs and the final fused features of each patient node are output.

[0038] To effectively utilize the graph structure information, this invention further introduces a heterogeneous graph attention neural network, which defines independent attention weights for different types of edges, achieving efficient fusion between multiple modalities and overcoming common problems such as heterogeneous input dimensions and time misalignment in medical multimodal data.

[0039] Specifically, the heterogeneous graph learning module employs a heterogeneous graph attention-based neural network (HGAN) to learn enhanced patient clinical representations that integrate various types of relational information from the heterogeneous graph. The module consists of L layers of stacked HGANs, with the target node type being HER nodes (patient nodes). The HGAN uses a message-passing architecture, passing and aggregating information before finally outputting the representation of the target node. .

[0040] (1) Targeting the patient node First, a multi-head attention mechanism is used to compute all neighbor nodes in a multimodal heterogeneous graph. Passed to the node Information: For the A neural network with a layered heterogeneous graph attention mechanism utilizes a multi-head attention mechanism to compute the information passed from all neighboring nodes to the target patient node. : , in, and These represent the target patient node and the neighboring nodes, respectively. This represents the edges between the target patient node and its neighboring nodes. Indicates aggregation, This indicates the number of heads in the multi-head attention mechanism. Indicates the first Output of each attention head: , in, This indicates that it is specific to different node types. Linear projection is used to capture information from different nodes. Indicates the upper level The neural network output of the layer heterogeneous graph attention mechanism includes all neighboring nodes. Passed to the target patient node Information, Represents learnable, edge-type specific features a matrix for capturing the dependency of different edges.

[0041] (2) The multi-head attention mechanism is used to calculate the aggregation weight of the information transmitted by all neighbor nodes to the target patient node in the multi-modal heterogeneous graph: First, the information weight output by the i-th attention head in the multi-head attention mechanism is calculated: : , wherein, represents a learnable matrix specific to the edge type, represents a prior tensor representing the importance of different relationships, and the superscript T represents transposition, represents the feature dimension, and are the key vector and the query vector, respectively: , , , wherein, and represent linear projection representations specific to different node types and , respectively, and represent the information transmitted by all neighbor nodes of the target patient node to the target patient node in the neural network output of the heterogeneous graph attention mechanism in the previous layer , and the information received by the current neighbor node when it is the target patient node in the previous layer ; The aggregation weight of the information transmitted by all neighbor nodes to the target patient node is finally obtained by fusing the information weight output by all attention heads: , wherein, represents a Softmax function, and represents the set of all neighbor nodes of the node .

[0042] (3) The information and the corresponding aggregation weight are multiplied and aggregated to obtain the representation of the target patient node aggregating all neighbor node information: The representation of the target patient node aggregating all neighbor node information is: .

[0043] ​​​(4) Finally, the self-features and neighbor information of the target patient node are aggregated by linear mapping and residual connection to obtain the final fusion features of each target patient node: Finally, the self-features and neighbor information of the target patient node are aggregated by linear mapping and residual connection to obtain the final representation:

[0044] S2.4, training of multi-task prediction based on the final fusion features by the multi-task prediction module.

[0045] In the embodiment, the final fusion features of each patient are input into the multi-task prediction module constructed by the hybrid expert model to perform multi-task prediction including mortality prediction, hospitalization duration prediction, and disease risk prediction. By combining the shared expert network and the task-specific gating mechanism, multiple related tasks such as mortality, hospitalization duration, and disease risk are jointly modeled, which not only captures the correlation between tasks, but also better meets the actual needs of "multi-task synchronous decision-making" in the clinic, providing strong technical support for early intervention and precise resource allocation in the ICU scenario.

[0046] Specifically, as shown in FIG. 6, the fusion feature representation of each patient's EHR node is input into the hybrid expert model in the multi-task prediction module to output the prediction values of mortality prediction, hospitalization duration prediction, and disease risk prediction. Figure 3 ​​​​​​​​​​​​​​​​​​​The gating network is responsible for calculating the loss function weight of each task , and This is to ensure that each task converges at an adaptive speed in the future model training process, preventing the overfitting of a single task.

[0047] Finally, the loss function formula of the model is: , Among them, are the cross-entropy loss function (binary classification task and multi-classification task) and the MSE loss function (regression task), respectively.

[0048] S3, input the newly collected patient data into the trained ICU multi-task prediction model for multi-task prediction.

[0049] In the embodiment, patient samples that have never participated in training are used for testing, and the model prediction results are compared with the actual results. In the testing stage, the new patient sample finds the K patients with the highest clinical feature similarity in the training set through KNN, forms a heterogeneous graph, and then inputs into the HGAN and hybrid expert model for multi-task prediction. After obtaining the prediction results of different new samples, various evaluation indicators can be used to objectively evaluate the prediction accuracy of the model. Among them, different evaluation indicators are selected for different tasks to measure the difference between the model prediction value and the actual observation value. For binary classification tasks, accuracy and recall rate are selected; for regression tasks, MSE is selected; for multi-classification tasks, PRAUC is selected.

[0050] In summary, the ICU multi-task auxiliary diagnosis and treatment method based on multi-modal data fusion provided by the embodiment of the application flexibly fuses complex data by constructing a multi-modal heterogeneous graph, realizes knowledge transfer by using a similar patient graph modeling, enhances reasoning ability, and excavates task association by means of a hybrid expert mechanism, achieves collaborative optimization, effectively improves the accuracy and generalization of diagnosis and treatment prediction, and provides an efficient and practical solution for ICU intelligent diagnosis and treatment.

[0051] Based on the same inventive concept, as shown in Figure 4 The embodiment of the application also provides an ICU multi-task auxiliary diagnosis and treatment device 400 based on multi-modal data fusion, which comprises a multi-modal data set construction module 410, a multi-task prediction model training module 420 and an ICU multi-task prediction module 430.

[0052] The multi-modal data set construction module 410 is used to construct a multi-modal data set comprising electronic medical record time series data and medical image data, and record the multi-task prediction labels of patients in the multi-modal data set.

[0053] The multi-task prediction model training module 420 is configured to train the constructed ICU multi-task prediction model including the multi-modal feature learning module, the multi-modal heterogeneous graph construction module, the heterogeneous graph learning module and the multi-task prediction module by using the multi-modal data set, including: inputting the electronic medical record time series data and the medical image data in the multi-modal data set into the multi-modal feature learning module to extract features to obtain multi-modal feature representations; constructing a multi-modal heterogeneous graph based on the multi-modal feature representations by using the multi-modal heterogeneous graph construction module, the node set in the multi-modal heterogeneous graph including the electronic medical record time series node and the medical image node of each patient, and the edge set including the inter-modal connection of the same patient and the connection between similar patients; fusing the information between different nodes based on the multi-modal heterogeneous graph by using the heterogeneous graph learning module and outputting the final fusion features of each patient node; and training the multi-task prediction based on the final fusion features by using the multi-task prediction module.

[0054] The ICU multi-task prediction module 430 is configured to input the newly collected patient data into the trained ICU multi-task prediction model to perform multi-task prediction.

[0055] Based on the same inventive concept, the embodiments of the present application also provide an electronic device including a memory and one or more processors, the memory being configured to store a computer program, and the processor being configured to implement the above-mentioned ICU multi-task auxiliary diagnosis and treatment method based on multi-modal data fusion when executing the computer program.

[0056] It should be noted that the ICU multi-task auxiliary diagnosis and treatment device based on multi-modal data fusion and the electronic device provided by the above embodiments belong to the same inventive concept as the ICU multi-task auxiliary diagnosis and treatment method based on multi-modal data fusion, and the specific implementation process is described in detail in the ICU multi-task auxiliary diagnosis and treatment method based on multi-modal data fusion, which will not be repeated here.

[0057] The above specific embodiments have described the technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only the most preferred embodiment of the present application, and is not used to limit the present application. Any modification, supplement and equivalent replacement, etc. made within the principle range of the present application should be included in the protection scope of the present application.

Claims

1. An ICU multitask auxiliary diagnosis and treatment method based on multi-modal data fusion, characterized in that, The method comprises the following steps: constructing a multi-modal data set comprising electronic medical record time series data and medical image data, and recording the multi-task prediction labels of the patients in the multi-modal data set; training the constructed ICU multi-task prediction model comprising a multi-modal feature learning module, a multi-modal heterogeneous graph construction module, a heterogeneous graph learning module, and a multi-task prediction module using the multi-modal data set, comprising: inputting the electronic medical record time series data and the medical image data in the multi-modal data set into the multi-modal feature learning module for feature extraction to obtain multi-modal feature representation; constructing a multi-modal heterogeneous graph based on the multi-modal feature representation through the multi-modal heterogeneous graph construction module, the node set in the multi-modal heterogeneous graph comprising the electronic medical record time series nodes and the medical image nodes of each patient, and the edge set comprising the inter-modal connection of the same patient and the connection between similar patients; fusing the information between different nodes based on the multi-modal heterogeneous graph through the heterogeneous graph learning module and outputting the final fusion feature of each patient node; training the multi-task prediction based on the final fusion feature through the multi-task prediction module; inputting the newly collected patient data into the trained ICU multi-task prediction model for multi-task prediction. 2.The ICU multitasking auxiliary diagnosis and treatment method based on multi-modal data fusion according to claim 1, characterized in that, The method comprises the following steps: The electronic medical record time series data and the medical image data are encoded by separate encoders, wherein the electronic medical record time series data is encoded by a Transformer model, and the medical image data is encoded by a ResNet model to obtain the encoded multi-modal feature representation. 3.The ICU multitasking auxiliary diagnosis and treatment method based on multi-modal data fusion according to claim 1, characterized in that, In the multi-modal heterogeneous graph, the connection between similar patients is constructed by a KNN algorithm, comprising: Using KNN algorithm to find the sample with the closest distribution of clinical indicators in the multi-modal data set Constructing connection edges between features of electronic medical record time series data based on similar patients : ， wherein, denotes a cosine similarity function, and denotes a feature representation of the electronic health record time series data of a patient and a patient , denotes selecting the K samples with the highest similarity, each pair denotes that a patient and a patient are similar in clinical features, denotes the total number of patients. 4.The ICU multitasking auxiliary diagnosis and treatment method based on multi-modal data fusion of claim 1, wherein, The method comprises the following steps: The heterogeneous graph learning module designs a multi-layer heterogeneous graph neural network based on a multi-head attention mechanism, first calculates the information transmitted by all neighbor nodes in the multi-modal heterogeneous graph to the target patient node using the multi-head attention mechanism, then calculates the aggregation weight of the information transmitted by all neighbor nodes in the multi-modal heterogeneous graph to the target patient node using the multi-head attention mechanism, multiplies each information and its corresponding aggregation weight, and aggregates to obtain the representation of the target patient node aggregated with all neighbor node information, and finally aggregates the self-feature of the target patient node and the neighbor node information through linear mapping and residual connection to obtain the final fusion feature of each target patient node.

5. The ICU multitasking auxiliary diagnosis and treatment method based on multi-modal data fusion according to claim 4, characterized in that, The method comprises the following steps: For the first Neural network of layer heterogeneous graph attention mechanism, all neighbor nodes pass information to the target patient node by using multi-head attention mechanism : ， wherein, and denote the target patient node and the neighbor node, respectively, denotes the edge between the target patient node and the neighbor node, denotes aggregation, denotes the number of heads of the multi-head attention mechanism, denotes the output of the th attention head: ， wherein, denotes a linear projection specific to different node types to capture information of different nodes, denotes all neighbor nodes of the previous layer layer heterogeneous graph attention mechanism’s neural network output passing information to the target patient node , denotes a learnable, edge type-specific matrix to capture dependency of different edges. 6.The ICU multitasking auxiliary diagnosis and treatment method based on multi-modal data fusion according to claim 5, characterized in that, The method comprises the following steps: First, the information weight of the output of the i-th attention head in the multi-head attention mechanism is calculated as follows: :​ ， wherein, represents a learnable, edge type-specific matrix, represents a prior tensor, representing the importance of different relations, the superscript represents the transpose, represents the feature dimension, and are the key and query vectors, respectively:​ ， ， wherein, and denote linear projection representations specific to different node types and , and denote all neighbor node passed information to the target patient node from the neural network output of the previous layer heterogeneous graph attention mechanism, and the current neighbor node receives all neighbor node passed information when it is the target patient node of the previous layer . The information weight of the final fusion of all attention head outputs gets the aggregation weight of the information passed by all neighbor nodes to the target patient node : ， wherein, denotes a Softmax function, denotes a set of all neighboring nodes of a node . 7.The ICU multitasking auxiliary diagnosis and treatment method based on multi-modal data fusion of claim 6, wherein, The multiplication and aggregation of each information and its corresponding aggregation weight obtain a representation of the target patient node aggregating all neighbor node information of the target patient node, and finally the final fusion feature of each target patient node is obtained by linear mapping and residual connection of the self characteristics of the target patient node and the neighbor node information, including: Target patient node The representation of all aggregated neighbor node information is: ， Finally, the target patient node is aggregated by linear mapping and residual connection with its own features and neighbor information to get the final representation: ​ ， in, This represents the final fusion feature of each target patient node. This indicates that it is specific to different node types. The linear projection representation, This represents the activation function. 8.The multi-modal data fusion based ICU multitasking auxiliary diagnosis and treatment method according to claim 1, characterized in that, The training of the multi-task prediction module based on the final fusion feature includes: Inputting the final fusion feature of each patient into the multi-task prediction module constructed by the hybrid expert model to perform multi-task prediction including mortality prediction, hospitalization duration prediction and disease risk prediction; In the hybrid expert model, the final fusion feature is first input into the shared layer to obtain the shared feature, and then the three expert networks are input in parallel, and the loss function weight of each task is calculated by the gate network to ensure the adaptive convergence of each task in the training process, and finally the trained ICU multi-task prediction model is obtained.

9. An ICU multitask auxiliary diagnosis and treatment device based on multi-modal data fusion, which is implemented by using the ICU multitask auxiliary diagnosis and treatment method based on multi-modal data fusion in any one of claims 1-8. Including: The multi-modal data set construction module, the multi-task prediction model training module and the ICU multi-task prediction module; The multi-modal data set construction module is used to construct a multi-modal data set including electronic medical record time series data and medical image data, and record the multi-task prediction label of the patient in the multi-modal data set; The multi-task prediction model training module is used to train the constructed ICU multi-task prediction model including a multi-modal feature learning module, a multi-modal heterogeneous graph construction module, a heterogeneous graph learning module and a multi-task prediction module using the multi-modal data set, including: inputting the electronic medical record time series data and the medical image data in the multi-modal data set into the multi-modal feature learning module to extract features to obtain multi-modal feature representation; constructing a multi-modal heterogeneous graph based on the multi-modal feature representation through the multi-modal heterogeneous graph construction module, the node set in the multi-modal heterogeneous graph including the electronic medical record time series node and the medical image node of each patient, and the edge set including the inter-modal connection of the same patient and the connection between similar patients; the final fusion feature of each patient node is output by the heterogeneous graph learning module based on the multi-modal heterogeneous graph fusion of information between different nodes; the training of the multi-task prediction module based on the final fusion feature for multi-task prediction; The ICU multi-task prediction module is used to input the newly collected patient data into the trained ICU multi-task prediction model for multi-task prediction.

10. An electronic device comprising a memory and one or more processors, the memory to store a computer program, characterized in that, The processor is used to implement the ICU multi-task auxiliary diagnosis and treatment method based on multi-modal data fusion according to any one of claims 1-8 when executing the computer program.

Images