Transformer-based representation learning model for uniform processing of multi-modal input for clinical diagnosis and prognosis

Abstract

In the diagnosis process, a clinician makes full use of multi-modal information such as chief complaint, medical images and laboratory test results. A deep learning model used for auxiliary diagnosis does not meet the requirement. In some aspects, a transformer-based representation learning model may be used as a clinical diagnostic aid for processing multi-modal inputs in a unified manner. The model does not learn modal-specific features, but uses an embedded layer to convert an image, an unstructured text and a structured text into visual lexical elements and text lexical elements. And using bi-directional blocks with intra-modal and inter-modal attention to learn an overall representation of radiographs, unstructured chief complaints and medical records, structured clinical information, such as laboratory test results and patient demographic information. The performance of the unified model in the aspect of lung disease recognition exceeds that of an only image model and that of a non-unified multi-modal diagnosis model (respectively improved by 12% and 9%), and the performance of the unified model in the aspect of COVID-19 patient adverse clinical outcome prediction exceeds that of the only image model and that of the non-unified multi-modal diagnosis model (respectively improved by 29% and 7%). A unified multi-mode transformer-based model is fully utilized, so that a patient triage process can be simplified, and a clinical decision-making process is promoted.

Description

Background Technology

[0001] In modern medicine, utilizing multimodal clinical information for medical diagnosis is common practice. For example, in addition to chest X-rays, thoracic surgeons need to consider each patient's demographic information (e.g., age and sex), chief complaint (e.g., present and past medical history), and laboratory test reports to make an accurate diagnostic decision. In practice, abnormal radiographic patterns are first correlated with the symptoms mentioned in the chief complaint or abnormal results in laboratory test reports. Then, relying on their extensive expertise and years of training, physicians make the best diagnosis by jointly interpreting this multimodal data. 1,2 Literature in various professional fields (including but not limited to radiology, dermatology, and ophthalmology) 3-10 In China, the importance of utilizing multimodal clinical information has been widely validated.

[0002] The aforementioned multimodal diagnostic process requires substantial expertise, which may be unavailable in areas with limited medical resources. Furthermore, simply increasing the workload of experienced physicians and radiologists will inevitably exhaust their time and energy, thereby increasing the risk of misdiagnosis. To meet the growing demand for precision medicine, machine learning technology... 11 It has become the de facto choice for automated intelligent medical diagnosis. Within machine learning technology, the unprecedented development of deep learning technology... 12,13 This enables machine learning models to detect diseases based on medical images at a level approaching or reaching that of human experts. 14-18 . Summary of the Invention

[0003] This paper discloses a method for providing a medical diagnosis to a patient, comprising: receiving one or more images of a patient and a text dataset associated with the patient; generating multiple tokens by: converting one or more images into one or more visual tokens; and converting the text dataset into one or more textual tokens; obtaining a medical diagnosis for the patient by feeding the multiple tokens into a trained machine learning model containing multiple bidirectional blocks with intra-modal and inter-modal attention; and providing a medical diagnosis to the patient.

[0004] In some implementations, medical diagnosis may include identified diseases, predictions of adverse clinical outcomes, or a combination thereof.

[0005] In any implementation thereof, the text dataset may contain: narrative text, one or more text field data, or a combination thereof.

[0006] In any implementation thereof, the trained machine learning model may further include one or more self-attention blocks.

[0007] In any implementation thereof, the trained machine learning model may further include a classification head. Attached Figure Description

[0008] The accompanying drawings illustrate certain features and advantages of this disclosure. These embodiments are not intended to limit the scope of the appended claims in any way.

[0009] Figure 1 An exemplary introduction to IRENE is depicted. a) IRENE is compared to previous non-uniform multimodal diagnostic paradigms. IRENE eliminates the verbose text structuring process, separate paths for modality-specific feature extraction, and multimodal feature fusion modules found in traditional non-uniform methods. Instead, IRENE performs multimodal diagnostics using a single unified Transformer. b) A scheme for splitting the original dataset into training, validation, and test sets for lung disease identification and COVID-19 adverse clinical outcome prediction is shown. (c, d) Experimental results are compared between image-only models, non-uniform early fusion methods, the multimodal Transformer (i.e., Perceiver), and IRENE in both tasks. p-values ​​between the average performance of IRENE and the multimodal Transformer are calculated using independent two-sample t-tests (two-sided). Specifically, each experiment is repeated ten times with a different random seed before p-values ​​are calculated. e) IRENE is compared to junior physicians (<7 years of experience) and senior physicians (more than 7 years of experience). This study included two junior physicians and two senior physicians, with average performance reported within each group. IRENE outperformed the junior physicians in diagnostic performance while demonstrating competitiveness comparable to senior specialists.

[0010] Figure 2The network architecture of IRENE is depicted. a) The overall workflow of IRENE in its first task, lung disease recognition, is shown. Input data comprises five parts: chief complaint (ChiComp), laboratory test results (LabTest), demographic information (sex and age), and radiographs. Our Multimodal Diagnostic Transformer (MDT) includes two bidirectional multimodal attention blocks and ten self-attention blocks. The training process is guided by lung disease annotations provided by human experts. b) How different types of clinical text are encoded into free-form embeddings is shown. Specifically, IRENE accepts unstructured chief complaints as part of the input. c) How radiographs are encoded into a series of image block lexical units is shown. d) The detailed design of the bidirectional multimodal attention block is presented, which includes two normalization layers (Norm), one bidirectional multimodal attention layer, and one multilayer perceptron (MLP). e) The detailed attention operations in the bidirectional multimodal attention layer are presented, where representations across multiple modalities are learned and fused simultaneously. f) The detailed architecture of the self-attention blocks is shown.

[0011] Figure 3 An exemplary attention analysis is depicted. a) The attention assigned to different types of input from COPD patients (i.e., chest X-rays, chief complaint (ChiComp), laboratory test results (LabTest), and demographic information) is presented. b) The relative importance of laboratory test items is shown. c) The importance of gender and age in making diagnostic decisions is compared. d) The attention assigned to individual pixels in chest X-rays is visualized. The left panel is the input chest X-ray. The right panel presents pixels with different attention values. e) The effect of cross-attention on the relevance and importance of high-ranking words (from chief complaint) and image patches (from chest X-rays) in a lung disease identification task is explored. Specifically, high-ranking words and image patches are defined as words and image patches whose cosine similarity scores with CLS words rank in the top 25%. f) The normalized importance of each word in the chief complaint is presented. g) The distribution of attention between each image patch and each of the top 3 words is visualized. The color bars in (d, g) show IRENE's confidence level for anomalous pixels, where bright colors represent high confidence and dark colors represent low confidence.

[0012] Figure 4 The impact of chief complaints and laboratory test results on each respiratory disease was described.

[0013] Figure 5 illustrates the comparison with baseline models in Table 1: Lung disease identification task. Baseline models include image-only models, early fusion methods, late fusion methods, and two recent Transformer-based multimodal classification models (i.e., GIT and Perceiver). 95% CI represents the 95% confidence interval. The evaluation metric is AUROC.

[0014] Figure 6 illustrates the comparison with baseline models in the task of predicting adverse clinical outcomes in COVID-19 patients, as shown in Table 2. We included five models in the comparison: image-only models, early fusion methods, late fusion methods, and two recent Transformer-based multimodal classification models (GIT and Perceiver). 95% CI represents a 95% confidence interval. The evaluation metric was AUPRC.

[0015] Figure 7 illustrates the ablation studies of IRENE performed by removing or replacing individual components as shown in Table 3. HA(N) indicates that there are N bidirectional multimodal attention blocks in the Multimodal Diagnostic Transformer (MDT), while the remaining blocks are self-attention blocks (a total of twelve blocks). The image represents the input ray photograph. Unidirectional means that we only compute text-to-image attention in the multimodal attention blocks. ChiComp represents the complaint. LabTest represents the laboratory test results. Lexicalization represents the lexicalization procedure for the complaint and laboratory test results. The evaluation metric is AUROC. Detailed Implementation

[0016] All publications cited in this application, including patent documents, scientific articles, and databases, are incorporated herein by reference in their entirety, and are substantially equivalent to each individual publication being incorporated separately by reference. If any definition set forth herein contradicts or is inconsistent with a definition set forth in a patent, application, published application, or other publication incorporated herein by reference, the definition set forth herein shall take precedence over the definition incorporated herein by reference.

[0017] The chapter titles used in this article are for structural purposes only and should not be construed as limiting the topics described.

[0018] Despite significant progress in AI-based medical image diagnosis in recent years, the method of jointly interpreting medical images and their associated clinical context remains controversial. Figure 1 As shown in a, current multimodal clinical decision support systems 19-23Most methods rely on a non-uniform approach to fuse information from multiple sources. When input datasets come from diverse sources, these methods first broadly categorize them into three basic modalities: images, narrative text (e.g., chief complaints including present and past medical history), and structured fields (e.g., demographic information and laboratory test results). Next, a text structuring process is introduced to transform the narrative text into structured lexical units. Then, data from different modalities are fed into different machine learning models to generate modality-specific features or predictions. Finally, a fusion module unifies these modality-specific features or predictions to make a final diagnostic decision. In practice, these non-uniform methods can be further classified as early fusion, depending on whether multiple input modalities are fused at the feature level or the prediction level. 19-22 or late fusion 23 method.

[0019] A prominent issue with early and late fusion methods is that they divide the multimodal diagnostic process into two relatively independent phases: modality-specific model training and diagnostic-oriented fusion. However, this design has a significant limitation: it cannot encode the connections and relationships between different modalities. Another significant drawback of these non-uniform methods is the cumbersome and still labor-intensive text structuring process, even with modern Natural Language Processing (NLP) tools. On the other hand, Transformer-based architectures... 24 It is expected to reshape natural language processing in a broad sense. 25 and computer vision 26 With convolutional neural networks 27 and word embedding algorithm 28, 29 In comparison, Transformer 24 It imposes almost no assumptions about the form of the input data, thus making it possible to learn higher-quality feature representations from multimodal input data. More importantly, the basic architectural components in the Transformer (i.e., self-attention blocks) remain almost invariant across different modalities. 25, 26 This provides an opportunity to build a unified yet flexible model for representation learning of multimodal clinical information.

[0020] This paper proposes a unified AI-based medical diagnostic model, IRENE, which aims to make decisions by jointly learning holistic representations of medical images, unstructured complaints, and structured clinical information. To our knowledge, IRENE is the first medical diagnostic approach to simultaneously learn holistic representations of multimodal clinical information using a single, unified AI model. Figure 1As shown in Figure a, the core of IRENE is the Unified Multimodal Diagnostic Transformer (MDT) and the Bidirectional Multimodal Attention Block. The MDT is a novel Transformer stack that directly generates diagnostic results from multimodal input data. This novel algorithm enables IRENE to employ a different approach than previous non-unified methods by progressively learning a holistic representation of multimodal clinical information while eliminating separate paths for learning modality-specific features. Furthermore, the MDT gives IRENE the ability to perform representation learning on unstructured raw text, avoiding the verbose text structuring steps in non-unified methods. To better handle the differences between modalities, IRENE introduces bidirectional multimodal attention to bridge the gap between lexical-level modality-specific features and higher-level diagnostic-oriented holistic representations by explicitly encoding the interconnections between different modalities. This explicit encoding process can be seen as a complement to the holistic multimodal representation learning process within the MDT.

[0021] like Figure 2 As shown in Figure a, the MDT mainly consists of an embedding layer, a bidirectional multimodal block, and a self-attention block. Due to the presence of the MDT, IRENE possesses the ability to simultaneously perform joint interpretation of multimodal clinical information. Specifically, a free-form embedding layer is employed to convert unstructured and structured text into unified text lexical units (see Figure a). Figure 2 b). Simultaneously, a similar lexicalization procedure is applied to each input image (see [link]). Figure 2 c). Next, two bidirectional multimodal blocks (see...) Figure 2 d) Stacking to learn mid-level representations for fusion across multimodalities. In addition to computing intra-modal attention between lexics from the same modality, these blocks also explicitly compute inter-modal attention between lexics from different modalities (see [link to relevant documentation]). Figure 2 e). These intramodal and intermodal attention operations align with routine clinical practice, where physicians need to uncover interconnections both within and across modalities. In reality, these connections are often hidden within local patterns, such as words in a chief complaint and image regions in a radiograph, and different local patterns may refer to the same lesion or the same disease. Therefore, such connections allow clinical evidence to corroborate each other and contribute to both clinical diagnosis and AI-based diagnosis. In bidirectional multimodal attention, each lexical can be viewed as a representation of a local pattern, and lexical-level intramodal and intermodal attention captures interconnections between local patterns from within the same modality and across different modalities, respectively. In contrast, previous non-uniform approaches base diagnosis on the respective global representations of input data in different modalities, thus failing to leverage potential local interconnections. Finally, ten self-attention blocks are stacked (see...). Figure 2 f), to learn multimodal representations.

[0022] IRENE and the Vision-Language Fusion Model 29-33 Both share some common features, aiming to learn joint multimodal representations. However, a most significant difference lies in the roles of different modalities. IRENE is designed for scenarios where multiple modalities provide complementary semantic information, which can be fused and used to improve prediction performance. In contrast, recent vision-language fusion methods... 31-33 It heavily relies on the extraction and utilization of common semantic information between different modalities in order to provide supervision for model training.

[0023] IRENE's effectiveness was validated on two tasks (see Figure 1 b): a) Lung disease identification and b) Prediction of adverse clinical outcomes in COVID-19 patients. In the first task, IRENE outperformed previous image-only diagnostic methods and non-uniform diagnostic methods by approximately 12% and 9%, respectively (see [link to relevant documentation]). Figure 1 c). In the second task, IRENE was used to predict adverse clinical events in COVID-19 patients, namely admission to the intensive care unit (ICU), mechanical ventilation (MV) treatment, and death. Unlike the first task, the second task relied more heavily on textual clinical information. In this case, IRENE significantly outperformed non-uniform methods by more than 7% (see [link to task description]). Figure 1 d). Of particular note is the nearly 10% improvement achieved by IRENE in mortality prediction, demonstrating its potential to assist physicians in taking immediate action to save COVID-19 patients. In lung disease identification, it collaborates with human experts (see...). Figure 1 e) In comparison, IRENE significantly outperformed junior physicians (with <7 years of experience) in the diagnosis of all eight diseases, and performed as well as senior physicians (with more than 7 years of experience) in six diseases.

[0024] discuss In multimodal medical diagnostics, IRENE is more effective than previous non-uniform early and late fusion paradigms.

[0025] These are the most prominent observations from our experimental results, and also in the tasks of lung disease identification and triage of COVID-19 patients. Specifically, for lung disease identification, IRENE outperforms previous early and late fusion methods by an average improvement of 9% and 10%, respectively. Simultaneously, IRENE achieves approximately 3% performance improvement for all eight diseases and significantly improves diagnostic performance for four diseases (bronchiectasis, pneumothorax, ILD, and tuberculosis) by more than 10% in AUROC. These outstanding performance advantages are closely related to several capabilities of IRENE. First, IRENE is built on a unified Transformer (MDT). The MDT directly generates diagnostic decisions based on multimodal input data and progressively and implicitly learns the overall multimodal representation. In contrast, traditional non-unified methods decompose the diagnostic problem into several components, which in most cases include data structuring, modality-specific model training, and diagnosis-oriented fusion. In practice, these components are difficult to optimize and may prevent the model from learning overall features and diagnosis-oriented features. Secondly, inspired by physicians' daily activities, IRENE applies intra-modal attention and bidirectional inter-modal attention to lexicalized multimodal data to leverage local interconnections between complementary modalities. In contrast, previous non-uniform paradigms directly utilize extracted global modality-specific representations or predictions for diagnosis. In practice, the lexical-level attention operation in the proposed bidirectional multimodal attention helps capture interconnections between local patterns of different modalities and encode them into a fused representation. Last but not least, IRENE aims to learn representations directly from unstructured raw text. Conversely, previous non-uniform methods rely on non-clinically pre-trained NLP models to provide word embeddings, inevitably deviating the diagnostic system from its intended function.

[0026] The superiority of these capabilities has been partially validated in the second task: predicting adverse clinical outcomes in COVID-19 patients. As shown in Table 2, IRENE achieves an average performance improvement of 7% compared to earlier fusion methods and an average improvement of 9% compared to later fusion methods. This performance improvement is slightly lower than that in the lung disease identification task because the MMC dataset lacks unstructured text that IRENE can utilize. Nevertheless, IRENE can still leverage its unified and bidirectional multimodal attention mechanism to better serve the goal of rapid triage of COVID-19 patients. For example, IRENE improves the performance of MV and mortality prediction by 7% and 10%, respectively. This significant performance improvement brought by the IRENE system is valuable in the real world for timely allocation of appropriate medical resources to patients, as medical resources are often limited during the COVID-19 pandemic.

[0027] IRENE offers a better Transformer-based approach for the joint interpretation of multimodal clinical information.

[0028] IRENE is compared with two typical Transformer-based models, GIT. 33 and Perceiver 30 In comparison, these two models fuse multimodal information for classification. GIT pre-trains on tens of millions of image-text pairs by utilizing common semantic information across different modalities as a supervision signal. However, these features have two significant drawbacks in medical diagnostic scenarios. First, accessing the same order of magnitude of multimodal medical data is much more difficult. Second, multimodal data in medical diagnostic scenarios provides complementary semantic information rather than shared semantic information. Therefore, large-scale multimodal pre-training using limited medical data (as in GIT) is impractical. These drawbacks are also reflected in the experimental results. For example, GIT's average performance is approximately 7% and 8% lower than IRENE in the lung disease identification task and the COVID-19 adverse outcome prediction task, respectively. These advantages demonstrate that the lexical-level bidirectional multimodal attention in IRENE can effectively utilize limited multimodal medical data and leverage complementary semantic information.

[0029] Perceiver simply concatenates the multimodal input data and uses the resulting 1D sequence as input, instead of learning a fusion representation between modality-specific low-level embeddings as in IRENE. This introduces a potential problem: the modality that constitutes the majority of the input will have a greater impact on the final diagnostic outcome. For example, since images tend to have far more terms than text, Perceiver inevitably assigns more weight to images (rather than text) when making predictions. However, images do not always play a more significant role in routine clinical decision-making. This is reflected to some extent in our experimental observations. For instance, Perceiver shows a significant performance improvement in identifying lung diseases compared to earlier fusion models (an average improvement of 2% in Table 1), with input radiographs used as the primary source of information. However, in the task of rapid triage of COVID-19 patients, Perceiver's performance is only comparable to that of earlier fusion methods. The fundamental reason is that CT images are less helpful in identifying lung diseases in this task than radiographs. In contrast, IRENE demonstrates satisfactory performance in both tasks by learning a global multimodal representation via bidirectional multimodal attention. Our method promotes the uniform mixing of features from different modalities, thereby preventing the learned representation from being dominated by high-dimensional inputs.

[0030] IRENE helps reduce the reliance on text structuring in traditional workflows.

[0031] In traditional non-uniform multimodal medical diagnostic methods, text structuring is a common approach for handling unstructured text. Recent non-uniform methods... 19-23 The text structuring process in traditional text processing relies heavily on manual rules and modern NLP tools. For example, text structuring requires human annotators to manually define lists of alternative spellings, synonyms, and abbreviations for structured labels. In addition to these preparatory tasks, specialized NLP tools are developed and applied to extract structured fields from unstructured text. Therefore, the text structuring steps are not only cumbersome but also costly in terms of labor and time. In contrast, IRENE eliminates these lengthy structuring steps by directly accepting unstructured clinical text as part of its input.

[0032] prospect In summary, despite NLP techniques, especially transformers, for the use of text-based electronic health records 35 or image 36 While recent AI diagnostic tools have made significant contributions, this study describes an AI framework that includes a unified multimodal diagnostic Transformer (MDT) and bidirectional multimodal attention blocks. This novel algorithm enables IRENE to employ a different approach than previous non-unified methods by progressively learning a holistic representation of multimodal clinical data while eliminating the separate paths used in non-unified techniques for learning modality-specific features. This approach will be greatly enhanced by recent advancements in large-scale language models. 37, 38 .

[0033] In real-world scenarios, IRENE could help streamline patient care processes, such as triage and differentiating between patients who may have a common cold and those with more severe conditions requiring urgent intervention. Furthermore, as algorithms become more sophisticated, these frameworks can serve as diagnostic aids for physicians, assisting in situations of diagnostic uncertainty or complexity, thus not only simulating but also enhancing physician reasoning. Our work is likely to have the most significant impact in areas where healthcare providers are relatively few in number and unevenly distributed among the population.

[0034] The following section highlights several limitations to consider when deploying IRENE in clinical workflows. First, the currently used datasets are limited in both size and diversity. To address this, more data needs to be collected from other healthcare institutions, medical devices, national and ethnic groups to train IRENE and enhance its generalization ability in broader clinical settings. Second, the clinical benefits of IRENE require further validation. Therefore, multi-institutional, multinational studies can further validate the clinical utility of IRENE in real-world scenarios. Third, it is important to adapt IRENE to constantly changing environments, such as responding to the rapidly mutating SARS-CoV-2 virus. To address this challenge, the model can be trained jointly on multiple cohorts or employ other machine learning techniques, such as online learning. Last but not least, IRENE fails to account for modality deficiencies, where one or more modalities may be unavailable. To address this, masked modeling can be used. 25 For example, during the training phase, some modalities can be randomly masked to simulate the absence of these modalities in clinical workflows.

[0035] method Image and text-based clinical data.

[0036] In the lung disease identification task, CXR images were acquired from West China Hospital. All CXRs were acquired as part of the patients' routine clinical care. To analyze the CXR images, all X-ray images were first de-identified to remove any patient-related information. CXR images include front and rear views. Three types of textual clinical data were present: unstructured chief complaints (i.e., present and past medical history), demographic information (age and sex), and laboratory test results. Specifically, chief complaints were unstructured, while demographic information and laboratory test results were structured. The maximum length of the chief complaint was set to 40 words. If a patient's chief complaint exceeded 40 words, only the first 40 words were taken; otherwise, zero padding was used to meet the length requirement. Each patient's laboratory test report contained 92 results (see Supplementary Note 2), most of which were from blood tests. Each test result was standardized using min-max scaling, such that each standardized value was within [0, 1], where the minimum and maximum values ​​in the min-max scaling were determined using the training set. Specifically, -1 indicates a missing value.

[0037] In the second task, namely the prediction of adverse clinical outcomes in COVID-19 patients, the available clinical data were divided into four categories: demographic information (age and sex), structured complaints including comorbidities (7) and symptoms (9), and laboratory test results (19). See Supplementary Note 3 for further details. In addition, median imputation was used to fill in missing values.

[0038] Approval has been obtained from the Institutional Review Board (IRB) / Ethics Committee of West China Hospital and all participating hospitals. All patients signed consent forms. This study was conducted in accordance with the U.S. Health Insurance Portability and Accountability Act (HIPAA). This study followed the principles of the Declaration of Helsinki and complied with the policies of the Chinese CDC regarding reportable infectious diseases and the Chinese Health Quarantine Law.

[0039] Baseline model Our experimental performance comparisons include five baseline models: a diagnostic model entirely based on medical images (labeled as image-only), a traditional non-uniform early and late fusion method using multimodal input data, and two recent state-of-the-art Transformer-based multimodal classification methods (i.e., GIT and Perceiver). Implementation details of these baseline models are as follows: Image only In lung disease identification tasks, diagnostic models based entirely on medical images are in ViT 26 Built upon this foundation, ViT is one of the most well-known and widely adopted Transformer-based deep neural networks for image understanding. Our ViT-like network architecture has 12 blocks, and each block includes a self-attention layer. 24 One multilayer perceptron (MLP) and two normalization layers 39 Each MLP contains two fully connected (FC) layers with 3,072 hidden nodes. The input size of the first FC layer is 768. A GeLU activation function is inserted between the two FC layers. 40 Add a dropout layer after each FC layer. 41 The dropout rate is set to 0.3. The output size of the second fully connected layer is also 768. Each input image is divided into several 16×16 image patches. The output CLS terms are used to perform the final classification. During training, binary cross-entropy loss is used as the cost function. It is worth noting that before the training phase, in MIMIC-CXR... 42 Supervised ViT pre-training is performed to obtain visual representations with stronger generalization capabilities. This is useful in tasks such as rapid triage of COVID-19 patients. 22In this study, pneumonia lesions are first segmented from CT scans. Then, multiple machine learning models (i.e., logistic regression, random forest, support vector machine, MLP, and LightGBM) are trained using image features extracted from the segmented lesion regions. Finally, the best model is selected based on the performance of these models on the validation set.

[0040] Non-unified early and late fusion Many methods exist that use typical non-uniform approaches to fuse multimodal input data for diagnostic purposes. To better adapt to different scenarios, different non-uniform models are employed for different tasks. Specifically, our first task (i.e., lung disease identification) utilizes an early fusion method reported in previous studies. 19 In practice, the ViT model extracts image features from ray photographs and uses the feature vectors of its CLS terms as a representation of the input image. Similar to the image-only baseline approach, it will be based on MIMIC-CXR before performing the formal task. 42 Supervised pre-training was applied to ViT to obtain more robust visual features. To handle three types of clinical data (i.e., chief complaint, demographic information, and lab test results), three independent MLPs were used to convert different types of textual clinical data into features, which were then concatenated with image representations. The underlying principle is that for cross-referencing, image and textual data should be represented in the same feature space. Since the chief complaint includes unstructured text, it first needs to be converted into structured items. To achieve this, an entity recognition model was trained to highlight relevant clinical symptoms in the chief complaint. Next, BERT was used... 25 The model extracts features from all these symptoms and applies average pooling to them to produce an overall representation of each patient's complaint. This overall feature set is then further transformed into a latent space similar to the latent space of the image representation using a three-layer MLP. This three-layer MLP has an input size of 768 and an output size of 512. It has 1,024 hidden nodes. A ReLU activation layer and a dropout layer with a dropout rate of 0.3 are added after each fully connected (FC) layer. Similarly, for lab test results, an MLP with the same architecture but independent weight parameters is applied to transform these test results into one-dimensional feature vectors. This lab test MLP has an input size of 92 and an output size of 512. The MLP model for demographic information has two FC layers with an input size of 2 and an output size of 512. The hidden layers have 512 nodes. The feature fusion module consists of a concatenation operation and a three-layer MLP with 1,024 hidden nodes. The output of the MLP in the feature fusion module is passed to the final classification layer for diagnostic decision-making. During the training phase, binary cross-entropy loss is used to jointly train the ViT-like model and all MLPs. As for the late-stage fusion baseline, it will be affected by... 23The predictions from the image-based and text-based classifiers are integrated. Specifically, a ViT model is trained with ray photographs and their associated labels. To construct the input for the text-based classifier, lab test results, demographic information, and a holistic representation of the subject's complaint (obtained by averaging extracted symptom features, similar to earlier fusion methods) are concatenated. The input is forward-propagated through a three-layer MLP with input and output dimensions of 862 and 8, respectively. The ViT model is then trained using an MLP with the same labels. Finally, the probabilities predicted by the image-based and text-based classifiers are averaged to obtain the final prediction.

[0041] In the second task, a proposed early fusion method was proposed. 22 The process involves concatenating image features, structured chief complaints (comorbidities and symptoms), and laboratory test results as input. Multiple machine learning models are then trained using the previously described artificial rules. 22 Select the optimal model. For late-stage fusion baselines, follow... 22 The schemes used for image features, structured claims, and laboratory test results were respectively trained on five machine learning models (logistic regression, random forest, support vector machine, MLP, and LightGBM). Then, the average of the predicted probabilities of these fifteen machine learning models was used as the prediction of adverse outcomes.

[0042] GIT GIT 33 It is a generative image-to-text Transformer for the unified vision-language task. GIT-Base was used as our baseline for comparison. GIT-Base's image encoder is a ViT-like transformer, and its text decoder consists of six standard transformer blocks. 24 In practice, the officially released pre-trained model is finely trained on a custom dataset. For fairness, the same fine-tuned hyperparameter settings as the IRENE model are used. In the lung disease identification task, each X-ray image is first forward-propagated through an image encoder to extract image features. Next, these image features are concatenated with the average word embeddings of the chief complaint (using BERT), along with feature vectors of demographic information and laboratory test results. The concatenated features are then passed to a text decoder for diagnostic prediction. In the COVID-19 patient adverse clinical outcome prediction task, the image features of CT slices are first averaged. Then, the averaged image features are concatenated with feature vectors of clinical comorbidities and symptoms, laboratory test results, and demographic information. The concatenated multimodal features are forward-propagated through a text decoder to predict adverse outcomes in COVID-19 patients.

[0043] Perceiver This is DeepMind's most recent and state-of-the-art Transformer-based model. 30 This is intended to solve the classification problem of multimodal input data. There is also a Perceiver... 30 A variant of Perceiver IO 43 It introduces output queries on top of Perceiver to handle additional task types. Since making diagnostic decisions can be viewed as a form of classification, we use Perceiver instead of Perceiver IO as one of our baseline models. Our Perceiver architecture follows the settings of ImageNet classification. 44,30 It features six cross-attention modules. Each cross-attention module is followed by a latent Transformer model with six self-attention blocks. The Perceiver's input consists of two arrays: a latent array and a byte array. According to... 30 The latent array is initialized using a truncated zero-mean normal distribution with a standard deviation of 0.02 and a cutoff boundary of [-2, 2]. The byte array includes multimodal data. In the lung disease recognition task, the input image is first flattened into a one-dimensional vector. This is then concatenated with the average word embedding of the chief complaint (using BERT) and one-dimensional feature vectors of the input demographic information and laboratory test results. This results in a long one-dimensional vector, which is treated as a byte array. In the COVID-19 adverse clinical outcome prediction task, the input image is flattened into a one-dimensional vector, and then this one-dimensional vector is concatenated with feature vectors of clinical comorbidities and symptoms, laboratory test results, and demographic information. The learning process of Perceiver can be summarized as follows: the latent array iteratively extracts higher-quality features from the input byte array through alternating cross-attention and latent self-attention computation, thus continuously evolving. Finally, the transformed latent array is used as a representation for diagnosis. Note that, similar to image-only and non-uniform baselines, Perceiver in MIMIC-CXR... 42 Pre-training is performed on the input. During pre-training, zero-padding is used in the byte array for each clinical text that is not present in the multimodal input.

[0044] IRENE In practice, multimodal input data (i.e., medical images and textual clinical information) is forward-propagated to the MDT to obtain predicted logit values. During the training phase, the binary cross-entropy loss between the logit value and the ground truth label is calculated. Specifically, in the first and second tasks, lung disease annotations (eight diseases) and actual adverse clinical outcomes (three clinical events) are used as ground truth labels, respectively.

[0045] MDT stands for Unified Transformer, which mainly consists of: two initial layers for embedding lexical units from the input image and text, respectively; two stacked bidirectional multimodal attention blocks for learning a fused mid-level representation by capturing interconnections between lexical units from the same modality and across modalities; ten stacked self-attention blocks for learning the overall multimodal representation and enhancing its discriminative power; and a classification head for generating predicted logistic values.

[0046] In the lung disease identification task (i.e., the first task), the multimodal input data includes five parts: radiographs, unstructured chief complaints including present and past medical history, laboratory test results, and the gender and age of each patient. These parts are represented as follows: , , , and . The data is passed to a convolutional layer, which produces a series of visual lexical units. Next, standard learnable 1D positional embeddings are added. 21,23 And discard it to each visual lexical unit to obtain a series of image block lexical units. At the same time, for Lexicalization is applied to encode each word in the unstructured subject. Specifically, a pre-trained BERT is used. 23 Model to generate The embedding feature vector of each word is then used to obtain a series of word units. .right A similar lexicalization procedure is applied, where a minimum-maximum scaling pair is first used. Each component is normalized. Then, each normalized component is passed to a shared linear projection layer to obtain a series of latent embeddings. .right and Perform linear projection to obtain the encoded feature vector. and Subsequently, These are pieced together to produce a series of clinical text lexical units. ,in In practice, and Set them to 40 and 92 respectively.

[0047] Regarding the task of predicting adverse clinical outcomes in COVID-19 patients, its multimodal input data also includes five parts: a set of CT slices, structured chief complaint (comorbidities and symptoms), laboratory test results, and the gender and age of each patient. These parts are represented as follows: , , , and In the first task, each CT slice was transformed into a series of image block terms. Unlike the first task, the chief complaint is structured. In order to... The components are converted into tokens, and a shared linear projection is performed on each component to generate a series of embeddings. Apply a linear projection layer to to obtain As for and For example, in the first task, a linear projection is performed to obtain the encoded... and Finally, Direct splicing to produce Clinical text terms ,in . and Set them to 16 and 19 respectively.

[0048] The first two layers of the MDT consist of two stacked bidirectional multimodal attention blocks. Assume the input to the first bidirectional multimodal attention block includes... and ,in Representative layer index, The set representing image block terms, and This represents a clinical text lexical package. The process of generating the query matrix, key matrix, and value matrix for each modality within the bidirectional multimodal attention block is as follows: , , in and These represent linear projection and layer normalization, respectively. The forward pass within a bidirectional multimodal attention block can be summarized as: , , in, and Capture modal intra-modal connections in the image and text modalities respectively. and We mine intermodal connections between images and text. Then, both intramodal and intermodal connections are encoded into latent representations. and middle. It was set to 1.0 because this value produced the best performance in our initial experiments. Attention Includes two matrix multiplications and a scaling softmax operation: , Where T represents the matrix transpose operator, d k This is the scaling hyperparameter, which is set to 64. Next, we introduce residual learning. 45 and will obtain Forward propagation to the subsequent normalization layer and MLP: , , Will and The input is passed to the next bidirectional multimodal attention block to obtain... and Then, and The lexical combinations in the code are used to generate a unified lexical bag, which is then passed to the subsequent self-attention block. 24 Multiple heads are allocated in both the bidirectional multimodal attention block and the self-attention block. 24 The number of heads is set to 12. This multi-head mechanism allows the model to perform attention operations simultaneously in multiple representation subspaces, and then aggregate the results.

[0049] Finally, average pooling is applied to the unified tokens generated from the final self-attention block to obtain a holistic multimodal representation for medical diagnosis. This representation is passed to a two-layer MLP to produce the final predicted logistic values. During the training phase, a binary cross-entropy loss is calculated between these logistic values ​​and their corresponding lung disease labels (first task) or actual adverse clinical outcomes (second task). The loss function value is calculated for each patient case. Specifically, in the first task, each patient case contains one radiograph and associated textual clinical information. In the second task, each patient case involves multiple CT slices, and these CT slices share the same textual clinical information. Each CT slice and its accompanying textual clinical information are forward-propagated to the MDT to obtain a holistic representation. Due to the multiple CT slices, multiple holistic representations are obtained for the same patient (equal to the number of CT slices). Average pooling is then performed on these holistic representations to calculate an average representation, which is finally passed to a two-layer MLP and a binary cross-entropy loss.

[0050] Implementation details.

[0051] For the lung disease recognition task, during the training phase, the size of each radiograph was resized to 256×256 pixels, and then a random portion of each image was cropped, with the area ratio of the cropped image patch to the original radiograph randomly determined between 0.09 and 1.0. The size of the cropped image patch was then resized to 224×224, followed by random horizontal flipping to increase the diversity of the training data. During the validation and testing phases, the size of each radiograph was first resized to 256×256 pixels, and then a square image patch was cropped from the center of the image. The size of the square cropped patch was 224×224. The processed radiographs were ultimately passed as input images to the image-only model, the non-uniform Chest, Perceiver, and IRENE. In the task of predicting adverse clinical outcomes in COVID-19 patients, the input images were CT scans. 46 The proposed lesion detection and segmentation method is based on a deep learning algorithm using a multi-view feature pyramid convolutional neural network. 47,48The neural network performs lesion detection, segmentation, and localization. It was trained and validated on 14,435 participants using chest CT images and definitive pathogen diagnoses. On a patient-by-patient basis, the algorithm demonstrated excellent sensitivity (1.00 [95% CI: 0.95, 1.00]) and an F1 score of 0.97 in detecting lesions based on CT images of COVID-19 pneumonia patients. It is speculated that adverse clinical outcomes in COVID-19 are closely related to the characteristics of pneumonia lesion regions. For each patient case, the 3D CT subvolume was cropped by calculating the minimum 3D bounding box surrounding all pneumonia lesions. Next, the size of all 3D subvolumes from different patients was reset to a uniform size of 224×224×64. Finally, 16 evenly spaced slices were taken from each 3D subvolume along its third dimension as samples.

[0052] Before performing the formal training procedure, in MIMIC-CXR 42 The MDT is pre-trained similarly to the baseline model. Similar to Perceiver, during pre-training, zero-padding is used for any missing textual clinical information in each multimodal input. During formal training, AdamW is used. 49 The default optimizer was chosen because it was empirically found to produce better performance on both the baseline model and IRENE. The initial learning rate was set to 3e-5, and the weights decayed to 1e-2. Each model was trained for 30 epochs, with the initial learning rate reduced by a factor of 10 at the 20th epoch. The batch size was set to 256 during the training phase for both tasks. Notably, in the COVID-19 adverse clinical outcome prediction task, a global feature representation was first extracted from 16 CT slices (cropped and sampled from the same CT volume). Next, average pooling was applied to these 16 global features to obtain an average representation representing all pneumonia lesion regions within the entire CT volume. A binary cross-entropy loss was then calculated based on this average representation. During the training phase, model performance was evaluated on the validation set, and the validation loss was calculated after each epoch. The model checkpoint that produced the lowest validation loss was saved and then tested on the test set. All ViT models employed learnable positional embeddings. PyTorch was used. 50 Implement IRENE, and through a hybrid precision strategy 51 NVIDIA Apex was used to accelerate the training phase. In practice, using four NVIDIA GPUs, the training phase for any task could be completed within a day.

[0053] The visual transformer employs a standard attention analysis strategy. For each layer in the transformer, attention weights are averaged across multiple heads (since multi-head self-attention is used in IRENE) to obtain an attention matrix. To account for residual connections, an identity matrix is ​​added to each attention matrix, and the resulting weight matrix is ​​normalized. Next, the weight matrices from different layers of the transformer are recursively multiplied. Finally, an attention map is obtained, which includes the similarity between each input term and a CLS term. Since CLS terms are used for diagnostic predictions, these similarities indicate a correlation between the input term and the prediction result, which can be used for visualization. For cross-attention results, Grad-CAM is used. 52 Visualization.

[0054] Nonparametric bootstrap sampling was used to calculate the 95% confidence interval. Specifically, 1,000 bootstrap samples were repeatedly drawn from the unseen test set. Each bootstrap sample was obtained by sampling with random replacement and was the same size as the test set. Then, AUROC (first task) or AUPRC (second task) was calculated based on each bootstrap sample, resulting in 1,000 AUROC or AUPRC values. Finally, these performance results were categorized, and the values ​​at the 2.5 percentile and 97.5 percentile were reported, respectively.

[0055] To demonstrate the statistical significance of our experimental results, we first repeated the IRENE and best-performing baseline (i.e., the Perceiver) experiments five times using different random seeds. Then, we used an independent two-sample t-test (two-tailed) to calculate the p-value between the mean performance of the IRENE and the best-performing baseline results.

[0056] The code can be obtained at https: / / github.com / RL4M / IRENE.

[0057] Example The following examples are included for illustrative purposes only and are not intended to limit the scope of this disclosure.

[0058] Transformer-based representation-learning models handle multimodal inputs in a unified manner and outperform non-unified multimodal models in two clinical-diagnostic tasks.

[0059] Example 1 – Dataset Features for Multimodal Diagnostics The first dataset focuses on lung diseases. Between November 27, 2008, and May 31, 2019, consecutive chest X-rays were retrospectively collected from 51,511 patients at West China Hospital, the largest tertiary medical center in western China, covering a population of 100 million. Each patient was associated with at least one X-ray, a brief unstructured chief complaint, present and past medical history, demographic information, and a complete laboratory test report. The dataset was constructed for eight lung diseases, including chronic obstructive pulmonary disease (COPD), bronchiectasis, pneumothorax, pneumonia, interstitial lung disease (ILD), tuberculosis, lung cancer, and pleural effusion. Based on previous research... 16 The standard procedure described herein extracts the discharge diagnosis from the discharge summary report and uses it as the true disease label. The discharge summary report is generated as follows: A preliminary report is written by a junior physician, and then the preliminary report is reviewed and confirmed by a senior physician. In case of any disagreement, the final decision is made by a department committee consisting of at least three senior physicians.

[0060] The constructed dataset comprises 72,283 data samples, of which 40,126 are normal. The disease distribution (i.e., the number of relevant cases) is as follows: COPD (4,912 cases), bronchiectasis (676 cases), pneumothorax (2,538 cases), pneumonia (21,409 cases), ILD (3,283 cases), tuberculosis (938 cases), lung cancer (2,651 cases), and pleural effusion (4,713 cases). The performance metric is the area under the receiver operating characteristic curve (AUROC). The dataset was split into training, validation, and test sets based on each patient's admission date. Specifically, the training set includes 44,628 patients admitted between November 27, 2008, and June 1, 2018. The validation set includes 3,325 patients admitted between June 2, 2018, and December 1, 2018. Finally, the trained and validated IRENE system was tested on 3,558 patients admitted between December 2, 2018, and May 31, 2019. Although this is a retrospective study, our data segmentation scheme follows the practices of prospective studies, thus creating a more challenging and realistic setting to validate the effectiveness of different multimodal medical diagnostic systems compared to data segmentation schemes based on random sampling.

[0061] The second dataset is MMC (i.e., the multimodal COVID-19 dataset). 19—Training and evaluating the IRENE model on this dataset involved collecting chest CT images and structured clinical information (e.g., chief complaints including comorbidities and symptoms, demographic information, laboratory test results, etc.) from COVID-19 patients. These CT images were associated with hospitalized patients with laboratory-confirmed COVID-19 infection between December 27, 2019, and March 31, 2020. In MMC, patients may experience three adverse events: ICU admission, mechanical ventilation (MV), and death. The training and validation sets were drawn from 17 hospitals, with the training set containing 1,164 labeled cases (70%) and the validation set containing 498 labeled cases (30%). Next, the training model that performed best on the validation set was selected and tested on a separate test set comprising 700 cases collected from 9 external medical centers. The distribution of the three events in the test set was as follows: ICU (155 cases), MV (94 cases), and death (59 cases). This is an imbalanced classification problem, where the majority of patients do not experience adverse outcomes. In this context, the area under the precision-recall curve (AUPRC) is used instead of AUROC as a performance metric, with AUPRC focusing more on identifying adverse events (i.e., ICU, MV, and death).

[0062] Example 2 – Lung Disease Identification Table 1 and Figure 3 Experimental results of IRENE and other methods on datasets for lung disease identification are presented. As shown in Table 1, IRENE significantly outperforms image-only models and traditional non-uniform early fusion methods in identifying lung diseases. 19 and late fusion methods 23 And two recent state-of-the-art Transformer-based multimodal methods (i.e., Perceiver) 30 and GIT 33Overall, IRENE achieved the highest mean AUROC (0.924, [95% CI: 0.921, 0.927]), approximately 12% higher than image-only models that only use radiographs as input (0.805, [95% CI: 0.802, 0.808]). IRENE maintained an advantage of at least 9% compared to diagnostic decisions made by non-uniform early fusion methods (0.835, [95% CI: 0.832, 0.839]) and late fusion methods (0.826, [95% CI: 0.823, 0.828]). Compared to GIT (0.848, [95% CI: 0.844, 0.850]), an advantage of over 7% was observed. Even compared to Perceiver, a Transformer-based multimodal classification model developed by DeepMind, IRENE still achieved competitive results, outperforming Perceiver by more than 6% (0.858 [95% CI: 0.855, 0.861]). When carefully examining each disease and comparing IRENE with the best results among all five previous baselines, IRENE showed the greatest improvement in bronchiectasis (12%), pneumothorax (10%), ILD (10%), and tuberculosis (9%) across all eight lung diseases.

[0063] IRENE was compared with human specialists, who were divided into two groups: one group consisted of two junior physicians (with <7 years of experience), and the other group consisted of two senior physicians (with ≥7 years of experience). For better comparison, Figure 1e presents the average performance within each group. Specifically, annotations by human experts were extracted from electronic discharge diagnostic records. Notably, none of the physicians from the reader study participated in data annotation. IRENE showed superiority over the primary group in all eight lung diseases, particularly in the diagnosis of bronchiectasis (primary, [FPR: 0.29, TPR: 0.58]), pneumonia (primary, [FPR: 0.37, TPR: 0.76]), ILD (primary, [FPR: 0.09, TPR: 0.63]), and pleural effusion (primary, [FPR: 0.35, TPR: 0.86]), where FPR and TPR represent the false positive rate and true positive rate, respectively. Compared to the senior physician group, IRENE showed an advantage in the diagnosis of pneumonia (senior, [FPR: 0.21, TPR: 0.80]), tuberculosis (senior, [FPR: 0.07, TPR: 0.17]), and pleural effusion (senior, [FPR: 0.25, TPR: 0.77]). Furthermore, IRENE performed comparably to senior physicians in COPD (senior, [FPR: 0.07, TPR: 0.76]), ILD (senior, [FPR: 0.09, TPR: 0.71]), and pneumothorax (senior, [FPR: 0.08, TPR: 0.79]), while showing slightly weaker performance in bronchiectasis (senior, [FPR: 0.12, TPR: 0.82]) and lung cancer (senior, [FPR: 0.08, TPR: 0.73]).

[0064] Example 3 – Prediction of adverse clinical outcomes in COVID-19 patients.

[0065] Triage of COVID-19 patients largely depends on the combined interpretation of chest CT scans and other non-radiological clinical information. In this context, IRENE demonstrates even greater advantages than in the lung disease identification task. As shown in Table 2, IRENE consistently achieves significant performance improvements in predicting the three adverse clinical outcomes (i.e., ICU admission, mechanical ventilation, and death) in COVID-19 patients. In terms of mean AUPRC, IRENE (0.592, [95% CI: 0.500, 0.682]) outperforms the image-only model (0.307, [95% CI: 0.237, 0.391]) and the early fusion model. 22 (0.521, [95% CI: 0.435, 0.614]) and late fusion model 23(0.503, [95% CI: 0.422, 0.598]) and improved by nearly 29%, 7%, and 9%, respectively. Regarding specific clinical outcomes, in predicting ICU admission, IRENE (0.712, [95% CI: 0.587, 0.834]) achieved approximately a 5% AUPRC gain compared to the non-uniform early fusion method (0.665, [95% CI: 0.548, 0.774]). Similarly, in predicting MV, IRENE achieved a performance improvement of over 6% compared to the early fusion model. Last but not least, IRENE (0.441, [95% CI: 0.270, 0.617]) significantly outperformed image-only models (0.192, [95% CI: 0.073, 0.333]), early fusion models (0.346, [95% CI: 0.174, 0.544]), and late fusion models (0.335, [95% CI: 0.168, 0.554]) in predicting mortality. An average advantage of over 6% was observed compared to the two Transformer-based multimodal models (i.e., GIT and Perceiver).

[0066] Example 4 – The impact of different modules and modes in IRENE.

[0067] To explore the impact of different modules and modalities, a comprehensive ablation experiment was conducted, and the results are reported in Table 3. First, the impact of bidirectional multimodal attention blocks (rows 0-2) was investigated. Replacing all bidirectional multimodal attention blocks with self-attention blocks resulted in a performance decrease of approximately 7% in lung disease recognition (from 0.924 to 0.858). This phenomenon confirms our intuition that learning progressively fused representations directly from raw data will degrade diagnostic performance. Conversely, simply increasing the number of bidirectional multimodal attention blocks from two to six did not bring a significant performance improvement (from 0.924 to 0.905), suggesting that using two consecutive bidirectional multimodal attention blocks may be the optimal choice in IRENE. Row 3 presents the results using unidirectional attention (i.e., text-to-image attention). Comparing rows 0 and 3, a 4% performance improvement was observed in the bidirectional design (from 0.884 to 0.924). Next, the impact of clinical text (rows 4 and 5) was investigated. The first observation is that utilizing supplemental narrative complaints significantly improves diagnostic performance, as removing complaints from the input data degrades model performance by 6% (from 0.924 to 0.860). In addition to complaints, the impact of laboratory test results (row 5) was investigated. Including laboratory test results resulted in approximately a 4% performance improvement (from 0.882 to 0.924). The effect of lexicalization was then explored. Modeling the patient's complaints and laboratory test results as a series of lexical units (row 0) performed better than directly passing the average representation (row 6) to the model. This improvement through lexicalization of complaints and laboratory test results validates the advantages of bidirectional multimodal attention at the lexical level, both intramodal and intermodal, which leverages the local interconnections between lexical units of clinical text and image block units of radiographs in the input data. Finally, the impact of input images in IRENE (row 7) leads to a sharp performance decline (from 0.924 to 0.543). This phenomenon indicates that input radiographs play a crucial role in the identification of lung diseases. The impact of chief complaints and laboratory test results on each respiratory disease was then explored. Figure 4 Performance degrades for each disease when chief complaints or lab test results are removed from the input. Specifically, including chief complaints is most helpful for diagnosing pneumothorax, lung cancer, and pleural effusion, while lab test results have the greatest impact on diagnosing bronchiectasis and tuberculosis. Clinical interpretations can be found in Supplementary Note 1.

[0068] Example 5 – Attention visualization results.

[0069] Figure 3 Attention visualization results for a COPD case are provided. Figure 3In this study, image modality (i.e., radiographs) plays a crucial role in the diagnostic process, accounting for nearly 80% of the final decision. Furthermore, the chief complaint is the second most important factor, accounting for approximately 16% of the weight. Figure 3 As shown in b, PaO2 (partial pressure of oxygen in arterial blood) and PaCO2 (partial pressure of carbon dioxide in arterial blood) are the two most important laboratory test items, which is consistent with the literature. 34 The observations reported in the article are consistent. Nevertheless, the total weight of the remaining 90 tests is quite large, and these weights are distributed almost evenly across these 90 lab tests. This may be because these lab tests may help rule out other diseases. Figure 3 c shows that, from IRENE's perspective, age is a more critical factor than gender. Figure 3 The d-image provides an attention map of the radiographs, suggesting that IRENE considers hilar enlargement, hyperdistension, and diaphragmatic flattening as the most important evidence for diagnosing COPD. Furthermore, IRENE also identifies large dark areas caused by bullae as relatively important evidence. Figure 3 e summarizes the experimental results with and without cross-attention, presenting the sum of similarity scores between important (top 25%) terms (i.e., words and image patches) and CLS terms. With cross-attention, the sum of similarity scores becomes larger, indicating that cross-attention improves the identification of important terms compared to the model without it. Figure 3 In her work, IRENE identified “phlegm,” “dyspnea,” and “year” as the three most important words in the chief complaint. Figure 3 g provides cross-attention maps between each of the top three most important words and the image. The word "sputum" is primarily associated with the trachea and lower lobes of the lungs in the image. The high attention area for the trachea is likely reasonable, as the trachea is often a site where sputum is likely to be produced. The high attention area in the left lower lobe shows reduced vascular scarring, while both the left and right lower lobes are overinflated. Lung overinflation and reduced vascular scarring are common symptoms of COPD, often accompanied by abnormal sputum production. Our model also associates the word "dyspnea" with a large area of ​​the lungs in the image, as dyspnea can be caused by a variety of lung abnormalities that can occur anywhere in the lungs. Finally, our model identifies the peribronchial region as the image region associated with the word "year," suggesting that "year" should be associated with chronic diseases such as chronic bronchitis, which is often part of COPD.

[0070] References 1. He, J. et al. The practical implementation of artificialintelligence technologies in medicine. Nature Medicine 25, 30-36, doi:10.1038 / s41591-018-0307-0 (2019). 2. Liang, H. et al. Evaluation and accurate diagnoses of pediatricdiseases using artificial intelligence. Nature Medicine 25, 433-438, doi:10.1038 / s41591-018-0335-9 (2019). 3. Boehm, K. M., Khosravi, P., Vanguri, R., Gao, J.&Shah, S. P.Harnessing multimodal data integration to advance precision oncology. Nat.Rev. Cancer 22, 114-126 (2022). 4. Li, J., Shao, J., Wang, C.&Li, W. The epidemiology and therapeuticoptions for the COVID-19. Precis. Clin. Med. 3, 71-84 (2020). 5. Comfere, N. I. et al. Provider-to-provider communication indermatology and implications of missing clinical information in skin biopsyrequisition forms: a systematic review. Int. J. Dermatol. 53, 549-557 (2014). 6. Shao, J. et al. Radiogenomic system for non-invasiveidentification of multiple actionable mutations and PD-L1 expression in non-small cell lung cancer based on CT images. Cancers (Basel) 14, 4823 (2022). 7. Huang, S. C., Pareek, A., Seyyedi, S., Banerjee, I.&Lungren, M. P.Fusion of medical imaging and electronic health records using deep learning:a systematic review and implementation guidelines. npj Digit. Med. 3, 136(2020). 8. Wang, C. et al. Non-Invasive measurement using deep learningalgorithm based on multi-source features fusion to predict PD-L1 expressionand survival in NSCLC. Front. Immunol. 13, 828560 (2022). 9. Zhang, K. et al. Clinically applicable AI system for accuratediagnosis, quantitative measurements, and prognosis of COVID-19 pneumoniausing computed tomography. Cell 181, 1423-1433.e1411 (2020). 10. Kermany, D. S. et al. Identifying Medical Diagnoses and TreatableDiseases by Image-Based Deep Learning. Cell 172, 1122-1131.e1129, doi:10.1016 / j.cell.2018.02.010 (2018). 11. Rajpurkar, P., Chen, E., Banerjee, O.&Topol, E. J. AI in healthand medicine. Nat. Med. 28, 31-38 (2022). 12. LeCun, Y., Bengio, Y.&Hinton, G. Deep learning. Nature 521, 436-444 (2015). 13. Schmidhuber, J. Deep learning in neural networks: an overview.Neural Netw. 61, 85-1173 (2015). 14. Wang, G. et al. A deep-learning pipeline for the diagnosis anddiscrimination of viral, non-viral and COVID-19 pneumonia from chest X-rayimages. Nat. Biomed. Eng. 5, 509-521 (2021). 15. Zhou, H.Y. et al. Generalized radiograph representation learningvia cross-supervision between images and free-text radiology reports. Nat.Mach. Intell. 4, 32-40 (2022). 16. Tang, Y. X. et al. Automated abnormality classification of chestradiographs using deep convolutional neural networks. npj Digit. Med. 3, 70(2020). 17. Wang, C. et al. Development and validation of an abnormality-derived deep-learning diagnostic system for major respiratory diseases. npjDigit. Med. 5, 124 (2022). 18. Rajpurkar, P. et al. ChexNet: Radiologist-level pneumoniadetection on chest x-rays with deep learning. arXiv preprint arXiv:1711.05225(2017). 19. Mei, X. et al. Artificial intelligence-enabled rapid diagnosis ofpatients with COVID-19. Nat. Med. 26, 1224-1228 (2020). 20. Yala, A., Lehman, C., Schuster, T., Portnoi, T.&Barzilay, R. Adeep learning mammography-based model for improved breast cancer riskprediction. Radiology 292, 60-66 (2019). 21. Zhang, K. et al. Deep-learning models for the detection andincidence prediction of chronic kidney disease and type 2 diabetes fromretinal fundus images. Nat. Biomed. Eng. 5, 533-545 (2021). 22. Xu, Q. et al. AI-based analysis of CT images for rapid triage ofCOVID-19 patients. npj Digit. Med. 4, 75 (2021). 23. Akselrod-Ballin, A. et al. Predicting breast cancer by applyingdeep learning to linked health records and mammograms. Radiology 292, 331-342(2019). 24. Vaswani, A. et al. Attention is all you need. Adv. Neural. Inf.Process. Syst. 30 (2017). 25. Devlin, J., Chang, M.-W., Lee, K.&Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. arXivpreprint arXiv:1810.04805 (2018). 26. Dosovitskiy, A. et al. An image is worth 16x16 words:Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929(2020). 27. LeCun, Y. et al. Handwritten digit recognition with a back-propagation network. Adv. Neural. Inf. Process. Syst. 2 (1989). 28. Mikolov, T., Chen, K., Corrado, G.&Dean, J. Efficient estimationof word representations in vector space. arXiv preprint arXiv:1301.3781(2013). 29. Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S.&Dean, J.Distributed representations of words and phrases and their compositionality.Adv. Neural. Inf. Process. Syst. 26 (2013). 30. Jaegle, A. et al. Perceiver: General perception with iterativeattention. In Proc. 38th International Conference on Machine Learning 4651-4663 (2021). 31. Li, J. et al. Align before fuse: Vision and languagerepresentation learning with momentum distillation. Adv. Neural. Inf.Process. Syst. 34, 9694-9705 (2021). 32. Su, W. et al. Vl-bert: Pre-training of generic visual-linguisticrepresentations. arXiv preprint arXiv:1908.08530 (2019). 33. Wang, J. et al. GIT: A generative image-to-text transformer forvision and language. arXiv preprint arXiv:2205.14100 (2022). 34. Pauwels, R. A., Buist, A. S., Calverley, P. M., Jenkins, C. R.&Hurd, S. S. Global strategy for the diagnosis, management, and prevention ofchronic obstructive pulmonary disease. NHLBI / WHO Global Initiative forChronic Obstructive Lung Disease (GOLD) Workshop summary. Am. J. Respir.Crit. Care Med. 163, 1256-1276 (2001). 35. Li, Y. et al. BEHRT: Transformer for Electronic Health Records.Sci Rep 10, 7155, doi:10.1038 / s41598-020-62922-y (2020). 36. Xia, K., Wang, J. Recent advances of transformers in medicalimage analysis: a comprehensive review. MedComm – Future Med. 2023; 2:e38(2023) 37. Wang, D., Feng, L., Ye, J., Zou, J., Zheng, Y. Accelerating theintegration of ChatGPT and other large-scale AI models into biomedicalresearch and healthcare. MedComm-Future Med. 2023; 2:e43. 38. Moor, M. et al. Foundation models for generalist medicalartificial intelligence. Nature 616, 259-265 (2023) 39. Ba, J. L., Kiros, J. R.&Hinton, G. E. Layer normalization. arXivpreprint arXiv:1607.06450 (2016). 40. Hendrycks, D.&Gimpel, K. Gaussian error linear units (gelus).arXiv preprint arXiv:1606.08415 (2016). 41. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I.&Salakhutdinov, R. Dropout: a simple way to prevent neural networks fromoverfitting. J. Mach. Learn. Res. 15, 1929-1958 (2014). 42. Johnson, A. E. et al. MIMIC-CXR, a de-identified publiclyavailable database of chest radiographs with free-text reports. Scientificdata 6, 1-8 (2019). 43. Jaegle, A. et al. Perceiver IO: A general architecture forstructured inputs&outputs. arXiv preprint arXiv:2107.14795 (2021). 44. Deng, J. et al. ImageNet: A large-scale hierarchical imagedatabase. In IEEE Conference on Computer Vision and Pattern Recognition 248-255 (2009). 45. He, K., Zhang, X., Ren, S.&Sun, J. Deep residual learning forimage recognition. In IEEE Conference on Computer Vision and PatternRecognition 770-778 (2016). 46. Ni, Q. et al. A deep learning approach to characterize 2019coronavirus disease (COVID-19) pneumonia in chest CT images. Eur. Radiol. 30,6517-6527 (2020). 47. Li, Z. et al. MVP-Net: multi-view FPN with position-awareattention for deep universal lesion detection. In International Conference onMedical Image Computing and Computer-Assisted Intervention 13-21 (2019). 48. Zhao, G. et al. Diagnose Like a Radiologist: Hybrid Neuro-Probabilistic Reasoning for Attribute-Based Medical Image Diagnosis. In IEEETrans. Pattern Anal. Mach. Intell. 44, 7400-7416 (2021). 49. Loshchilov, I.&Hutter, F. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017). 50. Paszke, A. et al. Pytorch: An imperative style, high-performance deep learning library. Adv. Neural. Inf. Process. Syst. 32 (2019). 51. Micikevicius, P. et al. Mixed precision training. arXiv preprintarXiv:1710.03740 (2017). 52. Selvaraju, RR et al. Grad-cam: Visual explanations from deepnetworks via gradient-based localization. In IEEE International Conference onComputer Vision 618-626 (2017). Table 1 | Comparison with baseline models in the lung disease identification task.

[0071] Baseline models include image-only models, early fusion methods, late fusion methods, and two recent Transformer-based multimodal classification models (GIT and Perceiver). 95% CI represents the 95% confidence interval. The evaluation metric is AUROC.

[0072]

[0073] Table 2 | Comparison with baseline models in a task predicting adverse clinical outcomes in COVID-19 patients. Five models were included in this comparison: image-only models, early fusion methods, late fusion methods, and two recent Transformer-based multimodal classification models (GIT and Perceiver). 95% CI represents a 95% confidence interval. The evaluation metric was AUPRC.

[0074]

[0075] Table 3 | Ablation studies of IRENE by removing or replacing individual components. HA(N) indicates that there are N bidirectional multimodal attention blocks in the Multimodal Diagnostic Transformer (MDT), while the remaining blocks are self-attention blocks (a total of twelve blocks). Image represents the input ray photograph. Unidirectional means that only text-to-image attention is calculated in the multimodal attention block. ChiComp represents the complaint. LabTest represents the laboratory test results. Lexicalization represents the lexicalization procedure used for the complaint and laboratory test results. The evaluation metric is AUROC.

[0076]

Claims

1. A method for providing a medical diagnosis to a patient, comprising: Receive one or more images of the patient and a text dataset associated with the patient; Multiple lexical units are generated in the following way: Convert the one or more images into one or more visual lexical units; as well as Convert the text dataset into one or more text-based lexical units; A medical diagnosis of the patient is obtained by inputting the multiple lexical units into a trained machine learning model, the machine learning model comprising multiple bidirectional blocks with intra-modal and inter-modal attention; Provide the medical diagnosis to the patient.

2. The method according to claim 1, wherein, The medical diagnosis includes identified diseases, predictions of adverse clinical outcomes, or combinations thereof.

3. The method according to any one of the preceding claims, wherein, The text dataset contains: narrative text, one or more text field data, or a combination thereof.

4. The method according to any one of the preceding claims, wherein, The trained machine learning model further includes one or more self-attention blocks.

5. The method according to any one of the preceding claims, wherein, The trained machine learning model further includes a classification head.

Images