System and method for automated identification of medical conditions from medical images and training of a model for identification of medical conditions from medical images

Abstract

Provided is a system and method for training a machine learning model to identify a medical condition in a medical image, and for using the trained model to identify the medical condition from the medical image. The method including: receiving a training dataset including a plurality of training medical images, the training medical images including annotations that identify the medical condition, the annotations including exam-level labels on a majority of the training medical images and the annotations including slice-level labels or region-level labels on a minority of the training medical images; training a semi-weakly supervised machine learning model using the annotated training dataset to identify one or more medical conditions from an input inference medical image.

Description

SYSTEM AND METHOD FOR AUTOMATED IDENTIFICATION OF MEDICAL CONDITIONS FROM MEDICAL IMAGES AND TRAINING OF A MODEL FOR IDENTIFICATION OF MEDICAL CONDITIONS FROM MEDICAL IMAGESTECHNICAL FIELD

[0001] The present invention relates to tools for labeling and analysis of medical imaging; and more particularly, to a system and method for automated identification of medical conditions from medical images and training of a model for identification of medical conditions from medical images.BACKGROUND

[0002] Machine learning (ML) has the potential to transform health care and medical imaging. Potential benefits include improved physician accuracy, prioritization of examinations with critical findings, alleviation of radiologist shortages, radiation dose reduction, and improved image quality. The training of medical imaging ML models has generally involved the annotation of large, curated datasets, which is often a very resource-intensive exercise. The annotation of such a large number of images can be a time-consuming and monotonous task, particularly for granular labels such as segmentation or bounding boxes. In addition, the recruitment of highly skilled expert radiologists as annotators can impose high financial costs as these professionals are in high demand and their time is valuable. The time-consuming and resource-intensive nature of medical dataset annotation limits the scalability of a manual approach. These concerns are compounded by concerns over label accuracy, particularly in complex imaging studies. Employing multiple independent annotations can be used to address these concerns, but challenges with respect to interrater reliability are ever-present.

[0003] It is therefore an object of the present invention to provide a system and method in which the above disadvantages are obviated or mitigated, and attainment of various desirable attributes is facilitated.SUMMARY

[0004] In an aspect, there is provided a computer-implemented method for generating a machine learning model to identify a medical condition in a medical image, the methodcomprising: receiving a training dataset comprising a plurality of training medical images, the training medical images comprising annotations that identify the medical condition, the annotations comprising exam-level labels on a majority of the training medical images and the annotations comprising slice-level labels or region-level labels, or both, on a minority of the training medical images; training a semi-weakly supervised machine learning model using the annotated training dataset to identify one or more medical conditions from an input inference medical image; and outputting the semi- weakly supervised machine learning model for use in identifying the one or more medical conditions from the input inference medical image.

[0005] In a particular case of the method, the medical condition is pulmonary embolism.

[0006] In another case of the method, the method further comprising performing preprocessing on the training medical images, the preprocessing comprising one or more of converting the training medical images to a standardized format, applying windowing to optimize contrast, and normalizing the image intensity ranges

[0007] In yet another case of the method, exam-level labels comprise an indication of a presence or an absence of a condition across an entire scan without specifying the location of the condition, wherein slice-level labels comprise identification of which specific slices contain the condition without specifying the location of the condition within the specific slices, and wherein region-level labels comprise identification of the condition at specific areas or organs within a slice.

[0008] In yet another case of the method, the slice-level labels or region-level labels, or both, are included on between 2.5% and 27.5% of the training images.

[0009] In yet another case of the method, the semi-weakly supervised machine learning model comprises an attention-based convolutional neural network (CNN)- recurrent neural network (RNN) model.

[0010] In yet another case of the method, the attention-based CNN-RNN model comprises feature extraction using a model that combines convolutional networks and transformers, or comprises batch normalization to capture z-axis temporal relationships within sequences of medical images, or both.

[0011] In another aspect, there is provided a system for generating a machine learning model to identify a medical condition in a medical image, the system comprising one or more processors and a data memory, the data memory comprising instructions for the one or more processors to execute: an input module to receive a training dataset comprising a plurality of training medical images, the training medical images comprising annotations that identify the medical condition, the annotations comprising exam-level labels on a majority of the training medical images and the annotations comprising slice-level labels or region-level labels, or both, on a minority of the training medical images; a training module to train a semi-weakly supervised machine learning model using the annotated training dataset to identify one or more medical conditions from an input inference medical image; and an output module to output the semi-weakly supervised machine learning model for use in identifying the one or more medical conditions from the input inference medical image.

[0012] In a particular case of the system, the medical condition is pulmonary embolism, the one or more processors further executing a preprocessing module to perform preprocessing on the training medical images, the preprocessing comprising one or more of converting the training medical images to a standardized format, applying windowing to optimize contrast, and normalizing the image intensity ranges.

[0013] In another case of the system, exam-level labels comprise an indication of a presence or an absence of a condition across an entire scan without specifying the location of the condition, wherein slice-level labels comprise identification of which specific slices contain the condition without specifying the location of the condition within the specific slices, and wherein region-level labels comprise identification of the condition at specific areas or organs within a slice.

[0014] In yet another case of the system, the slice-level labels or region-level labels, or both, are included on between 2.5% and 27.5% of the training images.

[0015] In yet another case of the system, the semi-weakly supervised machine learning model comprises an attention-based convolutional neural network (CNN)- recurrent neural network (RNN) model.

[0016] In yet another case of the system, the attention-based CNN-RNN model comprises feature extraction using a model that combines convolutional networks and transformers, or comprises batch normalization to capture z-axis temporal relationships within sequences of medical images, or both.

[0017] In another aspect, there is provided a computer-implemented method for identifying a medical condition in an inference medical image using a trained semi- weakly supervised machine learning model, the method comprising: receiving the inference medical image; identifying the medical condition in the received inference medical image using the trained semi-weakly supervised machine learning model, the semi-weakly supervised machine learning model trained using a training dataset comprising a plurality of training medical images, the training medical images comprising annotations that identify the medical condition, the annotations comprising exam-level labels on a majority of the training medical images and the annotations comprising slice-level labels or region-level labels, or both, on a minority of the training medical images; and outputting the identification of the medical condition.

[0018] In a particular case of the method, the method further comprising generating and outputting a visualization of the identified medical condition using Gradient-weighted Class Activation Mapping to highlight regions of the inference medical image that contributed most to the identification of the medical condition.

[0019] In another case of the method, the medical condition is pulmonary embolism.

[0020] In yet another case of the method, exam-level labels comprise an indication of a presence or an absence of a condition across an entire scan without specifying the location of the condition, wherein slice-level labels comprise identification of which specific slices contain the condition without specifying the location of the condition within the specific slices, and wherein region-level labels comprise identification of the condition at specific areas or organs within a slice.

[0021] In yet another case of the method, the slice-level labels or region-level labels, or both, are included on between 2.5% and 27.5% of the training images.

[0022] In yet another case of the method, the semi-weakly supervised machine learning model comprises an attention-based convolutional neural network (CNN)- recurrent neural network (RNN) model.

[0023] In yet another case of the method, the attention-based CNN-RNN model comprises feature extraction using a model that combines convolutional networks and transformers, or comprises batch normalization to capture z-axis temporal relationships within sequences of medical images, or both.

[0024] These and other aspects are contemplated and described herein. The foregoing summary sets out representative aspects of systems and methods to assist skilled readers in understanding the following detailed description.DESCRIPTION OF THE DRAWINGS

[0025] An embodiment of the present invention will now be described by way of example only with reference to the accompanying drawings, in which:

[0026] FIG. 1 is a block diagram showing a system for automated identification of medical conditions from medical images, according to an embodiment;

[0027] FIG. 2 is a flow chart showing a method for training a machine learning model for automated identification of medical conditions from medical images;

[0028] FIG. 3 is a flow chart showing a method for automated identification of medical conditions from medical images;

[0029] FIG. 4 is a diagram illustrating an example approach for data cleaning and pipeline splitting in accordance with the system of FIG. 1 ;

[0030] FIG. 5 is a diagram showing an end-to-end training pipeline of a model used in example experiments;

[0031] FIG. 6 is a chart illustrating the impact of label granularity on model performance for results of a test set in the example experiments;

[0032] FIG. 7 is a chart illustrating the impact of label granularity on model performance for results of an external validation dataset in the example experiments;

[0033] FIG. 8 is a chart illustrating ROC curves for PE detection on the training dataset in the example experiments;

[0034] FIG. 9 is a chart illustrating ROC curves for PE detection on the pooled external validation dataset in the example experiments;

[0035] FIG. 10 shows images of example cases highlighting a semi-weak learner model accurately detecting pulmonary emboli in both peripheral and central locations, in accordance with the system of FIG. 1 ;

[0036] FIG. 11 A is a diagram illustrating a slice-level model architecture for PE diagnosis, in accordance with the example experiments;

[0037] FIG. 11 B is a diagram illustrating an exam-level model architecture for PE diagnosis, in accordance with the example experiments;

[0038] FIG. 11 C is a diagram illustrating a hybrid model architecture for PE diagnosis, in accordance with the example experiments; and

[0039] FIGS. 12A and 12B are graphs showing experimental impact of label granularity on model performance as a function of AUC.DETAILED DESCRIPTION

[0040] Embodiments will now be described with reference to the figures. For simplicity and clarity of illustration, where considered appropriate, reference numerals may be repeated among the figures to indicate corresponding or analogous elements. In addition, numerous specific details are set forth in order to provide a thorough understanding of the embodiments described herein. However, it will be understood by those of ordinary skill in the art that the embodiments described herein may be practiced without these specific details. In other instances, well-known methods, procedures and components have not been described in detail so as not to obscure the embodiments described herein. Also, the description is not to be considered as limiting the scope of the embodiments described herein.

[0041] Any module, unit, component, server, computer, computing device, mechanism, terminal or other device exemplified herein that executes instructions may include or otherwise have access to computer readable media such as storage media, computerstorage media, or data storage devices (removable and / or non-removable) such as, for example, magnetic disks, optical disks, or tape. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Examples of computer storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by an application, module, or both. Any such computer storage media may be part of the device or accessible or connectable thereto. Any application or module herein described may be implemented using computer readable / executable instructions that may be stored or otherwise held by such computer readable media and executed by the one or more processors.

[0042] Embodiments of the present disclosure provide for the detection of medical conditions from medical images. In a particular case, detection of pulmonary embolism (PE) on computed tomography (CT) pulmonary angiography (CTPA). PEs are blood clots in the pulmonary arterial circulation and a potentially life-threatening condition. PE can vary dramatically in its presentation from large emboli occupying the central pulmonary arteries, to a small subsegmental embolus in the lung periphery. Larger PEs may span dozens of images while smaller PEs may only occupy a small number of voxels within a few images. Detecting smaller PEs is a substantial challenge due to the larger search space of the entire thorax covered by CTPA. Therefore, relying solely on exam-level labels may be insufficient. In order to overcome such challenges, embodiments of the present disclosure automate such detection using more granular annotations.

[0043] While the present disclosure is primarily directed to determining PE, the semi- weakly supervised approach can be used for detecting any suitable condition. The present embodiments are particularly suitable for imaging tasks where (1 ) the disease or condition manifests across multiple slices, requiring localization for accurate detection (e.g., PE, ICH, liver tumors, abdominal injuries); (2) detailed labeling is costlyand time-consuming but essential for high performance (e.g., mammography, cancer detection in CT / MRI); and (3) the dataset is large, but only a small proportion of detailed annotations can be feasibly obtained.

[0044] As illustrated in the example experiments described herein, semi-weakly supervised learning is particularly useful in the medical imaging space. Researchers in medical imaging have traditionally relied on strongly supervised learning, where fully annotated datasets are used to capture the intricate details of medical images, ensuring precise localization and high diagnostic accuracy. This approach has been traditionally used in the field of medical imaging because of the belief that detailed, fine-grained annotations are essential for training effective models. In this way, semi-weakly supervised learning, which uses a mix of detailed and coarse-grained labels, would not have tradionally been deemed suitable for medical imaging because researchers would have assumed that such approaches would not perform as well as fully annotated models due to not fully leveraging all available detailed information.

[0045] The present inventors determined that the wide variability in PE presentation allows for advantageous label granularity in the development of a machine learning (ML) model used to determine PE. In an example, a training dataset containing pulmonary embolism CT annotations, with slice and exam-level labels, can be used to train such a model. Other approaches for determining PE generally primarily rely on detailed annotations, like pixel or slice-level labels. Other approaches have explored full slice and exam-level labels. These approaches collectively underscore the prevailing assumption that granular annotations are needed for accurate PE detection.

[0046] In contrast, the present inventors challenged the above assumption and determined that semi-weakly supervised learning, which combines aspects of both weakly supervised and strongly supervised learning, can be used to make accurate determinations of PE. The present embodiments advantageously utilize exam-level labels and supplemented by a limited portion of slice-level labels, instead of extensive slice-level or pixel-level annotations. In example experiments conducted by the present inventors, the present embodiments were able to achieve high performance in PE diagnosis using models trained with a smaller number of slice level annotations thanother approaches. By reducing the need for meticulously hand-labeled data, the present embodiments streamline the training and development of ML models for PE detection, as well as for other medical imaging tasks. Such advancements produce significant cost savings and faster deployment of the models in clinical settings, which ultimately benefits patient care.

[0047] Accordingly, the present embodiments are able to significantly reduce annotation effort, achieving high performance with fewer detailed labels than other approaches. The strategic integration of primarily exam-level labels, along with a limited subset of slice-level labels, contributes to these significant advantages. Additionally, efficient data preprocessing techniques, in combination with the high efficacy of semi-weakly supervised learning for medical detection, ensures that the present embodiments are cost-effective and efficient. In this way, challenging traditional assumptions that extensive fine-grained labeling is necessary for high-performance medical imaging models.

[0048] FIG. 1 illustrates a schematic diagram of a system 100 for automated identification of medical conditions from medical images, according to various embodiments. As shown, the system 100 has a number of physical and logical components, including a processing unit (“PU”) 160, data memory 164, an interface module 168, a network module 176, non-volatile storage 180, and a local bus 184 enabling PU 160 to communicate with the other components. PU 160 can include one or more processors, a microprocessor, dedicated hardware, or other integrated processing circuits. Data memory 264 provides relatively responsive volatile storage to PU 260. The data memory 264 can store an operating system and programs, including computer-executable instructions for implementing the methods described herein, as well as any derivative or related data The interface module 268 enables input to be provided; for example, directly via a user input device, or indirectly, for example, via an external device. The interface module 168 also enables output to be provided; for example, directly via a user display, or indirectly, for example, sent over the network module 176. The network module 176 permits communication with other systems or computing devices; for example, over a local area network or over the Internet. In some cases, this data can be stored in one or more databases 188 that may be local orremote. In some cases, during operation of the system 100, the operating system, the programs and the data may be retrieved from the data memory 164 and or from the database 188 to facilitate execution. In other embodiments, any operating system, programs, or instructions can be executed in hardware, specialized microprocessors, logic arrays, or the like. In some cases, the processing can be distributed over multiple computing devices.

[0049] In an embodiment, the Pll 160 can be configured to execute a number of conceptual functional modules, for example, an input module 102, a preprocessing module 104, a training module 106, an output module 108, and a classification module 110. In further cases, functions of the above modules can be combined or executed on other modules. In some cases, functions of the above modules can be executed on remote computing devices, such as centralized servers and cloud computing resources communicating over the network module 176.

[0050] Turning to FIG. 2, a flowchart 200 for a method for training a machine learning model for automated identification of medical conditions from medical images, according to an embodiment, is shown.

[0051] At block 202, the input module 102 receives a training dataset that includes medical images that are annotated to identify the medical condition to be identified. For example, in the case of pulmonary embolism detection, the training dataset can include medical images that include a number of computed tomography pulmonary angiogram (CTPA) annotations. In some cases, the input module 102 can also receive external validation datasets. At least a majority of the training images, and in some cases all of the training images, include exam-level labels. A minority of the training images include more granular labels, such as slice-level labels or region-level labels, or both. These annotated training images are used by the training module 106 to train a semi-weakly supervised model. In various cases, the more granular labels can be included on less than 50% of the training images, less than 30% of the training images, less than 20% of the training images, less than 10% of the training images, less than 5% of the training images, or less than 3% of the training images. In a particular case, the more granular labels can be included on between 2.5% and 27.5% of the training images.

[0052] For the case of a computed tomography (CT) exam, such exams typically comprise hundreds of slices, each containing millions of pixels, and can be labeled at various levels of detail. Exam-level labels, as used in the present disclosure, refer to an indication of the presence or absence of a condition across the entire CT scan, without specifying its location. Slice-level labels, as used in the present disclosure, refer to labelling with more granularity by identifying which specific slices contain the condition; but do not provide for its exact location within those slices. Region-level labels, as used in the present disclosure, refer to labelling with markings of specific areas or organs within a slice, though they may still lack precise boundaries. Pixel-level or segmentation labels, as used in the present disclosure, provide the highest granularity and can outline the exact contours of the condition within the scan.

[0053] Exam-level labels, as provided in the input dataset of block 202, refer to annotations that provide a high-level summary of the findings within an entire medical imaging exam, rather than detailed annotations on individual images or slices within the exam. These labels can be binary or categorical indicators that describe the presence or absence of a condition across the entire exam. These labels do not focus on specific slices or regions within the scan but rather offer an overall assessment of the patient's condition as determined by the imaging study. Exam-level labels can include binaryindicators that indicate whether a particular condition or abnormality is present or absent across the entire imaging exam. For example, in the context of pulmonary embolism (PE) detection, an exam-level label can indicate whether the patient has PE (positive) or does not have PE (negative) based on the full CT pulmonary angiography (CTPA) scan. In some cases, the exam-level labels can include categorical information about the patient’s condition. For example, in an exam for abdominal injuries, the categorical information exam-level label can specify whether the injury is minor, moderate, or severe, reflecting the overall assessment derived from the imaging data. Thus, examlevel labels do not specify the exact location or extent of the condition within the imaging exam and do not provide detailed annotations; such as bounding boxes, pixel-level segmentation, or slice-level labels that indicate precisely where the condition is present.

[0054] At block 204, in some cases, the preprocessing module 104 can perform data cleaning and other preprocessing operations on the data in the received datasets. Forexample, conversion of the data to the Neuroimaging Informatics Technology Initiative (NlfTI) format. In some cases, the voxels of interest (VOI) can be segmented (e.g., with TotalSegmentator). In some cases, images with incoherent slice-level and exam-level labels, or those labeled as indeterminate due to impaired image quality, can be removed by the preprocessing module 104.

[0055] At block 206, in some cases, the preprocessing module 104 can perform data augmentation to prevent overfitting. In an example, one or more of the following can be applied:• random rotation (e.g., 0 to 10 degrees);• scaling and translation (e.g., up to 10%);• modifications to image brightness and contrast;• incorporation of random horizontal flips, motion blur, median blur, Gaussian blur, and Gaussian noise (e.g., variance of 0.004);• incorporation of random cutouts;• incorporation of random optical or grid distortions; and / or• combination of adjacent axial slices into a single 3-channel image.

[0056] At block 208, the training module 106 trains the semi-weakly supervised model using the preprocessed and augmented training dataset. In a particular case, the semi- weakly supervised model is an attention-based convolutional neural network (CNN)- recurrent neural network (RNN) model; however, any suitable model architecture can be used.

[0057] In the case of the attention-based CNN-RNN model, the semi-weakly supervised model can perform feature extraction using, for example, a CoAtNet model that combines convolutional networks and transformers. In some cases, the semi-weakly supervised model can perform batch normalization, which can include an attention layer, and / or can include bidirectional LSTM layers (e.g., three layers) to capture z-axis temporal relationships within sequences of medical images. The semi-weakly supervised model can use a fully connected layer to output a probability score (0 to 1 ),such as after sigmoid normalization. In some cases, a mask can be used in the semi- weakly supervised model to control the number of slice-level labels used; whereby the number and location of the masks in the pipeline can be determined to prevent label leakage.

[0058] Generally, semi-weakly supervised models are a hybrid approach between weakly supervised and strongly supervised learning. Weakly supervised models generally have multi-instance learning, where coarse-grained labels (e.g., exam-level or image-level annotations) are used to train the model. These labels provide general information about the presence or absence of a condition without specifying its exact location within the data. The model must infer which instances (e.g., slices in a CT scan) are relevant, making it suitable for scenarios where detailed labeling is minimal or unavailable. Strongly supervised models generally correspond to single-instance learning, where each instance (e.g., a specific image slice or pixel) is individually labeled with detailed annotations like bounding boxes or pixel-level segmentations. This allows the model to learn from precise localization, offering highly accurate predictions but requiring extensive detailed labeling. Semi-weakly supervised models, as used in block 208, are a blend of single-instance and multi-instance learning, where as little as possible fine-grained information is used while still achieving substantially valuable performance that would typically be achieved in strongly supervised models. By incorporating a small subset of fine-grained labels of detailed annotations (e.g., bounding boxes or segmentations) with a majority of coarse-grained labels (e.g., examlevel), the system 50 balances the benefits of detailed annotations with the efficiency of weaker supervision. Advantageously permitting significant reduction in the annotation burden while maintaining high performance.

[0059] At block 210, the output module 108 outputs the trained semi-weakly supervised model to the interface module 168, the network module 176, and / or the database 188 for use in making determinations of medical conditions.

[0060] Turning to FIG. 3, a flowchart 300 for a method for automated identification of medical conditions from medical images, according to an embodiment, is shown.

[0061] At block 302, the input module 102 receives the medical images to be classified as whether or not such images comprise the medical condition. In a particular case, the medical image can include CTPA images captured of a particular patient to determine whether the patient has a pulmonary embolism.

[0062] At block 304, the classification module 110 uses the trained semi-weakly supervised model trained by the training module 106 to perform a determination of whether the medical condition is present in the input medical images.

[0063] At block 306, the output module 108 outputs the determination to the interface module 168, the network module 176, and / or the database 188.

[0064] By utilizing less granular annotations, the system 100 can achieve substantially high performance for determining whether the medical condition is present with fewer detailed labels; thereby reducing the typically resource-intensive nature of medical image annotation.

[0065] In order to verify the substantial advantages of the system 100, the present inventors performed example experiments to detect pulmonary embolism (PE) on CTPA using a pulmonary embolism CT training dataset, followed by external validation using a pooled dataset. The example experiments explored the effect of varying slice-level label proportions from 0% to 100% in the training dataset with three model configurations: a weak learner using only examination-level labels, strong learner with all examination and slice-level labels, and semi-weak learner (per method 200 and 300) combining all examination-level labels with a subset of slice-level labels. Performance metrics included AUC, accuracy, sensitivity, specificity, PPV, and NPV.

[0066] The example experiments utilized a training dataset that included thousands of CTPA exams with image level labels and examination level labels. The example experiments also utilized two other datasets for external validation that totalled 65 exams with 38 positive examinations.

[0067] For preprocessing, the example experiments used extensive data cleaning. For example, conversion to Neuroimaging Informatics Technology Initiative (NlfTI) format. The voxels of interest (VOI) were segmented with TotalSegmentator, followed bywindow settings (width = 700, level = 100) for optimal contrast and normalization of image intensity ranges. Exams with incoherent slice-level and exam-level labels, or those labeled as indeterminate due to impaired image quality, were removed. The pooled external validation dataset was unchanged.

[0068] FIG. 4 is a diagram illustrating an example approach for data cleaning and pipeline splitting, as used in the example experiments. The diagram illustrates an example of preprocessing and splitting of the datasets into various sets for internal training, validation, and testing. The diagram shows the exclusion criteria (label mismatches, processing failures, misaligned orientations), the training dataset, and the test datasets.

[0069] Data augmentation was utilized in the example experiments to prevent overfitting. A suite of augmentations were used to enhance dataset diversity, including random rotation (0 to 10 degrees), scaling and translation (up to 10%), and modifications to image brightness and contrast. Random horizontal flips, motion blur, median blur, Gaussian blur, and Gaussian noise (variance of 0.004) were also used. Additionally, random cutouts were used and optical or grid distortions were applied. Adjacent axial slices were combined into a single 3-channel image for the models.

[0070] FIG. 5 is a diagram showing an end-to-end training pipeline of the model for the example experiments. The diagram illustrates the training pipeline for PE diagnosis using CoAtNet-0 as the feature extractor. The pipeline supports label granularity impact analysis by allowing slice-level classifier predictions to be masked. Strongly supervised learning (strong learner) uses all slice-level classifiers, while weakly supervised learning (weak learner) uses none. This pipeline enables experiments to assess the impact of varying amounts (n) of slice-level labels (0, 2.5, 5, 10, 20, 27.5, 35, 42.5, 50, 75, 100%) on model performance.

[0071] The end-to-end training pipeline shown in FIG. 5 was used in the example experiments to develop three learners: weakly, strongly, and semi-weakly supervised. To implement these models, transfer learning with a CoAtNetO model was used, pretrained on ImageNet. The CoAtNet model, which combines convolutional networks and transformers, served as the feature extractor. A batch normalization was applied,an attention layer was used, and three bidirectional LSTM layers were used to capture z-axis temporal relationships within CT scan sequences. Finally, a fully connected layer was used to output a probability score (0 to 1 ) after sigmoid normalization. The mask in in the pipeline controlled the number of slice-level labels used; setting it to 184 made it a strong learner, while setting it to 0 made it a weak learner. The number and location of masks were determined before each experiment to prevent label leakage.

[0072] The filtered training dataset was used for model training. The test models were trained on CT images resized to (184, 256, 256) using an Adam optimizer with an initial learning rate of 1e-4 and a batch size of 16. The binary cross-entropy loss function was used to evaluate the models’ performance. Each training process spanned 30 epochs, incorporating early stopping to prevent overfitting and a Cosine Annealing learning rate scheduler to dynamically adjust the learning rate (Tmax = 30, min_lr = 1e-6). All models were developed and trained using the same software and hardware.

[0073] FIGS. 11 A to 11 C are diagrams illustrating model architectures for the three different models for PE diagnosis. FIG. 11 A is a diagram showing a slice-level model. In such case, it is trained on slice input with CoAtNet-0 extracting slice-level embeddings, followed by GELLI and a fully connected (FC) layer for slice-level output. FIG. 11 B is a diagram showing an exam-level model. In this case, the model is trained on exam input with CoAtNet-0 extracting slice-level embeddings, processed by GELLI, Batch Normalization (BN), Self-Attention, Layer Normalization (LN), and bidirectional LSTM (Bi-LSTM) layers, concluding with an FC layer for exam-level output. FIG. 11 C is a diagram showing a hybrid model. This model combines slice-level and exam-level training, starting with slice input processed by CoAtNet-0, followed by parallel paths for slice-level and exam-level embeddings, using layers similar to the exam-level model for final outputs at both levels. A common approach for PE diagnosis involves using the slice-level model and then aggregating with LSTM for exam-level prediction. The hybrid model implemented by the system 100 improves training efficiency and ensures local slice-level information is retained. Example experiments determined that when sufficient slice-level information is available (e.g., n > 30), the model can perform substantially well by, for example, weighting the slice output loss at 0.95 and the exam output loss at 0.05.

[0074] In the example experiments, slice-level embeddings refer to training data of individual CT slices with specific slice annotations that can be used in a strongly supervised learner that uses detailed, fine-grained information. Exam-level embeddings refer to training data comprising entire CT scans with exam-level annotations that can be used in a weakly supervised learner that relies on broader, less detailed labels. A hybrid model is trained on whole CT scans using a mix of slice-level and exam-level annotations. This combination of elements of both strongly and weakly supervised learning provides a semi-weakly supervised learner that leverages information at multiple levels of granularity.

[0075] To analyze the impact of label granularity, the example experiments tested different percentages of slice-level labels used for model training. The weakly supervised model used only exam-level labels. The strongly supervised model used all exam and slice-level labels. Semi-weakly supervised models were trained with all examlevel labels and 0, 2.5, 5, 10, 20, 27.5, 35, 42.5, 50, 75, 100% of available slice-level labels. Five-fold cross-validation (CV) was used for training.

[0076] Youden's J index was used to identify an optimal threshold on the receiver operator curve (ROC) by maximizing a difference between true and false positive rates. This threshold was applied to the evaluation using a public and private test set. To assess generalizability, external validation was performed on pooled Aida and FUMPE datasets.

[0077] Performance of the models were primarily assessed based on average predictions across the five cross-validation (CV) models using area under the receiver operating characteristic curve (AUC), accuracy, sensitivity, specificity, positive predictive values (PPV), and negative predictive values (NPV). Model performance was also compared by conducting pairwise AUC comparisons using the DeLong test. Confidence intervals were calculated, employing the binomial method for accuracy, sensitivity, specificity, PPV, and NPV, and the fast DeLong method for AUC.

[0078] In the example experiments, model performance showed a significant initial improvement with just 2.5% of slice-level labels with the AUC increasing from 0.682 (0.652, 0.711 ) to 0.858 (0.836, 0.881 ) on the test set. Performance continued toimprove with increasing label availability. However, beyond 27.5% label availability, the gains in performance became less substantial showing marginal improvement. For example, the AUC was 0.928 (0.910, 0.945) with 27.5% of slice-level labels compared to 0.932 (0.915, 0.948) when using all slice-level labels (p=0.187). This trend was similarly reflected in accuracy metrics and the F1 score. Evaluation on the external dataset mirrored these findings, displaying the same trend in AUC, accuracy, and F1 score.

[0079] FIGS. 6 and 7 are charts illustrating the impact of label granularity on model performance. The charts illustrate the performance of the models in terms of AUC and F1 score across different datasets as a function of the percentage of slice-level labels used; FIG. 6 showing the results of the test set and FIG. 7 showing the results of the external validation dataset. The solid lines represent the performance of average predictions across 5-fold cross-validation and the shaded areas correspond to the 95% confidence intervals (Cl).TABLE 1TABLE 2

[0080] Table 1 shows the performance in detecting PE on the test set and Table 2 shows the performance in detecting PE on the external validation set. The acronyms used are: TP - true positive, FN - false negative, TN - true negative, FP - false positive, AUC - area under the receiver operating curve, Acc - accuracy, SEN -sensitivity, SPEC - specificity, PPV - positive predictive value, and NPV - negative predictive value.

[0081] FIGS. 8 and 9 show ROC curves for PE detection on the training dataset. ROC curves for models trained with different proportions of labeled data (0% to 100%) on the training dataset are shown in FIG. 8 and on the pooled external validation dataset in FIG. 9. The AUC values are shown in parentheses.

[0082] FIG. 10 illustrates example cases highlighting a semi-weak learner accurately detecting pulmonary emboli in both peripheral and central locations. The left image components indicate the location of the PE (arrows) while the right image components display the model’s attention map using element wise Gradient-weighted Class Activation Mapping (Grad-CAM).

[0083] In the example experiments, it was investigated whether labeling every slice is necessary for accurate PE exam-level classification. The experiments demonstrated that weakly supervised learning, using only exam-level labels, is limited for PE detection. The weakly supervised model achieved an AUC of just 0.656 (0.522, 0.790) on the test dataset, significantly lower than both strong and semi-weak learners. This is likely due to the need for localization of subtle emboli in PE diagnosis, which is more challenging than other tasks where weakly supervised approaches may work.

[0084] Detecting PE using CTPA can pose a significant challenge, even for experienced radiologists. Despite the high sensitivity of PE diagnosis on CTPA, the ML models employed by the system 100 are able to detect smaller PE, such as subsegmental pulmonary embolism (SPE). SPEs are often small, occupying only a few voxels on imaging, akin to searching for a needle in a haystack. In fact, the positive predictive value of SPE diagnosis was a mere 25%, underscoring the diagnostic complexity. Furthermore, interobserver agreement for SPE is notably lower compared to proximal PEs. Additionally, filling artifacts may mimic true thrombotic material, adding to the complexity to the differentiation between PE and mimics. Compounding these challenges are factors that contribute to poorer image quality, such as streak artifact, breathing motion, poor opacification of the pulmonary arterial tree, and suboptimal patient placement. These factors collectively degrade the sensitivity of PE detection, rendering it a considerably more challenging task compared to other pathologies that may require less granular annotation schema.

[0085] By incorporating a small number of labels, in a particular case, approximately 2.5% of total slice-level labels, the example experiments observed a significant performance boost. The AUC on the test dataset increased 0.682 from (0.652, 0.711 ) to 0.858 (0.836, 0.881 ), with similar improvements seen in the external validation dataset. Performance continued to improve with more slice-level labels but plateaued beyond 27.5% label availability, suggesting potentially diminishing returns with additional labeling.

[0086] Thus, the present embodiments provide an approach that is advantageous over previous approaches that assumed that, in PE detection, extensive fine-grained labeling is essential for high performance. FIG. 10 showcases example images where the semi- weak learner demonstrated its ability to correctly detected small PE.

[0087] Using semi-weakly supervised models, the present embodiments can thus achieve high performance with 3.64 fewer granular labels, comparable to strongly supervised models. This approach can maintain robust model performance and operational efficiency in clinical applications, only requiring accurately labeling a smallsubset of the data to yield highly effective models; thereby conserving both time and financial resources.TABLE 3

[0088] Table 3 shows tabular data for model performance in terms of AUC across different datasets in the example experiments as a function of slice-level label availability. Table 3 presents the AUC (95% Cl) for average predictions of 5 CV models trained with varying percentages of slice-level labels (0, 2.5, 5, 10, 20, 27.5, 35, 42.5, 50, 75%, 100%) across public test dataset, private test dataset, and external datasets. The values shown are AUC (95% Cl) for the corresponding percentage of slice-level labels used, along with p-values from the DeLong tests across the training replicates against the model using 100% of labels. Significant p-values (p < 0.05) are marked with an asterisk (*). means not available. The 95% Cl is calculated using DeLong Test across the average predictions of 5 CV experiments.TABLE 4TABLE 5TABLE 6

[0089] Table 4 shows performance in detecting PE on the public test set during the example experiments. Table 5 shows performance in detecting PE on the private test set during the example experiments. Table 6 shows performance in detecting PE on the external validation test set during the example experiments. The acronyms used are: TP - true positive, FN - false negative, TN - true negative, FP - false positive, AUC - area under the receiver operating curve, Acc - accuracy, SEN - sensitivity, SPEC - specificity, PPV - positive predictive value, and NPV - negative predictive value.

[0090] Accordingly, the example experiments illustrate that the use of semi-weakly supervised learning by the system 100, with a limited number of slice-level annotations, achieves substantially high performance for PE detection, comparable to strongly supervised models. Particularly, the example experiments explored fully supervised, weakly supervised, and semi-weakly supervised learning strategies and demonstrated that high diagnostic accuracy can be achieved with 6.7 times fewer slice-level labels; achieving performance comparable to models trained with 100% slice-level labels. In this way, the system 100 significantly reduces the need for extensive labeling and, in doing so, significantly streamlines model training and development.

[0091] For the particular example of using the system 100 to determine PEs, PEs can vary dramatically in their presentation from large emboli occupying the centralpulmonary arteries to a small subsegmental embolus in the lung periphery. Larger PEs may span dozens of images while smaller PEs may only occupy a small number of pixels within a few images. Detecting smaller PEs is particularly challenging due to the large search space of the entire thorax covered by CTPA. Therefore, in some cases, relying solely on exam-level labels may be insufficient, necessitating the use of more granular annotations.

[0092] FIGS. 12A and 12B show experimental impact of label granularity on model performance as a function of AUC. The graphs in FIGS. 12A and 12B illustrate the performance of the models in terms of AUC across different datasets (a private test set and an external validation) as a function of the percentage of slice-level labels used. The solid lines represent the performance of average predictions across 5-fold cross- validation, and the shaded areas correspond to the 95% confidence intervals (Cl). FIGS. 12A and 12B show AUC curves for central and peripheral PE detection on the private test set, with central PE results derived from a central PE subset and peripheral PE results from a peripheral PE subset. Similar to overall PE detection, both central and peripheral PE models benefited from increasing the proportion of slice-level labels.

[0093] For central PE, the initial weakly supervised model (0% slice-level labels) already had a relatively high AUC of 0.817 (0.776, 0.858). Introducing just 2.5% of slice-level labels substantially improved the AUC to 0.972 (0.953, 0.991 ), closely approaching the fully supervised model’s AUC of 0.987 (0.974, 1 .000) (p=0.05). In contrast, peripheral PE detection began with a lower baseline AUC of 0.647 (0.614, 0.680) under weakly supervised learning. Although adding 2.5% of slice-level labels improved performance to an AUC of 0.829 (0.802, 0.856), it required about 27.5% of slice-level labels to achieve near-peak performance (AUC 0.912 [0.891 , 0.933]) close to the fully supervised model’s AUC of 0.917 (0.898, 0.937) (p=0.119).

[0094] The results shown in FIGS. 12A and 12B illustrate that the system 100 is able to determine subtype-aware label budgeting. For each clinically relevant subtype (e.g., central vs. peripheral PE), the system 100 can fit a performance-annotation curve Ms(p) that relates a validation metric (e.g., AUROC) to an annotation amount p (defined as a positive-slice count per exam and / or a dataset-level share). From this curve, thesystem 100 can use a selected fixed minimum p* that meets a target criterion, either Msis a reference performance for subtype s obtained from a fully (or near-fully) labeled training regime or from a plateau estimate of Ms(p). Before training, the dataset is prepared so that each subtype s meets or exceeds p*. Empirically, central PE achieves target performance with substantially fewer positive slice annotations (e.g., 4 slices per exam or 2% of positive slices), whereas peripheral PE requires greater annotation density (e.g., 53.4 slices per exam or 27.5%). The learned policy reduces total annotation while preserving subtype performance.

[0095] In view of the above, the preprocessing module 104 can be used for preparing training labels for a volumetric medical-image classification by:• for each subtype s of a target condition, determining a minimum annotation requirement p* from a validation-based relationship between model performance and annotation amount;• selecting or measuring a training set such that, for each subtype s, the available slice-level labels meet or exceed *; and• training a machine learning classifier using semi-weak supervision on the prepared training set, as described herein.

[0096] In some cases, the subtypes include central and peripheral pulmonary embolism. In an example, central PE meets a requirement of approximately 0.2 positive slices per exam (or 2% of positive slices), and peripheral PE meets approximately 2.72 positive slices per exam or 27.5%; whereby the annotation amount is defined as a positive-slice count per exam and / or a dataset-level share of positive slices.TABLE 7

[0097] Table 7 shows exam-level AUC performance of models trained with varying proportions of slice-level labels for detecting all pulmonary emboli (All PE), central pulmonary emboli (Central PE), and peripheral pulmonary emboli (Peripheral PE). Values are presented as AUC (95% confidence interval) followed by the p-value from pairwise DeLong tests comparing each proportion to the fully supervised (100% slice labels) baseline. Asterisks (*) indicate statistically significant differences (p < 0.05).TABLE 8

[0098] Table 8 shows exam-level AUC performance of two different architectures (EfficientNet and Vision-Transformer) trained with varying proportions of slice-level labels for detecting all pulmonary emboli (All PE), central pulmonary emboli (Central PE), and peripheral pulmonary emboli (Peripheral PE). Values are presented as AUC (95% confidence interval) followed by p-values from pairwise DeLong tests comparing each proportion to the fully supervised (100% slice labels) baseline. Asterisks (*) indicate statistically significant differences (p < 0.05).

[0099] In some embodiments, the medical image can be a volumetric medical image, and the system 100 can update the semi-weakly supervised machine learning model parameters accordingly. To do so, the system 100 can identify a volume of interest (VOI) from provided organ or region coordinates and enlarge a z-extent by a fraction of its span while capping total depth. For training, the system 100 samples L center-slice indices by either (i) drawing a contiguous block or (ii) drawing L linearly-spaced indices across an oversampled span, selected stochastically each step. During training, intensities can be windowed using a specified width and center with random perturbations (e.g., ±5% center, ±10% width) to model scanner / protocol variability. Evaluation can use fixed windowing. To reduce positional overfitting, the depth order can be reversed and / or permuted while maintaining alignment of per-slice targets for auxiliary loss. In such cases, training minimizes a joint objective that includes an examlevel term (with label smoothing) and a masked per-slice auxiliary term that includes only slices flagged by a slice-selection indicator. Unlabeled slices can be ignored in order to prevent label leakage.

[0100] If an exam is labeled positive yet the sampled subsequence contains no slice with positive auxiliary label, the system 100 can override the exam label (or equivalently down-weight the exam term), reducing noise from coarse labels and mismatched sampling. Anatomical masks (e.g., lung) may be concatenated as an additional channel to the image tensor, improving focus on relevant anatomy. Training favors 2D per-slicegeometric and photometric transforms and can optionally use 3D transforms, while evaluation generally employs resize / pad only. Additionally, class imbalance can be handled via per-exam weights. Generally, any suitable slice encoder and sequence aggregator may be used (e.g., 2D / 3D CNNs, transformers, attention pooling, LSTM) Whereby CoAtNet and BiLSTM, described herein, are merely examples.

[0101] Although the invention has been described with reference to certain specific embodiments, various other aspects, advantages and modifications thereof will be apparent to those skilled in the art without departing from the spirit and scope of the invention as outlined in the claims appended hereto.

Claims

CLAIMS1 . A computer-implemented method for generating a machine learning model to identify a medical condition in a medical image, the method comprising: receiving a training dataset comprising a plurality of training medical images, the training medical images comprising annotations that identify the medical condition, the annotations comprising exam-level labels on a majority of the training medical images and the annotations comprising slice-level labels or region-level labels, or both, on a minority of the training medical images; training a semi-weakly supervised machine learning model using the annotated training dataset to identify one or more medical conditions from an input inference medical image; and outputting the semi-weakly supervised machine learning model for use in identifying the one or more medical conditions from the input inference medical image.

2. The method of claim 1 , wherein the medical condition is pulmonary embolism.

3. The method of claim 2, the method further comprising performing preprocessing on the training medical images, the preprocessing comprising one or more of converting the training medical images to a standardized format, applying windowing to optimize contrast, and normalizing the image intensity ranges4. The method of claim 1 , wherein exam-level labels comprise an indication of a presence or an absence of a condition across an entire scan without specifying the location of the condition, wherein slice-level labels comprise identification of which specific slices contain the condition without specifying the location of the condition within the specific slices, and wherein region-level labels comprise identification of the condition at specific areas or organs within a slice.

5. The method of claim 1 , wherein the slice-level labels or region-level labels, or both, are included on between 2.5% and 27.5% of the training images.

6. The method of claim 1 , wherein the semi-weakly supervised machine learning model comprises an attention-based convolutional neural network (CNN)- recurrent neural network (RNN) model.

7. The method of claim 6, wherein the attention-based CNN-RNN model comprises feature extraction using a model that combines convolutional networks and transformers, or comprises batch normalization to capture z-axis temporal relationships within sequences of medical images, or both.

8. The method of claim 1 , wherein the medical image comprises a volumetric medical image, and wherein the method further comprising updating the semi- weakly supervised machine learning model parameters by: determining a volume-of-interest and expanding a depth extent of the volume-of-interest by a fraction of its span, constructing a training input by sampling L slices from the volume-of-interest according to a stochastic rule, window-normalizing slice intensities with randomized window center and width, forming a per-slice auxiliary target vector aligned with the sampled L slices and masking unlabeled slices from an instance-level loss, and determining a joint loss comprising an examlevel term and the masked per-slice auxiliary term, and updating the model parameters to minimize joint loss.

9. A system for generating a machine learning model to identify a medical condition in a medical image, the system comprising one or more processors and a data memory, the data memory comprising instructions for the one or more processors to execute: an input module to receive a training dataset comprising a plurality of training medical images, the training medical images comprising annotations that identify the medical condition, the annotations comprising exam-level labels on a majority of the training medical images and theannotations comprising slice-level labels or region-level labels, or both, on a minority of the training medical images; a training module to train a semi-weakly supervised machine learning model using the annotated training dataset to identify one or more medical conditions from an input inference medical image; and an output module to output the semi-weakly supervised machine learning model for use in identifying the one or more medical conditions from the input inference medical image.

10. The system of claim 9, wherein the medical condition is pulmonary embolism, the one or more processors further executing a preprocessing module to perform preprocessing on the training medical images, the preprocessing comprising one or more of converting the training medical images to a standardized format, applying windowing to optimize contrast, and normalizing the image intensity ranges.11 . The system of claim 9, wherein exam-level labels comprise an indication of a presence or an absence of a condition across an entire scan without specifying the location of the condition, wherein slice-level labels comprise identification of which specific slices contain the condition without specifying the location of the condition within the specific slices, and wherein region-level labels comprise identification of the condition at specific areas or organs within a slice.

12. The system of claim 9, wherein the slice-level labels or region-level labels, or both, are included on between 2.5% and 27.5% of the training images.

13. The system of claim 9, wherein the semi-weakly supervised machine learning model comprises an attention-based convolutional neural network (CNN)- recurrent neural network (RNN) model.

14. The system of claim 13, wherein the attention-based CNN-RNN model comprises feature extraction using a model that combines convolutional networks andtransformers, or comprises batch normalization to capture z-axis temporal relationships within sequences of medical images, or both.

15. A computer-implemented method for identifying a medical condition in an inference medical image using a trained semi-weakly supervised machine learning model, the method comprising: receiving the inference medical image; identifying the medical condition in the received inference medical image using the trained semi-weakly supervised machine learning model, the semi-weakly supervised machine learning model trained using a training dataset comprising a plurality of training medical images, the training medical images comprising annotations that identify the medical condition, the annotations comprising exam-level labels on a majority of the training medical images and the annotations comprising slice-level labels or region-level labels, or both, on a minority of the training medical images; and outputting the identification of the medical condition.

16. The method of claim 15, further comprising generating and outputting a visualization of the identified medical condition using Gradient-weighted Class Activation Mapping to highlight regions of the inference medical image that contributed most to the identification of the medical condition.

17. The method of claim 15, wherein the medical condition is pulmonary embolism.

18. The method of claim 15, wherein exam-level labels comprise an indication of a presence or an absence of a condition across an entire scan without specifying the location of the condition, wherein slice-level labels comprise identification of which specific slices contain the condition without specifying the location of the condition within the specific slices, and wherein region-level labels comprise identification of the condition at specific areas or organs within a slice.

19. The method of claim 15, wherein the semi-weakly supervised machine learning model comprises an attention-based convolutional neural network (CNN)- recurrent neural network (RNN) model.

20. The method of claim 19, wherein the attention-based CNN-RNN model comprises feature extraction using a model that combines convolutional networks and transformers, or comprises batch normalization to capture z-axis temporal relationships within sequences of medical images, or both.

Images