Systems and methods for image based diagnosis of pneumonia and / or pneumonitis

Abstract

Systems and methods for image-based detection or diagnosis of pneumonia and / or pneumonitis can include a computer system acquiring a plurality of medical images of lungs of a plurality of individuals including a first set of individuals having pneumonia and a second set of individuals having pneumonitis, generating habitat maps of the medical images representing a segmentation of the medical images and depicting a plurality of image subregions determined based on similarity in imaging patterns across the plurality of medical images, determining a plurality of first features from the habitat maps, determining a plurality of second features indicative of how symmetric the plurality of first features across left and right lung, and generating, using the plurality of first features and the plurality of second features, a machine learning model configured to detect at least one of pneumonia or pneumonitis.

Description

SYSTEMS AND METHODS FOR IMAGE BASED DIAGNOSIS OF PNEUMONIA AND / OR PNEUMONITISCROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit and priority to U.S. Provisional Patent Application No. 63 / 517563 filed on August 3, 2023, and titled “SYSTEMS AND METHODS FOR IMAGE BASED DIAGNOSIS OF PNEUMONIA AND / OR PNEUMONITIS,” which is incorporated herein by reference in its entirety.FIELD OF THE DISCLOSURE

[0002] The present application relates generally to systems and methods for image-based diagnosis of pneumonia and / or pneumonitis. Specifically, systems and methods described herein employ habitat imaging to distinguish between pneumonia and pneumonitis.BACKGROUND OF THE DISCLOSURE

[0003] Immune checkpoint inhibitors (ICIs) have been a transformative force in oncology and have become a key part of the therapeutic arsenal for numerous cancers. Acute myeloid leukemia (AML), a highly lethal cancer which often requires allogeneic hematopoietic transplantation (allo-HCT) to achieve a durable remission, may sometimes respond to ICIs given in combination with hypomethylating agents. However, the use of ICIs to treat AML is associated with high rates of pneumonitis, which significantly increases mortality. Specifically, pneumonitis is a serious complication of checkpoint inhibitor immunotherapy, and the mortality ranges from 10-20% in non-small cell lung cancer cohorts (NSCLC), where ICIs are frequently used to nearly 50% in AML.

[0004] Pneumonitis is likely to be more amenable to treatment if detected early and distinguished from pneumonia. Pneumonia is a lung infection that causes the air sacs, or alveoli, of the lungs to fill up with fluid or pus. The treatments for pneumonitis, namely immunosuppressive therapies of appropriate intensity and duration, are substantively different from the treatments for pneumonia. Furthermore, overuse of antimicrobial agents in patients without pneumonia may alter the intestinal microbiome, potentially reducing the efficacy of ICIs.SUMMARY

[0005] According to one aspect, a system can include one or more processors and a memory storing computer instructions. The computer instructions when executed by the one or more processors can cause the system to acquire a plurality of medical images of lungs of a plurality of individuals, the plurality of individuals including a first set of individuals having pneumonia and a second set of individuals having pneumonitis, generate habitat maps of the medical images, the habitat maps representing a segmentation of the medical images and depicting a plurality of image sub-regions determined based on similarity in imaging patterns across the plurality of medical images, determine a plurality of first features from the habitat maps, determine a plurality of second features indicative of how symmetric the plurality of first features across left and right lungs, and generate, using the plurality of first features and the plurality of second features, a machine learning model configured to detect at least one of pneumonia or pneumonitis.

[0006] In some implementations, the one or more processors can be configured to cause the system to segment each of the plurality of medical images to determine corresponding left and right lung parenchyma. In some implementations, the one or more processors can be configured to cause the system to extract, from each image of the plurality of images, a corresponding first image depicting a corresponding left and right lung parenchyma, extract, from each image of the plurality of images, a corresponding second image depicting a corresponding mediastinum region, apply, for each image of the plurality of images, an entropy filter to the corresponding first image and the corresponding second image, and generate, for each image of the plurality of images, a corresponding composite image by combining the filtered corresponding first image and the filtered corresponding second image.

[0007] In some implementations, the one or more processors can be configured to cause the system to use hierarchical clustering to determine the plurality of image sub-regions. The one or more processors can cause the system to cluster pixels associated with each image of the plurality of images into superpixels, and apply the hierarchical clustering to the superpixels using superpixel characteristics across the plurality of images. In some implementations, the plurality of first features can include features indicative of spatial characteristics of the plurality of subregions.

[0008] In some implementations, the one or more processors can be configured to cause the system to rank the plurality of first features and the plurality of second features based oncorrelations of each feature with pneumonia infection and pneumonitis infection, and select a subset of the plurality of first features and the plurality of second features based on the raking, the selected subset used to generate the machine learning model. The one or more processors can cause the system to apply a Chi-square test to each feature of the plurality of first features and the plurality of second features to determine a corresponding probability of pneumonia or pneumonitis infection, and rank the plurality of first features and the plurality of second features based on corresponding probabilities of pneumonia or pneumonitis infection.

[0009] In some implementations, the machine learning model can include an ensemble of boosted decision trees. In some implementations, the one or more processors can be configured to cause the system to generate the machine learning model further using at least one of clinical parameters or blood analysis results of the individuals. In some implementations, the blood analysis results can include at least one of absolute white blood cell (WBC) count, absolute neutrophils count, absolute lymphocyte count, platelet count or bone marrow blast cell count. In some implementations, the clinical parameters can include at least one of age, gender, cough, fever or shortness of breath.

[0010] According to one aspect, a method can include acquiring a plurality of medical images of lungs of a plurality of individuals, the plurality of individuals including a first set of individuals having pneumonia and a second set of individuals having pneumonitis, generating habitat maps of the medical images, the habitat maps representing a segmentation of the medical images and depicting a plurality of image sub-regions determined based on similarity in imaging patterns across the plurality of medical images, determining a plurality of first features from the habitat maps; determining a plurality of second features indicative of how symmetric the plurality of first features across left and right lung, and generating, using the plurality of first features and the plurality of second features, a machine learning model configured to detect at least one of pneumonia or pneumonitis.

[0011] In some implementations, the method can include segmenting each of the plurality of medical images to determine corresponding left and right lung parenchyma. In some implementations, the method can include extracting, from each image of the plurality of images, a corresponding first image depicting a corresponding left and right lung parenchyma, extracting, from each image of the plurality of images, a corresponding second image depicting a corresponding mediastinum region, applying, for each image of the plurality of images, an entropy filter to the corresponding first image and the corresponding second image, andgenerating, for each image of the plurality of images, a corresponding composite image by combining the filtered corresponding first image and the filtered corresponding second image.

[0012] In some implementations, the method can include using hierarchical clustering to determine the plurality of image sub-regions. The method can include clustering pixels associated with each image of the plurality of images into superpixels, and applying the hierarchical clustering to the superpixels using superpixel characteristics across the plurality of images. In some implementations, the plurality of first features can include features indicative of spatial characteristics of the plurality of subregions.

[0013] In some implementations, the method can include ranking the plurality of first features and the plurality of second features based on correlations of each feature with pneumonia infection and pneumonitis infection, and selecting a subset of the plurality of first features and the plurality of second features based on the raking, the selected subset used to generate the machine learning model. The method can include applying a Chi-square test to each feature of the plurality of first features and the plurality of second features to determine a corresponding probability of pneumonia or pneumonitis infection, and ranking the plurality of first features and the plurality of second features based on corresponding probabilities of pneumonia or pneumonitis infection.

[0014] In some implementations, the machine learning model can include an ensemble of boosted decision trees. In some implementations, the method can include generating the machine learning model further using at least one of clinical parameters or blood analysis results of the individuals. In some implementations, the blood analysis results can include at least one of absolute white blood cell (WBC) count, absolute neutrophils count, absolute lymphocyte count, platelet count or bone marrow blast cell count. In some implementations, the clinical parameters can include at least one of age, gender, cough, fever or shortness of breath.

[0015] According to one aspect, a non-transitory computer-readable medium can store computer instructions. The computer instructions when executed by one or more processors cause the one or more processors to generate habitat maps of the medical images, the habitat maps representing a segmentation of the medical images and depicting a plurality of image sub-regions determined based on similarity in imaging patterns across the plurality of medical images, determine a plurality of first features from the habitat maps, determine a plurality ofsecond features indicative of how symmetric the plurality of first features across left and right lungs, and generate, using the plurality of first features and the plurality of second features, a machine learning model configured to detect at least one of pneumonia or pneumonitis.

[0016] According to one aspect, a system can include one or more processors and a memory storing computer instructions. The computer instructions when executed by the one or more processors can cause the system to receive a medical image of lungs of a an individual, generate a habitat map of the medical image, the habitat map representing a segmentation of the medical images and depicting a plurality of image sub-regions determined based on predefined imaging patterns, determine a plurality of image features from the habitat map, determine a plurality of second features indicative of how symmetric the plurality of first features across left and right lungs, and generate, using the plurality of first features and the plurality of second features, a machine learning model configured to detect at least one of pneumonia or pneumonitis.

[0017] According to one aspect, a method can include receiving, by a computer system including one or more processors and a memory, a medical image of lungs of a an individual; generating, by the computer system, a habitat map of the medical image, the habitat map representing a segmentation of the medical images and depicting a plurality of image subregions determined based on predefined imaging patterns; determining, by the computer system, a plurality of image features from the habitat map; determining, by the computer system, a plurality of second features indicative of how symmetric the plurality of first features across left and right lungs; and generating, by the computer system, using the plurality of first features and the plurality of second features, a machine learning model configured to detect at least one of pneumonia or pneumonitis.

[0018] According to one aspect, a non-transitory computer-readable medium can store computer instructions. The computer instructions when executed by one or more processors cause the one or more processors to receive a medical image of lungs of a an individual, generate a habitat map of the medical image, the habitat map representing a segmentation of the medical images and depicting a plurality of image sub-regions determined based on predefined imaging patterns, determine a plurality of image features from the habitat map, determine a plurality of second features indicative of how symmetric the plurality of first features across left and right lungs, and generate, using the plurality of first features and the plurality of second features, a machine learning model configured to detect at least one of pneumonia or pneumonitis.BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The foregoing and other objects, aspects, features, and advantages of the disclosure will become more apparent and better understood by referring to the following description taken in conjunction with the accompanying drawings, in which:

[0020] FIG. 1 is a diagram illustrating a high-level overview of various approaches for diagnosing pneumonia and / or pneumonitis, according to embodiments described herein;

[0021] FIG. 2 is a block diagram depicting an embodiment of a computer system, according to embodiments described herein;

[0022] FIG. 3 shows a flow diagram illustrating a method for generating a machine learning model for image-based detection or diagnosis of pneumonia and / or pneumonitis, according to embodiments described herein;

[0023] FIG. 4 is a diagram depicting an example implementation of the method of FIG. 3, according to embodiments described herein.

[0024] FIG. 5 shows a heatmap of superpixels of five identified sub-regions together with distribution plots (boxplots) of five representative features is shown, according to embodiments described herein;

[0025] FIG. 6 shows images of various patients at various stages of the method described in FIGS. 3 and 4, according to example embodiments herein;

[0026] FIGS. 7A-7C show performance results of different diagnostic models, according to embodiments described herein;

[0027] FIGS 8A-8D depict correlation heatmaps for features used to build different diagnostics models, according to embodiments described herein;

[0028] FIG. 9A shows prediction results for a clinical benchmark model, a habitat imagingbased model and a conventional radiomics model, according to embodiments described herein;

[0029] FIG. 9B shows a cascading model with a clinical benchmark model in the first layer and a habitat imaging-based model in the second layer, according to embodiments describedherein;

[0030] FIG. 10 shows feature importance scores and a confusion matrix for a radiomics model, according to embodiments described herein; and

[0031] FIG. 11 shows a flow chart illustrating a method for image-based detection or diagnosis of pneumonia and / or pneumonitis, according to embodiments described herein.DETAILED DESCRIPTION

[0032] Following below are more detailed descriptions of various concepts related to, and embodiments of, systems and methods for diagnosis of pneumonia and / or pneumonitis. It should be appreciated that various concepts introduced above and discussed in greater detail below may be implemented in any of numerous ways, as the disclosed concepts are not limited to any particular manner of implementation. Examples of specific implementations and applications are provided primarily for illustrative purposes.

[0033] Immune checkpoint inhibitors (ICI) may cause pneumonitis, resulting in potentially fatal lung inflammation. However, reliable diagnosis of pneumonitis is still a challenge. Specifically, distinguishing pneumonitis from pneumonia requires imperfect invasive testing with bronchoscopy and is resource-intensive and often inaccurate in usual clinical care. The prompt diagnosis of ICI pneumonitis and pneumonia is desired to ensure the appropriate administration of corticosteroids, both to promptly treat pneumonitis and to be withheld in cases of infection, since corticosteroids may worsen potential inflammation.

[0034] A major barrier to diagnosing pneumonitis is the difficulty in distinguishing pneumonitis from other pulmonary conditions, especially pneumonia. Various clinical and blood analysis parameters were explored to distinguish pneumonitis from pneumonia. However, the clinical and / or blood-based techniques are typically laborious and / or time consuming with suboptimal diagnosis results. For instance, bronchoalveolar lavage (BAL) biomarkers show clonal expensive of Thl7.1 cells, but do not necessarily distinguish between pneumonia and pneumonitis. Culture-based identification of pathogens can identify up to 60% of infections, but these results may require up to 48 hours and are more useful for ruling infection in, but not ruling infection out. Metagenomic approaches may increase the yield for the detection of bacterial organisms in immunocompromised hosts, but the diagnostic yield remains suboptimal for certain infections, and distinguishing colonization from true infectionis challenging. Furthermore, diagnosis of pneumonitis with histopathology in high-risk patients can lead to bleeding due to thrombocytopenia and / or lead to pulmonary deterioration after a biopsy procedure, and results of histopathological analyses are often delayed by several days after the biopsy procedure.

[0035] Radiomic approaches may allow for the prompt identification of pulmonary diseases, such as interstitial lung diseases. However, these approaches have not been tested to distinguish infectious pneumonia from ICI pneumonitis. One radiomic approach may be designed to profile the infected lung region as a whole entity. Such approach, however, may fall short with respect to characterizing phenotypically heterogeneous subareas of the lung that are infected or inflamed. In other words, phenotypical characteristics of various lung subregions can provide a basis or factors for detecting the type of lung infection. Radiomic analysis of the lung region as a whole entity would not reveal characteristics of various subregions and how such characteristics interact or correlate with each other.

[0036] In the current disclosure, habitat imaging is employed to identify image features that are correlated to the type of lung infection and use such features to build predictive or diagnosis machine learning models for distinguishing between pneumonia and ICI pneumonitis. Habitat imaging involves explicitly dividing a region-of-interest (ROI) into coherent sub-regions termed as habitats. Once the sub-regions are determined, embodiments described herein involve extracting various image features of the sub-regions. The extracted features can be assessed in terms of their correlations with the pneumonia infection and ICI pneumonitis infection, and used to generate machine learning models for image-based detection or diagnosis of pneumonia and / or ICI pneumonitis.

[0037] Referring now to FIG. 1, a diagram 10 depicting a high-level overview of various approaches for diagnosing pneumonia and / or pneumonitis is shown, according to embodiments described herein. FIG. 1 shows three different approaches for ICI pneumonitis diagnosis illustrated via three processing paths. The bottom processing path represents a diagnosis technique based on clinical data, such as clinical symptoms and blood analysis results. In the current disclosure, the clinical-metrics-based approach is used to derive a clinical benchmark model by selecting informative clinical and blood metrics to fit into a classification model. The clinical benchmark model is used to assess the performance of novel image-based diagnosis models described herein.

[0038] The top processing path in FIG. 1 describes an image-based approach for generating a classification model to distinguish between pneumonia and ICI pneumonitis. The imagebased approach includes acquisition and preprocessing of medical images of individuals or patients suffering from either pneumonia infection or ICI pneumonitis, habitat-based analysis of the medical images, extraction and assessment of features associated with sub-regions identified as part of the habitat-based analysis, and training a classification model for distinguishing between pneumonia and pneumonitis using at least a subset of the extracted features. The classification model is configured to determine whether a patient was more likely to have pneumonia or pneumonitis.

[0039] The middle processing path of FIG. 1 represents another approach combining the image-based approach in the top processing path and the clinical benchmark model depicted in the bottom processing path. Specifically, clinical metrics can be integrated into the imagebased approach to further evaluate the prediction performance. The final classification model in this case can include image features associated with sub-regions identified as part of the habitat-based analysis as well as clinical metrics.

[0040] The details of the various approaches depicted in FIG. 1 are described in further detail below in relation with FIGS. 3-X. The embodiments described herein include systems and methods for generating pneumonia and / or ICI pneumonitis detection models, or for identifying relevant features to train machine learning models, as well as system and methods for using such models to distinguish between pneumonia and / or ICI pneumonitis or to detect any one of these types of infection.A. Computing and Network Environment

[0041] It may be helpful to describe aspects of the operating environment as well as associated system components (e.g., hardware elements) in connection with the methods and systems described herein. The methods described herein, such as methods 300 and X00, can be implemented or executed by any of the devices, or any combination thereof, described in relation to FIG. 2.

[0042] Various operations described herein can be implemented on computer systems. FIG. 2 shows a simplified block diagram of a representative server system 200, client computing system 214, and network 226 usable to implement certain embodiments of the present disclosure. In various embodiments, server system 200 or similar systems can implementservices or servers described herein or portions thereof. Client computing system 214 or similar systems can implement clients described herein. Server system 200 can have a modular design that incorporates a number of modules 202 (e.g., blades in a blade server embodiment); while two modules 202 are shown, any number can be provided. Each module 202 can include processing unit(s) 204 and local storage 206.

[0043] Processing unit(s) 204 can include a single processor, which can have one or more cores, or multiple processors. In some embodiments, processing unit(s) 204 can include a general-purpose primary processor as well as one or more special-purpose co-processors such as graphics processors, digital signal processors, or the like. In some embodiments, some or all processing units 204 can be implemented using customized circuits, such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions that are stored on the circuit itself. In other embodiments, processing unit(s) 204 can execute instructions stored in local storage 206. Any type of processors in any combination can be included in processing unit(s) 204.

[0044] Local storage 206 can include volatile storage media (e.g., DRAM, SRAM, SDRAM, or the like) and / or non-volatile storage media (e.g., magnetic or optical disk, flash memory, or the like). Storage media incorporated in local storage 206 can be fixed, removable or upgradeable as desired. Local storage 206 can be physically or logically divided into various subunits such as a system memory, a read-only memory (ROM), and a permanent storage device. The system memory can be a read-and-write memory device or a volatile read-and- write memory, such as dynamic random-access memory. The system memory can store some or all of the instructions and data that processing unit(s) 204 need at runtime. The ROM can store static data and instructions that are needed by processing unit(s) 204. The permanent storage device can be a non-volatile read-and-write memory device that can store instructions and data even when module 202 is powered down. The term “storage medium” as used herein includes any medium in which data can be stored indefinitely (subject to overwriting, electrical disturbance, power loss, or the like) and does not include carrier waves and transitory electronic signals propagating wirelessly or over wired connections.

[0045] In some embodiments, local storage 206 can store one or more software programs to be executed by processing unit(s) 204, such as an operating system and / or programs implementing various methods or steps thereof such as methods 300 and 700 of FIGS. 3 and 7 or any other methods or processes described herein.

[0046] “ Software” refers generally to sequences of instructions that, when executed by processing unit(s) 204 cause server system 200 (or portions thereof) to perform various operations, thus defining one or more specific machine embodiments that execute and perform the operations of the software programs. The instructions can be stored as firmware residing in read-only memory and / or program code stored in non-volatile storage media that can be read into volatile working memory for execution by processing unit(s) 204. Software can be implemented as a single program or a collection of separate programs or program modules that interact as desired. From local storage 206 (or non-local storage described below), processing unit(s) 204 can retrieve program instructions to execute and data to process in order to execute various operations described above.

[0047] In some server systems 200, multiple modules 202 can be interconnected via a bus or other interconnect 208, forming a local area network that supports communication between modules 202 and other components of server system 200. Interconnect 208 can be implemented using various technologies including server racks, hubs, routers, etc.

[0048] A wide area network (WAN) interface 210 can provide data communication capability between the local area network (interconnect 208) and the network 226, such as the Internet. Technologies can be used, including wired (e.g., Ethernet, IEEE 1002.3 standards) and / or wireless technologies (e.g., Wi-Fi, IEEE 1002.11 standards).

[0049] In some embodiments, local storage 206 is intended to provide working memory for processing unit(s) 204, providing fast access to programs and / or data to be processed while reducing traffic on interconnect 208. Storage for larger quantities of data can be provided on the local area network by one or more mass storage subsystems 212 that can be connected to interconnect 208. Mass storage subsystem 212 can be based on magnetic, optical, semiconductor, or other data storage media. Direct attached storage, storage area networks, network-attached storage, and the like can be used. Any data stores or other collections of data described herein as being produced, consumed, or maintained by a service or server can be stored in mass storage subsystem 212. In some embodiments, additional data storage resources may be accessible via WAN interface 210 (potentially with increased latency).

[0050] Server system 200 can operate in response to requests received via WAN interface 1010. For example, one of modules 202 can implement a supervisory function and assign discrete tasks to other modules 202 in response to received requests. Work allocationtechniques can be used. As requests are processed, results can be returned to the requester via WAN interface 210. Such operation can generally be automated. Further, in some embodiments, WAN interface 210 can connect multiple server systems 200 to each other, providing scalable systems capable of managing high volumes of activity. Other techniques for managing server systems and server farms (collections of server systems that cooperate) can be used, including dynamic resource allocation and reallocation.

[0051] Server system 200 can interact with various user-owned or user-operated devices via a wide-area network such as the Internet. An example of a user-operated device is shown in FIG. 2 as client computing system 214. Client computing system 214 can be implemented, for example, as a consumer device such as a smartphone, other mobile phone, tablet computer, wearable computing device (e.g., smart watch, eyeglasses), desktop computer, laptop computer, and so on.

[0052] For example, client computing system 214 can communicate via WAN interface 210. Client computing system 214 can include computer components such as processing unit(s) 216, storage device 218, network interface 220, user input device 222, and user output device 224. Client computing system 214 can be a computing device implemented in a variety of form factors, such as a desktop computer, laptop computer, tablet computer, smartphone, other mobile computing device, wearable computing device, or the like.

[0053] Processing unit(s) 216 and storage device 218 can be similar to processing unit(s) 204 and local storage 206 described above. Suitable devices can be selected based on the demands to be placed on client computing system 214; for example, client computing system 214 can be implemented as a “thin” client with limited processing capability or as a high-powered computing device. Client computing system 214 can be provisioned with program code executable by processing unit(s) 216 to enable various interactions with server system 200.

[0054] Network interface 220 can provide a connection to the network 226, such as a wide area network (e.g., the Internet) to which WAN interface 210 of server system 200 is also connected. In various embodiments, network interface 220 can include a wired interface (e.g., Ethernet) and / or a wireless interface implementing various RF data communication standards such as Wi-Fi, Bluetooth, or cellular data network standards (e.g., 3G, 4G, LTE, etc.).

[0055] User input device 222 can include any device (or devices) via which a user can provide signals to client computing system 214; client computing system 214 can interpret thesignals as indicative of particular user requests or information. In various embodiments, user input device 222 can include any or all of a keyboard, touch pad, touch screen, mouse or other pointing device, scroll wheel, click wheel, dial, button, switch, keypad, microphone, and so on.

[0056] User output device 224 can include any device via which client computing system 214 can provide information to a user. For example, user output device 224 can include a display to display images generated by or delivered to client computing system 214. The display can incorporate various image generation technologies, e.g., a liquid crystal display (LCD), light-emitting diode (LED) including organic light-emitting diodes (OLED), projection system, cathode ray tube (CRT), or the like, together with supporting electronics (e.g., digital- to-analog or analog-to-digital converters, signal processors, or the like). Some embodiments can include a device such as a touchscreen that function as both input and output device. In some embodiments, other user output devices 224 can be provided in addition to or instead of a display. Examples include indicator lights, speakers, tactile “display” devices, printers, and so on.

[0057] Some embodiments include electronic components, such as microprocessors, storage and memory that store computer program instructions in a computer-readable storage medium. Many of the features described in this specification can be implemented as processes that are specified as a set of program instructions encoded on a computer-readable storage medium. When these program instructions are executed by one or more processing units, they cause the processing unit(s) to perform various operation indicated in the program instructions. Examples of program instructions or computer code include machine code, such as is produced by a compiler, and files including higher-level code that are executed by a computer, an electronic component, or a microprocessor using an interpreter. Through suitable programming, processing unit(s) 204 and 216 can provide various functionality for server system 200 and client computing system 214, including any of the functionality described herein as being performed by a server or client, or other functionality.

[0058] It will be appreciated that server system 200 and client computing system 214 are illustrative and that variations and modifications are possible. Computer systems used in connection with embodiments of the present disclosure can have other capabilities not specifically described here. Further, while server system 200 and client computing system 214 are described with reference to particular blocks, it is to be understood that these blocks are defined for convenience of description and are not intended to imply a particular physicalarrangement of component parts. For instance, different blocks can be but need not be located in the same facility, in the same server rack, or on the same motherboard. Further, the blocks need not correspond to physically distinct components. Blocks can be configured to perform various operations, e.g., by programming a processor or providing appropriate control circuitry, and various blocks might or might not be reconfigurable depending on how the initial configuration is obtained. Embodiments of the present disclosure can be realized in a variety of apparatus including electronic devices implemented using any combination of circuitry and software.

[0059] While the disclosure has been described with respect to specific embodiments, one skilled in the art will recognize that numerous modifications are possible. Embodiments of the disclosure can be realized using a variety of computer systems and communication technologies including but not limited to the specific examples described herein. Embodiments of the present disclosure can be realized using any combination of dedicated components and / or programmable processors and / or other programmable devices. The various processes described herein can be implemented on the same processor or different processors in any combination. Where components are described as being configured to perform certain operations, such configuration can be accomplished, e.g., by designing electronic circuits to perform the operation, by programming programmable electronic circuits (such as microprocessors) to perform the operation, or any combination thereof. Further, while the embodiments described above may make reference to specific hardware and software components, those skilled in the art will appreciate that different combinations of hardware and / or software components may also be used and that particular operations described as being implemented in hardware might also be implemented in software or vice versa.

[0060] Computer programs incorporating various features of the present disclosure may be encoded and stored on various computer-readable storage media; suitable media include magnetic disk or tape, optical storage media such as compact disk (CD) or DVD (digital versatile disk), flash memory, and other non-transitory media. Computer-readable media encoded with the program code may be packaged with a compatible electronic device, or the program code may be provided separately from electronic devices (e.g., via Internet download or as a separately packaged computer-readable storage medium).

[0061] Thus, although the disclosure has been described with respect to specific embodiments, it will be appreciated that the disclosure is intended to cover all modificationsand equivalents within the scope of the following claims.B. Generating Image-Based Detection Models for Diagnosis of Pneumonia and / or Pneumonitis

[0062] Referring now to FIG. 3, a flow diagram illustrating a method 300 for generating a machine learning model for image-based detection or diagnosis of pneumonia and / or pneumonitis is shown, according to embodiments described herein. In brief overview, the method 300 can include acquiring a plurality of medical images of lungs of a plurality of individuals individually suffering from either pneumonia or pneumonitis (STEP 302), determining coherent sub-regions of the plurality of medical images (STEP 304), determining a plurality of features the coherent sub-regions (STEP 306), and generating a machine learning model pneumonia and / or pneumonitis detection or diagnosis (STEP 308).

[0063] The method 300 can be executed by a computer system, such as the server system 200, the client computing system 214 or a combination thereof. The method 300 can be implemented as computer code instructions that are executed by one or more processors of the computer system. In some implementations, all the steps of the method 300 can be executed by the same set of processors. In some implementations, separate combinations of processors can execute different steps or different combinations of steps of method 300. In some implementations, the method 300 can be implemented using hardware circuits.

[0064] FIG. 4 shows a diagram 400 depicting an example implementation of the method 300 of FIG. 3, according to embodiments described herein. The diagram 400 depicts more detailed processing steps associated with the steps of method 300. In the following, the method 300 is described in relation with both FIGS. 3 and 4.

[0065] The method 300 can include the computer system acquiring a plurality of medical images of lungs of a plurality of individuals individually suffering from either pneumonia or pneumonitis (STEP 302). The plurality of individuals can include a first set of individuals having pneumonia and a second set of individuals having pneumonitis. The medical images can include CT scans of the plurality of individuals. In some implementations, the medical images can include CT scans, magnetic resonance (MR) images, ultrasound images, optical tomography (OT) images, electrical impedance tomography (EIT) images, multi-modality images, other types of medical images or a combination thereof. The medical images can be labeled or mapped to the type of infection. In other words, each medical image can beassociated with data indicative of the type of infection, e.g., pneumonia or ICI pneumonitis, affecting the lungs depicted in the image. The computer system can receive the medical images as input, e.g., via an input interface, or can retrieve the medical images from a database or a storage device.

[0066] The study referred to herein includes 125 enrolled patients. The medical images used in the study are CT scans of the 125 patients obtained using both Siemens and GE medical systems CT scanners. The CT scans have a slice thickness of 2.5mm and an in-plane spatial resolution of 0.98 to 1.2 mm. Initially, CT scans of 258 patients with AML who were started on ICI therapies between 2016 and 2018 were reviewed. A group of 97 patients with confirmed episodes of pneumonia and a group of 28 patients with confirmed pneumonitis were selected, and the corresponding CT scans were used in the study. All cases were reviewed by a multidisciplinary adjudication committee, who reviewed clinical history, representative laboratory, imaging, and microbiological data, and response to antimicrobial or antiinflammatory therapies.

[0067] Table 1 below provides a list of various metrics and characteristics of the 125 enrolled patients. In Table 1, ECOG refers to Eastern Cooperative Oncology Group, WBC refers to white blood cell, ANC refers to absolute neutrophil count, ALC refers to absolute lymphocyte count, COPD refers to chronic obstructive pulmonary disease, ILD refers to interstitial lung disease, and SCT refers to stem cell transplantation. Pneumonia was diagnosed in patients with1) consistent symptoms (e.g. fevers, cough) and consistent imaging (e.g. lobar pneumonia) and2) who had a clear response to antibiotics but not corticosteroids or had microbiological confirmation from a lower respiratory tract specimen of an organism known to cause pneumonia. Pneumonitis was diagnosed in patients with 1) consistent symptoms (e.g. cough, shortness of breath) and consistent imaging (e.g. diffuse ground-glass opacities) and 2) who had a clear response to corticosteroids but not antibiotics or had histopathological confirmation of pneumonitis. Pneumonitis was graded according to the Common Terminology Criteria for Adverse Events (CTCAE). The study was approved by the Institutional Review Board approved (PAI 8-0802).Table 1. Characteristics of the study cohort.

[0068] The computer system can segment each of the plurality of medical images to determine, for each image, the corresponding lung region, including the left lung parenchyma and the right lung parenchyma. In some implementations, the computer system can use a deep learning-based segmentation model determine the left and right lung parenchyma. The computer system can apply a morphological dilation and / or erosion to smoothen the boundaries of the segmented lung regions. In some implementations, radiologists and / or other physicianscan review and correct the lung ROI segmentations. In some implementations, the computer system can employ other segmentation techniques.

[0069] Referring now to FIGS. 3 and 4, the method 300 can include preprocessing the medical images as depicted in step 402 of FIG. 4. The computer system can extract, from each medical image, a corresponding first image depicting the corresponding left and right lung parenchyma and a corresponding second image depicting the corresponding mediastinum region. The first and second images can be referred to herein as the lung window (or lung view) and the mediastinum window (or mediastinum view), respectively. The computer system can apply one or more digital contrast-enhancing methods or techniques to generate the optimal viewing conditions for corresponding images for the lungs and mediastinum, which includes structures that exist in the center of the thorax other than the lungs, such as the heart, aorta, pulmonary vessels, and thoracic lymph nodes. These contrast-enhancing method(s) or technique(s) improve lesion conspicuity by manipulating the medical image data (e.g., CT images) to highlight or change the contrast of some anatomical structures.

[0070] The computer system can apply, for medical each image, an entropy filter (or entropy filtering) to the corresponding lung and mediastinum window images for each patient. The computer system can apply a local entropy filter to the lung and mediastinum window images to generate filtered images that capture or emphasize subtle variations in texture of the images under different window settings. In some implementations, the computer system can apply other types of filtering or processing to reveal texture variations. For example, the computer system can use wavelets or the cosine transform, among others, to determine texture variations.

[0071] The computer system can generate, for each pair of lung and mediastinum window images, a corresponding composite image by combining the lung and mediastinum window images as well as the corresponding filtered lung and mediastinum window images. The computer system can employ an image fusion techniques to combine the lung window image, mediastinum window image, the corresponding filtered images or a combination thereof to form the final composite image. The computer system can generate for each patient or for each original image a corresponding composite image.

[0072] The method 300 can include the computer system determining coherent sub-regions of the plurality of medical images (STEP 304). The computer system can use or apply habitat imaging techniques to determine the coherent sub-regions. In particular, the computer systemcan generate habitat maps of the original medical images. The habitat maps can be viewed as representing a segmentation of the medical images and depicting a plurality of image subregions determined based on similarity in imaging patterns across the plurality of medical images. The computer system can carry the segmentation into sub-regions or habitat detection by clustering or aggregating pixels or groups of pixels to identify or form sub-regions with similar imaging patterns. For a given sub-region, the computer system can assess the similarity in image characteristics or imaging patterns across images of the enrolled patients. In other words, while the sub-regions are defined for each patient and may have different sizes and / or different shapes for different patients, a sub-region represents similar image characteristics or imaging patterns across different patients.

[0073] Referring to step 404 of FIG. 4, the habitat detection can include a patient-level clustering and a population-level clustering. For the patient-level clustering, the computer system can over segment the lung region (e.g., left and right lungs) within each composite image of a corresponding patient into a large number of superpixels. As used herein, superpixels are the result of perceptual grouping of pixels, carry more information than pixels and align better with image edges than rectangular image blocks. Also, superpixels have a perceptual meaning since pixels belonging to a given superpixel share similar visual properties. In some implementations, the computer simple can use linear iterative clustering (SLIC) algorithm to over segment the composite images of the plurality of patients or individuals. In some implementations, the computer system can use other over segmentation techniques or methods. In general, the over segmentation includes the computer system clustering or aggregating neighboring pixels into image segments referred to as superpixels, such that each segment represents a group of neighboring pixels sharing similar visual properties and adheres to natural boundaries in the image.

[0074] The population-level clustering can include the computer system identifying superpixels of the lung region within each composite image and across composite images of plurality of patients having or sharing similar visual or image properties, and clustering or aggregating superpixels of the lung region within each composite image sharing similar visual properties into a corresponding sub-region. A sub-region can have different instances (e.g., different shape, different size and / or different number of clusters) within composite images of different patients. However, a sub-region depicts similar visual properties or attributes within a single composite image and across composite images of the plurality of patients orindividuals. The computer system can use a hierarchical clustering method or algorithm to identify sub-regions with similar imaging patterns, similar image properties or similar visual properties, and cluster each group of superpixels sharing similar image or visual properties into a corresponding sub-region. Each sub-region may be indicative of lung tissues having similar physiological or clinical properties.

[0075] In the population-level clustering process, the computer system can treat or consider each superpixel as an individual sample. The image features or visual properties of each superpixels can be defined as the average values of the corresponding image features or visual properties of the set of pixels forming the superpixel. For example, the intensity of a superpixel can be defined as the average intensity of the group of pixels forming the superpixel. In some implementations, a combination of statistical metrics such as mean, median, variance, standard deviation and / or other statistical metric(s) can be used to define the image or visual properties of a superpixel based on the corresponding image or visual properties of the pixels forming the superpixel. The computer system can carry out the hierarchical clustering method or algorithm using the feature values for the superpixels. In determining superpixels belonging to a given sub-region of the lung region, the computer system takes into consideration superpixels in the composite images of the plurality of patients or individuals. In other words, a sub-region of the lung region can be viewed as a class having separate instances in various composite images of the plurality of patients.

[0076] The computer system can determine the optimal number of sub-regions using both the gap criterion and the hierarchical structure of the clusters dendrogram. Gap criterion is a clustering evaluation technique that computes the difference of expected logarithm of the within cluster dispersion determined by Monte Carlo sampling and the logarithm of the within cluster dispersion computed from the sample data. The hierarchical structure of cluster dendrogram is a hierarchical clustering technique used by the computer system to determine the optimal number of clusters by pruning the tree like dendrogram to partition the data samples into clusters. Basically, samples that are very similar are densely packed in certain areas of the dendrogram and not in others, and this natural division (hierarchical structure of the clusters dendrogram) is used to decide the number of clusters in the data.

[0077] While steps 402 and 404 of FIG. 4 indicate that the habitats are determined based on the composite images, in general, the computer system can determine the habitats or the subregions of the lung region using the original medical images, filtered versions of the originalmedical images, the lung and mediastinum window images, filtered lung and mediastinum window images, the composite images or a combination thereof. For example, the computer system can apply the local entropy filter to the original medical images and carry out the patient-level clustering and the population-level clustering based on pixels and superpixels of the lung region. Using patient-level clustering and population-level clustering makes the determination of habitats or sub-region more computationally efficient. However, the computer system may apply hierarchical clustering at the pixel level and across images of the plurality of patients.

[0078] Referring to FIG. 5, a heatmap of superpixels of five identified sub-regions together with distribution plots (boxplots) of five representative features is shown, according to embodiments described herein. The representative features are used by the computer system to determine or identify the five habitat sub-regions. The representative features include intensity uniformity, texture of the entropy map, intensity, symmetricity and texture of the lung window. Table 4 below describes characteristics of the five habitat sub-regions in terms of the representative features. After computing the local entropies of the lung and mediastinum values, the computer system can compute features such as skewness, kurtosis, standard deviation, etc., to further characterized the intensity, texture and symmetricity of the medical images (e.g., CT images). Specifically, for each lung, entropy lung, mediastinum and entropy mediastinum, the computer system can compute 10 more features including, skewness, kurtosis, mean, median, ql = 0.25 quantile, q2 = 0.75 quantile, iqr = q2 - ql, standard deviation, variance and energy and provide the computed features as inputs to the hierarchical clustering algorithm to determine the habitat subregions. The hierarchical clustering algorithm takes in as input, the features computed from the fused images. The distribution plots (boxplots) depict how the different representative features vary across the different subregions, e.g., to show how the different features are able to capture some discriminating structures differentiating the different habitat subregions.Table 4. Image characteristics of habitat sub-regions

[0079] The optimal number of intra-lung sub-regions or habitats is determined to be equal to five according to hierarchical structure of the dendrogram. The representative features shown in FIG. 5 and Table 4 represent the imaging parameters that underline and differentiate the identified habitats. The distribution plots shown in FIG. 5 depict the detailed distributions of the representative features or imaging parameters (intensity uniformity, Intensity, texture of lung window, texture of entropy map, symmetricity of intensity and texture) in each of the habitats. The detailed phenotypical patterns associated with each habitat are summarized in Table 4.

[0080] It was observed that sub-region 5 corresponds to the uninfected lung parenchyma, sub-region 2 corresponds to certain lung parenchyma with elevated texture heterogeneity on entropy map, sub-region 1 corresponds to the consolidation, sub-regions 3 and 4 correspond to the transition zone to ground glass opacity (GGO) at different degrees. The detailed partitioning results of entire lung region after habitat analysis are presented in FIG. 6 for four selected patients, where detailed habitats are consistently defined to quantify the infection patterns.

[0081] FIG. 6 shows images of four patients at various stages of the method described in FIGS. 3 and 4, according to example embodiments herein. The four patients include one patient having pneumonia (i.e., patient 1) and three patients suffering from pneumonitis (i.e., patients 2, 3 and 4). The first and second columns depict lung window images and entropy filtered lung window images, respectively, of the four patients. The third and fourth columns depict mediastinum window images and entropy filtered mediastinum window images, respectively, of the four patients. The fifth column depicts composite or fused images of the four patients. The sixth and seventh columns depict superpixel images and habitat maps, respectively, of the four patients.

[0082] The method 300 can include the computer system determining a plurality of image features of the coherent sub-regions (STEP 306). In some implementations, the computer system can determine a plurality of first features from the habitat maps and a plurality of second features indicative of how symmetric the plurality of first features across left and right lungs. The plurality of first features can include multiregional spatial interaction (MSI) features determined or computed from the habitat maps to quantify overall lung parenchyma. The MSIfeatures refer to features capturing connections, intersections or between different subregions of the habitat maps. If two pixels from two subregions are connected to each other, then there is an interaction between these subregions. The MSI features can be configured or designed to quantify the spatial heterogeneity of infected lung patterns. For instance, the MSI features can capture or reveal information such as the absolute burden (total lesion, or habitat subregion, volume) and relative percentage (e.g., relative percentage of a habitat region is compared to other identified habitat subregions) of individual habitats as well as their interactions. Table 2 below provides a list and descriptions of 44 MSI features extracted from the habitat maps. The list shown in Table 2 is illustrative. Additional or alternative features can be extracted from the habitat maps.Table 2. Multiregional spatial interaction (MSI) features.

[0083] The computer system can also determine a plurality of second features quantifying the symmetric differencebetween the left and right lungs for various MSI features.Given an MSI feature denoted as “F,” the symmetric difference As-ymfor the feature F can be defined as:where FL, FW and FR denote the MSI feature F computed on the whole lung region, left lung and right lung, respectively. One strength of the MSI features is their clear interpretations. These features are designed to quantify the spatial heterogeneity of infected lung patterns.Table 3. Features depicting symmetric differences.

[0084] As depicted in step 406 of FIG. 4, the method 300 can include the computer systemranking the extracted features, e.g., plurality of first features and the plurality of second features, based on correlations of each feature with pneumonia infection and pneumonitis infection, and select a subset of the extracted features based on the raking. The computer system can apply a Chi-square test to each feature of the extracted features to determine a corresponding probability of pneumonia and / or a corresponding probability of pneumonitis infection, and rank the features based on the determined probabilities. The computer system can employ univariate feature ranking using Chi-square test to examine whether each feature is independent of the outcome. In particular, the computer system can apply a univariate Chi- square test to examine feature association with infection types (e.g., pneumonia and pneumonitis). The computer system can test each feature independently. The output of the univariate Chi-square test for each feature is the probability that differences in that feature between pneumonia and pneumonitis occurred by chance alone (p-value). The lower the p- value, the less likely a given result occurred by chance. A p-value of less than 0.05 is often considered to represent a statistically significant difference, and the interpretation of such a p- value is that the differences between features that we observed only had a 5% chance of being due to random variation in data as opposed to representing real differences. A small p-value indicates the corresponding feature is dependent (very important) on the outcome. The computer system can use the computed p-values of all the features to rank the individual features. For instance, the computer system can compute, for each feature, a corresponding feature importance score defied as score = -log(p), where p is the corresponding p-value for each feature. A higher score denotes greater importance of the feature.

[0085] The method 300 can include the computer system generating a machine learning model for pneumonia and / or pneumonitis detection or diagnosis (STEP 308). The computer model can select a subset of the extracted features, based on feature ranking and / or feature importance scores, for use to generate the machine learning model. In some implementations, the computer system can iteratively increase the number of selected features based on the features’ importance scores in order to identify the optimal diagnostic model. The computer system can use the top-ranked features to build an ensemble of boosted classification trees (or boosted decision trees). In some implementations, the machine learning model can include an ensemble of 100 boosted classification trees. In some implementations, the machine learning model can include another type of classifier, a neural network,

[0086] To avoid biased classification to the pneumonitis class due to data imbalance (i.e.,significantly larger number of samples in the pneumonia class), the computer system can employ synthetic minority oversampling technique (SMOTE) nested with leave-one-out cross validation (LOOCV) approach together to validate model performance. To avoid information leakage, the computer system can first leave out one sample as the test set before applying SMOTE on the training set to train a prediction model. This process can be repeated n-times (n equals to total number of samples) until every data sample is left out as a test sample.

[0087] As discussed above with regard to FIG. 1, a benchmark model is built using clinical and blood-based metrics. Clinical variables or metrics include cough, fever, shortness of breath at both baseline and at time of syndrome together with age and sex. Blood-based variables or metrics include absolute white blood cells (WBC) count, absolute neutrophils count (ANC), absolute lymphocyte count (ALC) and platelet count at both baseline and time of syndrome together with bone marrow blast cells count at baseline. The log transformations of WBC and platelets at both baseline and at time of syndrome are considered. Given the clinical and blood measures, a similar SMOTE and LOOCV machine learning strategy was adopted to build the benchmark model. The diagnostic performance of the benchmark model is compared to the habitat imaging model. Furthermore, the computer system can integrate the clinical / blood benchmark model and the habitat imaging model into a composite model. For instance, the composite model can include a cascade of the clinical / blood benchmark model and the habitat imaging model. In some implementations, the computer system can generate the machine learning model using the habitat features (e.g., features listed in Tables 1 and 2), clinical variables or metrics and / or blood variables or metrics (e.g., blood analysis results or metrics) of the patients. In particular, the computer system can rank, or compute importance scores of, the habitat features and the clinical / blood metrics, and select a subset from the total features for use to generate the machine learning model for detection or diagnosis of pneumonia and / or pneumonitis.

[0088] The performances of the clinical benchmark model, the habitat imaging model and the composite model are evaluated and compared. Furthermore, another radiomics-based prediction model is built using classical radiomics features extracted from the whole lung regions (e.g., without habitat analysis), and the corresponding performance is evaluated and compared to other models described herein.

[0089] FIGS. 7A-7C show performance results of different diagnostic models, according to embodiments described herein. FIG. 7A shows the feature importance and confusion matrixfor the clinical / blood benchmark model. FIG. 7B shows the feature importance and confusion matrix for the habitat imaging-based model. FIG. 7C shows the feature importance and confusion matrix for the composite prediction / diagnosis model.

[0090] The importance scores for various clinical and / or blood variables or metrics are shown in FIG. 7A. The clinical benchmark model is built using the features depicted with the corresponding importance scores. The top ranked features include sex, cough, ANC at time of syndrome, baseline platelets and platelets at time of syndrome. This clinical benchmark model has an accuracy of 68%, sensitivity (i.e., true positive of pneumonitis) of 29%, and specificity (true negative of pneumonitis) of 79%.

[0091] As discussed above, MSI features characterizing the overall infection patterns are extracted, based on the habitat map for individual patients, as well as their symmetricity between left and right lungs, which resulted in a total of 88 features. The correlation among these habitat features are presented in FIGS. 8A and 8C. Specifically, FIGS 8A-8D depict the correlation heatmaps for features used to build the different diagnostics models. FIG. 8A shows the correlation heatmap for the clinical / blood features. FIG. 8B shows the correlation heatmap for the MSI features extracted from the habitat maps. FIG. 8B shows the correlation heatmap for the MSI features measuring the symmetricity between the left and right lung, FIG. 8D shows the correlation heatmap for the combination of the whole features. Note that FIG. 8D shows only the correlation among the clinical / blood metrics and the MSI 1 - MSI 20 features due to the large number of features.

[0092] The machine learning model or the classifier to differentiate pneumonia from pneumonitis can be built using LOOCV and the features with importance scores depicted in FIG. 7B. The top ranked habitat features are MSI41 and MSI85, which relates to the interaction between habitat 5 (normal lung parenchyma) and habitat 2 (heterogeneously abnormal lung parenchyma). Using this approach, it was found that pneumonia had elevated asymmetric interaction, suggesting more asymmetry in the radiographic pattern between left and right lung. The habitat model achieved an accuracy of 78%, sensitivity of 43%, and specificity of 89%, based on the cross-validated confusion matrix in FIG. 7B.

[0093] For comparison purposes and as discussed above, a conventional radiomics model was built and the corresponding performance was evaluated. The feature importance and confusion matrix for the conventional radiomics model is shown in FIG. 10. The conventionalradiomics model was found to have a significantly lower performance (accuracy, 58%; sensitivity, 64%; specificity, 57%). The prediction results for the clinical benchmark model, the habitat imaging-based model and the conventional radiomics model are shown in FIG. 9A.

[0094] The composite model of FIG. 9B is a cascading model with clinical benchmark model in the first layer and habitat imaging-based model in the second layer. The predicted infection type stratification is simulated as shown in FIG. 9B. Setting up a rule where pneumonia is detected if both models predicted pneumonia, and pneumonitis is detected if any model predicts pneumonitis, the cascading model has 85% accuracy in predicting pneumonia and 39% accuracy in predicting pneumonitis, which has a greater than 5% increase in pneumonia detection and greater than 10% increase in pneumonitis compared to benchmark clinical and blood model.

[0095] FIG. 7C shows the feature importance and confusion matrix for another composite prediction / diagnosis model, where the habitat features are mixed with clinical and blood metrics together to re-fit a prediction model. The ranked feature importance table for the mixed features is shown in FIG 7C. The ranked feature importance table shows that the habitat features are consistently more important than the clinical and blood metrics. The composite model achieved the optimal performance with overall accuracy of 81%, sensitivity of 29% and specificity of 96%.

[0096] Pre-test and post-test probabilities of detecting pneumonitis using different models were evaluated. The pre-test and post-test probabilities are depicted in Table 4. The pre-test probability (i.e., prevalence) of pneumonitis was observed to be 22.4%. With the clinical benchmark prediction model using clinical and blood metrics, the post-test probability increased to 28.5% if the model diagnosed pneumonitis. When using our habitat imagingbased model, the post-test probability was increased to 53% if the model diagnosed pneumonitis. Synergistic effects can be observed when combining habitat imaging with clinical and blood metrics into a composite model, which achieved the best post-test probability of 67.7% if the model diagnosed pneumonitis; a 3-fold increase from pre-test probability. By contrast, the classical radiomics model had a lower performance with post-test probability of 30.1%.Table 4. Pre-test and post-test probability comparison among different diagnostic models.

[0097] Recognizing the difficulty in distinguishing pneumonia and pneumonitis in the absence of definitive biomarkers, an imaging-based pipeline was developed that could distinguish these two types of infection with a much-improved accuracy using a well- characterized cohort of patients with AML undergoing ICI therapy. The proposed imaging marker has significantly outperformed a benchmark model based on clinical blood metrics. Further, a synergy between the imaging markers and blood markers was observed, and their integration into a joint model has achieved the best prediction. All in all, the pilot study serves as proof-of-concept to demonstrate that machine learning of radiographic scans can offer complementary values on top of existing clinical biomarkers for improved management of immune-related adverse events (irAE).

[0098] Diagnosing pneumonitis in real-time is challenging, and the prompt differentiation of pneumonitis from other conditions, such as pneumonia or cancer progression, is not always possible from imaging information alone. Additional tests may be desired, but these results may further delay the prompt administration of definitive therapy toward pneumonia or pneumonitis, potentially leading to patient harm. For example, prompt administration of antibiotics for community acquired pneumonia decreases in-hospital mortality. Improving the ability of interpreting radiologists to diagnose pneumonitis may lead to improved patient outcomes. Tools such as the one we highlight in this work can potentially augment the capabilities of interpreting so to help radiologists make definitive image-based diagnoses. Approaches that combine artificial intelligence imaging tools with clinical radiologists often exceed the accuracy seen with human evaluations, as has been shown when determining the probability that a lung nodule is malignant or whether reticular abnormalities represent interstitial lung disease. One can envision that this tool may reduce the uncertainty seen when trying to differentiate pneumonia and pneumonitis in real time, but further studies are needed to validate this.

[0099] Pneumonitis is a serious complication of checkpoint inhibitor immunotherapy, andthe mortality ranges from 10-20% in non-small cell lung cancer cohorts (NSCLC), where ICIs are frequently used to nearly 50% in AML. Pneumonitis is likely to be more amenable to treatment if detected early and distinguished from pneumonia. The treatments for pneumonitis, namely immunosuppressive therapies of appropriate intensity and duration, are substantively different from the treatments for pneumonia. Furthermore, overuse of antimicrobial agents in patients without pneumonia may alter the intestinal microbiome, potentially reducing the efficacy of ICIs. In our original report, 28 / 31 cases of pneumonitis were treated with both corticosteroids and antibiotic therapies. While pneumonia was five-fold more common in our cohort than pneumonitis, promptly distinguishing these two conditions will benefit all patients undergoing ICI therapy for cancer, regardless of the underlying rates of these two conditions. Further, diagnosis of pneumonitis with histopathology in nearly all cases due to the concern for bleeding due to thrombocytopenia or the concern for pulmonary deterioration after a biopsy procedure.

[0100] Radiographic patterns associated with immune checkpoint inhibitor related pneumonitis may resemble interstitial lung diseases seen in the general population, including organizing pneumonia, interstitial pneumonitis, and others. The patterns that may be seen in these diseases is highly variable from case to case, as others have shown. Radiomics has been used to predict the risk of developing ICI-induced pneumonitis based on baseline CT scans from 2 patients who developed pneumonitis and 30 who did not, but not to differentiate pneumonitis from other lung diseases. In this study, we have implemented the habitat imaging algorithm to differentiate pneumonia and pneumonitis. Compared to conventional radiomics, the key strength of our habitat imaging analysis is that it explicitly accounts for the spatial heterogeneity of the infected lung and partitions the whole lung regions into phenotypically distinct sub-regions. By analyzing these sub-regions individually as well as their interactions, we have demonstrated its superior performance in separating pneumonitis from pneumonia. Analogous to the superior multiregional gene sequencing over conventional cocktail sequencing, a fine-grained spatial analysis enabled by habitat imaging can reveal new insights to improve the pneumonitis diagnosis. By contrast, traditional radiomics extracts features (including texture) from the entire lung region but cannot capture the degree of intra-lung infection heterogeneity. This may explain why our habitat imaging approach outperformed conventional radiomics.

[0101] Our study has several strengths. This is the first tool of its kind and is positioned toaddress a significant problem that hinders the treatment of all patients undergoing ICI therapy. The tool was developed by incorporating CT images that used diverse acquisition protocols, and thus can be more easily applied and validated in external cohorts. Also, another strength of our study is the strict selection of patients with AML under immunotherapy. AML patients do not have solid malignancies in their lung regions to confound the imaging analysis, which is different from solid tumors (e.g., NSCLC).

[0102] Several limitations must be considered. First, the results presented in this manuscript would benefit from an external cohort for model validation. Second, while all cases of pneumonia and pneumonitis in this study were confirmed by an expert multidisciplinary cohort, the accuracy of this tool needs to be confirmed in a prospective cohort where the appropriate testing, especially CT imaging of the chest and universal BAL, are performed promptly and systematically. Third, it is likely that blood and clinical markers that associate with pneumonitis will vary from cohort to cohort, which would make an approach that only utilizes imaging more attractive. Fourth, other lung processes such as disease progression or radiation injury are more applicable to solid tumors but not seen in the current cohort treated for AML, and therefore this tool must be validated before applying in patients with solid tumors such as non-small cell lung cancer. Also, it is possible that pneumonia and pneumonitis may co-exist in some patients, and it is not uncommon for more than on serious adverse event to manifest concurrently in AML patients 30. There is no “gold standard” to diagnose pneumonia, and it remains a clinical diagnosis. Therefore, it is possible that the multidisciplinary adjudication of pneumonia and pneumonitis were erroneous in some instances.

[0103] In conclusion, the developed tool could accurately distinguish pneumonia and pneumonitis in AML patients treated with ICI inhibitors. If validated, the proposed approach holds great promise to improve the clinical care of cancer patients treated with ICIs by improving our ability to differentiate pneumonitis from other lung diseases in a prompt fashion.

[0104] While embodiments described herein focus on distinguishing pneumonitis from pneumonia, the same approach can be applied to or extended to distinguish pneumonitis from lung cancer progression or from non-pneumonitis conditions. More generally, the model can be extended or trained to distinguish any two lung related diseases so far as the diseases cause some patterns (e.g., lesions) in the lung regions.C. Image-Based Detection of Pneumonia and / or Pneumonitis

[0105] Referring to FIG. I l a flow chart illustrating a method 700 for image-based detection or diagnosis of pneumonia and / or pneumonitis, according to embodiments described herein. In brief overview, the method can include receiving a medical image of lungs of a patient (STEP 702), determining a plurality of sub-regions of the lungs of the patient based on predefined image properties (STEP 704), determining predefined image features of the subregions (STEP 708), and determining whether the patient has pneumonia or pneumonitis using a trained diagnosis model and the determined image features of the sub-regions (STEP 708).

[0106] The method 700 can be executed by a computer system, such as the server system 200, the client computing system 214, components thereof (e.g., one or more processors) or a combination thereof. The method 700 can be implemented as computer code instructions that are executed by one or more processors of the computer system. In some implementations, all the steps of the method 700 can be executed by the same set of processors. In some implementations, separate combinations of processors can execute different steps or different combinations of steps of method 700. In some implementations, the method 700 can be implemented using hardware circuits.

[0107] The method 700 can include the computer system receiving one or more medical images of lungs of a patient (STEP 702). The medical image(s) can include CT scans of the plurality of individuals. In some implementations, the medical images can include CT scans, magnetic resonance (MR) images, ultrasound images, optical tomography (OT) images, electrical impedance tomography (EIT) images, multi-modality images, other types of medical images or a combination thereof. In some implementations, the type of medical images used can be similar to a type of images used to train the trained diagnosis model. The computer system can receive the medical images as input, e.g., via an input interface, or can retrieve the medical images from a database or a storage device.

[0108] The computer system can generate, for each pair of lung and mediastinum window images, a corresponding composite image by combining the lung and mediastinum window images as well as the corresponding filtered lung and mediastinum window images. The computer system can employ image fusion techniques to combine the lung window image, mediastinum window image, the corresponding filtered images or a combination thereof to form the final composite image. The computer system can generate for each original image of the patient a corresponding composite image.

[0109] The computer system can segment each of the plurality of medical images to determine, for each image, the corresponding lung region, including the left lung parenchyma and the right lung parenchyma. In some implementations, the computer system can use a deep learning-based segmentation model (e.g., similar to the model discussed in relation with STEP 302) to determine the left and right lung parenchyma. The computer system can apply a morphological dilation and / or erosion to smoothen the boundaries of the segmented lung regions. In some implementations, radiologists and / or other physicians can review and correct the lung ROI segmentations. In some implementations, the computer system can employ other segmentation techniques.

[0110] The computer system can employ similar preprocessing to the medical image(s) of the patient as depicted in step 402 of FIG. 4. The computer system can extract, from each medical image of the patient, a corresponding first image depicting the corresponding left and right lung parenchyma and a corresponding lung view and a corresponding mediastinum view. The computer system can apply one or more digital contrast-enhancing methods or techniques to generate the optimal viewing conditions for corresponding images for the lungs and mediastinum.[OHl] The computer system can apply, for each medical image, an entropy filter (or entropy filtering) to the corresponding lung and mediastinum window images for each patient. The computer system can apply a local entropy filter to the lung and mediastinum window images of the patient to generate filtered images that capture or emphasize subtle variations in texture of the images under different window settings. In some implementations, the computer system can apply other types of filtering or processing, e.g., wavelets or cosine transform, to reveal texture variations.

[0112] The computer system can generate, for each pair of lung and mediastinum window images of the patient, a corresponding composite image by combining the lung and mediastinum window images as well as the corresponding filtered lung and mediastinum window images. The computer system can employ image fusion techniques to combine the lung window image, mediastinum window image, the corresponding filtered images or a combination thereof to form the final composite image. The computer system can generate for each original image a corresponding composite image.

[0113] The method 700 can include the computer system determining a plurality of sub-regions of the lungs of the patient based on predefined image properties (STEP 704). The computer system can use or apply habitat imaging techniques to determine the coherent subregions. In particular, the computer system can generate habitat maps of the original medical images. The computer system can carry the segmentation into sub-regions or habitat detection by clustering or aggregating pixels or groups of pixels to identify or form sub-regions with similar imaging patterns. The habitat detection can include the computer system applying a patient-level clustering and population level clustering as discussed above in relation to FIG. 3. The computer system can use characteristics of the sub-regions of the lungs determined during the training phase of the diagnosis model to perform the habitat detection. In other words, computer system can use characteristics of the habitats identified during the training phase of the diagnosis model to habitat maps of the patient. The computer system can use any of the techniques described in relation with step 304 of FIG. 3 to generate the habitat mapping.

[0114] The method 700 can include the computer system determining predefined image features of the sub-regions (STEP 708) and determining whether the patient has pneumonia or pneumonitis using a trained diagnosis model and the determined image features of the subregions (STEP 708). The computer system can extract or determine the image features discussed in relation with step 306 of FIG. 3 from the habitat map(s) of the patient. The features can include MSI features as shown above in Table 2 and features quantifying the symmetric difference between the left and right lungs for the various MSI features as shown above in Table 3. In some implementations, the computer system can extract or determine similar image features as those used to train the diagnosis model.

[0115] The computer system can provide the determined or extracted features as input to the trained diagnosis model and execute the diagnosis model. The diagnosis model can provide, as output, an indication pneumonitis or an indication of pneumonia. In some implementations, the diagnosis model can provide, as output, an indication pneumonitis or an indication of other lung disease. In some implementations, the diagnosis model can provide, as output, an indication of a probability or likelihood of pneumonitis or an indication of a probability or likelihood of pneumonia (or other lung disease).

[0116] Non-limiting examples of various embodiments are disclosed herein. Features from one embodiments disclosed herein may be combined with features of another embodiment disclosed herein as someone of ordinary skill in the art would understand.

[0117] As utilized herein, the terms “approximately,” “about,” “substantially” and similar terms are intended to have a broad meaning in harmony with the common and accepted usage by those of ordinary skill in the art to which the subject matter of this disclosure pertains. It should be understood by those of skill in the art who review this disclosure that these terms are intended to allow a description of certain features described without restricting the scope of these features to the precise numerical ranges provided. Accordingly, these terms should be interpreted as indicating that insubstantial or inconsequential modifications or alterations of the subject matter described and are considered to be within the scope of the disclosure.

[0118] For the purpose of this disclosure, the term “coupled” means the joining of two members directly or indirectly to one another. Such joining may be stationary or moveable in nature. Such joining may be achieved with the two members or the two members and any additional intermediate members being integrally formed as a single unitary body with one another or with the two members or the two members and any additional intermediate members being attached to one another. Such joining may be permanent in nature or may be removable or releasable in nature.

[0119] It should be noted that the orientation of various elements may differ according to other exemplary embodiments, and that such variations are intended to be encompassed by the present disclosure. It is recognized that features of the disclosed embodiments can be incorporated into other disclosed embodiments.

[0120] It is important to note that the constructions and arrangements of apparatuses or the components thereof as shown in the various exemplary embodiments are illustrative only. Although only a few embodiments have been described in detail in this disclosure, those skilled in the art who review this disclosure will readily appreciate that many modifications are possible (e.g., variations in sizes, dimensions, structures, shapes and proportions of the various elements, values of parameters, mounting arrangements, use of materials, colors, orientations, etc.) without materially departing from the novel teachings and advantages of the subject matter disclosed. For example, elements shown as integrally formed may be constructed of multiple parts or elements, the position of elements may be reversed or otherwise varied, and the nature or number of discrete elements or positions may be altered or varied. The order or sequence of any process or method steps may be varied or re-sequenced according to alternative embodiments. Other substitutions, modifications, changes and omissions may also be made in the design, operating conditions and arrangement of the various exemplary embodimentswithout departing from the scope of the present disclosure.

[0121] While various inventive embodiments have been described and illustrated herein, those of ordinary skill in the art will readily envision a variety of other mechanisms and / or structures for performing the function and / or obtaining the results and / or one or more of the advantages described herein, and each of such variations and / or modifications is deemed to be within the scope of the inventive embodiments described herein. More generally, those skilled in the art will readily appreciate that, unless otherwise noted, any parameters, dimensions, materials, and configurations described herein are meant to be exemplary and that the actual parameters, dimensions, materials, and / or configurations will depend upon the specific application or applications for which the inventive teachings is / are used. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific inventive embodiments described herein. It is, therefore, to be understood that the foregoing embodiments are presented by way of example only and that, within the scope of the appended claims and equivalents thereto, inventive embodiments may be practiced otherwise than as specifically described and claimed. Inventive embodiments of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent, is included within the inventive scope of the present disclosure.

[0122] Also, the technology described herein may be embodied as a method, of which at least one example has been provided. The acts performed as part of the method may be ordered in any suitable way unless otherwise specifically noted. Accordingly, embodiments may be constructed in which acts are performed in an order different than illustrated, which may include performing some acts simultaneously, even though shown as sequential acts in illustrative embodiments.

[0123] The indefinite articles “a” and “an,” as used herein in the specification and in the claims, unless clearly indicated to the contrary, should be understood to mean “at least one.” As used herein in the specification and in the claims, “or” should be understood to have the same meaning as “and / or” as defined above. For example, when separating items in a list, “or” or “and / or” shall be interpreted as being inclusive, i.e., the inclusion of at least one, but also including more than one, of a number or list of elements, and, optionally, additional unlisteditems. Only terms clearly indicated to the contrary, such as “only one of’ or “exactly one of’ will refer to the inclusion of exactly one element of a number or list of elements. In general, the term “or” as used herein shall only be interpreted as indicating exclusive alternatives (i.e. “one or the other but not both”) when preceded by terms of exclusivity, such as “either,” “one of,” “only one of,” or “exactly one of.”

[0124] As used herein in the specification and in the claims, the phrase “at least one,” in reference to a list of one or more elements, should be understood to mean at least one element selected from any one or more of the elements in the list of elements, but not necessarily including at least one of each and every element specifically listed within the list of elements and not excluding any combinations of elements in the list of elements. This definition also allows that elements may optionally be present other than the elements specifically identified within the list of elements to which the phrase “at least one” refers, whether related or unrelated to those elements specifically identified. Thus, as a non-limiting example, “at least one of A and B” (or, equivalently, “at least one of A or B,” or, equivalently “at least one of A and / or B”) can refer, in one embodiment, to at least one, optionally including more than one, A, with no B present (and optionally including elements other than B); in another embodiment, to at least one, optionally including more than one, B, with no A present (and optionally including elements other than A); in yet another embodiment, to at least one, optionally including more than one, A, and at least one, optionally including more than one, B (and optionally including other elements); etc.

Claims

What is claimed is:

1. A system comprising: one or more processors; and a memory storing computer instructions, the computer instructions when executed by the one or more processors cause the system to: acquire a plurality of medical images of lungs of a plurality of individuals, the plurality of individuals including a first set of individuals having pneumonia and a second set of individuals having pneumonitis; generate habitat maps of the medical images, the habitat maps representing a segmentation of the medical images and depicting a plurality of image sub-regions determined based on similarity in imaging patterns across the plurality of medical images; determine a plurality of first features from the habitat maps; determine a plurality of second features indicative of how symmetric the plurality of first features across left and right lungs; and generate, using the plurality of first features and the plurality of second features, a machine learning model configured to detect at least one of pneumonia or pneumonitis.

2. The system of claim 1, wherein the one or more processors are configured to cause the system to segment each of the plurality of medical images to determine corresponding left and right lung parenchyma.

3. The system of claim 1, wherein the one or more processors are configured to cause the system to: extract, from each image of the plurality of images, a corresponding first image depicting a corresponding left and right lung parenchyma; extract, from each image of the plurality of images, a corresponding second image depicting a corresponding mediastinum region; apply, for each image of the plurality of images, an entropy filter to the corresponding first image and the corresponding second image; and generate, for each image of the plurality of images, a corresponding composite image by combining the filtered corresponding first image and the filtered corresponding second image.

4. The system of claim 1, wherein the one or more processors are configured to cause the system to use hierarchical clustering to determine the plurality of image sub-regions.

5. The system of claim 4, wherein the one or more processors are configured to cause the system to: cluster pixels associated with each image of the plurality of images into superpixels; and apply the hierarchical clustering to the superpixels using superpixel characteristics across the plurality of images.

6. The system of claim 1, wherein the plurality of first features include features indicative of spatial characteristics of the plurality of subregions.

7. The system of claim 1, wherein the one or more processors are configured to cause the system to: rank the plurality of first features and the plurality of second features based on correlations of each feature with pneumonia infection and pneumonitis infection; and select a subset of the plurality of first features and the plurality of second features based on the raking, the selected subset used to generate the machine learning model.

8. The system of claim 7, wherein the one or more processors are configured to cause the system to: apply a Chi-square test to each feature of the plurality of first features and the plurality of second features to determine a corresponding probability of pneumonia or pneumonitis infection; and rank the plurality of first features and the plurality of second features based on corresponding probabilities of pneumonia or pneumonitis infection.

9. The system of claim 1, wherein the machine learning model includes an ensemble of boosted decision trees.

10. The system of claim 1, wherein the one or more processors are configured to cause the system to generate the machine learning model further using at least one of clinical parameters or blood analysis results of the individuals.

11. The system of claim 10, wherein the blood analysis results include at least one of absolute white blood cell (WBC) count, absolute neutrophils count, absolute lymphocyte count, platelet count or bone marrow blast cell count.

12. The system of claim 10, wherein the clinical parameters include at least one of age, gender, cough, fever or shortness of breath.

13. A method compri sing : acquiring a plurality of medical images of lungs of a plurality of individuals, the plurality of individuals including a first set of individuals having pneumonia and a second set of individuals having pneumonitis; generating habitat maps of the medical images, the habitat maps representing a segmentation of the medical images and depicting a plurality of image sub-regions determined based on similarity in imaging patterns across the plurality of medical images; determining a plurality of first features from the habitat maps; determining a plurality of second features indicative of how symmetric the plurality of first features across left and right lungs; and generating, using the plurality of first features and the plurality of second features, a machine learning model configured to detect at least one of pneumonia or pneumonitis.

14. The method of claim 13, comprising: segmenting each of the plurality of medical images to determine corresponding left and right lung parenchyma. extracting, from each segmented image of the plurality of images, a corresponding first image depicting a corresponding left and right lung parenchyma; extracting, from each segmented image of the plurality of images, a corresponding second image depicting a corresponding mediastinum region; applying, for each image of the plurality of images, an entropy filter to the corresponding first image and the corresponding second image; andgenerating, for each image of the plurality of images, a corresponding composite image by combining the filtered corresponding first image and the filtered corresponding second image, the habitat maps generated using the corresponding composite images.

15. The method of claim 13, wherein the generating the habitat maps includes using hierarchical clustering to determine the plurality of image sub-regions.

16. The method of claim 15, comprising: clustering pixels associated with each image of the plurality of images into superpixels; and applying the hierarchical clustering to the superpixels using superpixel characteristics across the plurality of images.

17. The method of claim 13, comprising: ranking the plurality of first features and the plurality of second features based on correlations of each feature with pneumonia infection and pneumonitis infection; and selecting a subset of the plurality of first features and the plurality of second features based on the raking, the selected subset used to generate the machine learning model.

18. The method of claim 17, comprising: applying a Chi-square test to each feature of the plurality of first features and the plurality of second features to determine a corresponding probability of pneumonia or pneumonitis infection; and ranking the plurality of first features and the plurality of second features based on corresponding probabilities of pneumonia or pneumonitis infection.

19. The method of claim 13, wherein the machine learning model includes an ensemble of boosted decision trees.

20. The method of claim 13, comprising generating the machine learning model further using at least one of clinical parameters or blood analysis results of the individuals.

21. A non-transitory computer-readable medium storing computer instructions, the computer instructions when executed by one or more processors cause the one or more processors to:acquire a plurality of medical images of lungs of a plurality of individuals, the plurality of individuals including a first set of individuals having pneumonia and a second set of individuals having pneumonitis; generate habitat maps of the medical images, the habitat maps representing a segmentation of the medical images and depicting a plurality of image sub-regions determined based on similarity in imaging patterns across the plurality of medical images; determine a plurality of first features from the habitat maps; determine a plurality of second features indicative of how symmetric the plurality of first features across left and right lungs; and generate, using the plurality of first features and the plurality of second features, a machine learning model configured to detect at least one of pneumonia or pneumonitis.

22. A system comprising: one or more processors; and a memory storing computer instructions, the computer instructions when executed by the one or more processors cause the system to: receive a medical image of lungs of a an individual; generate a habitat map of the medical image, the habitat map representing a segmentation of the medical images and depicting a plurality of image sub-regions determined based on predefined imaging patterns; determine a plurality of image features from the habitat map; determine a plurality of second features indicative of how symmetric the plurality of first features across left and right lungs; and generate, using the plurality of first features and the plurality of second features, a machine learning model configured to detect at least one of pneumonia or pneumonitis.

23. A method comprising: receiving, by a computer system including one or more processors and a memory, a medical image of lungs of a an individual; generating, by the computer system, a habitat map of the medical image, the habitat map representing a segmentation of the medical images and depicting a plurality of image sub-regions determined based on predefined imaging patterns;determining, by the computer system, a plurality of image features from the habitat map; determining, by the computer system, a plurality of second features indicative of how symmetric the plurality of first features across left and right lungs; and generating, by the computer system, using the plurality of first features and the plurality of second features, a machine learning model configured to detect at least one of pneumonia or pneumonitis.

24. A non-transitory computer-readable medium storing computer instructions, the computer instructions when executed by one or more processors cause the one or more processors to: receive a medical image of lungs of a an individual; generate a habitat map of the medical image, the habitat map representing a segmentation of the medical images and depicting a plurality of image sub-regions determined based on predefined imaging patterns; determine a plurality of image features from the habitat map; determine a plurality of second features indicative of how symmetric the plurality of first features across left and right lungs; and generate, using the plurality of first features and the plurality of second features, a machine learning model configured to detect at least one of pneumonia or pneumonitis.

Images