Systems and methods for generating radiology report

Abstract

To provide improved systems and methods for generating a radiology report for interpretation by a clinician.SOLUTION: Medical images of a three-dimensional region of interest of a subject are obtained. The medical images are segmented by assigning labels corresponding to tissue types for each of a plurality of sets of one or more pixels in the medical images. Masks are generated for the tissue types based on the segmentation labels. A visual representation of tumor tissue is generated based on a corresponding mask for the tumor tissue. A visual representation of a skeleton of the subject is also generated based on the medical images. The visual representation of the tumor tissue and the visual representation of the skeleton of the subject is displayed in a single image, in a same spatial orientation as in the set of medical images, and at a same relative size as in the set of medical images.SELECTED DRAWING: Figure 6

Description

[Background technology]

[0001] Medical imaging is routinely used to diagnose and treat disease. See Hussain, S. et al., Modern Diagnostic Imaging Technique Applications and Risk Factors in the Medical Field: A Review, BioMed Research International, 2022:5164970 (2022), which is incorporated herein by reference in its entirety. However, manual interpretation of medical imaging is time-consuming, expensive, and subjective. Furthermore, radiology images are difficult to interpret for non-expert clinicians and patients. Radiology reports from trained radiologists can be time-consuming and often lack important quantitative measurements due to the time it takes to perform such measurements. Summary of the Invention

[0002] In view of the above background, what is needed in the art are improved methods and systems for generating radiology reports. As disclosed herein, the present disclosure provides improved systems and methods for generating radiology reports for interpretation by clinicians. Advantageously, the systems and methods described herein provide simple radiology reports with easily interpretable visual images of tumor burden, e.g., as assessed from a CT scan and its quantitative measurements. In some embodiments, when longitudinal imaging data of the tumor is available, the systems and methods described herein further include visualization of the tumor from medical imaging and / or past quantitative measurements for a clearer picture of how the patient's cancer is progressing, e.g., in light of ongoing treatment.

[0003] In one aspect, the present disclosure provides a method for visualizing cancer in a subject in a computer system including one or more processors and a memory coupled to the one or more processors, the memory including one or more programs configured to be executed by the one or more processors.

[0004] In some embodiments, a method includes acquiring a first set of medical images of a three-dimensional region of interest of a subject, the first set of medical images being collected at a first time point using a first medical imaging modality, the three-dimensional region of interest of the subject including a plurality of tissue types, the plurality of tissue types including tumor tissue and non-cancerous tissue of the subject.

[0005] In some such embodiments, the first set of medical images includes a computed tomography (CT) scan of a three-dimensional region of the subject.

[0006] In some such embodiments, the cancer is non-small cell lung cancer (NSCLC) and the plurality of tissue types comprises non-cancerous lung tissue. In some such embodiments, the cancer is pancreatic cancer and the plurality of tissue types comprises non-cancerous pancreatic tissue. In some such embodiments, the plurality of tissue types comprises lymphatic tissue. In some such embodiments, the plurality of tissue types comprises vascular tissue.

[0007] In some embodiments, the method includes segmenting the set of medical images by assigning, to each respective set of one or more pixels in a first plurality of sets of one or more pixels in the set of medical images, a label corresponding to a respective tissue type among a plurality of tissue types based on one or more corresponding pixel values ​​for the respective set of one or more pixels. In some embodiments, the method includes, for each respective tissue type of the plurality of tissue types, generating a corresponding mask based on the respective set of one or more pixels assigned the label corresponding to the respective tissue type.

[0008] The method includes generating a visual representation of tumor tissue of the subject based on a corresponding mask of the tumor tissue, hi some such embodiments, a corresponding mesh surface is generated for the tumor tissue based on an outer boundary of the corresponding mask for the tumor tissue, and smoothing edges of the corresponding mesh surface for the tumor tissue.

[0009] In some such embodiments, each respective set of one or more pixels in the first plurality of sets of one or more pixels corresponds to a respective voxel in a uniform three-dimensional grid of voxels defined for the set of medical images and each tissue type in the plurality of tissue types, and the corresponding mask includes, for each respective voxel in the three-dimensional grid of voxels, a binary indication of whether the tissue represented in the voxel is of the respective tissue type. In some such embodiments, the corresponding mask of the tumor tissue includes a plurality of groups of non-zero voxels, and each respective group of non-zero voxels in the plurality of groups of non-zero voxels is separated from every other respective group of non-zero voxels in the plurality of groups of non-zero voxels by at least one zero voxel. In some such embodiments, generating the visual representation of the tumor tissue includes generating a mesh surface corresponding to each group of non-zero voxels in the plurality of groups of non-zero voxels. In some such embodiments, the visual representation of the tumor tissue excludes representations of each group of non-zero voxels having a total volume below a first volume threshold.

[0010] In some such embodiments, the method also includes determining a volume of the cancer in the subject based on a volume contained within the mesh surface of the tumor tissue, hi some such embodiments, the volume is determined after smoothing the edges of the corresponding mesh surface of the tumor tissue.

[0011] In some embodiments, the method also includes generating a visual representation of the subject's anatomy within the region of interest based on the first set of medical images.

[0012] In some such embodiments, this includes determining a corresponding tissue density for each respective set of one or more pixels in a second plurality of sets of one or more pixels in the set of medical images. In some such embodiments, a corresponding mask of the skeleton is generated based on each set of one or more pixels of the second plurality of sets of one or more pixels having a tissue density that satisfies one or more sets of bone density criteria. In some such embodiments, the first set of medical images includes computed tomography (CT) scans, and the set of one or more bone density criteria includes a first criterion satisfied by a measure of central tendency for pixel intensities of the respective set of one or more pixels greater than a first threshold of 300-500 Hounsfield units (HU). In some such embodiments, the first set of medical images includes computed tomography (CT) scans, and the set of one or more bone density criteria includes a second criterion satisfied by a measure of central tendency for pixel intensities of the respective set of one or more pixels less than a second threshold of 1750-2500 Hounsfield units (HU).

[0013] In some such embodiments, the corresponding mesh surface of the skeleton is generated based on the outer boundary of the corresponding mask of the skeleton, hi some such embodiments, the edges of the corresponding mesh surface of the skeleton are smoothed.

[0014] In some such embodiments, each respective set of one or more pixels in the second plurality of sets of one or more pixels corresponds to a respective voxel in a second three-dimensional grid of voxels defined for the set of medical images, and the corresponding mask for the skeleton includes, for each voxel in the second three-dimensional grid of voxels, a binary indication of whether the tissue represented in the voxel is skeletal tissue or not. In some such embodiments, the visual representation of the skeleton excludes representations of respective groups of non-zero voxels in the three-dimensional grid of voxels that have a total volume below a second volume threshold.

[0015] The method also includes displaying a visual representation of the tumor tissue, and optionally a visual representation of the subject's skeleton, in a single image in the same spatial orientation as the set of medical images and at the same relative size as the set of medical images.

[0016] In some such embodiments, the method also includes generating a report including a single image including a visual representation of the tumor tissue and a visual representation of the subject's skeleton. In some such embodiments, the report further includes one or more measurements for the cancerous tissue. In some such embodiments, the one or more measurements for the cancerous tissue include at least one measurement selected from a volume for the cancerous tissue, a cross-sectional length of the cancerous tissue, a density measurement for the cancerous tissue, a density of non-cancerous tissue within the plurality of tissues, a distance from the cancerous tissue to non-cancerous tissue within the plurality of tissues, and a distance from the cancerous tissue to a vascular structure in a three-dimensional region of interest of the subject. In some such embodiments, the report further includes a change in the measurements of the cancerous tissue over time. In some such embodiments, the report further includes, for each respective time point in the plurality of time points, a respective visual representation of the tumor tissue at each time point. In some such embodiments, the report further includes a timeline indicating the timing of one or more events related to cancer in the subject. In some such embodiments, the events in the one or more events related to cancer in the subject are selected from a cancer imaging study, a medical intervention for cancer, a cancer diagnosis, a cancer prognosis, and cancer progression or regression. In some such embodiments, the report further includes a prognosis of cancer in the subject. In some such embodiments, the report is displayed on a user interface on a second computer system that includes one or more processors, a memory coupled to the one or more processors, and a display.

[0017] In some such embodiments, the single image further includes a visual representation of a reference shape in the single image corresponding to the volume of the subject's three-dimensional region of interest, with a size in the same proportion as the visual representation of the tumor tissue and the visual representation of the subject's skeleton relative to the set of medical images.

[0018] In some such embodiments, the single image further includes a visual representation of a reference axial, coronal, or sagittal slice of the subject's three-dimensional region of interest from the first set of medical images, hi some such embodiments, the reference axial, coronal, or sagittal slice bisects the tumor tissue.

[0019] In some such embodiments, the single image includes a three-dimensional representation of the subject's tumor tissue and skeleton. In some such embodiments, the single image includes a two-dimensional representation of the subject's tumor tissue and skeleton.

[0020] In some such embodiments, the single image further includes a visual representation of the non-cancerous tissue of interest in the same spatial orientation and at the same relative size as the set of medical images.

[0021] It should be noted that the various embodiments described above can be combined with any other embodiment described herein. The features and advantages described herein are not exhaustive, and many additional features and advantages will be apparent to those skilled in the art, particularly in view of the drawings, specifications, and claims. Furthermore, it should be noted that the language used herein has been chosen primarily for readability and instructional purposes, and may not be chosen to describe or enclose the subject matter of the present invention. [Brief explanation of the drawings]

[0022] The drawings illustrate, by way of example, embodiments of the disclosed system and method, and it is to be expressly understood that the description and drawings are for illustrative purposes only, as an aid to understanding, and are not intended as a definition of the limits of the disclosed system and method.

[0023] [Figure 1A] 1A, 1B, and 1C illustrate a computer system for generating a radiology report according to some embodiments of the present disclosure. [Figure 1B] Same as above. [Figure 1C] Same as above.

[0024] [Figure 2A] 2A, 2B, 2C, 2D, 2E, 2F, and 2G collectively provide a flowchart of an exemplary method for generating a radiology report, according to some embodiments of the present disclosure. [Figure 2B] Same as above. [Figure 2C] Same as above. [Figure 2D] Same as above. [Figure 2E] Same as above. [Figure 2F] Same as above. [Figure 2G] Same as above.

[0025] [Figure 3] FIG. 3 illustrates a process for generating a radiology report according to some embodiments of the present disclosure.

[0026] [Figure 4] FIG. 4 shows a visual representation of a pancreatic tumor 402 and pancreas 404 generated from a CT scan, according to some embodiments of the present disclosure.

[0027] [Figure 5] FIG. 5 shows a visual representation of a pancreatic tumor 502 and pancreas 504 generated from a CT scan, bisected by sagittal and coronal sections from the CT scan, according to some embodiments of the present disclosure.

[0028] [Figure 6] FIG. 6 shows a visual representation of a pancreatic tumor 602, pancreas 604, and skeleton 606 generated from a CT scan, according to some embodiments of the present disclosure.

[0029] [Figure 7] FIG. 7 shows an example of a radiology report generated from a CT scan, according to some embodiments of the present disclosure, having a visual representation of non-small cell lung cancer (NSCLC) in relation to the respiratory tract and skeleton at a first time point, as well as a visual representation of the NSCLC over time.

[0030] [Figure 8] FIG. 8 shows an example of a radiology report having a visual representation of a pancreatic tumor in relation to a patient's pancreas and skeleton at a first time point, as well as a visual representation of the pancreatic tumor over time generated from a CT scan, according to some embodiments of the present disclosure.

[0031] [Figure 9] FIG. 9 shows an example of a radiology report generated from a CT scan with a visual representation of a lung tumor in relation to a patient's skeleton, as well as a visual representation of a lung tumor islet alone, according to some embodiments of the present disclosure.

[0032] [Figure 10] FIG. 10 shows a three-dimensional visual representation of a pancreatic tumor, pancreas, and vasculature surrounding the pancreas generated from a CT scan, as well as two-dimensional representations of the pancreatic tumor, pancreas, and vasculature overlaid on axial slices from the CT scan at different depths, in accordance with some embodiments of the present disclosure.

[0033] [Figure 11] FIG. 11 shows a three-dimensional visual representation of a pancreatic tumor, pancreas, and surrounding vasculature generated from a CT scan, according to some embodiments of the present disclosure.

[0034] Like reference numbers refer to corresponding parts throughout the several views of the drawings. DETAILED DESCRIPTION OF THE INVENTION

[0035] Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0036] Oncologists have very limited time to prepare for appointments with cancer patients. However, radiology data is complex, and oncologists spend extensive time before each appointment collecting and reviewing previous radiology scans and reports, interpreting the patient's most recent scans, and determining the best direction for future therapy to understand issues such as the trajectory of the cancer and the effectiveness of current therapy. Advantageously, in some embodiments, the systems and methods described herein collect relevant data into a single display, such as a PDF or user interface, helping clinicians save significant time and money associated with cancer therapy.

[0037] Additionally, patients often struggle to understand the results reported by their physicians. Advantageously, in some embodiments, the systems and methods described herein contextualize that information into easy-to-understand visualizations, along with information from the patient's journey to the report they keep after their visit.

[0038] definition

[0039] As used herein, the term "measure of central tendency" refers to the central or representative value of a distribution of values. Non-limiting examples of measures of central tendency include the arithmetic mean, weighted mean, median, central area, central hinge, trimean, geometric mean, geometric median, Winsorized mean, median, and mode of a distribution of values.

[0040] As used herein, the term "subject" refers to any living or non-living organism, including, but not limited to, a human (e.g., a male human, a female human, a fetus, a pregnant woman, a child, or the like), a non-human mammal, or a non-human animal. Any human or non-human animal can serve as a subject, including, but not limited to, a mammal, a reptile, a bird, an amphibian, a fish, a ungulate, a ruminant, a bovine (e.g., a cow), an equine (e.g., a horse), a caprine and ovine (e.g., a sheep, a goat), a porcine (e.g., a pig), a camelid (e.g., a camel, a llama, an alpaca), a monkey, an ape (e.g., a gorilla, a chimpanzee), a ursidae (e.g., a bear), a fowl, a dog, a cat, a mouse, a rat, a fish, a dolphin, a whale, and a shark. In some embodiments, the subject is a male or female (e.g., a man, a woman, or a child) of any age.

[0041] As used herein, the terms "cancer," "cancerous tissue," or "tumor" refer to an abnormal mass of tissue whose growth exceeds and is out of step with that of normal tissue. Cancers or tumors can be defined as "benign" or "malignant" depending on the following characteristics: degree of cellular differentiation, including morphology and functionality, growth rate, local invasion, and metastasis. "Benign" tumors are well differentiated, have characteristically slower growth than malignant tumors, and may remain localized to the site of origin. In addition, in some cases, benign tumors do not have the ability to infiltrate, invade, or metastasize to distant sites. "Malignant" tumors may be poorly differentiated (dysplastic) and have characteristically rapid growth accompanied by progressive infiltration, invasion, and destruction of surrounding tissue. Furthermore, malignant tumors may have the ability to metastasize to distant sites. Thus, cancer cells are cells found within an abnormal mass of tissue whose growth is out of step with that of normal tissue. Thus, a "tumor sample" refers to a biological sample obtained from or derived from a tumor of a subject, as described herein.

[0042] As used herein, the term "classification" may refer to any number or other letter associated with a particular characteristic of a sample. For example, in some embodiments, the term "classification" may refer to a tissue type in a subject, e.g., a tissue type determined from a set of medical images such as a CT scan. The term classification may also refer to the type of cancer in a subject, the stage of cancer in a subject, the prognosis of cancer in a subject, tumor burden, the presence of tumor metastasis in a subject, etc. Classifications may be binary (e.g., positive or negative) or have more levels of classification (e.g., a 1-10 or 0-1 scale). The terms "cutoff" and "threshold" may refer to a predetermined number used in a calculation. For example, a cutoff density may refer to the density above which tissue is excluded from being classified as skeletal tissue. A threshold may be a value above or below which a particular classification applies. Either of these terms may be used in either of these contexts.

[0043] As used herein, an "effective amount" or a "therapeutically effective amount" is an amount sufficient to affect beneficial or desired clinical results during treatment. An effective amount can be administered to a subject in one or more doses. In terms of treatment, an effective amount is an amount sufficient to alleviate, improve, stabilize, reverse, or delay the progression of a disease, or otherwise reduce the pathological consequences of a disease. An effective amount is generally determined by a physician on a case-by-case basis and is within the skill of a person skilled in the art. Several factors are typically considered when determining the appropriate dosage to achieve an effective amount. These factors include the age, sex, and weight of the subject, the condition being treated, the severity of the condition, and the form and effective concentration of the therapeutic agent being administered.

[0044] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the description of the invention and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly dictates otherwise. As used herein, the term "and / or" refers to and includes any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms "comprises" and / or "comprising" as used herein specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Furthermore, to the extent the terms "including," "includes," "having," "has," "with," or variations thereof are used in either the detailed description and / or the claims, such terms are intended to be as inclusive as the term "comprising."

[0045] As used herein, the term "if" can be interpreted to mean "when" or "upon," "in response to detecting," or "in response to determining," depending on the context. Similarly, the phrase "when determined" or "when [a described condition or event] is detected" can be interpreted to mean "upon determining," or "in response to determining," or "upon detection [of [a described condition or event]," or "in response to detecting [a described condition or event]," depending on the context.

[0046] Also, while terms such as "first," "second," etc. may be used herein to describe various elements, it will be understood that these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, a first subject may be referred to as a second subject, and similarly, a second subject may be referred to as a first subject, without departing from the scope of the present disclosure. A first subject and a second subject are both subjects, but are not the same subject. Furthermore, the terms "subject," "user," and "patient" are used interchangeably herein.

[0047] Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the present disclosure, including example systems, methods, techniques, instruction sequences, and computing machine program products embodying illustrative implementations. However, the following illustrative discussion is not intended to be exhaustive or to limit implementations to the precise form disclosed. Many modifications and variations are possible in light of the above teachings. Features described herein are not limited by the illustrated order of acts or events, as some acts may occur in different orders and / or contemporaneously with other acts or events.

[0048] The implementations provided herein are chosen and described to best explain the principles and their practical applications, thereby enabling those skilled in the art to best utilize the various embodiments with various modifications suited to the particular uses contemplated. In some instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments. In other instances, it will be apparent to those skilled in the art that the present disclosure may be practiced without one or more of the specific details.

[0049] It will be understood that in the development of any such actual implementation, numerous implementation-specific decisions will be made to achieve the designer's specific goals, such as adherence to use case and business-related constraints, and that these specific goals will vary from implementation to implementation and from designer to designer. It will further be understood that such a design effort may be complex and time-consuming, but would nevertheless be a routine undertaking for one of ordinary skill in the art having the benefit of this disclosure.

[0050] Exemplary System Embodiments

[0051] Improved systems and methods for generating radiology reports are described herein. Exemplary details of such systems, devices, and / or processes according to the present disclosure are described below in connection with Figures 1-3. Product examples of these systems, devices, and / or processes are further described in connection with Figures 4-11.

[0052] 1A-1C collectively illustrate a computer system for generating radiology reports according to some embodiments of the present disclosure. In a typical embodiment, computer system 100 includes one or more computers. For purposes of illustration in FIG. 1A, computer system 100 is depicted as a single computer that includes all of the functionality of computer system 100 of the present disclosure. However, the present disclosure is not so limited. The functionality of computer system 100 may be spread across any number of networked computers and / or reside on each of several networked computers and / or virtual machines. Those skilled in the art will appreciate that a wide variety of different computer topologies are possible for computer system 100, and all such topologies are within the scope of the present disclosure.

[0053] 1A , computer system 100 includes one or more processing units (CPUs) 59, a network or other communications interface 84, a user interface 78 (e.g., including an optional display 82 and an optional keyboard 80 or other form of input device), memory 92 (e.g., random access memory, persistent memory, or a combination thereof), one or more magnetic disk storage and / or persistent devices 90 optionally accessed by one or more controllers 88, one or more communications buses 12 for interconnecting the aforementioned components, and a power supply 79 for powering the aforementioned components. To the extent that components of memory 92 are not persistent, data in memory 92 may be seamlessly shared with non-volatile memory 90 or portions of memory 92 that are non-volatile or persistent using known computing techniques, such as caching. Memory 92 and / or memory 90 may include mass storage located remotely relative to central processing unit 59. In other words, some data stored in memory 92 and / or memory 90 may actually be hosted on a computer that is external to computer system 100, but that can be accessed electronically by computer system 100 over network 102 (e.g., the Internet, an intranet, or other form of network or electronic cable) using network interface 84. In some embodiments, computer system 100 utilizes models that execute from memory associated with one or more graphics processing units to improve system speed and performance. In some alternative embodiments, computer system 100 utilizes models that execute from memory 92 rather than memory associated with the graphics processing units.

[0054] The memory 92 of the computer system 100 stores: Any operating system 30 containing procedures for handling various basic system services; · a communications module 34 that connects to and communicates with other network devices (e.g., local networks such as routers providing Internet connections, network storage devices, network routing devices, server systems, other computer systems 100, and / or other connected devices) coupled to one or more communications networks via a network interface 84 (e.g., wired or wireless); A patient data store 40 containing one or more types of medical data and / or output from the methods described herein for one or more patients 41. In some embodiments, the medical data includes one or more types of data selected from the following: One or more medical images 42 of a region of interest in a subject having cancer. In some embodiments, the medical images include multiple images of a region of interest in the subject collected at a single time point that represent a three-dimensional volume in the subject. For example, FIG. 1B shows a medical image store 42-1 that includes a series of CT scans 42-1-i (42-1-1, ..., 42-1-D) acquired at different times for a patient 41-1, each CT scan 42-1-i including multiple image slices 42-1-1-j (e.g., 42-1-1-1, ..., 42-1-1-B of CT scan 42-1-1) of a three-dimensional region of interest in the subject. Imaging features 43 (e.g., medical images 42) extracted from a medical image set of a region of interest in a subject having cancer. In some embodiments, such features may be determined at the pixel level, e.g., pixel features 81-1-1-1, ..., 81-1-1-F for image 43-1-1-1 of CT scan 43-1-1 illustrated in FIG. 1B, or at the voxel level, e.g., voxel features 83-1-1-1, ..., 83-1-1-F for an established voxel grid 82 for image 43-1-1-1 of CT scan 43-1-1 illustrated in FIG. 1B. A tissue classification 44 of one or more medical image sets, generated by analyzing the imaging features 43 using the segmentation module 50. For example, as shown in FIG. 1C , the tissue classification 44-1 provides a classification of tissue type at each voxel 83-1-1-i of the three-dimensional voxel grid 82 by evaluating the voxel features 84-1-1-i using the segmentation module 50. A tissue mask 45 of one or more imaged tissues, e.g., generated based on tissue classification 44. For example, FIG. 1C shows tumor tissue mask 85-1-1-1 and non-cancerous tissue masks 85-1-1-2, ..., 85-1-1-K generated from voxel classifications 83-1-1-1, ..., 83-1-1-H. In some embodiments, tissue mask 85 includes values ​​85-ijkl for each voxel in a three-dimensional voxel grid established for a region of interest in the subject that indicate whether the tissue represented in the voxel is of that type of tissue. For example, as shown in FIG. 1C, tumor tissue mask 85-1-1-1 includes voxel values ​​85-1-1-1-1, ..., 85-1-1-1-I. In some embodiments, the voxel value is a binary value, e.g., either 1 (e.g., indicating that the tissue represented by the voxel is of the type of tissue represented by the mask), or 0 (e.g., indicating that the tissue represented by the voxel is not of the type of tissue represented by the mask). a mesh surface 46 for visualizing one or more tissue types imaged in a subject with cancer, generated, for example, by processing the tissue mask 45 using the visualization module 60; medical records of one or more patients; and · A patient report 48 generated by analyzing imaging data, e.g., medical images 42, and optionally other medical information, e.g., medical records 47; a segmentation module 50 including a segmentation algorithm 61 and / or a tissue classification algorithm 62 for segmenting different tissues in a medical image (e.g., medical image 42) and / or for determining tissue types for cancerous and / or non-cancerous tissues, for further visualization of such tissues, for example, in a subject with cancer; a visualization module 60 including a masking algorithm 61 and / or a visualization algorithm 62 for generating a visual representation of cancerous and / or non-cancerous tissue identified in a medical image (e.g., medical image 42); and A report module 70 for generating a report including a visual representation (e.g., based on mesh surface 46) of one or more cancerous and / or non-cancerous tissues identified in a medical image (e.g., medical image 42), and optionally including other biometric personal and / or medical information about the subject, e.g., from medical records 47.

[0055] In some embodiments, one or more of the above-identified data elements or modules of computer system 100 are stored in one or more of the aforementioned memory devices and correspond to sets of instructions for implementing the functions described above. The above-identified data, modules, or programs (e.g., sets of instructions) need not be implemented as separate software programs, procedures, or modules; thus, various subsets of these modules may be combined or otherwise rearranged in various implementations. In some implementations, memory 92 and / or 90 optionally stores some of the modules and data structures described above. Additionally, in some embodiments, memory 92 and / or 90 stores additional modules and data structures not described above. Details of the above-identified modules and data structures are described below with reference to FIGS. 2A-11.

[0056] 2A-2K collectively provide a flowchart of an exemplary method 200 for generating a radiology report, according to some embodiments. In some embodiments, method 200 is performed on a computer system 100 including one or more processors (e.g., CPU 59) and memory (e.g., memory 90 or memory 92). In some embodiments, a user initiates instructions on computer system 100 to begin a process for generating a radiology report. In some embodiments, computer system 100 performs the steps as shown in FIG. 2.

[0057] Referring to block 202, in some embodiments, a method includes acquiring a first set of medical images of a three-dimensional region of interest of a subject, the first set of medical images being collected at a first time point using a first medical imaging modality, the three-dimensional region of interest of the subject including multiple tissue types, the multiple tissue types including tumor tissue and non-cancerous tissue of the subject. In some embodiments, the set of medical images are cross-sectional imaging scans of the subject, such as CT scans, PET scans, or a hybrid thereof. For a review of CT and PET medical imaging, see, e.g., Al-Sharify, Z. T. et al., "A critical review on medical imaging techniques (CT and PET scans) in the medical field," OP Conf. Mater. Sci. Eng., 870:012043 (2020), the contents of which are incorporated herein by reference.

[0058] In some embodiments, the obtaining is performed by system 100 in response to receiving user input requesting a report on cancer in a subject. In some embodiments, the system queries electronic medical data, e.g., specific to the subject (e.g., an individual's electronic medical record) or common to medical imaging datasets associated with a healthcare entity (e.g., a healthcare provider, clinic, hospital, hospital system, healthcare payer, etc.), for the most recent medical imaging dataset related to the cancer. In some embodiments, the system further queries the electronic medical data to determine whether earlier medical imaging data for the same cancer is also available.

[0059] Referring to block 204, in some embodiments, the first set of medical images includes a computed tomography (CT) scan of a three-dimensional region of the subject. In some embodiments, the medical imaging dataset includes a magnetic resonance imaging (MRI) dataset. Typically, a CT scan includes 4 to 640 individual images, each corresponding to a different plane of the three-dimensional region of the subject. The images are collected on a detector, typically a flat panel, charge-coupled device (CCD), or active pixel sensor (APS), such as a complementary metal-oxide semiconductor (CMOS) or scientific CMOS (sCMOS) image sensor. Typically, the detector includes a 1000 x 1000 to 3000 x 3000 pixel array, although arrays with larger or smaller pixel arrays can be used in the systems and methods described herein.

[0060] However, the methods and systems described herein are not limited by the use of CT scans. Indeed, many imaging modalities can be used with these methods and systems. A review of various imaging modalities that can be used to visualize a subject's tumor can be found, for example, in Rowe SP, and Pomper MG., "Molecular imaging in oncology: Current impact and future directions," CA Cancer J Clin. (2021), the disclosure of which is incorporated herein by reference.

[0061] In some embodiments, the medical imaging data includes a plurality of X-ray image sets, a plurality of computed tomography (CT) image sets, a plurality of positron emission tomography (PET) image sets, a plurality of magnetic resonance imaging (MRI) image sets, a plurality of single photon emission computed tomography (SPECT) image sets, a plurality of mammography image sets, a plurality of sonography (ultrasound) image sets, or a combination thereof. For a review of medical imaging techniques used in diagnosis, see, for example, Hussain, S. et al., Modern Diagnostic Imaging Technique Applications and Risk Factors in the Medical Field: A Review, BioMed Research International, 2022:5164970 (2022), which is incorporated herein by reference in its entirety.

[0062] Many medical imaging modalities used in diagnosis generate datasets composed of multiple images. For example, a CT scan uses a rotating x-ray tube to capture multiple images at different angles that can be processed to generate a two-dimensional or three-dimensional tomographic image of the area of ​​interest. Depending on the particular implementation and area of ​​interest being imaged, a CT scan can generate tens to thousands of images. Similarly, an MRI scan acquires image slices of the area of ​​interest. Depending on the particular implementation and area of ​​interest being imaged, an MRI scan can generate tens to thousands of images.

[0063] In some embodiments, the medical imaging dataset includes an ultrasound dataset. A discussion of the use of ultrasound imaging in clinical cancer therapy is described, for example, in Sarikaya, I., “Biology of Cancer and PET Imaging: Pictorial Review,” J. Nucl. Med. Technol., 16(1):531 (2022), the disclosure of which is incorporated herein by reference. In some embodiments, the medical imaging dataset includes a position emission tomography (PET) dataset. A discussion of the use of PET imaging in clinical cancer therapy is described, for example, in Diaz-Alejo JF, et. al., “Ultrasounds in cancer therapy: A summary of their use and unexplored potential,” Oncol. Rev., 50(2):81-89 (2022), the disclosure of which is incorporated herein by reference. In some embodiments, the medical imaging dataset includes an X-ray dataset.

[0064] In some embodiments, the first set of medical images has at least 8 images. In some embodiments, the first set of medical images has at least 32 images. In some embodiments, the first set of medical images has at least 128 images. In some embodiments, the first set of medical images has at least 4 images, at least 8 images, at least 16 images, at least 32 images, at least 64 images, at least 128 images, at least 256 images, at least 320 images, at least 640 images, at least 1280 images, at least 2560 images, or more. In some embodiments, the first set of medical images has 100,000 images or less. In some embodiments, the first set of medical images has 50,000 images or less. In some embodiments, the first set of medical images has 10,000 images or less. In some embodiments, the first set of medical images has 2500 images or less. In some embodiments, the first set of medical images has 1500 images or less.

[0065] In some embodiments, the first set of medical images has between 4 and 2560 images. In some embodiments, the first set of medical images has between 4 and 1280 images. In some embodiments, the first set of medical images has between 4 and 640 images. In some embodiments, the first set of medical images has between 4 and 320 images. In some embodiments, the first set of medical images has between 4 and 256 images. In some embodiments, the first set of medical images has between 4 and 128 images. In some embodiments, the first set of medical images has between 4 and 64 images. In some embodiments, the first set of medical images has between 8 and 2560 images. In some embodiments, the first set of medical images has between 8 and 1280 images. In some embodiments, the first set of medical images has between 8 and 640 images. In some embodiments, the first set of medical images has between 8 and 320 images. In some embodiments, the first set of medical images has between 8 and 256 images. In some embodiments, the first set of medical images has between 8 and 128 images. In some embodiments, the first set of medical images has between 8 and 64 images. In some embodiments, the first set of medical images has between 16 and 2560 images. In some embodiments, the first set of medical images has between 16 and 1280 images. In some embodiments, the first set of medical images has between 16 and 640 images. In some embodiments, the first set of medical images has between 16 and 320 images. In some embodiments, the first set of medical images has between 16 and 256 images. In some embodiments, the first set of medical images has between 16 and 128 images. In some embodiments, the first set of medical images has between 16 and 64 images. In some embodiments, the first set of medical images has between 64 and 2560 images. In some embodiments, the first set of medical images has between 64 and 1280 images. In some embodiments, the first set of medical images has between 64 and 640 images. In some embodiments, the first set of medical images has between 64 and 320 images.In some embodiments, the first set of medical images has between 64 and 256 images. In some embodiments, the first set of medical images has between 64 and 128 images. In some embodiments, the first set of medical images has between 128 and 2560 images. In some embodiments, the first set of medical images has between 128 and 1280 images. In some embodiments, the first set of medical images has between 128 and 640 images. In some embodiments, the first set of medical images has between 128 and 320 images. In some embodiments, the first set of medical images has between 128 and 256 images. In some embodiments, the first set of medical images has between 256 and 2560 images. In some embodiments, the first set of medical images has between 256 and 1280 images. In some embodiments, the first set of medical images has between 256 and 640 images. In some embodiments, the first set of medical images has between 256 and 320 images.

[0066] In some embodiments, the images in the set of medical images have at least 250,000 pixels. In some embodiments, the images in the set of medical images have at least 1 million pixels. In some embodiments, the images in the set of medical images have at least 2.5 million pixels. In some embodiments, the images in the set of medical images have at least 5 million pixels. In some embodiments, the images in the set of medical images have 1 billion pixels or less. In some embodiments, the images in the set of medical images have 100 million pixels or less. In some embodiments, the images in the set of medical images have 10 million pixels or less. In some embodiments, the images in the set of medical images have 5 million pixels or less.

[0067] In some embodiments, the images in the set of medical images have between 250,000 and 1 billion pixels. In some embodiments, the images in the set of medical images have between 250,000 and 100 million pixels. In some embodiments, the images in the set of medical images have between 250,000 and 10 million pixels. In some embodiments, the images in the set of medical images have between 250,000 and 5 million pixels. In some embodiments, the images in the set of medical images have between 1 million and 1 billion pixels. In some embodiments, the images in the set of medical images have between 1 million and 100 million pixels. In some embodiments, the images in the set of medical images have between 1 million and 10 million pixels. In some embodiments, the images in the set of medical images have between 1 million and 5 million pixels. In some embodiments, the images in the set of medical images have between 2.5 million and 1 billion pixels. In some embodiments, the images in the set of medical images have between 2.5 million and 100 million pixels. In some embodiments, the images in the set of medical images have between 2.5 million and 10 million pixels, hi some embodiments, the images in the set of medical images have between 2.5 million and 5 million pixels.

[0068] In some embodiments, the first set of medical images has at least 5 million pixels. In some embodiments, the first set of medical images has at least 10 million pixels. In some embodiments, the first set of medical images has at least 50 million pixels. In some embodiments, the first set of medical images has at least 100 million pixels. In some embodiments, the first set of medical images has at least 250 million pixels. In some embodiments, the first set of medical images has at least 500 million pixels. In some embodiments, the first set of medical images has at least 1 billion pixels. In some embodiments, the first set of medical images has 100 billion pixels or less. In some embodiments, the first set of medical images has 50 billion pixels or less. In some embodiments, the first set of medical images has 10 billion pixels or less. In some embodiments, the first set of medical images has 5 billion pixels or less. In some embodiments, the first set of medical images has 1 billion pixels or less.

[0069] In some embodiments, the first set of medical images has between 5 million and 100 billion pixels. In some embodiments, the first set of medical images has between 5 million and 50 billion pixels. In some embodiments, the first set of medical images has between 5 million and 10 billion pixels. In some embodiments, the first set of medical images has between 5 million and 5 billion pixels. In some embodiments, the first set of medical images has between 5 million and 1 billion pixels. In some embodiments, the first set of medical images has between 10 million and 100 billion pixels. In some embodiments, the first set of medical images has between 10 million and 50 billion pixels. In some embodiments, the first set of medical images has between 10 million and 10 billion pixels. In some embodiments, the first set of medical images has between 10 million and 5 billion pixels. In some embodiments, the first set of medical images has between 10 million and 1 billion pixels. In some embodiments, the first set of medical images has between 50 million and 100 billion pixels. In some embodiments, the first set of medical images has between 50 million and 50 billion pixels. In some embodiments, the first set of medical images has between 50 million and 10 billion pixels. In some embodiments, the first set of medical images has between 50 million and 5 billion pixels. In some embodiments, the first set of medical images has between 50 million and 1 billion pixels. In some embodiments, the first set of medical images has between 100 million and 100 billion pixels. In some embodiments, the first set of medical images has between 100 million and 50 billion pixels. In some embodiments, the first set of medical images has between 100 million and 100 billion pixels. In some embodiments, the first set of medical images has between 100 million and 50 billion pixels. In some embodiments, the first set of medical images has between 100 million and 10 billion pixels. In some embodiments, the first set of medical images has between 100 million and 5 billion pixels. In some embodiments, the first set of medical images has between 100 million and 1 billion pixels. In some embodiments, the first set of medical images has between 250 million and 100 billion pixels. In some embodiments, the first set of medical images has between 250 million and 50 billion pixels.In some embodiments, the first set of medical images has between 250 million and 10 billion pixels. In some embodiments, the first set of medical images has between 250 million and 5 billion pixels. In some embodiments, the first set of medical images has between 250 million and 1 billion pixels.

[0070] In some embodiments, the plurality of non-cancerous tissues includes organ tissues proximal to the tumor. In some embodiments, the organ tissue is the tissue of origin of the case. For example, referring to block 206, in some embodiments, the cancer is non-small cell lung cancer (NSCLC) and the plurality of tissue types includes non-cancerous lung tissue. Similarly, referring to block 208, in some embodiments, the cancer is pancreatic cancer and the plurality of tissue types includes non-cancerous pancreatic tissue. However, this is not always the case, particularly when the cancer is a metastatic cancer originating from the tumor being imaged or tissue distal to the tumor. In any event, in some embodiments, it is advantageous for a physician to consider the proximity of organ systems to the tumor when evaluating methods for treating the cancer and / or whether the tumor is operable. Thus, in some embodiments, non-cancerous tissue, e.g., a nearby organ or portion thereof, is presented along with the tumor in the reports described herein.

[0071] In yet other embodiments, the tumor being imaged is any solid tumor or tumors, such as carcinoma, lymphoma, blastoma, glioblastoma, sarcoma, leukemia, breast cancer, squamous cell carcinoma, lung cancer, small cell lung cancer, non-small cell lung cancer (NSCLC), adenocarcinoma of the lung, squamous cell carcinoma of the lung, head and neck cancer, cancer of the peritoneum, hepatocellular carcinoma, gastric or stomach cancer, pancreatic cancer, ovarian cancer, cervical cancer, liver cancer, bladder cancer, hepatoma, colon cancer, colorectal cancer, endometrial or uterine cancer, salivary gland cancer, kidney or renal cancer, liver cancer, prostate cancer, vulvar cancer, thyroid cancer, liver cancer, B-cell lymphoma, low-grade / follicular non-Hodgkin's lymphoma (NHL), small lymphocytic (SL) NHL, intermediate-grade / follicular NHL, intermediate-grade diffuse NHL, high-grade immunoblastic NHL, high-grade lymphoblastic NHL, high-grade small non-cleaving cell NHL, bulky disease NHL, mantle cell lymphoma, AIDS-related lymphoma, Waldenström's macroglobulinemia, chronic lymphocytic leukemia (CLL), acute lymphoblastic leukemia (ALL), hairy cell leukemia or chronic myeloblastic leukemia.

[0072] In some embodiments, the plurality of tissues includes one or more tissues from supporting structures proximal to the tumor, such as arteries, veins, blood vessels, lymphatic vessels, etc. For example, in some embodiments, it is advantageous for a physician to consider the proximity of such structures when evaluating how to treat cancer and / or whether the tumor is operable. Accordingly, in some embodiments, nearby supporting structures are displayed along with the tumor in the reports described herein. Thus, with reference to block 210, in some embodiments, the plurality of tissue types includes lymphatic tissue. Similarly, with reference to block 212, in some embodiments, the plurality of tissue types includes vascular tissue.

[0073] Referring to block 214, in some embodiments, the method 200 includes segmenting the set of medical images by assigning, to each respective set of one or more pixels in the first plurality of sets of one or more pixels in the set of medical images, a label corresponding to a respective tissue type among the plurality of tissue types based on one or more corresponding pixel values ​​for the respective set of one or more pixels. In some embodiments, the medical images are pre-segmented, and the method includes obtaining pre-segmentation results, such as, for example, a tissue mask for the tumor and, optionally, tissue masks for one or more other tissues identified in the imaging dataset.

[0074] Each pixel in a medical imaging dataset having multiple images corresponds to a unique location in a three-dimensional grid that corresponds to the volume of the imaged subject. Thus, in some embodiments, each pixel is assigned to a location in three-dimensional space. In some embodiments, the set of medical images is segmented by evaluating, for each pixel, features of the medical image, such as, for example, raw image intensities, filtered image intensities, and / or a computational combination of filtered and / or unfiltered intensities.

[0075] Given the size of medical imaging sets, such as CT imaging sets, which typically have hundreds of images each with a million or more pixels, evaluating the dataset pixel by pixel is time-consuming. Furthermore, this strategy is susceptible to pixel-to-pixel variability, which is common in medical imaging. In some embodiments, to address these issues, a three-dimensional voxel grid is established, with each voxel representing a plurality of pixels that define a volume in the three-dimensional space represented in the imaging dataset. For example, each voxel in the grid may consist of a 2-pixel by 2-pixel cube, a 3-pixel by 3-pixel cube, a 4-pixel by 4-pixel cube, a 5-pixel by 5-pixel cube, or larger cubes. The use of voxels reduces the time and computational burden of segmenting medical image sets and accounts for pixel-to-pixel variability. In some embodiments, the feature value of a voxel is a measure of central tendency of the feature values ​​of each pixel within the voxel.

[0076] In some embodiments, the three-dimensional voxel grid has at least 100 voxels. In some embodiments, the three-dimensional voxel grid has at least 1000 voxels. In some embodiments, the three-dimensional voxel grid has at least 10,000 voxels. In some embodiments, the three-dimensional voxel grid has at least 100,000 voxels. In some embodiments, the three-dimensional voxel grid has at least 1 million voxels. In some embodiments, the three-dimensional voxel grid has at least 10 million voxels. In some embodiments, the three-dimensional voxel grid has at least 100 million voxels. In some embodiments, the three-dimensional voxel grid has no more than 100 billion voxels. In some embodiments, the three-dimensional voxel grid has no more than 10 billion voxels. In some embodiments, the three-dimensional voxel grid has no more than 1 billion voxels. In some embodiments, the three-dimensional voxel grid has no more than 100 million voxels. In some embodiments, the three-dimensional voxel grid has 10 million voxels or less.

[0077] In some embodiments, the three-dimensional voxel grid has between 100 voxels and 100 billion voxels. In some embodiments, the three-dimensional voxel grid has between 1000 voxels and 100 billion voxels. In some embodiments, the three-dimensional voxel grid has between 10,000 voxels and 100 billion voxels. In some embodiments, the three-dimensional voxel grid has between 100,000 voxels and 100 billion voxels. In some embodiments, the three-dimensional voxel grid has between 1 million voxels and 100 billion voxels. In some embodiments, the three-dimensional voxel grid has between 10 million voxels and 100 billion voxels. In some embodiments, the three-dimensional voxel grid has between 10 million voxels and 100 billion voxels. In some embodiments, the three-dimensional voxel grid has between 100 million voxels and 100 billion voxels. In some embodiments, the three-dimensional voxel grid has between 1 billion voxels and 100 billion voxels. In some embodiments, the three-dimensional voxel grid has between 100 voxels and 10 billion voxels. In some embodiments, the three-dimensional voxel grid has between 1000 voxels and 10 billion voxels. In some embodiments, the three-dimensional voxel grid has between 10,000 voxels and 10 billion voxels. In some embodiments, the three-dimensional voxel grid has between 100,000 voxels and 10 billion voxels. In some embodiments, the three-dimensional voxel grid has between 100,000 voxels and 10 billion voxels. In some embodiments, the three-dimensional voxel grid has between 1 million voxels and 10 billion voxels. In some embodiments, the three-dimensional voxel grid has between 10 million voxels and 10 billion voxels. In some embodiments, the three-dimensional voxel grid has between 10 million voxels and 10 billion voxels. In some embodiments, the three-dimensional voxel grid has between 1 billion voxels and 10 billion voxels. In some embodiments, the three-dimensional voxel grid has between 100 voxels and 1 billion voxels. In some embodiments, the three-dimensional voxel grid has between 1000 voxels and 1 billion voxels. In some embodiments, the three-dimensional voxel grid has between 10,000 voxels and 1 billion voxels. In some embodiments, the three-dimensional voxel grid has between 100,000 voxels and 1 billion voxels. In some embodiments, the three-dimensional voxel grid has between 100,000 voxels and 1 billion voxels.In some embodiments, the three-dimensional voxel grid has between 1 million voxels and 1 billion voxels. In some embodiments, the three-dimensional voxel grid has between 10 million voxels and 1 billion voxels. In some embodiments, the three-dimensional voxel grid has between 100 million voxels and 1 billion voxels. In some embodiments, the three-dimensional voxel grid has between 100 voxels and 100 million voxels. In some embodiments, the three-dimensional voxel grid has between 1000 voxels and 100 million voxels. In some embodiments, the three-dimensional voxel grid has between 10,000 voxels and 100 million voxels. In some embodiments, the three-dimensional voxel grid has between 10,000 voxels and 100 million voxels. In some embodiments, the three-dimensional voxel grid has between 100,000 voxels and 100 million voxels. In some embodiments, the three-dimensional voxel grid has between 100,000 voxels and 100 million voxels. In some embodiments, the three-dimensional voxel grid has between 1 million voxels and 100 million voxels. In some embodiments, the three-dimensional voxel grid has between 10 million voxels and 100 million voxels.

[0078] Examples of radiometric features useful for image segmentation include, but are not limited to, shape features, linear features, gray level co-occurrence matrix (GLCM) features, gray level run length matrix (GLRLM) features, gray level size zone matrix (GLSZM), gray level dependency matrix (GLDM) features, and neighborhood gray tone difference matrix (NGTDM) features.

[0079] Examples of shape features useful for segmenting medical images are listed in Table 1. Examples of primary features useful for segmenting medical images are listed in Table 2. Examples of GLCM features useful for segmenting medical images are listed in Table 3. Examples of GLRLM features useful for segmenting medical images are listed in Table 4. Examples of GLSZM features useful for segmenting medical images are listed in Table 5. Examples of GLDM features useful for segmenting medical images are listed in Table 6. Examples of NGTDM features useful for segmenting medical images are listed in Table 7. [Table 1-1]

Table 1-2

[0080]

Table 2

Table 3

Table 4

Table 5

Table 6

Table 7

[0081] Methods for segmenting medical imaging datasets are known in the art. For example, traditionally, tissues and organ structures present in medical images have been segmented using signal thresholding techniques that rely on differences in tissue density to distinguish them. See, for example, Hu S. et al., “Automatic lung segmentation for accurate quantitation of volumetric X-ray CT images,” IEEE Trans. Imaging, 20(6):490-98 (2001), the disclosure of which is incorporated herein by reference. More recently, machine learning models have been used to improve and / or replace traditional threshold tissue segmentation techniques. For example, Skourt BA et al., “Lung CT Image Segmentation Using Deep Neural Networks,” Procedia Computer Science, 127:109-13 (2018), the disclosure of which is incorporated herein by reference, describe the use of a U-Net deep learning model to segment tissues in CT imaging data. Other examples of the use of machine learning models for tissue segmentation are described in M. Havaei et al., “Brain tumor segmentation with Deep Neural Networks,” Med. Image Anal., 35:18-31 (2017), Akkus Z. et al., “Deep Learning for Brain MRI Segmentation: State of the Art and Future Directions,” Journal of Digital Imaging, 30(4):449-59 (2017), and Badinarayanan V. et al., “SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation,” arXiv1511.00561 (2015), the disclosures of which are incorporated herein by reference.Similarly, U.S. Patent No. 10,991,097, the disclosure of which is incorporated herein by reference, describes a machine learning model for identifying cell types present in a slide. Similar methods can be applied to identify cell types in medical imaging.

[0082] Non-limiting examples of models that may be used for image segmentation include neural networks, support vector machines, naive Bayes models, nearest neighbor models, boosted tree models, random forest models, and clustering models. In some embodiments, the model is a machine learning model or algorithm. In some embodiments, the model is an unsupervised learning algorithm. One example of an unsupervised learning algorithm is cluster analysis.

[0083] In some embodiments, the model is supervised machine learning. Non-limiting examples of supervised learning algorithms include, but are not limited to, logistic regression, neural networks, support vector machines, naive Bayes algorithms, nearest neighbor algorithms, random forest algorithms, decision tree algorithms, boosted tree algorithms, multinomial logistic regression algorithms, linear models, linear regression, gradient boosting, mixture models, hidden Markov models, Gaussian NB algorithms, linear discriminant analysis, or any combination thereof. In some embodiments, the model is a multinomial classifier algorithm. In some embodiments, the model is a two-stage stochastic gradient descent (SGD) model. In some embodiments, the model is a deep neural network (e.g., a deep extensive sample-level classifier).

[0084] In some embodiments, an untrained model (e.g., an “untrained classifier” and / or an “untrained neural network”) includes a machine learning model or algorithm, e.g., a classifier or neural network, that has not been trained on a target dataset. In some embodiments, “training a model” (e.g., “training a neural network”) refers to the process of training an untrained or partially trained model (e.g., an “untrained or partially trained neural network”). For example, consider the case of a plurality of training samples including a corresponding plurality of medical images (e.g., of a medical dataset). The plurality of medical images, along with corresponding measured representations of one or more features for each medical image (hereinafter, the training dataset), are applied as collective inputs to the untrained or partially trained model to train the untrained or partially trained model on representations that identify features associated with morphological classes, thereby obtaining a trained model. Furthermore, it will be understood that the term “untrained model” does not exclude the possibility that transfer learning techniques may be used in such training of an untrained or partially trained model. For example, Fernandes et al., 2017, “Transfer Learning with Partial Observability Applied to Cervical Cancer Screening,” Pattern Recognition and Image Analysis:8 thIberian Conference Proceedings, pp. 243-250, incorporated herein by reference in its entirety for all purposes, provides a non-limiting example of such transfer learning. In instances where transfer learning is used, the untrained model described above is provided with additional data beyond that of the primary training dataset. That is, in a non-limiting example of a transfer learning embodiment, the untrained model receives (i) a plurality of images and measured representations for each respective image (the primary training dataset), and (ii) additional data. In some embodiments, this additional data is in the form of parameters (e.g., coefficients, weights, and / or hyperparameters) learned from another auxiliary training dataset. Furthermore, while a description of a single auxiliary training dataset is disclosed, it will be understood that there is no limit to the number of auxiliary training datasets that may be used to supplement the primary training dataset in training the untrained model in this disclosure. For example, in some embodiments, two or more auxiliary training datasets, three or more auxiliary training datasets, four or more auxiliary training datasets, or five or more auxiliary training datasets are used to supplement the primary training dataset through transfer learning, where each such auxiliary dataset is different from the primary training dataset. In such embodiments, any method of transfer learning may be used. For example, consider the case where, in addition to the primary training dataset, there are a first auxiliary training dataset and a second auxiliary training dataset: The parameters learned from the first auxiliary training dataset (by applying a first model to the first auxiliary training dataset) may be applied to the second auxiliary training dataset using transfer learning techniques (e.g., a second model that is the same as or different from the first model), resulting in a trained intermediate model, the parameters of which are then applied to the primary training dataset, which, together with the primary training dataset itself, is applied to the untrained model.Alternatively, a first set of parameters learned from a first auxiliary training dataset (by application of a first model to the first auxiliary training dataset) and a second set of parameters learned from a second auxiliary training dataset (by application of the same or different second model from the first model to the second auxiliary training dataset) may each be applied individually to distinct instances of the primary training dataset (e.g., by separate and independent matrix multiplications), and both such applications of parameters to separate instances of the primary training dataset, along with the primary training dataset itself (or some reduced form of the primary training dataset, such as principal components or regression coefficients learned from the primary training set), may then be applied to the untrained model to train the untrained model. In some cases, knowledge about objects associated with morphological classes derived from the auxiliary training datasets is used in combination with the objects and / or class-labeled images in the primary training dataset to train the untrained model.

[0085] Neural network algorithms, also known as artificial neural networks (ANNs), include convolutional and / or residual neural network algorithms (deep learning algorithms). Neural networks are machine learning algorithms that can be trained to map input data sets to output data sets. A neural network includes a group of interconnected nodes organized into multiple layers of nodes. For example, a neural network architecture may include at least an input layer, one or more hidden layers, and an output layer. A neural network may include any total number of layers and any number of hidden layers, where the hidden layers function as trainable feature extractors that enable mapping a set of input data to an output value or set of output values. As used herein, a deep learning algorithm (DNN) may be a neural network that includes multiple hidden layers, e.g., two or more hidden layers. Each layer of a neural network may include a number of nodes (or "neurons"). Nodes may receive input directly from input data or from the output of a node in a previous layer and perform a specific operation, e.g., a summation operation. In some embodiments, the connections from inputs to nodes are associated with parameters (e.g., weights and / or weight coefficients). In some embodiments, the node receives an input x i and their associated parameters. In some embodiments, the weighted sum is offset by a bias b. In some embodiments, the output of a node or neuron can be gated using an activation function f, which can be a threshold or a linear or nonlinear function. The activation function can be, for example, a rectified linear unit (ReLU) activation function, a leaky ReLU activation function, or other functions such as a saturated hyperbolic tangent function, an identity function, a binary step function, a logistic function, an arctangent function, a soft sine function, a parametric rectified linear unit function, an exponential linear unit function, a soft plus function, a bent identity function, a soft exponential function, a sinusoidal function, a sine function, a Gaussian function, or a sigmoid function, or any combination thereof.

[0086] The weight coefficients, bias values, and thresholds, or other computational parameters of a neural network, can be "taught" or "learned" during a training phase using one or more sets of training data. For example, the parameters can be trained using input data from a training data set and gradient descent or backpropagation techniques so that the output values ​​calculated by the ANN match the examples contained in the training data set. The parameters can be obtained from the learning process of a backpropagation neural network.

[0087] Any of a variety of neural networks may be suitable for use in implementing the methods disclosed herein. Examples include, but are not limited to, feedforward neural networks, radial basis function networks, recurrent neural networks, residual neural networks, convolutional neural networks, residual convolutional neural networks, etc., or any combination thereof. In some embodiments, the machine learning utilizes pre-trained and / or transfer-trained ANNs or deep learning architectures. Convolutional and / or residual neural networks may be used to analyze images of a subject according to the present disclosure.

[0088] For example, a deep neural network model includes an input layer, multiple individually parameterized (e.g., weighted) convolutional layers, and an output scorer. Each parameter (e.g., weight) of the convolutional layer and the input layer contributes to multiple parameters (e.g., weights) associated with the deep neural network model. In some embodiments, at least 100 parameters, at least 1000 parameters, at least 10,000 parameters, at least 100,000 parameters, at least 1 million parameters, at least 10 million parameters, or more parameters are associated with the deep neural network model. As such, deep neural network models cannot be solved by the human mind, and therefore require the use of a computer. In other words, given the inputs to the model, the model output, in such embodiments, must be determined using a computer rather than a human mind. See, for example, Krizhevsky et al., 2012, “Imagenet classification with deep convolutional neural networks,” in Advances in Neural Information Processing Systems 2, Pereira, Burges, Bottou, Weinberger, eds., pp. 1097-1105, Curran Associates, Inc.; Zeiler, 2012, “ADADELTA: an adaptive learning rate method,” CoRR, vol. abs / 1212.5701; and Rumelhart et al., 1988, “Neurocomputing: Foundations of research,” ch. Learning Representations by Back-propagating Errors, pp. 696-699, Cambridge, MA, USA: MIT Press, each of which is incorporated by reference herein in its entirety for all purposes.

[0089] Neural network algorithms, including convolutional neural network algorithms, suitable for use as models are disclosed, for example, in Vincent et al., 2010, “Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion,” J Mach Learn Res 11, pp. 3371-3408; Larochelle et al., 2009, “Exploring strategies for training deep neural networks,” J Mach Learn Res 10, pp. 1-40; and Hassoun, 1995, Fundamentals of Artificial Neural Networks, Massachusetts Institute of Technology, each of which is incorporated herein by reference in its entirety for all purposes. Additional exemplary neural networks suitable for use as models are disclosed in Duda et al., 2001, Pattern Classification, Second Edition, John Wiley & Sons, Inc., New York, and Hastie et al., 2001, The Elements of Statistical Learning, Springer-Verlag, New York, each of which is incorporated herein by reference in its entirety for all purposes. Additional exemplary neural networks suitable for use as models are also described in Draghici, 2003, Data Analysis Tools for DNA Microarrays, Chapman & Hall / CRC, and Mount, 2001, Bioinformatics: sequence and genome analysis, Cold Spring Harbor Laboratory Press, Cold Spring Harbor, New York, each of which is incorporated herein by reference in its entirety for all purposes.

[0090] Support vector machine. In some embodiments, the model is a support vector machine (SVM). Suitable SVM algorithms for use as models are described, for example, in Cristianini and Shawe-Taylor, 2000, "An Introduction to Support Vector Machines," Cambridge University Press, Cambridge; Boser et al., 1992, "A training algorithm for optimal margin classifiers," Proceedings of the 5th Annual ACM Workshop on Computational Learning Theory, ACM Press, Pittsburgh, Pa., pp. 142-152; Vapnik, 1998, Statistical Learning Theory, Wiley, New York, Mount; 2001, Bioinformatics: sequence and genome analysis, Cold Spring Harbor Laboratory Press, Cold Spring Harbor, NY; Duda, Pattern Classification, Second Edition, 2001, John Wiley & Sons, Inc., pp. 259, 262-265; and Hastie, 2001, The Elements of Statistical Learning, Springer, New York. York, and Furey et al., 2000, Bioinformatics 16, 906-914, each of which is incorporated herein by reference in its entirety for all purposes. When used for classification, SVMs separate a given set of binary labeled data with a hyperplane that is maximally distant from the labeled data. In cases where linear separation is not possible, SVMs can work in conjunction with "kernel" techniques that automatically achieve a nonlinear mapping to the feature space. The hyperplane that the SVM finds in the feature space can correspond to a nonlinear decision boundary in the input space. In some embodiments, multiple parameters (e.g., weights) associated with the SVM define the hyperplane.In some embodiments, the hyperplane is defined by at least 10, at least 20, at least 50, or at least 100 parameters, and the SVM model must be computed because it cannot be solved by the human mind.

[0091] Naive Bayes algorithm. In some embodiments, the model is a naive Bayes algorithm. A naive Bayes classifier suitable for use as the model is described, for example, in Ng et al., 2002, "On discriminative vs. generative classifiers: A comparison of logistic regression and naive Bayes," Advances in Neural Information Processing Systems, 14, incorporated herein by reference in its entirety for all purposes. A naive Bayes classifier is any classifier in the family of "probabilistic classifiers" based on applying Bayes' theorem with a strong (naive) independence assumption between features. In some embodiments, they are combined with kernel density estimation. See, for example, Hastie et al., 2001, The elements of statistical learning: data mining, inference, and prediction, eds. Tibshirani and Friedman, Springer, New York, incorporated herein by reference in its entirety for all purposes.

[0092] Nearest neighbor algorithm. In some embodiments, the model is a nearest neighbor algorithm. The nearest neighbor model can be memory-based and does not involve model fitting. In the nearest neighbor case, given a query point x0 (first image), the training point x0 that is closest in distance to x0 is found. (r), r, ..., k (here, training images) are identified, and then the point x is classified using its k nearest neighbors. In some embodiments, the distance to these neighbors is a function of the values ​​of the discrimination set. In some embodiments, the Euclidean distance in feature space is

number

[0093] The k-nearest neighbor model is a non-parametric machine learning method in which the input consists of the k closest training examples in feature space. The output is class membership. An object is classified by the multiple votes of its neighbors, and the object is assigned to the most common class among its k nearest neighbors (k is a positive integer, typically small). If k=1, the object is simply assigned to the class of its single nearest neighbor. See Duda et al. 2001, Pattern Classification, Second Edition, John Wiley & Sons, incorporated herein by reference in its entirety for all purposes. In some embodiments, the number of distance calculations required to solve a k-nearest neighbor model is such that it cannot be performed by the human mind, and therefore a computer is used to solve the model for a given input.

[0094] Random forests, decision trees, and boosted tree algorithms. In some embodiments, the model is a decision tree. Decision trees suitable for use as models are generally described in Duda, 2001, Pattern Classification, John Wiley & Sons, Inc., New York, pp. 395-396, which is incorporated herein by reference in its entirety for all purposes. Tree-based methods partition the feature space into a set of rectangles and then fit a model (such as a constant) to each. In some embodiments, the decision tree is a random forest regression. One specific algorithm that can be used is classification and regression trees (CART). Other specific decision tree algorithms include, but are not limited to, ID3, C4.5, MART, and random forest. CART, ID3, and C4.5 are described in Duda, 2001, Pattern Classification, John Wiley & Sons, Inc., New York, pp. 396-408 and pp. 411-412, which is incorporated herein by reference in its entirety for all purposes. CART, MART, and C4.5 are described in Hastie et al., 2001, The Elements of Statistical Learning, Springer-Verlag, New York, Chapter 9, incorporated herein by reference in its entirety for all purposes. Random forests are described in Breiman, 1999, "Random Forests--Random Features," Technical Report 567, Statistics Department, UC Berkeley, September 1999, incorporated herein by reference in its entirety for all purposes. In some embodiments, the decision tree model includes at least 10, at least 20, at least 50, or at least 100 parameters (e.g., weights and / or decisions) that cannot be solved by the human mind and therefore must be computed by a computer.

[0095] Linear discriminant analysis algorithm. Linear discriminant analysis (LDA), normal discriminant analysis (NDA), or discriminant function analysis may be a generalization of Fisher's linear discriminant, a method used in statistics, pattern recognition, and machine learning to find linear combinations of features that characterize or separate two or more classes of objects or events. The resulting combinations can be used as models (e.g., linear classifiers) in some embodiments of the present disclosure.

[0096] Mixture models and hidden Markov models. In some embodiments, the model is a mixture model such as that described in McLachlan et al., Bioinformatics 18(3):413-422, 2002. In some embodiments, particularly those that include a temporal component, the model is a hidden Markov model such as that described in Schliep et al., 2003, Bioinformatics 19(1):i255-i263.

[0097] Clustering. In some embodiments, the model is an unsupervised clustering model. In some embodiments, the model is a supervised clustering model. Clustering algorithms suitable for use as models are described, for example, in Duda and Hart, Pattern Classification and Scene Analysis, 1973, John Wiley & Sons, Inc., New York (hereinafter "Duda 1973"), pages 211-256, incorporated herein by reference in its entirety for all purposes. The problem of clustering can be described as one of discovering natural groupings within a dataset. To identify natural groupings, two challenges can be addressed. First, a way to measure the similarity (or dissimilarity) between two samples can be determined. This metric (e.g., a similarity metric) can be used to ensure that samples within one cluster are more similar to each other than to samples in other clusters. Second, a mechanism for partitioning the data into clusters using the similarity metric can be determined. One way to begin a clustering investigation is to define a distance function and calculate a matrix of distances between all pairs of samples in the training dataset. If distance is a good measure of similarity, the distance between reference entities in the same cluster may be significantly smaller than the distance between reference entities in different clusters. However, clustering may not use a distance metric. For example, a nonmetric similarity function, s(x,x'), can be used to compare two vectors, x and x'. s(x,x') is a symmetric function whose value increases when x and x' are "similar" in some respect. Once a method for measuring "similarity" or "dissimilarity" between points in a dataset is selected, clustering can use a criterion function to measure the clustering quality of any partition of the data. A partition of the dataset that maximizes the criterion function can be used to cluster the data.Specific exemplary clustering techniques that may be used in the present disclosure include, but are not limited to, hierarchical clustering (agglomerative clustering using nearest neighbor, longest distance, average linkage, centroid, or sum-of-squares algorithms), k-means clustering, fuzzy k-means clustering algorithm, and Jarvis-Patrick clustering. In some embodiments, the clustering comprises unsupervised clustering (e.g., the number of clusters is not predetermined and / or cluster assignments are not predetermined).

[0098] Ensemble of models and boosting. In some embodiments, an ensemble of models (two or more) is used. In some embodiments, boosting techniques such as AdaBoost are used in conjunction with many other types of learning algorithms to improve model performance. In this approach, the outputs of any of the models disclosed herein, or their equivalents, are combined into a weighted sum that represents the final output of the boosted model. In some embodiments, multiple outputs from the models are combined using any measure of central tendency known in the art, including, but not limited to, the mean, median, mode, weighted mean, weighted median, weighted mode, etc. In some embodiments, multiple outputs are combined using a voting method. In some embodiments, each model in the ensemble of models is weighted or unweighted.

[0099] Those skilled in the art will readily recognize other models that are applicable to the systems and methods of the present disclosure. In some embodiments, the systems, methods, and devices of the present disclosure utilize two or more models to provide an assessment with greater accuracy (e.g., to arrive at an assessment given one or more inputs). For example, in some embodiments, each respective model arrives at a corresponding assessment when provided with a respective data set. Thus, each respective model can independently arrive at a result, and the results of each respective model are then collectively validated by model comparison or amalgamation. This results in a cumulative result from the models. However, the present disclosure is not limited in this respect.

[0100] Referring to block 216, in some embodiments, method 200 includes, for each respective tissue type of the plurality of tissue types, generating a corresponding mask based on a respective set of one or more pixels assigned a label corresponding to the respective tissue type. In some embodiments, the corresponding mask is a binary construct having, for each location of the three-dimensional grid representing the volume of interest, either a first value (e.g., "1") indicating that the tissue is of the tissue type represented by the mask, or a second value (e.g., "0") indicating that the tissue is not of the tissue type represented by the mask.

[0101] Referring to block 217, in some embodiments, method 200 includes generating a visual representation of the tumor tissue of interest based on the corresponding mask of the tumor tissue. In some embodiments, referring to block 218, this includes generating a corresponding mesh surface of the tumor tissue based on an outer boundary of the corresponding mask of the tumor tissue and smoothing the edges of the corresponding mesh surface of the tumor tissue. For example, FIG. 4 shows a visualization of the smoothed masks of a pancreatic tumor 402 and a pancreas 404.

[0102] Referring to block 220, in some embodiments, each respective set of one or more pixels in the first plurality of sets of one or more pixels corresponds to a respective voxel in a uniform three-dimensional grid of voxels defined for the set of medical images and each tissue type in the plurality of tissue types, and the corresponding mask includes, for each respective voxel in the three-dimensional grid of voxels, a binary indication of whether the tissue represented in the voxel is of the respective tissue type.

[0103] In some embodiments, the three-dimensional grid is the same as a three-dimensional voxel grid established for segmenting medical images. In other embodiments, the three-dimensional grid represents a resampling of the input volume to regular intervals, for example, intervals representing the spacing of tissue in the subject. Exemplary methods for resampling medical imaging data are described in Li A. et al., "Methods for efficient, high-quality volume resampling in the frequency domain," IEEE Visualization, Austin, TX, USA, pp. 3-10, (2004) doi:10.1109 / VISUAL.2004.70, the disclosure of which is incorporated herein by reference.

[0104] Referring to block 222, in some embodiments, the corresponding mask of tumor tissue includes multiple groups of non-zero voxels, where each respective group of non-zero voxels in the multiple groups of non-zero voxels is separated from every other respective group of non-zero voxels in the multiple groups of non-zero voxels by at least one zero voxel. For example, many cancers appear as multiple tumor islands separated in space within the body. For example, panel 914 of radiology report 902 provides a visual representation of a pancreatic cancer with three tumor islands 918, 920, and 922, as shown in FIG. 9 .

[0105] Referring to block 224, in some embodiments, generating the visual representation of the tumor tissue includes generating a mesh surface corresponding to each group of non-zero voxels in the plurality of groups of non-zero voxels. In some embodiments, the system identifies each tumor island represented in the tumor tissue mask and generates a separate mask for each island, for example, as illustrated in panel 914 of the radiology report 902 illustrated in FIG.

[0106] Referring to block 226, in some embodiments, the visual representation of the tumor tissue excludes representations of each group of non-zero voxels that have a total volume below a first volume threshold. That is, in some embodiments, the system does not show tumor islands that are below the threshold volume. For example, panel 914 of radiology report 902 provides a visual representation of a pancreatic cancer with three tumor islands 918, 920, and 922, as shown in FIG. 9 . Tumor island 922 has a minimum total volume of 0.19 mL. However, smaller tumor islands represented in the tumor tissue mask may not be displayed in this panel because they have volumes below the threshold volume, e.g., 0.01 mL, 0.05 mL, 0.1 mL, etc. In some embodiments, this simplifies the image provided to the physician while still reporting all substantial tumor mass. In some embodiments, this accounts for background noise in the segmentation process that may result in misidentification of small tumor islands in the data.

[0107] Referring to block 228, in some embodiments, the method 200 includes determining a volume of cancer in the subject based on the volume contained within the mesh surface of the tumor tissue. Referring to block 230, in some embodiments, the volume is determined after smoothing the edges of the corresponding mesh surface of the tumor tissue.

[0108] Referring to block 231, in some embodiments, method 200 includes generating a visual representation of a subject's skeleton within the region of interest based on the first set of medical images. For example, FIG. 6 shows a visual representation of a pancreatic tumor 602 and pancreas 604 positioned relative to a visual representation of a patient's skeleton 606. Visualizing the subject's skeleton provides a physician with a spatial context of the tumor to inform treatment and / or surgical intervention decisions.

[0109] Referring to block 232, in some embodiments, generating the visual representation of the skeleton includes determining a corresponding tissue density for each respective set of one or more pixels in the second plurality of sets of one or more pixels in the set of medical images. In some embodiments, the second plurality of sets of one or more pixels is a three-dimensional voxel grid established for segmenting tissue in the medical images. In other embodiments, the second plurality of sets of one or more pixels is a three-dimensional voxel grid that is different from the three-dimensional voxel grid established for segmenting tissue in the medical images.

[0110] Referring to block 234, in some embodiments, method 200 includes generating a corresponding mask of the skeleton based on each set of one or more pixels of the second plurality of sets of one or more pixels having tissue densities that satisfy one or more sets of bone density criteria. That is, in some embodiments, bone is identified in the set of medical images by simply thresholding the intensity values ​​of the image, rather than through segmentation performed on tumor tissue and one or more non-cancerous tissues. This is because healthy bone has a substantially greater density than other tissues. An exemplary scheme for thresholding intensity to distinguish soft tissue from bone is provided in Chougule VN et al., “Clinical Case Study: Spine Modeling for Minimum Invasive Surgeries (MISS) using Rapid Prototyping,” Procedia Engineering 97:212-19 (2014), the disclosure of which is incorporated herein by reference.

[0111] If the second plurality of sets of one or more pixels is a three-dimensional voxel grid that is different from the three-dimensional voxel grid established for segmenting tissue in the medical image, one or more of the resulting tissue masks are resampled to ensure that all imaged tissues are set to the same coordinate system.

[0112] Referring to block 236, in some embodiments, the first set of medical images includes computed tomography (CT) scans, and the set of one or more bone mineral density criteria includes a first criterion satisfied by a measure of central tendency for pixel intensities of a respective set of one or more pixels being greater than a first threshold of 300-500 Hounsfield Units (HU). In some embodiments, the first threshold is between 200 and 600 HU. In some embodiments, the first threshold is between 200 and 550 HU. In some embodiments, the first threshold is between 200 and 500 HU. In some embodiments, the first threshold is between 200 and 475 HU. In some embodiments, the first threshold is between 200 and 450 HU. In some embodiments, the first threshold is between 200 and 425 HU. In some embodiments, the first threshold is between 200 and 400 HU. In some embodiments, the first threshold is between 250 and 600 HU. In some embodiments, the first threshold is between 250 and 550 HU. In some embodiments, the first threshold is 250 to 500 HU. In some embodiments, the first threshold is 250 to 475 HU. In some embodiments, the first threshold is 250 to 450 HU. In some embodiments, the first threshold is 250 to 425 HU. In some embodiments, the first threshold is 250 to 400 HU. In some embodiments, the first threshold is 275 to 600 HU. In some embodiments, the first threshold is 275 to 550 HU. In some embodiments, the first threshold is 275 to 500 HU. In some embodiments, the first threshold is 275 to 475 HU. In some embodiments, the first threshold is 275 to 450 HU. In some embodiments, the first threshold is 275 to 425 HU. In some embodiments, the first threshold is 275 to 400 HU. In some embodiments, the first threshold is 300 to 600 HU. In some embodiments, the first threshold is 300 to 550 HU. In some embodiments, the first threshold is 300 to 500 HU. In some embodiments, the first threshold is 300 to 475 HU.In some embodiments, the first threshold is 300 to 450 HU. In some embodiments, the first threshold is 300 to 425 HU. In some embodiments, the first threshold is 300 to 400 HU. In some embodiments, the first threshold is 325 to 600 HU. In some embodiments, the first threshold is 325 to 550 HU. In some embodiments, the first threshold is 325 to 500 HU. In some embodiments, the first threshold is 325 to 475 HU. In some embodiments, the first threshold is 325 to 450 HU. In some embodiments, the first threshold is 325 to 425 HU. In some embodiments, the first threshold is 325 to 400 HU. In some embodiments, the first threshold is 350 to 600 HU. In some embodiments, the first threshold is 350 to 550 HU. In some embodiments, the first threshold is 350 to 500 HU. In some embodiments, the first threshold is 350-475 HU. In some embodiments, the first threshold is 350-450 HU. In some embodiments, the first threshold is 350-425 HU. In some embodiments, the first threshold is 350-400 HU. In some embodiments, the first threshold is 200, 225, 250, 275, 300, 325, 350, 375, 400, 425, 450, 475, 500, 525, 550, 575, or 600 HU. In some embodiments, the first threshold is 200±5%, 225±5%, 250±5%, 275±5%, 300±5%, 325±5%, 350±5%, 375±5%, 400±5%, 425±5%, 450±5%, 475±5%, 500±5%, 525±5%, 550±5%, 575±5%, or 600±5% HU. In some embodiments, the first threshold is 200±10%, 225±10%, 250±10%, 275±10%, 300±10%, 325±10%, 350±10%, 375±10%, 400±10%, 425±10%, 450±10%, 475±10%, 500±10%, 525±10%, 550±10%, 575±10%, or 600±10% HU.

[0113] Referring to block 238, in some embodiments, the first set of medical images includes computed tomography (CT) scans, and the set of one or more bone mineral density criteria includes a second criterion met by a measure of central tendency for pixel intensities of a respective set of one or more pixels being less than a second threshold of 1750-2500 Hounsfield Units (HU). In some embodiments, the second threshold is 1600-3000 HU. In some embodiments, the second threshold is 1600-2900 HU. In some embodiments, the second threshold is 1600-2800 HU. In some embodiments, the second threshold is 1600-27000 HU. In some embodiments, the second threshold is 1600-2600 HU. In some embodiments, the second threshold is 1600-2550 HU. In some embodiments, the second threshold is 1600-2500 HU. In some embodiments, the second threshold is 1600-2450 HU. In some embodiments, the second threshold is 1600 to 2400 HU. In some embodiments, the second threshold is 1600 to 2350 HU. In some embodiments, the second threshold is 1600 to 2300 HU. In some embodiments, the second threshold is 1650 to 3000 HU. In some embodiments, the second threshold is 1650 to 2900 HU. In some embodiments, the second threshold is 1650 to 2800 HU. In some embodiments, the second threshold is 1650 to 27000 HU. In some embodiments, the second threshold is 1650 to 2600 HU. In some embodiments, the second threshold is 1650 to 2550 HU. In some embodiments, the second threshold is 1650 to 2500 HU. In some embodiments, the second threshold is 1650 to 2450 HU. In some embodiments, the second threshold is 1650 to 2400 HU. In some embodiments, the second threshold is 1650 to 2350 HU. In some embodiments, the second threshold is 1650 to 2300 HU. In some embodiments, the second threshold is 1700 to 3000 HU. In some embodiments, the second threshold is 1700 to 2900 HU.In some embodiments, the second threshold is 1700 to 2800 HU. In some embodiments, the second threshold is 1700 to 27000 HU. In some embodiments, the second threshold is 1700 to 2600 HU. In some embodiments, the second threshold is 1700 to 2550 HU. In some embodiments, the second threshold is 1700 to 2500 HU. In some embodiments, the second threshold is 1700 to 2450 HU. In some embodiments, the second threshold is 1700 to 2400 HU. In some embodiments, the second threshold is 1700 to 2350 HU. In some embodiments, the second threshold is 1700 to 2300 HU. In some embodiments, the second threshold is 1725 to 3000 HU. In some embodiments, the second threshold is 1725 to 2900 HU. In some embodiments, the second threshold is 1725 to 2800 HU. In some embodiments, the second threshold is 1725 to 27,000 HU. In some embodiments, the second threshold is 1725 to 2600 HU. In some embodiments, the second threshold is 1725 to 2550 HU. In some embodiments, the second threshold is 1725 to 2500 HU. In some embodiments, the second threshold is 1725 to 2450 HU. In some embodiments, the second threshold is 1725 to 2400 HU. In some embodiments, the second threshold is 1725 to 2350 HU. In some embodiments, the second threshold is 1725 to 2300 HU. In some embodiments, the second threshold is 1750 to 3,000 HU. In some embodiments, the second threshold is 1750 to 2900 HU. In some embodiments, the second threshold is 1750 to 2800 HU. In some embodiments, the second threshold is 1750 to 27,000 HU. In some embodiments, the second threshold is between 1750 and 2600 HU. In some embodiments, the second threshold is between 1750 and 2550 HU. In some embodiments, the second threshold is between 1750 and 2500 HU. In some embodiments, the second threshold is between 1750 and 2450 HU. In some embodiments, the second threshold is between 1750 and 2400 HU.In some embodiments, the second threshold is 1750 to 2350 HU. In some embodiments, the second threshold is 1750 to 2300 HU. In some embodiments, the second threshold is 1775 to 3000 HU. In some embodiments, the second threshold is 1775 to 2900 HU. In some embodiments, the second threshold is 1775 to 2800 HU. In some embodiments, the second threshold is 1775 to 27000 HU. In some embodiments, the second threshold is 1775 to 2600 HU. In some embodiments, the second threshold is 1775 to 2550 HU. In some embodiments, the second threshold is 1775 to 2500 HU. In some embodiments, the second threshold is 1775 to 2450 HU. In some embodiments, the second threshold is 1775 to 2400 HU. In some embodiments, the second threshold is 1775 to 2350 HU. In some embodiments, the second threshold is 1775 to 2300 HU. In some embodiments, the second threshold is 1800 to 3000 HU. In some embodiments, the second threshold is 1800 to 2900 HU. In some embodiments, the second threshold is 1800 to 2800 HU. In some embodiments, the second threshold is 1800 to 27000 HU. In some embodiments, the second threshold is 1800 to 2600 HU. In some embodiments, the second threshold is 1800 to 2550 HU. In some embodiments, the second threshold is 1800 to 2500 HU. In some embodiments, the second threshold is 1800 to 2450 HU. In some embodiments, the second threshold is 1800 to 2400 HU. In some embodiments, the second threshold is 1800 to 2350 HU. In some embodiments, the second threshold is 1800 to 2300 HU. In some embodiments, the second threshold is between 1850 and 3000 HU. In some embodiments, the second threshold is between 1850 and 2900 HU. In some embodiments, the second threshold is between 1850 and 2800 HU. In some embodiments, the second threshold is between 1850 and 27000 HU. In some embodiments, the second threshold is between 1850 and 2600 HU.In some embodiments, the second threshold is 1850 to 2550 HU. In some embodiments, the second threshold is 1850 to 2500 HU. In some embodiments, the second threshold is 1850 to 2450 HU. In some embodiments, the second threshold is 1850 to 2400 HU. In some embodiments, the second threshold is 1850 to 2350 HU. In some embodiments, the second threshold is 1850 to 2300 HU. In some embodiments, the second threshold is 1900 to 3000 HU. In some embodiments, the second threshold is 1900 to 2900 HU. In some embodiments, the second threshold is 1900 to 2800 HU. In some embodiments, the second threshold is 1900 to 27000 HU. In some embodiments, the second threshold is 1900 to 2600 HU. In some embodiments, the second threshold is 1900 to 2550 HU. In some embodiments, the second threshold is 1900 to 2500 HU. In some embodiments, the second threshold is 1900 to 2450 HU. In some embodiments, the second threshold is 1900 to 2400 HU. In some embodiments, the second threshold is 1900 to 2350 HU. In some embodiments, the second threshold is 1900 to 2300 HU. In some embodiments, the second threshold is 1600, 1625, 1650, 1675, 1700, 1725, 1750, 1775, 1800, 1825, 1850, 1875, or 1900 HU. In some embodiments, the second threshold is 1600±5%, 1625±5%, 1650±5%, 1675±5%, 1700±5%, 1725±5%, 1750±5%, 1775±5%, 1800±5%, 1825±5%, 1850±5%, 1875±5%, or 1900±5% HU. In some embodiments, the second threshold is 1600±10%, 1625±10%, 1650±10%, 1675±10%, 1700±10%, 1725±10%, 1750±10%, 1775±10%, 1800±10%, 1825±10%, 1850±10%, 1875±10%, or 1900±10% HU.

[0114] Referring to block 240, in some embodiments, the method then includes generating a corresponding mesh surface of the skeleton based on an outer boundary of the corresponding mask of the skeleton. Referring to block 242, in some embodiments, the method includes smoothing edges of the corresponding mesh surface of the skeleton.

[0115] Referring to block 244, in some embodiments, each respective set of one or more pixels of the second plurality of sets of one or more pixels corresponds to a respective voxel in a second three-dimensional grid of voxels defined for the set of medical images, and the corresponding mask of the skeleton includes, for each voxel of the second three-dimensional grid of voxels, a binary indication of whether the tissue represented in the voxel is skeletal tissue or not.

[0116] Referring to block 246, in some embodiments, the visual representation of the skeleton excludes representations of respective groups of non-zero voxels within the three-dimensional grid of voxels that have a total volume below a second volume threshold. In some embodiments, this simplifies the visual representation for the clinician while still providing sufficient spatial context for the tumor location in the subject.

[0117] Referring to block 247, in some embodiments, method 200 includes displaying a visual representation of tumor tissue and a visual representation of the subject's skeleton in a single image (i) in the same spatial orientation as the set of medical images, and (ii) at the same relative size as the set of medical images. For example, radiology report 702 includes panel 706 showing a three-dimensional representation of tumor 710, as well as the patient's partial skeleton 708 and trachea / primary bronchi 712, to provide a spatial perspective of the NSCLC tumor. Illustration of the tumor in relation to non-cancerous tissue and skeleton assists physicians in evaluating treatment options, including the feasibility of surgical intervention.

[0118] Referring to block 248, in some embodiments, method 200 includes generating a report including a single image that includes a visual representation of the tumor tissue and a visual representation of the subject's skeleton. For example, radiology reports 702, 802, and 902 illustrated in Figures 7, 8, and 9, respectively, each include panels 706, 806, and 930, showing the tumor in context with the subject's partial skeleton to aid in clinical evaluation of potential therapeutic interventions.

[0119] Referring to block 250, in some embodiments, the report further includes one or more measurements for cancerous tissue. For example, radiology reports 702, 802, and 902 illustrated in FIGS. 7, 8, and 9, respectively, each include one or more measurements for cancerous tissue. Report 702 provides, for example, in panels 716, 722, and 726, the total tumor volume for the cancer at each time point for which imaging data is available. Report 802 provides additional tumor measurements in panels 816 and 818, including various diameter and density measurements for the tumor at each time point for which imaging data is available. Report 902 provides tumor volume and diameter measurements in panel 914. Providing common measurements for cancerous tissue on the radiology reports described herein simplifies the review and evaluation process for clinicians, who would otherwise need to generate these calculations themselves.

[0120] Referring to block 252, in some embodiments, the one or more measurements for the cancerous tissue include at least one measurement selected from a volume for the cancerous tissue, including a cross-sectional length of the cancerous tissue, a density measurement of the cancerous tissue, a density of non-cancerous tissue within the plurality of tissues, a distance from the cancerous tissue to non-cancerous tissue within the plurality of tissues, and a distance from the cancerous tissue to a vascular structure in the three-dimensional region of interest of the subject. Referring to block 254, in some embodiments, the report further includes a change in the measurement of the cancerous tissue over time, for example, a difference between a measurement of the tumor (e.g., tumor volume) at a first time point corresponding to a first imaging analysis and a measurement of the tumor (e.g., tumor volume) at a second time point corresponding to a second imaging analysis. Reporting the difference in measurements allows for easy assessment of tumor progression without the need for the clinician to perform further measurements or calculations.

[0121] Referring to block 256, in some embodiments, the report further includes, for each respective time point in the plurality of time points, a respective visual representation of the tumor tissue at the respective time point. For example, radiology reports 702 and 802 provide visual representations of each tumor from the multiple image assessments in panels 716, 722, and 726, and panels 816 and 818, respectively.

[0122] Referring to block 258, in some embodiments, the report further includes a timeline showing the timing of one or more events related to the cancer in the subject. For example, the radiology report 702 shown in FIG. 7 shows a related medical timeline 713 of a patient's NSCLC, from the date of initial cancer diagnosis (April 2018) to the most recent related medical event (January 2019). The timeline includes event points 714, such as initial diagnosis, medical imaging analysis, and therapeutic intervention (e.g., initiation of a specific therapy). The combined display of the timeline 713 with medical interventions and timeline panels 716, 722, and 726 facilitates rapid assessment by the clinician. For example, it can be quickly recognized that the combined IO therapy initiated on June 11, 2018, did not halt tumor progression because the total volume significantly increased from 19.5 mL on May 23, 2018 to 188.6 mL on September 18, 2018.

[0123] Referring to block 260, in some embodiments, the event in the one or more events related to cancer in the subject is selected from cancer imaging, cancer medical intervention, cancer diagnosis, cancer prognosis, and cancer progression or regression. Referring to block 262, in some embodiments, the report further includes a prognosis of cancer in the subject. In some embodiments, the prognosis is a predicted survival time. In some embodiments, the prognosis is a chance of survival for a defined time interval (e.g., 6 months, 12 months, 2 years, 5 years, or 10 years survival). In some embodiments, the prognosis is a predicted response to therapy.

[0124] Referring to block 264, in some embodiments, the report is displayed on a user interface on a second computer system that includes one or more processors, memory coupled to the one or more processors, and a display. In some embodiments, the report is an interactive report. For example, the report 902 illustrated in FIG. 9 includes multiple user interface (UI) affordances 906, 908, 910, 912, etc., allowing a clinician to easily navigate between images of the patient's tumor at different time points.

[0125] Referring to block 266, in some embodiments, the single image further includes a visual representation of a reference shape in the single image corresponding to the volume of the subject's three-dimensional region of interest, in the same proportional size as the visual representation of the tumor tissue and the visual representation of the subject's skeleton relative to the set of medical images. For example, reports 702, 802, and 902 include visual representations of reference volumes to provide context regarding the size of the tumor in panels 716, 722, and 726, panels 816 and 818, and panel 914, respectively.

[0126] Referring to block 268, in some embodiments, the single image further includes a visual representation of a reference axial, coronal, or sagittal slice of the subject's three-dimensional region of interest from the first set of medical images. Referring to block 270, in some embodiments, the reference axial, coronal, or sagittal slice bisects the tumor tissue. For example, FIG. 5 shows a visual representation of a pancreatic tumor 502 and pancreas 504 bisected by a coronal plane 506 and a sagittal plane 508 from a CT scan. Co-presentation of the tumor representation and the reference axial, coronal, or sagittal slice of the medical imaging set provides additional context to a clinician evaluating a potential therapeutic intervention.

[0127] Referring to block 272, in some embodiments, the single image includes a three-dimensional representation of the subject's tumor tissue and skeleton. Programs for generating three-dimensional representations are known in the art. For example, PyVista is a helper library for the Visualization Toolkit (VTK) open source software.

[0128] Referring to block 274, in some embodiments, the single image includes a two-dimensional representation of the subject's tumor tissue and skeleton. Programs for generating two-dimensional representations are known in the art. In addition to VTK, Matplotlib can be used to generate the two-dimensional representation. See, for example, Hunter, JD, "Matplotlib: A 2D graphics environment," Computing in Science & Engineering, 9(3):90-95 (2007), the disclosure of which is incorporated herein by reference.

[0129] Referring to block 276, in some embodiments, the single image further includes a visual representation of non-cancerous tissue of interest in the same spatial orientation and at the same relative size as the set of medical images.

[0130] 3 also illustrates a method for visualizing cancer in a subject, the method being implemented in a computer system including one or more processors and a memory coupled to the one or more processors, the memory including one or more programs configured to be executed by the one or more processors.

[0131] Method 300 includes acquiring (302) a first set of medical images of a three-dimensional region of interest of a subject from a first medical database from a most recent medical imaging evaluation for cancer performed at a first time. In some embodiments, the acquisition is in response to receiving a request, e.g., made by a clinician, for a radiology report of cancer in the subject. In other embodiments, the acquisition is automated by the system, e.g., according to a schedule for generating a radiology report after a medical imaging evaluation.

[0132] In some embodiments, method 300 also includes querying 304 a second medical database to identify one or more medical imaging assessments of the cancer performed before the first time point. In some embodiments, the second medical database is the same medical database as the first medical database, e.g., an electronic repository of the patient's medical imaging data or a medical record. In some embodiments, the second medical database is different from the first medical database.

[0133] The method 300 includes generating (306) a radiology report of cancer in the subject based on the first set of medical images, the report including a first image showing a visual representation of the cancer and a visual representation of at least a portion of the skeleton for the subject. In some embodiments, the first image further includes a visual representation of non-cancerous tissue of the subject based on the first set of medical images. In some embodiments, the first set of medical images includes a computed tomography (CT) scan of a three-dimensional region of the subject.

[0134] In some embodiments, the method 300 includes obtaining information for one or more medical imaging evaluations performed before the first time point from a second medical database and adding (310) second images to the radiology report. The second images show a visual representation of the cancer based on each of the one or more medical imaging evaluations performed before the first time point. In some embodiments, the second images further include a reference shape corresponding to a volume of a three-dimensional region of interest of the subject.

[0135] In some embodiments, method 300 includes identifying 306 one or more events related to cancer in the subject from the subject's medical record and adding 312 a timeline to the radiology report, the timeline including an indication of a corresponding date for each respective event in the one or more events related to cancer in the subject. In some embodiments, each event in the one or more events related to cancer in the subject is selected from an initial diagnosis, a medical imaging evaluation, and a therapeutic intervention.

[0136] In some embodiments, the first image is generated by a process that includes segmenting the first set of medical images by assigning, to each respective set of one or more pixels in the first plurality of sets of one or more pixels in the set of medical images, a label corresponding to a respective tissue type among the plurality of tissue types based on one or more corresponding pixel values ​​of the respective set of one or more pixels, e.g., as described above. In some embodiments, the process also includes generating, for each respective tissue type of the plurality of tissue types, a corresponding mask based on each respective set of one or more pixels that have been assigned a label corresponding to the respective tissue type, e.g., as described herein. In some embodiments, the process also includes generating a visual representation of tumor tissue of the subject based on the corresponding mask of the tumor tissue, e.g., as described herein. In some embodiments, the process also includes generating a visual representation of the subject's skeleton within the region of interest based on the first set of medical images, e.g., as described above. In some embodiments, the process also includes displaying the visual representation of the tumor tissue and the visual representation of the subject's skeleton in a single image, in the same spatial orientation and with the same relative size as the set of medical images, e.g., as described herein.

[0137] In some embodiments, generating the visual representation of the tumor tissue includes generating a corresponding mesh surface for the tumor tissue based on an outer boundary of a corresponding mask for the tumor tissue, and smoothing edges of the corresponding mesh surface for the tumor tissue.

[0138] In some embodiments, the method 300 also includes determining a volume of the cancer in the subject based on the volume contained within the mesh surface of the tumor tissue, in some embodiments, the volume is determined after smoothing the edges of the corresponding mesh surface of the tumor tissue.

[0139] In some embodiments, generating the visual representation of the skeleton includes, for each respective set of one or more pixels in the second plurality of sets of one or more pixels in the set of medical images, determining a corresponding tissue density, e.g., as described above. In some embodiments, this also includes generating a corresponding mask of the skeleton based on each set of one or more pixels in the second plurality of sets of one or more pixels having a tissue density that meets one or more sets of bone density criteria, e.g., as described herein. In some embodiments, this also includes generating a corresponding mesh surface of the skeleton based on an outer boundary of the corresponding mask of the skeleton.

[0140] In some embodiments, the first set of medical images includes computed tomography (CT) scans, and the set of one or more bone density criteria includes a first criterion satisfied by a measure of central tendency for pixel intensities of a respective set of one or more pixels greater than a first threshold of 300 to 500 Hounsfield Units (HU).

[0141] In some embodiments, the first set of medical images includes computed tomography (CT) scans, and the set of one or more bone density criteria includes a second criterion satisfied by a measure of central tendency of pixel intensities of a respective set of one or more pixels being less than a second threshold of 1750 to 2500 Hounsfield Units (HU).

[0142] Example Implementation

[0143] Figure 7 - NSCLC report with tumor timeline and chronology

[0144] 7 illustrates an exemplary radiology report generated using the methods and systems described herein, according to some embodiments. Specifically, FIG. 7 illustrates a radiology report 702 for Jane Doe, a patient with non-small cell lung cancer (NSCLC). The report includes biometric information 704 about the patient, obtained, for example, from medical records 47. The biometric information provides the physician with targeted information relevant to clinical treatment decisions, such as the patient's age / date of birth, previous diagnoses of cancer, biometric information for the medical imaging file from which the tumor visualization 706 was prepared, and the date the report was generated.

[0145] The report 702 also includes a panel 706 showing a three-dimensional representation of the tumor 710, as well as a partial skeleton 708 and trachea / primary bronchi 712 of the patient to provide a spatial perspective of the NSCLC tumor. The report 702 also shows a related medical timeline 713 of the patient's NSCLC, from the date of initial diagnosis of the cancer (April 2018) to the most recent related medical event of the cancer (January 2019). The timeline includes points 714 of events such as initial diagnosis, medical imaging analysis, and therapeutic intervention (e.g., initiation of specific therapy).

[0146] The report 702 also includes time series images 716, 722, and 726 of visual representations of the NSCLC (e.g., 718, 724) for visualizing the progression of the cancer over time. Each image is generated from a different medical imaging assessment, as indicated immediately above the image. To facilitate rapid assessment of cancer progression, the time series images do not include a visual representation of the subject's skeleton or proximal non-cancerous tissue (e.g., trachea / primary bronchi). However, the time series images show a reference volume 720 at the bottom left of the panel to provide perspective on the size of the tumor, as well as the total volume of the tumor islands. In some embodiments, the time series of images includes a visual representation of the subject's skeleton and / or one or more non-cancerous tissues / organs.

[0147] The combined display of timeline 713 with medical intervention and time series panels 716, 722, and 726 facilitates rapid assessment by clinicians. For example, it can be quickly recognized that combination IO therapy initiated on June 11, 2018, did not halt tumor progression because the tumor significantly increased from a total volume of 19.5 mL on May 23, 2018, to 188.6 mL on September 18, 2018. The tumor then continued to grow to a volume of 217.7 mL on January 8, 2019.

[0148] Figure 8 - Pancreatic cancer report with tumor timeline

[0149] 8 illustrates another exemplary radiology report generated using the methods and systems described herein, according to some embodiments. Specifically, FIG. 8 illustrates a radiology report 802 for Jane Doe, a patient with pancreatic cancer. The report includes biometric information 804 about the patient, obtained, for example, from medical records 47. The biometric information provides physicians with targeted information relevant to clinical treatment decisions, such as the patient's age / date of birth, previous diagnoses of cancer, biometric information for the medical imaging file from which the tumor visualization 806 was prepared, and the date the report was generated.

[0150] Report 802 also includes a single image 806 showing a three-dimensional representation of tumor 814, as well as a partial skeleton 808 of the patient and pancreas 812 to provide a spatial perspective of the tumor. Unlike report 702, report 802 does not include an associated medical timeline. However, report 802 does include side-by-side images 816 and 818 of the pancreatic cancer and pancreas at two time points, along with a reference cube for size. Images 816 and 818 also provide several measured characteristics of the cancer, including volume, maximum diameter, maximum axial diameter, maximum coronal diameter, maximum sagittal diameter, mean tumor density, 10th percentile tumor density, and 90th percentile tumor density.

[0151] The side-by-side images 816 and 818 of the pancreatic cancer and pancreas at the two time points facilitate a clinician's rapid assessment, for example, that the tumor has shrunk in volume from 13 mL in July 2021 to 4.6 mL in October 2021, as well as in all other reported size metrics, and that the cancer is responding to the treatment regimen.

[0152] Figure 9 - Non-Small Cell Lung Cancer (NSCLC) Report with Tumor Timeline

[0153] FIG. 9 illustrates another exemplary radiology report generated using the methods and systems described herein, according to some embodiments. Specifically, FIG. 9 illustrates an interactive radiology report 902 displayed on a computer. The interactive radiology report includes multiple user interface (UI) affordances 906, 908, 910, 912, etc., allowing a clinician to easily navigate between images of a patient's tumor at different times. For example, the interactive radiology report 902 illustrated in FIG. 9 shows an image of an NSCLC tumor from a medical imaging evaluation performed on October 7, 2021, as indicated by the bold and underlined affordance 910. By selecting another one of the UI affordances, the user can navigate to images of the tumor at different times. For example, by selecting affordance 908, the computer replaces the current user interface display with a user interface showing one of the visual representations of the tumor generated from a medical imaging evaluation performed on July 14, 2021, providing the user with information about how the tumor progressed between July 14, 2021, and October 7, 2021. Selecting UI affordance 906 navigates to a user interface showing the visual representation of the tumor as the medical imaging evaluation immediately preceding the currently displayed medical imaging evaluation. Similarly, selecting UI affordance 912 navigates to a user interface showing the visual representation of the tumor as the medical imaging evaluation immediately following the currently displayed medical imaging evaluation.

[0154] Report 902 includes a panel 914 showing a three-dimensional visual representation of three tumor islands 918, 920, and 922, along with a reference volume 920 for perspective on the size of the tumor islands. Panel 914 also provides several measured characteristics of each tumor island, including volume, maximum diameter, maximum axial diameter, maximum coronal diameter, and maximum sagittal diameter, for quick reference by the clinician. Report 902 also includes a panel 930 showing a three-dimensional representation of tumor 934, as well as a partial skeleton 932 of the patient to provide a spatial perspective of the tumor. While the information and visualization illustrated in FIG. 9 does not include a direct comparison of tumors over time or information regarding tumor characteristics, in some embodiments, one or more such comparisons are added to the user interface display to further facilitate comparison of tumors over time.

[0155] Easy navigation between tumor representations from different medical imaging assessments facilitates rapid assessment by clinicians, for example, allowing them to quickly recognize that a cancer is responding or not responding to therapy over time.

[0156] Multiple instances may be provided for components, operations, or structures described herein as a single instance. Finally, boundaries between various components, operations, and data stores are somewhat arbitrary, and particular operations are illustrated in the context of specific exemplary configurations. Other forms of functionality are contemplated and may fall within the scope of the implementations. In general, structures and functions presented as separate components in the exemplary configurations may be implemented as combined structures or components. Similarly, structures and functions presented as a single component may be implemented as separate components. These and other variations, modifications, additions, and improvements are within the scope of the implementations.

[0157] Also, while terms such as "first," "second," and the like may be used herein to describe various elements, it will be understood that these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, a first attribute can be referred to as a second attribute, and similarly, a second attribute can be referred to as a first attribute, without changing the meaning of the description, so long as all occurrences of "first attribute" are consistently renamed and all occurrences of "second attribute" are consistently renamed. A first attribute and a second attribute are both attributes, but are not the same attribute unless otherwise specified.

[0158] The terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the claims. When used in describing implementations and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly dictates otherwise. The term "and / or" as used herein refers to and includes any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms "comprises" and / or "comprising" as used herein specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0159] The term "if" as used herein may be interpreted to mean "when," "upon," or, depending on the context, "in response to determining," or "in accordance with determining," or "in response to detecting," that a stated condition precedent is true. Similarly, the phrases "if it is determined that the stated condition precedent is true," or "if the stated condition precedent is true," or "when the stated condition precedent is true," may be interpreted to mean "when determining," or "in response to determining," or "in accordance with determining," or "when detecting," or "in response to detecting," that the stated condition precedent is true, depending on the context.

[0160] The foregoing description has included example systems, methods, techniques, instruction sequences, and computing machine program products embodying example implementations. For purposes of explanation, numerous specific details have been set forth in order to provide an understanding of various implementations of the inventive subject matter. However, it will be apparent to those skilled in the art that implementations of the inventive subject matter may be practiced without these specific details. In general, well-known example instructions, protocols, structures, and techniques have not been shown in detail.

[0161] The foregoing description is set forth with reference to specific implementations for purposes of explanation. However, the illustrative discussion above is not intended to be exhaustive or to limit the implementations to the precise form disclosed. Many modifications and variations are possible in light of the above teachings. The implementations were chosen and described in order to best explain the principles and practice so that those skilled in the art can best utilize various implementations with various modifications suited to their implementation and particular applications.

Claims

1. 1. A method of visualizing cancer in a subject, comprising: A computer system including one or more processors and a memory coupled to the one or more processors, the memory including one or more programs configured to be executed by the one or more processors, the one or more programs comprising: A) acquiring a first set of medical images of a three-dimensional region of interest of the subject, the first set of medical images is acquired at a first time point using a first medical imaging modality; the three-dimensional region of interest of the subject comprises a plurality of tissue types; obtaining the plurality of tissue types, wherein the plurality of tissue types comprises tumor tissue and non-cancerous tissue of the subject; B) segmenting the first set of medical images by assigning, to each respective set of one or more pixels in a first plurality of sets of one or more pixels in the set of medical images, a label corresponding to a respective tissue type among the plurality of tissue types based on one or more corresponding pixel values ​​for the respective set of one or more pixels; C) for each respective tissue type in the plurality of tissue types, generating a corresponding mask based on a respective set of one or more pixels assigned the label corresponding to the respective tissue type; D) generating a visual representation of the tumor tissue of the subject based on the corresponding mask for the tumor tissue; E) generating a visual representation of the subject's anatomy within the region of interest based on the first set of medical images; F) displaying the visual representation of the tumor tissue and the visual representation of the skeleton of the subject in a single image in the same spatial orientation and at the same relative size as the set of medical images.

2. The method of claim 1 , wherein the first set of medical images comprises a computed tomography (CT) scan of the three-dimensional region of the subject.

3. 3. The method of claim 1 or 2, wherein the cancer is non-small cell lung cancer (NSCLC) and the multiple tissue types include non-cancerous lung tissue.

4. 3. The method of claim 1 or 2, wherein the cancer is pancreatic cancer and the multiple tissue types include non-cancerous pancreatic tissue.

5. The method of any one of claims 1 to 4, wherein the plurality of tissue types comprises lymphoid tissue.

6. The method of any one of claims 1 to 5, wherein the plurality of tissue types comprises vascular tissue.

7. 7. The method of claim 1, wherein generating the visual representation of the tumor tissue comprises generating a corresponding mesh surface for the tumor tissue based on an outer boundary of the corresponding mask for the tumor tissue, and smoothing edges of the corresponding mesh surface for the tumor tissue.

8. each respective set of one or more pixels in the first plurality of sets of one or more pixels corresponds to a respective voxel within a uniform three-dimensional grid of voxels defined for the set of medical images; 8. The method of claim 1, wherein for each tissue type in the plurality of tissue types, the corresponding mask comprises, for each respective voxel in the three-dimensional grid of voxels, a binary indication of whether the tissue represented in the voxel is of the respective tissue type.

9. 9. The method of claim 8, wherein the corresponding mask of the tumor tissue includes multiple groups of non-zero voxels, and each respective group of non-zero voxels in the multiple groups of non-zero voxels is separated from every other respective group of non-zero voxels in the multiple groups of non-zero voxels by at least one zero voxel.

10. 10. The method of claim 9, wherein generating the visual representation of the tumor tissue comprises generating a corresponding mesh surface for each respective group of non-zero voxels in the plurality of groups of non-zero voxels.

11. 11. The method of claim 9 or 10, wherein the visual representation of the tumor tissue excludes representations of respective groups of non-zero voxels having a total volume below a first volume threshold.

12. The method of any one of claims 7 to 11, further comprising determining the volume of the cancer in the subject based on the volume of the tumor tissue contained within the mesh surface.

13. The method of claim 12 , wherein the volume is determined after smoothing edges of the corresponding mesh surface of the tumor tissue.

14. generating the visual representation of the skeleton, determining a corresponding tissue density for each respective set of one or more pixels in the second plurality of sets of one or more pixels in the set of medical images; generating a corresponding mask of the skeleton based on each set of one or more pixels of the second plurality of sets of one or more pixels having a tissue density that meets a set of one or more bone density criteria; and generating a corresponding mesh surface for the skeleton based on an outer boundary of the corresponding mask for the skeleton.

15. 15. The method of claim 14, wherein the first set of medical images comprises computed tomography (CT) scans, and the set of one or more bone mineral density criteria comprises a first criterion met by a measure of central tendency for pixel intensities of each set of the one or more pixels being greater than a first threshold of 300 to 500 Hounsfield Units (HU).

16. 16. The method of claim 14 or 15, wherein the first set of medical images comprises computed tomography (CT) scans, and the set of one or more bone mineral density criteria comprises a second criterion met by a measure of central tendency for the pixel intensities of the respective set of one or more pixels being less than a second threshold of 1750 to 2500 Hounsfield Units (HU).

17. The method of any one of claims 14 to 16, further comprising smoothing edges of the corresponding mesh surface to the skeleton.

18. each respective set of one or more pixels in the second plurality of sets of one or more pixels corresponds to a respective voxel within a second three-dimensional grid of voxels defined for the set of medical images; 18. The method of any one of claims 14 to 17, wherein the corresponding mask for the skeleton comprises, for each respective voxel in the second three-dimensional grid of voxels, a binary indication of whether the tissue represented in the voxel is skeletal tissue or not.

19. The method of claim 18 , wherein the visual representation of the skeleton excludes representations of respective groups of non-zero voxels within the three-dimensional grid of voxels that have a total volume below a second volume threshold.

20. 20. The method of any one of claims 1 to 19, wherein said displaying F) comprises generating a report including said single image comprising said visual representation of said tumor tissue and said visual representation of said skeleton of said subject.

21. 21. The method of claim 1, wherein the single image further comprises a visual representation of a reference shape of the single image corresponding to the volume of the three-dimensional region of interest of the subject, in the same proportions of size as the visual representation of the tumor tissue and the visual representation of the skeleton of the subject relative to the set of medical images.

22. 22. The method of any one of claims 1 to 21, wherein the single image further comprises a visual representation of a reference axial, coronal, or sagittal slice of the three-dimensional region of interest of the subject from the first set of medical images.

23. 23. The method of claim 22, wherein the reference axial, coronal, or sagittal slice bisects the tumor tissue.

24. The method of any one of claims 20 to 22, wherein the report further comprises one or more measurements for the cancerous tissue.

25. the one or more measurements for the cancerous tissue are: the volume of the cancerous tissue; the cross-sectional length of the cancer tissue; a density measurement value of the cancerous tissue; the density of non-cancerous tissue within the plurality of tissues; the distance from the cancerous tissue to the non-cancerous tissue within the plurality of tissues; and and a distance from the cancerous tissue to a vascular structure in the three-dimensional region of interest of the object.

26. The method of any one of claims 20 to 25, wherein the report further comprises changes in measurements of the cancerous tissue over time.

27. The method of any one of claims 20 to 26, wherein the report further comprises, for each respective time point in the plurality of time points, a respective visual representation of the tumor tissue at the respective time point.

28. 28. The method of any one of claims 20 to 27, wherein the report further comprises a timeline indicating the timing of one or more events associated with the cancer in the subject.

29. an event in the one or more events associated with the cancer in the subject, an imaging study of the cancer; medical intervention for said cancer; diagnosing said cancer; the prognosis of said cancer, and 29. The method of claim 28, selected from the group consisting of progression or regression of the cancer.

30. 30. The method of any one of claims 20 to 29, wherein the report further comprises a prognosis for the cancer in the subject.

31. 31. The method of any one of claims 20 to 30, wherein the report is displayed on a user interface on a second computer system comprising one or more processors, a memory coupled to the one or more processors, and a display.

32. The method of any one of claims 1 to 31, wherein the single image comprises a three-dimensional representation of the tumor tissue and skeleton of the subject.

33. The method of any one of claims 1 to 31, wherein the single image comprises a two-dimensional representation of the tumor tissue and skeleton of the subject.

34. 34. The method of any one of claims 1 to 33, wherein the single image further comprises a visual representation of the non-cancerous tissue of the subject in the same spatial orientation and at the same relative size as the set of medical images.

35. 1. A method of visualizing cancer in a subject, comprising: A computer system including one or more processors and a memory coupled to the one or more processors, the memory containing one or more programs configured to be executed by the one or more processors, In response to receiving a request for a radiology report of cancer in a subject, A) obtaining from a first medical database a first set of medical images of a three-dimensional region of interest of the subject from a most recent medical imaging evaluation of the cancer performed at a first time point; B) querying a second medical database to identify one or more medical imaging evaluations of the cancer performed before the first time point; C) generating a radiology report of the cancer in the subject based on the first set of medical images, the radiology report including a first image showing a visual representation of the cancer and a visual representation of at least a portion of a skeleton for the subject.

36. 36. The method of claim 35, wherein the first image further comprises a visual representation of non-cancerous tissue of the subject based on the first set of medical images.

37. obtaining information from the second medical database for one or more medical imaging assessments performed prior to the first time point; 37. The method of claim 35 or 36, further comprising adding a second image to the radiology report, the second image displaying a visual representation of the cancer based on each medical imaging evaluation of the one or more medical imaging evaluations performed before the first time point.

38. 38. The method of claim 37, wherein the second image further comprises a reference shape corresponding to a volume in the three-dimensional region of interest of the object.

39. identifying one or more events related to the cancer in the subject from the subject's medical record; 39. The method of any one of claims 35-38, further comprising adding a timeline to the radiology report, the timeline including an indication of a corresponding date for each respective event in the one or more events related to the cancer in the subject.

40. 40. The method of claim 39, wherein each event in the one or more events associated with the cancer in the subject is selected from an initial diagnosis, a medical imaging evaluation, and a therapeutic intervention.

41. The first image is i) segmenting the first set of medical images by assigning, to each respective set of one or more pixels in a first plurality of sets of one or more pixels in the set of medical images, a label corresponding to a respective tissue type among the plurality of tissue types based on one or more corresponding pixel values ​​for the respective set of one or more pixels; ii) for each respective tissue type in the plurality of tissue types, generating a corresponding mask based on a respective set of one or more pixels assigned the label corresponding to the respective tissue type; iii) generating a visual representation of the tumor tissue of the subject based on the corresponding mask for the tumor tissue; iv) generating a visual representation of the subject's anatomy within the region of interest based on the first set of medical images; and v) displaying the visual representation of the tumor tissue and the visual representation of the skeleton of the subject in a single image, in the same spatial orientation as the set of medical images and at the same relative size as the set of medical images.

42. 42. The method of any one of claims 35 to 41, wherein generating the visual representation of the tumor tissue comprises generating a corresponding mesh surface for the tumor tissue based on an outer boundary of the corresponding mask for the tumor tissue, and smoothing edges of the corresponding mesh surface for the tumor tissue.

43. 43. The method of claim 42, further comprising determining the volume of the cancer in the subject based on the volume of the tumor tissue contained within the mesh surface.

44. 44. The method of claim 43, wherein the volume is determined after smoothing edges of the corresponding mesh surface of the tumor tissue.

45. generating the visual representation of the skeleton, determining a corresponding tissue density for each respective set of one or more pixels in the second plurality of sets of one or more pixels in the set of medical images; generating a corresponding mask of the skeleton based on each set of one or more pixels of the second plurality of sets of one or more pixels having a tissue density that meets a set of one or more bone density criteria; and generating a corresponding mesh surface for the skeleton based on an outer boundary of the corresponding mask for the skeleton.

46. 46. ​​The method of claim 45, wherein the first set of medical images comprises computed tomography (CT) scans, and the set of one or more bone mineral density criteria comprises a first criterion met by a measure of central tendency for pixel intensities of each set of the one or more pixels being greater than a first threshold of 300 to 500 Hounsfield Units (HU).

47. 47. The method of claim 46, wherein the first set of medical images comprises computed tomography (CT) scans, and the set of one or more bone mineral density criteria comprises a second criterion met by a measure of central tendency for the pixel intensities of the respective set of one or more pixels being less than a second threshold of 1750 to 2500 Hounsfield Units (HU).

48. The method of any one of claims 35 to 47, wherein the first set of medical images comprises a computed tomography (CT) scan of the three-dimensional region of the subject.

49. 1. A computer system comprising: one or more processors; A computer system comprising: a memory for storing one or more programs, said one or more programs comprising instructions for performing a method according to any one of claims 1 to 48.

50. 49. A non-transitory computer readable storage medium storing one or more programs for execution by a computer system using one or more processors, the one or more programs comprising instructions for performing a method according to any one of claims 1 to 48.

Images