Method and system for performing real-time radiology

Abstract

The present disclosure provides methods and systems directed to performing real-time and / or AI-assisted radiology. A method for processing images of a body part of a subject includes: The method may include (a) acquiring an image of a part of the subject's body; (b) using a trained algorithm to classify the image, or a derivative thereof, into one of a plurality of categories, where the classifying includes applying an image processing algorithm; (c) if the image is classified into a first category of the plurality of categories, sending the image to a first radiologist for radiological evaluation, or (ii) if the image is classified into a second category of the plurality of categories, sending the image to a second radiologist for radiological evaluation; and (d) receiving a recommendation from the first radiologist or the second radiologist to examine the subject based at least in part on the radiological analysis.

Description

[Technical Field]

[0001] cross reference

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 958,859, filed January 9, 2020, which is incorporated herein by reference in its entirety. [Background technology]

[0002]

[0002] Breast cancer is the most common cancer among women in the United States, with more than 250,000 new diagnoses in 2017 alone. Approximately 1 in 8 women will be diagnosed with breast cancer at some point in their lifetime. Despite improvements in treatment, more than 40,000 women die from breast cancer each year in the United States. Great strides have been made in reducing breast cancer mortality, in part due to widespread uptake of screening mammography. Breast cancer screening can help identify early-stage cancers, which have a much better prognosis and lower treatment costs compared to late-stage cancers. This difference can be very significant: women with localized breast cancer have a 5-year survival rate of nearly 99%, while women with metastatic breast cancer have a 5-year survival rate of 27%.

[0003]

[0003] Despite these documented benefits, uptake of screening mammography is hindered in part by poor patient experience, including long delays in obtaining an appointment, unclear pricing, long wait times to receive results, and confusing reports. Furthermore, the problems arising from a lack of pricing transparency are exacerbated by wide variations in costs across healthcare facilities. Similarly, turnaround times for receiving results are inconsistent across healthcare facilities. Additionally, significant variations in radiologist performance result in patients experiencing widely varying standards of care depending on location and income. Summary of the Invention

[0004]

[0004] The present disclosure provides methods and systems for performing radiological evaluations of subjects by using artificial intelligence to stratify medical image data into individualized radiological workflows for further screening and / or diagnostic evaluation. Such subjects may include subjects with disease (e.g., cancer) and subjects without disease (e.g., cancer). Screening may be directed toward cancer, such as breast cancer. Stratification may be based on disease-related or other assessments (e.g., estimated difficulty of the case).

[0005]

[0005] In one aspect, the present disclosure provides a method for detecting a subject's body part, the method comprising: (a) acquiring at least one image of the subject's body part; (b) using a trained algorithm to classify the at least one image, or a derivative thereof, into one of a plurality of categories, the classifying step comprising applying an image processing algorithm to the at least one image, or a derivative thereof; and (c) classifying the at least one image, or a derivative thereof, in (b), such that: (i) if the at least one image is classified into a first category of the plurality of categories, at least one and (d) receiving a radiological evaluation of the subject from the first radiologist or the second radiologist based at least in part on the radiological analysis of the at least one image or a derivative thereof.

[0006] In some embodiments, (b) includes classifying the at least one image or derivative thereof as normal, equivocal, or suspicious. In some embodiments, the method further includes sending the at least one image or derivative thereof to a classifier based on the classification of the at least one image or derivative thereof in (b). In some embodiments, (c) includes sending the at least one image or derivative thereof to a first radiologist of the first plurality of radiologists or a second radiologist of the second plurality of radiologists for radiological evaluation. In some embodiments, the at least one image or derivative thereof is a medical image.

[0007] In some embodiments, the trained algorithm is configured to classify at least one image, or a derivative thereof, as normal, equivocal, or suspicious with at least about 80% sensitivity. In some embodiments, the trained algorithm is configured to classify at least one image, or a derivative thereof, as normal, equivocal, or suspicious with at least about 80% specificity. In some embodiments, the trained algorithm is configured to classify at least one image, or a derivative thereof, as normal, equivocal, or suspicious with at least about 80% positive predictive value. In some embodiments, the trained algorithm is configured to classify at least one image, or a derivative thereof, as normal, equivocal, or suspicious with at least about 80% negative predictive value.

[0008]

[0008] In some embodiments, the trained machine learning algorithm is configured to identify at least one region of at least one image or a derivative thereof that contains or is suspected of containing abnormal tissue.

[0009] In some embodiments, the trained algorithm classifies at least one image, or a derivative thereof, as normal, equivocal, or suspicious for indicating cancer. In some embodiments, the cancer is breast cancer. In some embodiments, the at least one image, or a derivative thereof, is a three-dimensional image of a subject's breast. In some embodiments, the trained machine learning algorithm is trained using at least about 100 independent training samples that include images that are indicative of or suspected of indicating cancer.

[0010] In some embodiments, the trained algorithm is trained using a first plurality of independent training samples including positive images indicative of or suspected of being indicative of cancer, and a second plurality of independent training samples including negative images not indicative of or suspected of being indicative of cancer. In some embodiments, the trained algorithm comprises a supervised machine learning algorithm. In some embodiments, the supervised machine learning algorithm comprises a deep learning algorithm, a support vector machine (SVM), a neural network, or a random forest.

[0011] In some embodiments, the method further includes monitoring the subject, wherein the monitoring includes evaluating images of a region of the subject's body at a plurality of time points, the evaluating being based at least in part on classifying at least one image, or a derivative thereof, at each of the plurality of time points as normal, equivocal, or suspicious. In some embodiments, differences in the evaluation of the images of the subject's body at the plurality of time points are indicative of one or more clinical indicators selected from the group consisting of: (i) a diagnosis of the subject, (ii) a prognosis of the subject, and (iii) the effectiveness or ineffectiveness of a course of treatment for the subject.

[0012] In some embodiments, (c) further comprises: (i) sending at least one image or a derivative thereof to a first radiologist of the first set of radiologists for radiological evaluation and producing a screening result based at least in part on whether the at least one image is classified as suspicious; (ii) sending at least one image or a derivative thereof to a second radiologist of the second set of radiologists for radiological evaluation and producing a screening result based at least in part on whether the at least one image is classified as equivocal; or (iii) sending at least one image or a derivative thereof to a third radiologist of the third set of radiologists for radiological evaluation and producing a screening result based at least in part on whether the at least one image is classified as normal. In some embodiments, (c) further comprises, if the at least one image is classified as suspicious, sending at least one image or a derivative thereof to a first radiologist of the first set of radiologists for radiological evaluation and producing a screening result. In some embodiments, (c) further comprises, if at least one image is classified as equivocal, sending the at least one image or a derivative thereof to a second radiologist of the second set of radiologists for radiological evaluation to produce a screening result. In some embodiments, (c) further comprises, if at least one image is classified as normal, sending the at least one image or a derivative thereof to a third radiologist of the third set of radiologists for radiological evaluation to produce a screening result. In some embodiments, the screening result for the subject is produced in the same clinic visit as the step of acquiring the at least one image or a derivative thereof. In some embodiments, the first set of radiologists is located at an on-site clinic, and the at least one image or a derivative thereof is acquired at the on-site clinic.

[0013] In some embodiments, the second set of radiologists includes radiologists, who are trained to classify at least one image, or a derivative thereof, as normal or suspicious with greater accuracy than the trained algorithm. In some embodiments, a third set of radiologists is located remotely from the on-site clinic, and the at least one image is acquired at the on-site clinic. In some embodiments, a third radiologist in the third set of radiologists performs a radiological evaluation of at least one image, or a derivative thereof, of a batch containing a plurality of images, the batch being selected to improve efficiency of the radiological evaluation.

[0014] In some embodiments, the method further includes performing a diagnostic procedure for the subject based at least in part on the screening results to produce a diagnostic result for the subject. In some embodiments, the diagnostic result for the subject is produced in the same clinic visit as the step of acquiring the at least one image. In some embodiments, the diagnostic result for the subject is produced within about one hour of the step of acquiring the at least one image.

[0015] In some embodiments, at least one image or a derivative thereof is sent to a first radiologist, a second radiologist, or a third radiologist based at least in part on additional characteristics of the part of the subject's body, in some embodiments, the additional characteristics include anatomy, tissue characteristics (e.g., tissue density or physical properties), the presence of foreign bodies (e.g., implants), a type of finding, a medical condition (e.g., predicted by an algorithm, such as a machine learning algorithm), or a combination thereof.

[0016]

[0016] In some embodiments, the at least one image or a derivative thereof is sent to the first radiologist, the second radiologist, or the third radiologist based at least in part on additional characteristics of the first radiologist, the second radiologist, or the third radiologist (e.g., the personal ability of the first radiologist, the second radiologist, or the third radiologist to perform a radiological assessment of the at least one image or a derivative thereof).

[0017] In some embodiments, (c) further includes generating an alert based at least in part on sending the at least one image or a derivative thereof to a first radiologist or sending the at least one image or a derivative thereof to a second radiologist. In some embodiments, the method further includes sending the alert to the subject or the subject's clinical caregiver. In some embodiments, the method further includes sending the alert to the subject through a patient mobile application. In some embodiments, the alert is generated in real time with (b) or near real time with (b).

[0018] In some embodiments, applying the image processing algorithm includes identifying a region of interest in at least one image or a derivative thereof and labeling the region of interest to produce at least one labeled image. In some embodiments, the method further includes storing the at least one labeled image in a database. In some embodiments, the method further includes storing the at least one image or one or more of its derivatives and a classification in a database. In some embodiments, the method further includes generating a presentation of the at least one image based at least in part on the at least one image or one or more of its derivatives and the classification. In some embodiments, the method further includes storing the presentation in a database.

[0019] In some embodiments, (c) is performed in real time with (b) or near real time with (b). In some embodiments, the at least one image includes multiple images acquired from the subject, the multiple images being acquired using different modalities or at different points in time. In some embodiments, the classifying step includes processing clinical health data of the subject.

[0020] In another aspect, the present disclosure provides a computer system for processing at least one image of a part of a body of a subject: a database configured to store at least one image of a part of the body of the subject; and one or more computer processors operably coupled to the database, the one or more computer processors performing the following steps: (a) classifying the at least one image, or a derivative thereof, into one of a plurality of categories using a trained algorithm, wherein the classifying step includes applying an image processing algorithm to the at least one image, or a derivative thereof, in (a); and (b) classifying the at least one image, or a derivative thereof, into one of a plurality of categories using a trained algorithm. and one or more computer processors individually or collectively programmed to perform the steps of: (i) sending at least one image or a derivative thereof to a first radiologist for radiological evaluation if the at least one image is classified into a first category of the plurality of categories; or (ii) sending at least one image or a derivative thereof to a second radiologist for radiological evaluation if the at least one image is classified into a second category of the plurality of categories; and (c) receiving a radiological evaluation of the subject from the first radiologist or the second radiologist based at least in part on the radiological analysis of the at least one image or a derivative thereof.

[0021] In some embodiments, (a) includes classifying at least one image or derivative thereof as normal, equivocal, or suspicious. In some embodiments, the one or more computer processors are individually or collectively programmed to further perform the step of sending at least one image or derivative thereof to a classifier based on the classification of the at least one image or derivative thereof in (a). In some embodiments, (b) includes sending at least one image or derivative thereof to a first radiologist of the first plurality of radiologists or a second radiologist of the second plurality of radiologists for radiological evaluation. In some embodiments, the at least one image or derivative thereof is a medical image.

[0022] In some embodiments, the trained algorithm is configured to classify at least one image or a derivative thereof as normal, equivocal, or suspicious with at least about 80% sensitivity. In some embodiments, the trained algorithm is configured to classify at least one image or a derivative thereof as normal, equivocal, or suspicious with at least about 80% specificity. In some embodiments, the trained algorithm is configured to classify at least one image or a derivative thereof as normal, equivocal, or suspicious with at least about 80% positive predictive value. In some embodiments, the trained algorithm is configured to classify at least one image or a derivative thereof as normal, equivocal, or suspicious with at least about 80% negative predictive value. In some embodiments, the trained machine learning algorithm is configured to identify at least one region of at least one image or a derivative thereof that contains abnormal tissue or is suspected of containing abnormal tissue.

[0023] In some embodiments, the trained algorithm classifies at least one image, or a derivative thereof, as normal, equivocal, or suspicious for indicating cancer. In some embodiments, the cancer is breast cancer. In some embodiments, the at least one image, or a derivative thereof, is a three-dimensional image of a subject's breast. In some embodiments, the trained machine learning algorithm is trained using at least about 100 independent training samples that include images that are indicative of or suspected of indicating cancer.

[0024] In some embodiments, the trained algorithm is trained using a first plurality of independent training samples that include positive images indicative of or suspected of being indicative of cancer, and a second plurality of independent training samples that include negative images that do not include or are not suspected of being indicative of cancer. In some embodiments, the trained algorithm comprises a supervised machine learning algorithm. In some embodiments, the supervised machine learning algorithm comprises a deep learning algorithm, a support vector machine (SVM), a neural network, or a random forest.

[0025] In some embodiments, the one or more computer processors are individually or collectively programmed to further perform the step of monitoring the subject, the monitoring step including evaluating images of a part of the subject's body at a plurality of time points, the evaluating step being based at least in part on classifying at least one image, or a derivative thereof, at each of the plurality of time points as normal, equivocal, or suspicious. In some embodiments, differences in the evaluation of the images of the subject's body at the plurality of time points are indicative of one or more clinical indicators selected from the group consisting of: (i) a diagnosis of the subject, (ii) a prognosis of the subject, and (iii) the effectiveness or ineffectiveness of a course of treatment for the subject.

[0026]

[0026] In some embodiments, (b) further includes (i) sending at least one image or derivative thereof to a first radiologist of the first set of radiologists for radiological evaluation and generating a screening result based at least in part on whether the at least one image or derivative thereof is classified as suspicious; (ii) sending at least one image or derivative thereof to a second radiologist of the second set of radiologists for radiological evaluation and generating a screening result based at least in part on whether the at least one image or derivative thereof is classified as equivocal; or (iii) sending at least one image or derivative thereof to a third radiologist of the third set of radiologists for radiological evaluation and generating a screening result based at least in part on whether the at least one image or derivative thereof is classified as normal. In some embodiments, (b) further comprises, if at least one image is classified as suspicious, sending the at least one image or a derivative thereof to a first radiologist of the first set of radiologists for radiological evaluation to produce a screening result. In some embodiments, (b) further comprises, if at least one image is classified as equivocal, sending the at least one image or a derivative thereof to a second radiologist of the second set of radiologists for radiological evaluation to produce a screening result. In some embodiments, (b) further comprises, if at least one image is classified as normal, sending the at least one image or a derivative thereof to a third radiologist of the third set of radiologists for radiological evaluation to produce a screening result. In some embodiments, the screening result for the subject is produced in the same clinic visit as the step of acquiring the at least one image. In some embodiments, the first set of radiologists is located at an on-site clinic, and the at least one image is acquired at the on-site clinic.

[0027] In some embodiments, the second set of radiologists includes radiologists who are trained to classify at least one image, or a derivative thereof, as normal or suspicious with greater accuracy than the trained algorithm. In some embodiments, a third set of radiologists is located remotely from the on-site clinic, and at least one image is acquired at the on-site clinic. In some embodiments, a third radiologist in the third set of radiologists performs a radiological evaluation of at least one image, or a derivative thereof, of a batch containing a plurality of images, the batch being selected to improve efficiency of the radiological evaluation.

[0028] In some embodiments, the one or more computer processors are individually or collectively programmed to further obtain a diagnostic result for the subject from a diagnostic procedure performed on the subject based at least in part on the screening results. In some embodiments, the diagnostic result for the subject is produced in the same clinic visit as the step of acquiring the at least one image. In some embodiments, the diagnostic result for the subject is produced within about one hour of the step of acquiring the at least one image.

[0029] In some embodiments, at least one image or a derivative thereof is sent to a first radiologist, a second radiologist, or a third radiologist based at least in part on additional characteristics of the part of the subject's body, in some embodiments, the additional characteristics include anatomy, tissue characteristics (e.g., tissue density or physical properties), the presence of foreign bodies (e.g., implants), a type of finding, a medical condition (e.g., predicted by an algorithm, such as a machine learning algorithm), or a combination thereof.

[0030]

[0030] In some embodiments, the at least one image or a derivative thereof is sent to the first radiologist, the second radiologist, or the third radiologist based at least in part on additional characteristics of the first radiologist, the second radiologist, or the third radiologist (e.g., the personal ability of the first radiologist, the second radiologist, or the third radiologist to perform a radiological assessment of the at least one image or a derivative thereof).

[0031] In some embodiments, (b) further includes generating an alert based at least in part on sending the at least one image or a derivative thereof to a first radiologist or sending the at least one image or a derivative thereof to a second radiologist. In some embodiments, the one or more computer processors are individually or collectively programmed to further send the alert to the subject or the subject's clinical caregiver. In some embodiments, the one or more computer processors are individually or collectively programmed to further send the alert to the subject through a patient mobile application. In some embodiments, the alert is generated in real time with (a) or near real time with (a).

[0032] In some embodiments, applying the image processing algorithm includes identifying a region of interest in the at least one image or a derivative thereof and labeling the region of interest to produce at least one labeled image. In some embodiments, the one or more computer processors are individually or collectively programmed to further store the at least one labeled image in a database. In some embodiments, the one or more computer processors are individually or collectively programmed to further store one or more of the at least one image or its derivatives and a classification in a database. In some embodiments, the one or more computer processors are individually or collectively programmed to further generate a presentation of the at least one image or its derivative based at least in part on the one or more of the at least one image and the classification. In some embodiments, the one or more computer processors are individually or collectively programmed to further store the presentation in a database.

[0033] In some embodiments, (b) is performed in real time with (a) or near real time with (a). In some embodiments, the at least one image includes multiple images acquired from the subject, the multiple images being acquired using different modalities or at different points in time. In some embodiments, the classifying step includes processing clinical health data of the subject.

[0034]

[0034] Another aspect of the present disclosure provides a non-transitory computer-readable medium containing machine-executable code that, when executed by one or more computer processors, implements any of the methods described above or elsewhere in this specification.

[0035] Another aspect of the present disclosure provides a system comprising one or more computer processors and a computer memory coupled thereto, the computer memory including machine-executable code that, when executed by the one or more computer processors, implements any of the methods described above or elsewhere herein.

[0036]

[0036] Additional aspects and advantages of the present disclosure will become readily apparent to those skilled in the art from the following detailed description. In the following detailed description, merely exemplary embodiments of the present disclosure are shown and described. As will be understood, the present disclosure is capable of other and different embodiments, and its several details are capable of modifications in various obvious respects, all without departing from the present disclosure. Accordingly, the drawings and description should be regarded as illustrative in nature, and not as restrictive.

[0037] Incorporation by Reference

[0037] All publications, patents, and patent applications mentioned in this specification are incorporated herein by reference to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference. To the extent that the publications and patents or patent applications incorporated by reference conflict with the disclosure contained herein, the present specification is intended to supersede and / or take precedence over such conflicting material.

[0038] The novel features of the invention are set forth with particularity in the appended claims. The features and advantages of the present invention will be better understood by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the invention are utilized, and the accompanying drawings (also referred to herein as "drawings" and "figures"), in which: [Brief explanation of the drawings]

[0039] [Figure 1]

[0039] FIG. 1 is an exemplary workflow diagram of a method for sending a case for radiological review (eg, by a radiologist or radiographer) according to the disclosed embodiments. [Figure 2]

[0040] FIG. 1 illustrates an example method of using a triage engine configured to stratify subjects undergoing mammography screening by classifying the subject's mammography data into one of three different workflows: normal, equivocal, and suspicious, according to disclosed embodiments. [Figure 3A]

[0041] 1 is a diagram of an example user interface for a real-time radiology system including a view from the perspective of a mammography technician or technician assistant, according to a disclosed embodiment; [Figure 3B] FIG. 1 is an illustration of an example user interface for a real-time radiology system including a view from a radiologist's perspective, in accordance with a disclosed embodiment; [Figure 3C] FIG. 1 is an illustration of an example user interface for a real-time radiology system including a view from a billing officer's perspective, in accordance with a disclosed embodiment; [Figure 3D] 1 is a diagram of an example user interface for a real-time radiology system including a view from the perspective of an ultrasound technician or technician assistant, in accordance with a disclosed embodiment; [Figure 4]

[0042] FIG. 1 is a diagram of a computer system programmed or configured to implement the methods provided herein. [Figure 5]

[0043] 1 is an exemplary plot of the detection frequency of breast cancer tumors of various sizes (ranging from 2 mm to 29 mm) detected using a real-time radiology system, according to disclosed embodiments. [Figure 6]

[0044] 1 is an exemplary plot of positive predictive value (PPV1) versus callback rate from screening mammography, according to disclosed embodiments. [Figure 7]

[0045] 10 is an exemplary plot comparing batch (including control, BI-RADs, and density) interpretation time (left) and percentage improvement in interpretation time over the control (right) for a first set of radiologists, a second set of radiologists, and the total set of radiologists overall, in accordance with disclosed embodiments. [Figure 8]

[0046] 1 is a receiver operating characteristic (ROC) curve illustrating the performance of a DNN on a binary classification task evaluated on a test dataset, according to disclosed embodiments. [Figure 9]

[0047] FIG. 1 is a diagram of an example schematic of patient flow through a clinic using an AI-enabled real-time radiology system and a patient mobile application (app), according to disclosed embodiments. [Figure 10]

[0048] FIG. 1 is a diagram of a schematic example of an AI-assisted radiological evaluation workflow, according to the disclosed embodiments. [Figure 11]

[0049] FIG. 1 is a diagram of an example triage software system developed using machine learning for screening mammography to enable more timely report delivery and follow-up for suspicious cases (e.g., as occurs in a batch reading setting), according to disclosed embodiments. [Figure 12A]

[0050] FIG. 1 is an example of a composite 2D mammography (SM) image derived from a digital breast tomosynthesis (DBT) examination for Breast Imaging Reporting and Data System (BI-RADS) breast density category (A) almost entirely fatty, according to a disclosed embodiment. [Figure 12B]FIG. 1 is an example of a synthetic 2D mammography (SM) image derived from a DBT examination for BI-RADS breast density category (B) scattered areas of fibroglandular density, in accordance with disclosed embodiments. [Figure 12C] FIG. 1 is an example of a synthetic 2D mammography (SM) image derived from a DBT examination for BI-RADS breast density category (C) heterogeneously dense, in accordance with the disclosed embodiments. [Figure 12D] FIG. 1 is an example of a synthetic 2D mammography (SM) image derived from a DBT examination for a BI-RADS breast density category (D) extremely dense, in accordance with the disclosed embodiments. [Figure 13A]

[0051] 13A-13D are comparative images of the same breast under the same pressure, according to disclosed embodiments: Figure 13A is a full-field digital mammography (FFDM) image. [Figure 13B] FIG. 13B is a composite 2D mammography (SM) image. [Figure 13C] FIG. 13C is a zoomed-in view of the FFDM image with the original site indicated by the white square. [Figure 13D] Figure 13D is a zoomed-in area of the SM image with the original area indicated by a white square. Figures 13C and 13D are intended to highlight the differences in texture and contrast that can occur between the two image types. [Figure 14A]

[0052] 1 is a confusion matrix for the Breast Imaging Reporting and Data System (BI-RADS) breast density task evaluated against a full-field digital mammography (FFDM) test set, according to disclosed embodiments. The number of test samples (trials) in each bin is shown in parentheses. [Figure 14B]1 shows a confusion matrix for a binary density task (BI-RADS C+D, which is high density, vs. BI-RADS A+B, which is non-high density) evaluated on a full-field digital mammography (FFDM) test set, according to disclosed embodiments. The number of test samples (tests) in each bin is shown in parentheses. [Figure 15A]

[0053] 1 is a confusion matrix for the Breast Imaging Reporting and Data System (BI-RADS) breast density task without adaptation evaluated against the Site1SM test set, with the number of test samples (trials) in each bin shown in parentheses, in accordance with disclosed embodiments. [Figure 15B] 1 is a confusion matrix for a binary density task (BI-RADS C+D, which is dense, versus BI-RADS A+B, which is non-dense) without adaptation, evaluated against the Site1SM test set, according to a disclosed embodiment. The number of test samples (trials) in each bin is shown in parentheses. [Figure 15C] 1 is a confusion matrix for the BI-RADS breast density task with adaptation by matrix calibration of 500 training samples, evaluated against the Site1SM test set, in accordance with a disclosed embodiment. The number of test samples (tests) in each bin is shown in parentheses. [Figure 15D] 1 is a diagram of a confusion matrix for a binary density task (dense vs. non-dense) with adaptation by matrix calibration of 500 training samples, evaluated against the Site1SM test set, in accordance with a disclosed embodiment. The number of test samples (tests) in each bin is shown in parentheses. [Figure 16A]

[0054] FIG. 1 is a confusion matrix for the Breast Imaging Reporting and Data System (BI-RADS) breast density task without adaptation evaluated against the Site2SM test set, in accordance with disclosed embodiments. [Figure 16B]FIG. 10 is a confusion matrix for a binary density task (BI-RADS C+D, which is dense, versus BI-RADS A+B, which is non-dense) without adaptation, evaluated against the Site2SM test set, in accordance with disclosed embodiments. [Figure 16C] FIG. 10 is a confusion matrix for the BI-RADS breast density task with adaptation by matrix calibration of 500 training samples evaluated against the Site2SM test set, in accordance with disclosed embodiments. [Figure 16D] 1 is a confusion matrix for a binary density task (dense vs. non-dense) with adaptation by matrix calibration of 500 training samples, evaluated against the Site2SM test set, according to a disclosed embodiment. The number of test samples (tests) in each bin is shown in parentheses. [Figure 17A]

[0055] FIG. 10 is a diagram of the impact of the amount of training data on the performance of adaptive methods, as measured by macroAUC, for the Site1 dataset, in accordance with disclosed embodiments. [Figure 17B] FIG. 10 is a diagram of the impact of the amount of training data on the performance of adaptive methods, as measured by linearly weighted Cohen's Kappa coefficient, for the Site1 dataset, in accordance with disclosed embodiments. [Figure 17C] FIG. 10 is a diagram of the impact of the amount of training data on the performance of adaptive methods, as measured by macroAUC, for the Site2SM dataset, in accordance with disclosed embodiments. [Figure 17D] FIG. 10 is a diagram of the impact of the amount of training data on the performance of adaptive methods, as measured by linearly weighted Cohen's Kappa coefficient, for the Site2SM dataset, in accordance with disclosed embodiments. [Figure 18]

[0056] FIG. 1 is a diagram of an example of a schematic of a real-time radiological assessment workflow. [Figure 19]

[0057] FIG. 1 is a diagram of an example of a schematic of a real-time radiological assessment workflow. [Figure 20]

[0058] FIG. 1 is a schematic example of an AI-assisted radiological evaluation workflow in a teleimaging setting. DETAILED DESCRIPTION OF THE INVENTION

[0040]

[0059] While various embodiments of the present invention have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Numerous variations, changes, and substitutions may occur to those skilled in the art without departing from the invention. It is understood that various alternatives can be employed to the embodiments of the invention described herein.

[0041]

[0060] As used in this specification and claims, the singular forms "a," "an," and "the" include the plural forms unless expressly stated otherwise. For example, the term "a nucleic acid" includes "nucleic acids," including mixtures thereof.

[0042]

[0061] As used herein, the term "subject" generally refers to an entity or medium having testable or detectable genetic information. A subject may be a person, an individual, or a patient. A subject may be a vertebrate, such as a mammal. Non-limiting examples of mammals include humans, monkeys, livestock, game animals, rodents, and pets. A subject may have cancer or be suspected of having cancer. A subject may exhibit symptoms indicative of the subject's health or physiological state or condition, such as the subject's cancer (e.g., breast cancer). Alternatively, a subject may be asymptomatic with respect to such health or physiological state or condition.

[0043]

[0062] Breast cancer is the most common cancer among women in the United States, with more than 250,000 new diagnoses in 2017 alone. Approximately 1 in 8 women will be diagnosed with breast cancer at some point in their lifetime. Despite improvements in treatment, more than 40,000 women die from breast cancer each year in the United States. Significant progress has been made in reducing breast cancer mortality, due in part to widespread uptake of screening mammography. Breast cancer screening can help identify early-stage cancers, which have a much better prognosis and lower treatment costs compared with late-stage cancers. This difference can be very significant: women with localized breast cancer have a 5-year survival rate of nearly 99%, while women with metastatic breast cancer have a 5-year survival rate of 27%.

[0044]

[0063] Despite these documented benefits, uptake of screening mammography is hindered in part by poor patient experience, including long delays in obtaining an appointment, unclear pricing, long wait times to receive results, and confusing reports. Furthermore, problems arising from a lack of pricing transparency are exacerbated by wide variations in costs across healthcare facilities. Similarly, turnaround times for receiving results are inconsistent across healthcare facilities. Additionally, significant variations in radiologist performance result in patients experiencing widely varying standards of care depending on location and income.

[0045]

[0064] The present disclosure provides methods and systems for performing real-time radiology on subjects by using artificial intelligence to layer medical image data into individualized radiology workflows for further screening and / or diagnostic evaluation. Such subjects may include subjects with and without cancer. Screening may be directed to cancer, such as breast cancer.

[0046]

[0065] FIG. 1 illustrates an exemplary workflow of a method for sending a case for radiological review (e.g., by a radiologist, radiologist, or radiographer) according to disclosed embodiments. In one aspect, the present disclosure provides a method 100 for processing at least one image of a part of a subject's body. Method 100 may include acquiring an image of the part of the subject's body (as per act 102). Then, method 100 may include classifying the image, or a derivative thereof, into one of a plurality of categories using a trained algorithm (as per act 104). For example, classifying may include applying an image processing algorithm to the image, or a derivative thereof. Next, method 100 may include determining whether the image has been classified into a first category or a second category of the plurality of categories (as per act 106). If the image has been classified into the first category, method 100 may include sending the image to a first radiologist for radiological evaluation (as per act 108). If the image is classified in the second category, method 100 may include sending the image to a second radiologist for radiological evaluation (as per operation 110). Method 100 may then include receiving a suggestion to examine the subject (e.g., from the first radiologist or the second radiologist, or from another radiologist or physician) based on the radiological evaluation of the image (as per operation 112).

[0047]

[0066] FIG. 2 illustrates an example method for using a triage engine configured to stratify subjects undergoing mammography screening by classifying their mammography data into one of three different workflows: normal, uncertain, and suspicious, according to disclosed embodiments. First, a dataset including the patient's electronic health record (EHR) and medical images is provided. Next, the AI-based triage engine processes the EHR and medical images to analyze the dataset and classify it as likely normal, possibly suspicious, or likely suspicious. Next, the patient's dataset is processed by one of three workflows: a normal workflow, an uncertain workflow, and a suspicious workflow, respectively, based on the dataset's classification as normal, uncertain, or suspicious. Each of the three workflows may include radiologist review or further AI-based analysis (e.g., by a trained algorithm). The normal workflow may include AI-based (optionally cloud-based) confirmation that the patient's dataset is normal, which completes routine screening. For example, a group of radiologists may review normal workflow cases in a high volume and efficiently. Alternatively, the normal workflow may include an AI-based (optionally cloud-based) determination that a patient's dataset is suspicious, which then orders a rapid radiologist review of the patient's dataset. For example, a second group of radiologists may review fewer, less efficiently, suspicious workflow cases (e.g., a radiologist performs a more detailed radiological evaluation). Similarly, the uncertain and suspicious workflows may also include a rapid radiologist review of the patient's dataset. In some embodiments, different sets of radiologists are used to review different workflows, as described elsewhere herein. In some embodiments, the same set of radiologists is used to review different workflows (e.g., at different points in time depending on the priority of the cases for radiological evaluation).

[0048]

[0067] 3A-3D show diagrams of example user interfaces for a real-time radiology system, including views from the perspectives of a mammography technician or technician assistant (FIG. 3A), a radiologist (FIG. 3B), a billing specialist (FIG. 3C), and an ultrasound technician or technician assistant (FIG. 3D), according to disclosed embodiments. The views may include a heat map showing which areas have been identified as suspicious by the AI algorithm. The mammography technician or technician assistant may ask the patient several questions and evaluate the patient's responses to determine whether the patient is suitable for real-time radiology evaluation. The radiologist may read or interpret the patient's medical images (e.g., mammography images) according to the disclosed real-time radiology methods and systems. The billing specialist may estimate the diagnosis cost based on the patient's suitability for real-time radiology evaluation. The mammography / ultrasound technician or technician assistant may inform the patient to wait for the results of the real-time radiology evaluation. The user interface may provide a notification to the technician or technician assistant that the acquired image is of poor quality (e.g., generated by an AI-based algorithm) so that the technician or technician assistant can make corrections to the acquired image or repeat the image acquisition.

[0049] Medical Image Acquisition

[0068] The medical images can be acquired or derived from a human subject (e.g., a patient). The medical images may be stored in a database, such as a computer server (e.g., a cloud-based server), a local server, a local computer, or a mobile device (such as a smartphone or tablet). The medical images may be acquired from subjects with cancer, from subjects suspected of having cancer, or from subjects without or not suspected of having cancer.

[0050]

[0069] Medical images may be obtained before and / or after treatment of a subject with cancer. Medical images may be obtained from a subject during a treatment or treatment regime. Multiple sets of medical images may be obtained from a subject to monitor the effectiveness of treatment over time. Medical images may be obtained from subjects with known or suspected cancer (e.g., breast cancer) where a definitive positive or negative diagnosis is not available through clinical testing. Medical images may be obtained from subjects suspected of having cancer. Medical images may be obtained from subjects experiencing unexplained symptoms such as fatigue, nausea, weight loss, aches and pains, weakness, or bleeding. Medical images may be obtained from subjects with explained symptoms. Medical images may be obtained from subjects at risk for developing cancer due to factors such as family history, age, hypertension or pre-hypertension, diabetes or pre-diabetes, overweight or obesity, environmental exposures, lifestyle risk factors (e.g., smoking, alcohol consumption, or drug use), or the presence of other risk factors.

[0051]

[0070] Medical images may be obtained using one or more imaging modalities, such as mammography scans, computed tomography (CT) scans, magnetic resonance imaging (MRI) scans, ultrasound scans, digital X-ray scans, positron emission tomography (PET) scans, PET-CT scans, nuclear medicine scans, thermography scans, ophthalmology scans, optical coherence tomography scans, electrocardiogram scans, endoscopic scans, fluoroscopy scans, bone densitometry scans, optical scans, or any combination thereof. Medical images may be pre-processed using image processing techniques or deep learning to enhance image characteristics (e.g., contrast, brightness, sharpness), remove noise or artifacts, filter frequency ranges, compress images to small file sizes, or sample or crop images. Medical images may be raw or reconstructed (e.g., to create a 3D volume from multiple 2D images). Images may be processed to compute maps correlated to tissue properties or functional behavior, such as in functional MRI (fMRI) or resting-state fMRI. The image may be overlaid with additional information showing information such as a heat map or fluids. The image may be created from a composite of images from several scans of the same subject, or from several subjects.

[0052]

[0071] Pre-trained Algorithms

[0072] After obtaining a dataset including multiple medical images of one or more subject body parts, a trained algorithm can be used to process the dataset and classify the images as normal, equivocal, or suspicious. For example, a trained algorithm can be used to determine regions of interest (ROIs) in multiple medical images of a subject and process the ROIs to classify the images as normal, equivocal, or suspicious. The trained algorithm may be configured to classify images as normal, equivocal, or suspicious with at least about 50%, at least about 55%, at least about 60%, at least about 65%, at least about 70%, at least about 75%, at least about 80%, at least about 85%, at least about 90%, at least about 95%, at least about 96%, at least about 97%, at least about 98%, at least about 99%, or greater than 99% accuracy for at least about 25, at least about 50, at least about 100, at least about 150, at least about 200, at least about 250, at least about 300, at least about 350, at least about 400, at least about 450, at least about 500, or more than about 500 independent samples.

[0053]

[0073] The trained algorithm may include a supervised machine learning algorithm. The trained algorithm may include a classification and regression tree (CART) algorithm. The supervised machine learning algorithm may include, for example, a random forest, a support vector machine (SVM), a neural network (e.g., a deep neural network (DNN)), or a deep learning algorithm. The trained algorithm may include an unsupervised machine learning algorithm.

[0054]

[0074] The trained algorithm may be configured to accept multiple input variables and produce one or more output values based on the multiple input variables. The multiple input variables may include features extracted from one or more datasets including medical images of parts of the subject's body. For example, the input variables may include the number of potentially cancerous or suspicious regions of interest (ROIs) in the medical image dataset. The potentially cancerous or suspicious regions of interest (ROIs) may be identified or extracted from the medical image dataset using various image processing techniques, such as image segmentation. The input variables may also include several images from a 3D volume or slices in multiple visits over time. The multiple input variables may also include clinical health data of the subject.

[0055]

[0075] In some embodiments, the clinical health data includes one or more quantitative measurements of a subject, such as age, weight, height, body mass index (BMI), blood pressure, heart rate, glucose level, etc. As another example, the clinical health data can include one or more categorical measurements, such as race, ethnicity, medication or other clinical treatment history, smoking history, alcohol consumption history, daily activity or exercise level, genetic test results, blood test results, imaging results, and screening results.

[0056]

[0076] The trained algorithm may include one or more modules configured to perform image processing on one or more images (e.g., radiological images) to produce detection or segmentation of the one or more images. The trained algorithm may include a classifier (e.g., a linear classifier, a logistic regression classifier, etc.) such that each of one or more output values includes one of a fixed number of possible values to demonstrate classification of a dataset including medical images by the classifier. The trained algorithm may include a binary classifier such that each of one or more output values includes one of two values (e.g., {0, 1}, {positive, negative}, {high risk, low risk}, or {suspicious, normal}) to demonstrate classification of a dataset including medical images by the classifier. The trained algorithm may be another type of classifier, such that one or more output values each include one of three or more values (e.g., {0, 1, 2}; {positive, negative, or neutral}; {high risk, medium risk, or low risk}; or {suspicious, normal, or unclassifiable}), indicating the classification of a dataset including medical images by the classifier. The output values may include a descriptive label, a numerical value, or a combination thereof. Some of the output values may include a descriptive label. Such a descriptive label may provide an identification, indication, likelihood, or risk of a disease or disorder condition for the subject, and may include, for example, positive, negative, high risk, medium risk, low risk, suspicious, normal, or unclassifiable. Such a descriptive label may provide an identification of a follow-up diagnostic procedure or treatment for the subject, and may include, for example, a therapeutic intervention appropriate for treating cancer or other conditions, the duration of the therapeutic intervention, and / or the administration method of the therapeutic intervention. Such an instructional label may provide the identity of secondary clinical tests that may be appropriate to perform on the subject, and may include, for example, imaging tests, blood tests, computed tomography (CT) scans, magnetic resonance imaging (MRI) scans, ultrasound scans, digital x-rays, positron emission tomography (PET) scans, PET-CT scans, or any combination thereof. As another example, such an instructional label may provide a prognosis for the subject's cancer.As another example, such descriptive labels may provide a relative assessment of a subject's cancer (e.g., estimated stage or tumor burden). Some descriptive labels may be mapped to numerical values, for example, by mapping "positive" to 1 and "negative" to 0.

[0057]

[0077] Some of the output values may include numeric values, such as binary values, integers, or continuous values. Such binary output values may include, for example, {0, 1}, {positive, negative}, or {high risk, low risk}. Such integer output values may include, for example, {0, 1, 2}. Such continuous output values may include, for example, probability values at least 0 and less than or equal to 1. Such continuous output values may include, for example, the center coordinates of an ROI. Such continuous output values may indicate the subject's cancer prognosis. Some numeric values may be mapped to descriptive labels, for example, by mapping 1 to "positive" and 0 to "negative." Arrays or maps of numeric values, such as cancer probability maps, may be created.

[0058]

[0078] A portion of the output values may be assigned based on one or more cutoff values. For example, if a dataset including medical images indicates that a subject has cancer (e.g., breast cancer) at least 50% of the time, a binary classification of the dataset including medical images may assign an output value of "positive" or 1. For example, if a dataset including medical images indicates that a subject has cancer less than 50% of the time, a binary classification of the dataset including medical images may assign an output value of "negative" or 0. In this case, a single cutoff value of 50% is used to classify the dataset including medical images into one of two possible binary output values. Examples of single cutoff values include about 1%, about 2%, about 5%, about 10%, about 15%, about 20%, about 25%, about 30%, about 35%, about 40%, about 45%, about 50%, about 55%, about 60%, about 65%, about 70%, about 75%, about 80%, about 85%, about 90%, about 91%, about 92%, about 93%, about 94%, about 95%, about 96%, about 97%, about 98%, and about 99%.

[0059]

[0079] As another example, a classification of a dataset including medical images may assign an output value of "positive" or 1 if the dataset including medical images indicates that the subject has at least about a 50%, at least about 55%, at least about 60%, at least about 65%, at least about 70%, at least about 75%, at least about 80%, at least about 85%, at least about 90%, at least about 91%, at least about 92%, at least about 93%, at least about 94%, at least about 95%, at least about 96%, at least about 97%, at least about 98%, at least about 99% or more chance of having cancer. If the dataset including the medical images indicates that the subject has more than about 50%, more than about 55%, more than about 60%, more than about 65%, more than about 70%, more than about 75%, more than about 80%, more than about 85%, more than about 90%, more than about 91%, more than about 92%, more than about 93%, more than about 94%, more than about 95%, more than about 96%, more than about 97%, more than about 98%, or more than about 99% chance of having cancer, the classification of the sample may be assigned an output value of "positive" or 1.

[0060]

[0080] A classification of a dataset including medical images may assign an output value of "negative" or 0 if the dataset including medical images indicates that the subject has less than about 50%, less than about 45%, less than about 40%, less than about 35%, less than about 30%, less than about 25%, less than about 20%, less than about 15%, less than about 10%, less than about 9%, less than about 8%, less than about 7%, less than about 6%, less than about 5%, less than about 4%, less than about 3%, less than about 2%, or less than about 1% chance of having cancer. A classification of a dataset including medical images may assign an output value of "negative" or 0 if the dataset including medical images indicates that the subject has about a 50% or less, about a 45% or less, about a 40% or less, about a 35% or less, about a 30% or less, about a 25% or less, about a 20% or less, about a 15% or less, about a 10% or less, about a 9% or less, about a 8% or less, about a 7% or less, about a 6% or less, about a 5% or less, about a 4% or less, about a 3% or less, about a 2% or less, or about a 1% or less chance of having cancer.

[0061]

[0081] If a dataset including medical images cannot be classified as "positive," "negative," 1, or 0, the classification of the dataset including medical images may assign an output value of "neutral" or 2. In this case, a set of two cutoff values is used to classify the dataset including medical images into one of three possible output values. Example sets of cutoff values include {1%, 99%}, {2%, 98%}, {5%, 95%}, {10%, 90%}, {15%, 85%}, {20%, 80%}, {25%, 75%}, {30%, 70%}, {35%, 65%}, {40%, 60%}, and {45%, 55%}. Similarly, a set of n cutoff values may be used to classify a dataset including medical images into one of n+1 possible output values, where n is any positive integer.

[0062]

[0082] The trained algorithm may be trained using multiple independent training samples. Each of the independent training samples may include a dataset including medical images from a subject, an associated dataset (e.g., labels or annotations) obtained by analyzing the medical images, and one or more known output values corresponding to the dataset including medical images (e.g., difficulty of interpreting the images, time taken to interpret the images, clinical diagnosis, prognosis, defect, efficacy of treatment for the subject's cancer). The independent training sample may include a dataset including medical images and associated datasets and outputs obtained or derived from multiple different subjects. The independent training sample may include a dataset including medical images and associated datasets and outputs obtained from the same subject at multiple different points in time (e.g., periodically, such as weekly, monthly, or yearly). The independent training sample may be associated with the presence of cancer or disease (e.g., a training sample including a dataset including medical images and associated datasets and outputs obtained or derived from multiple subjects known to have cancer or disease). The independent training sample may be associated with the absence of cancer or disease (e.g., a training sample including a dataset containing medical images and associated datasets and output obtained or derived from multiple subjects who are known not to have been previously diagnosed with cancer or who have received negative test results for cancer or disease).

[0063]

[0083] The trained algorithm may be trained using at least about 50, at least about 100, at least about 250, at least about 500, at least about 1,000, at least about 5,000, at least about 10,000, at least about 15,000, at least about 20,000, at least about 25,000, at least about 30,000, at least about 35,000, at least about 40,000, at least about 45,000, at least about 50,000, at least about 100,000, at least about 150,000, at least about 200,000, at least about 250,000, at least about 300,000, at least about 350,000, at least about 400,000, at least about 450,000, or at least about 500,000 independent training samples. The independent training samples may include datasets including medical images associated with the presence of a disease (e.g., cancer) and / or datasets including medical images associated with the absence of a disease (e.g., cancer). The trained algorithm may be trained using independent training samples associated with the presence of about 500,000 or less, about 450,000 or less, about 400,000 or less, about 350,000 or less, about 300,000 or less, about 250,000 or less, about 200,000 or less, about 150,000 or less, about 100,000 or less, about 50,000 or less, about 25,000 or less, about 10,000 or less, about 500 or less, about 250 or less, about 100 or less, or about 50 or less diseases (e.g., cancer). In some embodiments, the dataset including medical images is unrelated to the samples used to train the trained algorithm.

[0064]

[0084] The trained algorithm may be trained using a first number of independent training samples associated with the presence of a disease (e.g., cancer) and a second number of independent training samples associated with the absence of the disease (e.g., cancer). The first number of independent training samples associated with the presence of a disease (e.g., cancer) may be less than or equal to the second number of independent training samples associated with the absence of the disease (e.g., cancer). The first number of independent training samples associated with the presence of a disease (e.g., cancer) may be equal to the second number of independent training samples associated with the absence of the disease (e.g., cancer). The first number of independent training samples associated with the presence of a disease (e.g., cancer) may be greater than the second number of independent training samples associated with the absence of the disease (e.g., cancer).

[0065]

[0085] The trained algorithm may perform a quantification test with an accuracy of at least about 50%, at least about 55%, at least about 60%, at least about 65%, at least about 70%, at least about 75%, at least about 80%, at least about 81%, at least about 82%, at least about 83%, at least about 84%, at least about 85%, at least about 86%, at least about 87%, at least about 88%, at least about 89%, at least about 90%, at least about 91%, at least about 92%, at least about 93%, at least about 94%, at least about 95%, at least about 96%, at least about 97%, at least about 98%, at least about 99% or higher. The medical image classification algorithm may be configured to classify medical images for at least about 50, at least about 100, at least about 250, at least about 500, at least about 1,000, at least about 5,000, at least about 10,000, at least about 15,000, at least about 20,000, at least about 25,000, at least about 30,000, at least about 35,000, at least about 40,000, at least about 45,000, at least about 50,000, at least about 100,000, at least about 150,000, at least about 200,000, at least about 250,000, at least about 300,000, at least about 350,000, at least about 400,000, at least about 450,000, or at least about 500,000 independent test samples. The accuracy of classifying medical images by the trained algorithm may be calculated as the percentage of independent test samples that are correctly identified or classified as normal or suspicious (e.g., images from subjects known to have cancer or subjects with negative clinical test results for cancer).

[0066]

[0086] The trained algorithm may be configured to classify medical images with a positive predictive value (PPV) of at least about 5%, at least about 10%, at least about 15%, at least about 20%, at least about 25%, at least about 30%, at least about 35%, at least about 40%, at least about 50%, at least about 55%, at least about 60%, at least about 65%, at least about 70%, at least about 75%, at least about 80%, at least about 81%, at least about 82%, at least about 83%, at least about 84%, at least about 85%, at least about 86%, at least about 87%, at least about 88%, at least about 89%, at least about 90%, at least about 91%, at least about 92%, at least about 93%, at least about 94%, at least about 95%, at least about 96%, at least about 97%, at least about 98%, at least about 99% or more. The PPV of classifying medical images using a trained algorithm may be calculated as the percentage of medical images identified or classified as suspicious that correspond to subjects with a truly abnormal condition (e.g., cancer).

[0067]

[0087] The trained algorithm may be configured to classify medical images with a negative predictive value (NPV) of at least about 5%, at least about 10%, at least about 15%, at least about 20%, at least about 25%, at least about 30%, at least about 35%, at least about 40%, at least about 50%, at least about 55%, at least about 60%, at least about 65%, at least about 70%, at least about 75%, at least about 80%, at least about 81%, at least about 82%, at least about 83%, at least about 84%, at least about 85%, at least about 86%, at least about 87%, at least about 88%, at least about 89%, at least about 90%, at least about 91%, at least about 92%, at least about 93%, at least about 94%, at least about 95%, at least about 96%, at least about 97%, at least about 98%, at least about 99% or more. The NPV of classifying medical images using a trained algorithm may be calculated as the percentage of medical images that are identified or classified as normal, which corresponds to subjects who do not truly have an abnormal condition (e.g., cancer).

[0068]

[0088] The trained algorithm may be configured to interpret the medical images as being at least about 5%, at least about 10%, at least about 15%, at least about 20%, at least about 25%, at least about 30%, at least about 35%, at least about 40%, at least about 50%, at least about 55%, at least about 60%, at least about 65%, at least about 70%, at least about 75%, at least about 80%, at least about 81%, at least about 82%, at least about 83%, at least about 84%, at least about 85%, at least about 86%, at least about 87%, at least about 88%, at least about 89%, at least about 100%, at least about 101%, at least about 102%, at least about 103%, at least about 104%, at least about 105%, at least about 106%, at least about 107%, at least about 108%, at least about 109%, at least about 110%, at least about 111%, at least about 112%, at least about 113%, at least about 114%, at least about 115%, at least about 116%, at least about 117%, at least about 118%, at least about 119%, at least about 120%, at least about 121%, at least about 122%, at least about 123%, at least about 124%, at least about 125%, at least about 126%, at least about 127%, at least about 128%, at least about 129%, at least about 130%, at least about 131%, at least about 132%, at least about 133%, at least about 134%, at least about 135%, at least about 136%, at least about 137%, at least about 138%, at least about 139%, at least about 140%, at least The trained algorithm may be configured to classify medical images with a clinical sensitivity of at least about 90%, at least about 91%, at least about 92%, at least about 93%, at least about 94%, at least about 95%, at least about 96%, at least about 97%, at least about 98%, at least about 99%, at least about 99.1%, at least about 99.2%, at least about 99.3%, at least about 99.4%, at least about 99.5%, at least about 99.6%, at least about 99.7%, at least about 99.8%, at least about 99.9%, at least about 99.99%, at least about 99.999% or more. Clinical sensitivity of classifying medical images using the trained algorithm may be calculated as the percentage of medical images obtained from subjects known to have a condition (e.g., cancer) that are correctly identified or classified as suspicious for that condition.

[0069]

[0089] The trained algorithm may be configured to interpret the medical images as being at least about 5%, at least about 10%, at least about 15%, at least about 20%, at least about 25%, at least about 30%, at least about 35%, at least about 40%, at least about 50%, at least about 55%, at least about 60%, at least about 65%, at least about 70%, at least about 75%, at least about 80%, at least about 81%, at least about 82%, at least about 83%, at least about 84%, at least about 85%, at least about 86%, at least about 87%, at least about 88%, at least about 89%, at least about 100%, at least about 101%, at least about 102%, at least about 103%, at least about 104%, at least about 105%, at least about 106%, at least about 107%, at least about 108%, at least about 109%, at least about 110%, at least about 111%, at least about 112%, at least about 113%, at least about 114%, at least about 115%, at least about 116%, at least about 117%, at least about 118%, at least about 119%, at least about 120%, at least about 121%, at least about 122%, at least about 123%, at least about 124%, at least about 125%, at least about 126%, at least about 127%, at least about 128%, at least about 129%, at least about 130%, at least about 131%, at least about 132%, at least about 133%, at least about 134%, at least about 135%, at least about 136%, at least about 137%, at least about 138%, at least about 139%, at least about 140%, at least The trained algorithm may be configured to classify medical images with a clinical specificity of at least about 90%, at least about 91%, at least about 92%, at least about 93%, at least about 94%, at least about 95%, at least about 96%, at least about 97%, at least about 98%, at least about 99%, at least about 99.1%, at least about 99.2%, at least about 99.3%, at least about 99.4%, at least about 99.5%, at least about 99.6%, at least about 99.7%, at least about 99.8%, at least about 99.9%, at least about 99.99%, at least about 99.999%, or greater. Clinical specificity for classifying medical images using the trained algorithm may be calculated as the percentage of medical images obtained from subjects without a condition (e.g., subjects with negative laboratory tests for cancer) that are correctly identified or classified as normal for that condition.

[0070]

[0090] The trained algorithm may be configured to classify medical images with an Area-Under-Curve (AUC) of at least about 0.50, at least about 0.55, at least about 0.60, at least about 0.65, at least about 0.70, at least about 0.75, at least about 0.80, at least about 0.81, at least about 0.82, at least about 0.83, at least about 0.84, at least about 0.85, at least about 0.86, at least about 0.87, at least about 0.88, at least about 0.89, at least about 0.90, at least about 0.91, at least about 0.92, at least about 0.93, at least about 0.94, at least about 0.95, at least about 0.96, at least about 0.97, at least about 0.98, at least about 0.99, or more. The AUC may be calculated as the integral of a receiver operating characteristic (ROC) curve (e.g., the area under the ROC curve) associated with a trained algorithm in classifying a dataset containing medical images as normal or suspicious.

[0071]

[0091] The trained algorithm may be adjusted or fine-tuned to improve one or more of its cancer-identifying performance, accuracy, PPV, NPV, clinical sensitivity, clinical specificity, or AUC. The trained algorithm may be adjusted or fine-tuned by adjusting the parameters of the trained algorithm (e.g., a set of cutoff values used to classify a dataset containing medical images as described elsewhere herein, or parameters or weights of a neural network). The trained algorithm may be continuously adjusted or fine-tuned during the training process or after the training process is completed.

[0072]

[0092] After the trained algorithm is initially trained, a subset of inputs may be identified as most influential or most important to include for achieving high-quality classification. For example, a subset of features of a dataset including medical images may be identified as most influential or most important to include for achieving high-quality classification or cancer identification. The features of a dataset including medical images, or a subset thereof, may be ranked based on a classification metric that indicates the influence or importance of each individual feature toward achieving high-quality classification or cancer identification. Such metrics can be used to reduce, possibly significantly, the number of input variables (e.g., predictor variables) that can be used to train the trained algorithm to a desired performance level (e.g., based on a desired minimum accuracy, PPV, NPV, clinical sensitivity, clinical specificity, AUC, or a combination thereof). For example, if training a trained algorithm with a plurality of variables that includes tens to hundreds of input variables to the trained algorithm results in greater than 99% classification accuracy, instead training the trained algorithm with only a selected subset of such most influential or most significant input variables from the plurality of variables, such as about 5 or less, about 10 or less, about 15 or less, about 20 or less, about 25 or less, about 30 or less, about 35 or less, about 40 or less, about 45 or less, about 50 or less, or about 100 or less, may result in a reduced, but still acceptable, classification accuracy (e.g., at least about 50%, at least about 55%, At least about 60%, at least about 65%, at least about 70%, at least about 75%, at least about 80%, at least about 81%, at least about 82%, at least about 83%, at least about 84%, at least about 85%, at least about 86%, at least about 87%, at least about 88%, at least about 89%, at least about 90%, at least about 91%, at least about 92%, at least about 93%, at least about 94%, at least about 95%, at least about 96%, at least about 97%, at least about 98%, or at least about 99%.The subset may be selected by rank ordering the input variables across the plurality of input variables and selecting a predetermined number (e.g., about 5 or less, about 10 or less, about 15 or less, about 20 or less, about 25 or less, about 30 or less, about 35 or less, about 40 or less, about 45 or less, about 50 or less, or about 100 or less) of input variables having the best classification metrics.

[0073] Identifying or monitoring cancer

[0093] After processing a dataset including multiple medical images of a part of the subject's body using a trained algorithm to classify the images as normal, equivocal, or suspicious, cancer may be identified or monitored in the subject. The identification may be based, at least in part, on the classification of the images as normal, equivocal, or suspicious, multiple features extracted from the dataset including the medical images, and / or clinical health data of the subject. The identification may be performed by a radiologist, multiple radiologists, or a trained algorithm.

[0074]

[0094] Cancer may be identified within a subject with an accuracy of at least about 50%, at least about 55%, at least about 60%, at least about 65%, at least about 70%, at least about 75%, at least about 80%, at least about 81%, at least about 82%, at least about 83%, at least about 84%, at least about 85%, at least about 86%, at least about 87%, at least about 88%, at least about 89%, at least about 90%, at least about 91%, at least about 92%, at least about 93%, at least about 94%, at least about 95%, at least about 96%, at least about 97%, at least about 98%, at least about 99%, or more. Accuracy for identifying cancer may be calculated as the percentage of subjects of an independent test (e.g., subjects known to have cancer or subjects with a negative clinical test result for cancer) who are correctly identified or classified as having or not having cancer.

[0075]

[0095] Cancer may be identified in a subject with a positive predictive value (PPV) of at least about 5%, at least about 10%, at least about 15%, at least about 20%, at least about 25%, at least about 30%, at least about 35%, at least about 40%, at least about 50%, at least about 55%, at least about 60%, at least about 65%, at least about 70%, at least about 75%, at least about 80%, at least about 81%, at least about 82%, at least about 83%, at least about 84%, at least about 85%, at least about 86%, at least about 87%, at least about 88%, at least about 89%, at least about 90%, at least about 91%, at least about 92%, at least about 93%, at least about 94%, at least about 95%, at least about 96%, at least about 97%, at least about 98%, at least about 99% or more. The PPV for identifying cancer may be calculated as the percentage of subjects from an independent test who are identified or classified as having cancer that represent subjects who truly have cancer.

[0076]

[0096] Cancer may be identified in a subject with a negative predictive value (NPV) of at least about 5%, at least about 10%, at least about 15%, at least about 20%, at least about 25%, at least about 30%, at least about 35%, at least about 40%, at least about 50%, at least about 55%, at least about 60%, at least about 65%, at least about 70%, at least about 75%, at least about 80%, at least about 81%, at least about 82%, at least about 83%, at least about 84%, at least about 85%, at least about 86%, at least about 87%, at least about 88%, at least about 89%, at least about 90%, at least about 91%, at least about 92%, at least about 93%, at least about 94%, at least about 95%, at least about 96%, at least about 97%, at least about 98%, at least about 99% or more. The NPV of identifying cancer using a trained algorithm may be calculated as the percentage of subjects from an independent test who are identified or classified as not having cancer, which corresponds to subjects who truly do not have cancer.

[0077]

[0097] The cancer is at least about 5%, at least about 10%, at least about 15%, at least about 20%, at least about 25%, at least about 30%, at least about 35%, at least about 40%, at least about 50%, at least about 55%, at least about 60%, at least about 65%, at least about 70%, at least about 75%, at least about 80%, at least about 81%, at least about 82%, at least about 83%, at least about 84%, at least about 85%, at least about 86%, at least about 87%, at least about 88%, at least about 89%, at least about 90%, The cancer may be identified within a subject with a clinical sensitivity of at least about 91%, at least about 92%, at least about 93%, at least about 94%, at least about 95%, at least about 96%, at least about 97%, at least about 98%, at least about 99%, at least about 99.1%, at least about 99.2%, at least about 99.3%, at least about 99.4%, at least about 99.5%, at least about 99.6%, at least about 99.7%, at least about 99.8%, at least about 99.9%, at least about 99.99%, at least about 99.999%, or more. Clinical sensitivity for identifying cancer may be calculated as the percentage of subjects for which an independent test associated with the presence of cancer (e.g., subjects known to have cancer) are correctly identified or classified as having cancer.

[0078]

[0098] Cancer is at least about 5%, at least about 10%, at least about 15%, at least about 20%, at least about 25%, at least about 30%, at least about 35%, at least about 40%, at least about 50%, at least about 55%, at least about 60%, at least about 65%, at least about 70%, at least about 75%, at least about 80%, at least about 81%, at least about 82%, at least about 83%, at least about 84%, at least about 85%, at least about 86%, at least about 87%, at least about 88%, at least about 89%, at least about 90%, at least about 91%, at least about 92%, at least about 93%, at least about 94%, at least about 95%, at least about 96%, at least about 97%, at least about 98%, at least about 99%, at least about 100%, at least about 101%, at least about 102%, at least about 103%, at least about 104%, at least about 105%, at least about 106%, at least about 107%, at least about 108%, at least about 109%, at least about 110%, at least about 111%, at least about 112%, at least about 113%, at least about 114%, at least about 115%, at least about 116%, at least about 117%, at least about 118%, at least about 119%, at least about 120%, at least about 121%, at least about 122%, at least about 123%, at least about 124%, at least about 125%, at least about 126%, at least about 127%, at least about 128%, at least about 129%, at least about 130%, at least about 131%, at least about 132%, at least about 133%, The clinical specificity for identifying cancer may be calculated as the percentage of subjects with an independent test associated with the absence of cancer (e.g., subjects with a negative clinical test result for cancer) that are correctly identified or classified as not having cancer.

[0079]

[0099] In some embodiments, a subject may be identified as being at risk for cancer. After identifying the subject as being at risk for cancer, a clinical intervention may be selected for the subject based at least in part on the cancer for which the subject was identified as being at risk. In some embodiments, the clinical intervention is selected from a plurality of clinical interventions (e.g., clinically indicated for various types of cancer).

[0080]

[0100] In some embodiments, the trained algorithm may determine that the subject is at least about 5%, at least about 10%, at least about 15%, at least about 20%, at least about 25%, at least about 30%, at least about 35%, at least about 40%, at least about 50%, at least about 55%, at least about 60%, at least about 65%, at least about 70%, at least about 75%, at least about 80%, at least about 81%, at least about 82%, at least about 83%, at least about 84%, at least about 85%, at least about 86%, at least about 87%, at least about 88%, at least about 89%, at least about 90%, at least about 91%, at least about 92%, at least about 93%, at least about 94%, at least about 95%, at least about 96%, at least about 97%, at least about 98%, at least about 99% or more at risk for cancer.

[0081]

[0101] The trained algorithm may be configured to determine whether the subject is at least about 50%, at least about 55%, at least about 60%, at least about 65%, at least about 70%, at least about 75%, at least about 80%, at least about 81%, at least about 82%, at least about 83%, at least about 84%, at least about 85%, at least about 86%, at least about 87%, at least about 88%, at least about 89%, at least about 90%, at least about 91%, at least about 92%, at least about 93%, at least about 94%, at least about 95%, at least about 96%, at least about 97%, at least about 98%, at least about 99%, at least about 100%, at least about 101%, at least about 102%, at least about 103%, at least about 104%, at least about 105%, at least about 106%, at least about 107%, at least about 108%, at least about 109%, at least about 110%, at least about 111%, at least about 112%, at least about 113%, at least about 114%, at least about 115%, at least about 116%, at least about 117%, at least about 118%, at least about 119%, at least about 120%, at least about 121%, at least about 122%, at least about 123%, at least about 124%, at least about 125%, at least about 126%, at least about 127%, at least about 128%, at least about 129%, at least about 130%, at least about 131%, at least about 132%, at least about 133%, at least about 134%, at least about 135%, at least about 136%, at least about 137%, at least about 138%, at least about The risk of cancer may be determined with an accuracy of at least about 94%, at least about 95%, at least about 96%, at least about 97%, at least about 98%, at least about 99%, at least about 99.1%, at least about 99.2%, at least about 99.3%, at least about 99.4%, at least about 99.5%, at least about 99.6%, at least about 99.7%, at least about 99.8%, at least about 99.9%, at least about 99.99%, at least about 99.999% or higher.

[0082]

[0102] Once a subject is identified as having cancer, the subject may optionally be given a therapeutic intervention (e.g., prescribed an appropriate course of treatment to treat the subject's cancer). The therapeutic intervention may include prescribing an effective amount of a drug, further testing or evaluation of the cancer, further monitoring of the cancer, or a combination thereof. If the subject is currently being treated for cancer with a course of treatment, the therapeutic intervention may include a subsequent, different course of treatment (e.g., increasing the effectiveness of the treatment because the current course of treatment is ineffective).

[0083]

[0103] Therapeutic intervention may include recommending a secondary laboratory test for the subject to confirm the cancer diagnosis, which may include an imaging test, a blood test, a computed tomography (CT) scan, a magnetic resonance imaging (MRI) scan, an ultrasound scan, a chest x-ray, a positron emission tomography (PET) scan, a PET-CT scan, or any combination thereof.

[0084]

[0104] The classification of an image as normal, equivocal, or suspicious, a plurality of features extracted from a dataset including medical images, and / or a subject's clinical health data may be evaluated over a period of time to monitor a patient (e.g., a subject with cancer or a subject being treated for cancer). In some cases, the classification of a patient's medical images may change over the course of treatment. For example, features from a dataset of a patient whose risk of cancer is reduced by an effective treatment may shift to resemble the profile or distribution of a healthy subject (e.g., a subject without cancer). Conversely, features from a dataset of a patient whose risk of cancer is increased by an ineffective treatment may shift to resemble the profile or distribution of a subject at higher risk of cancer or a subject with a more advanced cancer.

[0085]

[0105] A subject's cancer may be monitored by monitoring a course of treatment for treating the subject's cancer. Monitoring may include assessing the subject's cancer at two or more time points. The assessing may be based on at least a classification of an image as normal, equivocal, or suspicious, a plurality of features extracted from a dataset including medical images, and / or clinical health data of the subject determined at each of the two or more time points.

[0086]

[0106] In some embodiments, the differences in classification of images as normal, equivocal, or suspicious, the plurality of features extracted from a dataset including medical images, and / or the subject's clinical health data determined between two or more time points may indicate one or more clinical indicators, such as (i) a diagnosis of cancer in the subject, (ii) a prognosis of cancer in the subject, (iii) an increased risk of cancer in the subject, (iv) a decreased risk of cancer in the subject, (v) the effectiveness of a course of treatment to treat cancer in the subject, and (vi) the ineffectiveness of a course of treatment to treat cancer in the subject.

[0087]

[0107] In some embodiments, differences in classification of images as normal, equivocal, or suspicious, features extracted from a dataset including medical images, and / or clinical health data of a subject determined between two or more time points may indicate a diagnosis of cancer in the subject. For example, if no cancer is detected in the subject at an earlier time point but cancer is detected in the subject at a later time point, the difference indicates a diagnosis that the subject has cancer. Clinical action or decision may be made based on this indication of a diagnosis that the subject has cancer, such as prescribing a new therapeutic intervention for the subject. Clinical action or decision may include recommending a secondary laboratory test for the subject to confirm the cancer diagnosis. The secondary laboratory test may include an imaging test, blood test, computed tomography (CT) scan, magnetic resonance imaging (MRI) scan, ultrasound scan, chest x-ray, positron emission tomography (PET) scan, PET-CT scan, or any combination thereof.

[0088]

[0108] In some embodiments, differences in classification of images as normal, equivocal, or suspicious, multiple features extracted from a dataset including medical images, and / or clinical health data of a subject determined between two or more time points may indicate a prognosis for the subject's cancer.

[0089]

[0109] In some embodiments, differences in classification of images as normal, equivocal, or suspicious, features extracted from a dataset including medical images, and / or clinical health data of a subject determined between two or more time points may indicate an elevated risk of cancer for the subject. For example, if cancer is detected in the subject at both an earlier and a later time point and the difference is positive (e.g., increasing from the earlier time point to the later time point), the difference may indicate an elevated risk of cancer for the subject. Clinical action or decision may be made based on this indication of an elevated risk of cancer, such as prescribing a new therapeutic intervention for the subject or switching therapeutic interventions (e.g., terminating a current treatment and prescribing a new treatment). Clinical action or decision may include recommending a secondary laboratory test for the subject to confirm the elevated risk of cancer. This secondary laboratory testing may include imaging tests, blood tests, computed tomography (CT) scans, magnetic resonance imaging (MRI) scans, ultrasound scans, chest x-rays, positron emission tomography (PET) scans, PET-CT scans, or any combination thereof.

[0090]

[0110] In some embodiments, differences in classification of images as normal, equivocal, or suspicious, features extracted from a dataset including medical images, and / or clinical health data of a subject determined between two or more time points may indicate a reduced risk of cancer for the subject. For example, if cancer is detected in the subject at both an earlier and a later time point and the difference is negative (e.g., decreasing from the earlier time point to the later time point), the difference may indicate a reduced risk of cancer for the subject. Clinical action or decision may be made based on this indication that the subject has a reduced risk of cancer (e.g., continuing or terminating a current therapeutic intervention). Clinical action or decision may include recommending a secondary laboratory test for the subject to confirm the reduced risk of cancer. The secondary laboratory test may include an imaging test, blood test, computed tomography (CT) scan, magnetic resonance imaging (MRI) scan, ultrasound scan, chest x-ray, positron emission tomography (PET) scan, PET-CT scan, or any combination thereof.

[0091]

[0111] In some embodiments, differences in classification of images as normal, equivocal, or suspicious, features extracted from a dataset including medical images, and / or clinical health data of a subject determined between two or more time points may indicate the effectiveness of a course of treatment for treating cancer in the subject. For example, if cancer is detected in the subject at an earlier time point but not at a later time point, the difference indicates the effectiveness of a course of treatment for treating cancer in the subject. Clinical action or decision may be made based on this indication of the effectiveness of a course of treatment for treating cancer in the subject, such as continuing or terminating a current therapeutic intervention for the subject. Clinical action or decision may include recommending a secondary laboratory test for the subject to confirm the effectiveness of the course of treatment for treating cancer. The secondary laboratory test may include an imaging test, a blood test, a computed tomography (CT) scan, a magnetic resonance imaging (MRI) scan, an ultrasound scan, a chest x-ray, a positron emission tomography (PET) scan, a PET-CT scan, or any combination thereof.

[0092]

[0112] In some embodiments, differences in classification of images as normal, equivocal, or suspicious, features extracted from a dataset including medical images, and / or clinical health data of a subject determined between two or more time points may indicate the ineffectiveness of a course of treatment for treating cancer in the subject. For example, if cancer is detected in the subject at both an earlier time point and a later time point, and the difference is positive or zero (e.g., increases or remains constant from the earlier time point to the later time point), and effective treatment was indicated at the earlier time point, the difference may indicate the ineffectiveness of a course of treatment for treating cancer in the subject. Clinical action or decision may be made based on this indication of the ineffectiveness of a course of treatment for treating cancer in the subject, such as terminating a current therapeutic intervention and / or switching (e.g., prescribing) a different new therapeutic intervention for the subject. Clinical action or decision may include recommending a secondary laboratory test for the subject to confirm the ineffectiveness of a course of treatment for treating cancer. This secondary laboratory testing may include imaging tests, blood tests, computed tomography (CT) scans, magnetic resonance imaging (MRI) scans, ultrasound scans, chest x-rays, positron emission tomography (PET) scans, PET-CT scans, or any combination thereof.

[0093]

[0113] Printing disease reports

[0114] After a cancer has been identified in a subject or an elevated risk of the disease or cancer has been monitored, a report may be electronically output that indicates (e.g., identifies or provides an indication of) the disease or cancer in the subject. The subject may not exhibit the disease or cancer (e.g., the disease or cancer is asymptomatic, such as due to complications). The report may be presented on a graphical user interface (GUI) of the user's electronic device. The user may be the subject, a caregiver, a doctor, a nurse, or another medical worker.

[0094]

[0115] The report may include one or more clinical indicators, such as (i) a diagnosis of cancer in the subject, (ii) a prognosis of the subject's disease or cancer, (iii) an elevated risk of the subject's disease or cancer, (iv) a reduced risk of the subject's disease or cancer, (v) the effectiveness of a course of treatment for treating the subject's disease or cancer, (vi) the ineffectiveness of a course of treatment for treating the subject's disease or cancer, (vii) the location and / or level of suspicion of the disease or cancer, and (viii) a measure of the effectiveness of a proposed course of diagnosis of the disease or cancer. The report may also include one or more clinical actions or decisions made based on such one or more clinical indicators. Such clinical actions or decisions may be directed to therapeutic intervention or further clinical evaluation or testing of the subject's disease or cancer.

[0095]

[0116] For example, a clinical indication of a diagnosis of disease or cancer in a subject may involve the clinical action of prescribing a new therapeutic intervention for the subject. As another example, a clinical indication of an increased risk of disease or cancer in a subject may involve the clinical action of prescribing a new therapeutic intervention for the subject or switching therapeutic interventions (e.g., terminating a current treatment and prescribing a new treatment). As another example, a clinical indication of a decreased risk of disease or cancer in a subject may involve the clinical action of continuing or terminating a current therapeutic intervention for the subject. As another example, a clinical indication of the effectiveness of a course of treatment for treating disease or cancer in a subject may involve the clinical action of continuing or terminating a current therapeutic intervention for the subject. As another example, a clinical indication of the ineffectiveness of a course of treatment for treating disease or cancer in a subject may involve the clinical action of terminating a current therapeutic intervention for the subject and / or switching to (e.g., prescribing) a different new therapeutic intervention. As another example, a clinical indication of a disease or cancer site may accompany the clinical action of prescribing a new diagnostic test, particularly any particular parameter of that test that may be the target of the indication.

[0096] Computer Systems

[0117] The present disclosure provides computer systems programmed to implement the methods of the present disclosure. Figure 4 shows a computer system 401 programmed or configured to, for example, train and test a trained algorithm, process medical images using the trained algorithm to classify the images as normal, equivocal, or suspicious, identify or monitor cancer in a subject, and electronically output a report indicating cancer in the subject.

[0097]

[0118] The computer system 401 may coordinate various aspects of the analysis, calculation, and generation of the present disclosure, such as training and testing the trained algorithms, processing medical images using the trained algorithms to classify the images as normal, equivocal, or suspicious, identifying or monitoring cancer in a subject, and electronically outputting a report indicating cancer in a subject. The computer system 401 may be a user's electronic device or a computer system located remotely relative to the electronic device. The electronic device may be a mobile electronic device.

[0098]

[0119] The computer system 401 includes a central processing unit (CPU, also referred to herein as a "processor" and a "computer processor") 405, which may be a single-core or multi-core processor, or multiple processors for parallel processing. The computer system 401 also includes memory or memory locations 410 (e.g., random access memory, read-only memory, flash memory), electronic storage 415 (e.g., a hard disk), a communication interface 420 (e.g., a network adapter) for communicating with one or more other systems, and peripheral devices 425, such as cache, other memory, data storage, and / or electronic display adapters. The memory 410, storage 415, interface 420, and peripheral devices 425 communicate with the CPU 405 through a communication bus (solid lines), such as a motherboard. The storage 415 may be a data storage device (or data repository) for storing data. The computer system 401 can be operatively coupled to a computer network ("network") 430 with the aid of the communication interface 420. Network 430 may be the Internet, an Internet and / or an extranet, or an intranet and / or an extranet in communication with the Internet.

[0099]

[0120] In some cases, network 430 is a telecommunications and / or data network. Network 430 can include one or more computer servers that can enable distributed computing, such as cloud computing. For example, one or more computer servers can enable cloud computing on network 430 (the “cloud”) to perform various aspects of the analysis, calculation, and generation of the present disclosure, such as training and testing trained algorithms, processing medical images using the trained algorithms to classify the images as normal, equivocal, or suspicious, identifying or monitoring cancer in a subject, and electronically outputting a report indicating cancer in the subject. Such cloud computing can be achieved by cloud computing platforms such as Amazon Web Services (AWS), Microsoft Azure, Google Cloud Platform, and IBM cloud. Network 430, in some cases, with the assistance of computer system 401, can implement a peer-to-peer network, which can enable devices coupled to computer system 401 to act as clients or servers.

[0100]

[0121] The CPU 405 may include one or more computer processors and / or one or more graphics processing units (GPUs). The CPU 405 is capable of executing sequences of machine-readable instructions, which may be embodied as a program or software. The instructions may be stored in a memory location, such as the memory 410. The instructions may be sent to the CPU 405, which may then be programmed or configured to implement the methods of the present disclosure. Examples of operations performed by the CPU 405 may include fetch, decoder, execute, and writeback.

[0101]

[0122] The CPU 405 may be part of a circuit, such as an integrated circuit. One or more other components of the system 401 may be included in the circuit. In some cases, the circuit is an application specific integrated circuit (ASIC).

[0102]

[0123] Storage device 415 may store files such as drivers, libraries, and saved programs. Storage device 415 may store user data such as, for example, user settings and user programs. Computer system 401 may include one or more additional data storage devices that are external to computer system 401, such as, in some cases, located on a remote server that communicates with computer system 401 over an intranet or the Internet.

[0103]

[0124] Computer system 401 can communicate with one or more remote computer systems through network 430. For example, computer system 401 can communicate with a user's remote computer system. Examples of remote computer systems include a personal computer (e.g., a portable PC), a slate or tablet PC (e.g., an Apple® iPad®, a Samsung® Galaxy Tab), a telephone, a smartphone (e.g., an Apple® iPhone®, an Android-enabled device, a Blackberry®), or a personal digital assistant. A user can access computer system 401 through network 430.

[0104]

[0125] The methods described herein may be implemented by machine (e.g., computer processor) executable code stored on an electronic storage location of the computer system 401, such as on memory 410 or electronic storage 415. The machine-executable or machine-readable code may be provided in the form of software. During use, the code is executable by the processor 405. In some cases, the code may be retrieved from storage 415 and stored in memory 410 for easy access by the processor 405. In some situations, the electronic storage 415 may be excluded, and the machine-executable instructions are stored in memory 410.

[0105]

[0126] The code may be pre-compiled and configured for use on a machine having a processor configured to execute the code, or may be compiled during run-time. The code may be supplied in a programming language that may be selected so that the code can be executed in a pre-compiled or as-compiled manner.

[0106]

[0127] Aspects of the systems and methods provided herein, such as computer system 401, can be embodied as programming. Various aspects of the technology may be thought of as a “product” or “article of manufacture,” typically in the form of machine (or processor) executable code and / or associated data executed on or embodied in some type of machine-readable medium. The machine-executable code can be stored in electronic storage, such as memory (e.g., read-only memory, random-access memory, flash memory) or a hard disk. “Storage”-type media can include any or all of the tangible memory of a computer, processor, etc., or its associated modules, such as various semiconductor memories, tape drives, disk drives, etc., which can provide non-transitory storage for software programming at any time. All or portions of the software may be communicated from time to time over the Internet or various other telecommunications networks. Such communication can, for example, enable the software to be loaded from one computer or processor to another, e.g., from a management server or host computer to an application server computer platform. Thus, other types of media that may carry software elements include optical, electrical, and electromagnetic waves, such as those used between physical interfaces between local devices, through wired and optical landline networks, and on various air-links. Physical elements, such as wired or wireless links, optical links, that carry such waves may also be considered media that carry software. As used herein, unless qualified as a non-transitory, tangible "storage" medium, the term computer or machine "readable medium" refers to any medium that participates in providing instructions to a processor for execution.

[0107]

[0128] Thus, machine-readable media such as computer-executable code may take forms including, but not limited to, tangible storage media, carrier wave media, or physical transmission media. Non-volatile storage media include, for example, optical or magnetic disks shown in the drawings, any of the storage devices in any computer, such as those that may be used to implement a database, etc. Volatile storage media include dynamic memory such as the main memory of such a computer platform. Tangible transmission media include coaxial cables, copper wire, and optical fiber, including wiring that comprises a bus within a computer system. Carrier wave transmission media may take the form of electric or electromagnetic signals, or acoustic or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communications. Thus, common forms of computer readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tape, any other magnetic medium, CD-ROMs, DVDs or DVD-ROMs, any other optical medium, punched card paper tape, any other physical storage medium with a pattern of holes, RAM, ROM, PROMs and EPROMs, FLASH-EPROMs, any other memory chips or cartridges, carrier waves that transport data or instructions, cables or links that transport such carrier waves, or any other medium from which a computer can read programming code and / or data. Many such forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution.

[0108]

[0129] The computer system 401 includes or is capable of communicating with an electronic display 435 that includes a user interface (UI) 440 for providing, for example, visual displays indicating the training and testing of a trained algorithm, visual displays of image data indicating classification as normal, equivocal, or suspicious, identification of a subject as having cancer, or an electronic report (e.g., a diagnostic or radiology report) indicating cancer in a subject. Examples of UIs include, but are not limited to, graphical user interfaces (GUIs) and web-based user interfaces.

[0109]

[0130] The methods and systems of the present disclosure may be implemented by one or more algorithms. The algorithms are implemented in software when executed by the central processing unit 405. The algorithms may, for example, train and test trained algorithms, process medical images using the trained algorithms to classify images as normal, equivocal, or suspicious, identify or monitor cancer in a subject, and electronically output a report indicating cancer in a subject.

[0110] Example

[0131] [Example]

[0111] Improving patient care with real-time radiology

[0132] Using the systems and methods of the present disclosure, real-time radiology screening and diagnostic workflow was performed on multiple patients. For example, on the first day of the real-time radiology clinic, the patients received immediate results for normal cases, leaving them feeling relieved and calm.

[0112]

[0133] In another example, the day after a real-time radiology clinic visit, another patient received a suspicious finding during screening and had a diagnostic follow-up within three hours of the suspicious finding. The patient was told by the radiologist that the finding was benign and not suspected to be cancerous. The patient was very relieved and happy to avoid the anxiety of waiting for a final diagnostic result. On average anywhere in the United States, such a process can take two to eight weeks. Even in certain clinics with rapid workflows, the process can take one to two weeks without the assistance of real-time radiology.

[0113]

[0134] As another example, on another day at a real-time radiology clinic, an AI-based real-time radiology system detected a 3 mm breast cancer tumor, which was confirmed as cancerous by biopsy five days later. FIG. 5 shows an exemplary plot of the detection frequency of breast cancer tumors of various sizes (ranging from 2 mm to 29 mm) detected by radiologists, according to disclosed embodiments. A real-time radiology system can provide life-saving clinical impact by reducing time to treatment. The cancer may continue to grow until the patient undergoes their next screening or diagnostic procedure, at which time removal or treatment may become more life-threatening, painful, expensive, and less successful.

[0114]

[0135] In another example of a real-time radiology clinic, a patient received a diagnostic follow-up procedure for a suspicious finding within an hour. A biopsy was required, but because the patient was taking aspirin, the biopsy was completed the next clinic day. The biopsy confirmed the cancer detected by real-time radiology. The radiology workup period was reduced from eight clinic days to one day, and the time to diagnosis was reduced from one month to one week.

[0115]

[0136] The clinical impact of a real-time radiology system may be measured by screening mammography metrics such as PPV1 and callback rate. PPV1 generally refers to the percentage of consultations with abnormal initial interpretations by radiologists that result in a tissue diagnosis of cancer within one year. Callback rate generally refers to the percentage of consultations with abnormal initial interpretations (e.g., "recall rate"). Over a six-week span, the real-time radiology clinic processed 796 patient cases using AI-based analytics, of which 94 were flagged to be reviewed by a radiologist in real time. A total of four cases were diagnosed with cancer, of which three were confirmed as cancer (e.g., by biopsy).

[0116]

[0137] 6 shows an exemplary plot of positive predictive value (PPV1) from screening mammography versus callback rate, according to disclosed embodiments. A prospective study had a callback rate of 11.8% with a PPV1 of 3.2%. In contrast, an intermediate radiologist had a callback rate of 11.6% with a PPV1 of 4.4%.

[0117]

[0138] 7 shows an exemplary plot comparing the interpretation time (including Bi-RADS Assessment and density) for reading images in AI-sorted batches (left) for a first set of radiologists, a second set of radiologists, and the total set of radiologists overall (right) and the percentage improvement in interpretation time relative to a control reading randomly shuffled batches, according to disclosed embodiments. This figure shows that an AI-driven workflow can improve radiologist productivity to a statistically significant extent (ranging from approximately 13% to 21%).

[0118]

[0139] [Example]

[0119] Classification of suspicious findings in screening mammography using deep neural networks

[0140] Deep learning can be applied to a variety of computer vision and image processing applications. For example, deep learning can be used to automatically learn image features relevant to a given task, ranging from classification and detection to segmentation. Computational models based on deep neural networks (DNNs) have been developed for radiology applications, such as screening mammography, and are used to identify suspicious and potentially abnormal or high-risk lesions, improving radiologist productivity. In some cases, deep learning models can rival or even exceed human-level performance. Additionally, deep learning can be used to help improve the performance of general radiologists to approach that of breast imaging experts. For example, general radiologists typically have lower cancer detection rates and much higher recall rates than fellowship-trained breast radiologists.

[0120]

[0141] Deep learning can be used to interpret screening mammograms, including distinguishing between malignant and benign findings. DNN models have been trained for this task to identify missed cancers or reduce false positive callbacks, especially for non-expert readers.

[0121]

[0142] The DNN model was trained using the publicly accessible Digital Database for Screening Mammography (DDSM) dataset (eng.usf.edu / cvprg / Mammography / Database.html). DDSM contains 2,620 cases with over 10,000 digitized scan film mammography images. The images were evenly divided into normal mammograms and mammograms with suspicious findings. Normal mammograms were confirmed over a 4-year follow-up period. Suspicious findings were further divided into biopsy-verified benign findings (51%) and biopsy-verified malignant findings (49%). All cases with apparently benign findings not followed up with biopsy as part of routine clinical care were excluded from the dataset. As a result, distinguishing benign from malignant findings in this dataset can be more challenging than in a typical clinical mammography screening scenario.

[0122]

[0143] The DDSM dataset was split into a subset containing a training dataset, a validation dataset, and a testing dataset. Using the training dataset, the DNN was trained to distinguish between benign or normal breast regions and malignant breast regions. The dataset contained annotations pointing out the location of tumors within the image, which can be crucial in guiding the deep learning process.

[0123]

[0144] The performance of the DNN for this binary classification task was evaluated against the test dataset through the use of receiver operating characteristic (ROC) curves (as shown in Figure 8). The DNN model was used to distinguish between malignant and benign findings with high accuracy, as indicated by the area under the ROC curve (AUC) of 0.89. In contrast, radiologists can typically achieve 84.4% sensitivity and 90.8% specificity for the task of cancer detection in screening mammograms. The DNN model was used to distinguish between malignant and benign findings with 79.2% sensitivity and 80.0% specificity for the more challenging cases found in the DDSM dataset. The performance gap relative to radiologists is due in part to the relatively small dataset size and can be mitigated by incorporating a larger training dataset. Furthermore, the DNN model can be further configured to outperform typical radiologists in terms of accuracy, sensitivity, specificity, AUC, positive predictive value, negative predictive value, or a combination thereof.

[0124]

[0145] A highly accurate DNN model was developed by training on a limited public benchmark dataset. Although the dataset is arguably more challenging than those in clinical settings, the DNN model was able to distinguish between malignant and benign findings with near-human-level performance.

[0125]

[0146] Similar DNN models may be trained using the clinical mammography dataset from the Joanne Knight Breast Health Center in St. Louis, in partnership with Washington University in St. Louis. This dataset contains a large medical record database, including over 100,000 patients, 4,000 biopsy-confirmed cancer cases, and over 400,000 imaging sessions consisting of 1.5 million images. The dataset may be manually or automatically labeled (e.g., by building annotations) to optimize the deep learning process. Because DNN performance improves significantly with the size of the training dataset, this unique, vast, and rich dataset may dramatically improve the sensitivity and specificity of DNN models compared to DNN models trained on DDSM data. Such highly accurate DNN models offer an opportunity for transformative improvements in breast cancer screening, ensuring all women have access to expert-level care.

[0126]

[0147] [Example]

[0127] Artificial Intelligence (AI)-Driven Radiology Clinic for Early Cancer Detection

[0148] Introduction

[0149] Breast cancer is the most prevalent disease among women in the United States, with more than 250,000 new diagnoses in 2017 alone. Approximately 1 in 8 women will be diagnosed with breast cancer at some point in their lifetime. Despite improvements in treatment, more than 40,000 women die from breast cancer each year in the United States. Widespread uptake of screening mammography has made significant progress in reducing some breast cancer mortality rates (a 39% decrease since 1989). Breast cancer screening can help identify early-stage cancers, which have a much better prognosis and lower treatment costs compared to late-stage cancers. This difference can be very significant: women with localized breast cancer have a 5-year survival rate of nearly 99%, while women with metastatic breast cancer have a 5-year survival rate of 27%.

[0128]

[0150] Despite these proven benefits, only about half of women currently receive mammograms at the rates recommended by the American College of Radiology. This low mammography utilization can result in significant burdens for patients and for the health care system in the form of increased expenses and costs. Screening mammography uptake is hampered in part by poor patient experience, including long delays in obtaining an appointment, unclear pricing, long wait times to receive results, and confusing reports. Furthermore, problems arising from a lack of pricing transparency are exacerbated by wide variations in costs across health care providers. Similarly, turnaround times for receiving results are inconsistent across health care providers.

[0129]

[0151] Additionally, significant variation in radiologist performance results in patients experiencing widely varying standards of care depending on location and income. For example, cancer detection rates are more than twice as high for radiologists in the 90th percentile compared with those in the 10th percentile. False-positive rates (e.g., the percentage of healthy patients incorrectly recalled for follow-up visits) vary even more significantly between these two groups. Aggregating all screening tests performed in the United States, approximately 96% of patients who are called back are false-positives. Given the significant societal and personal burden of cancer, often coupled with poor patient experience, inconsistent screening performance, and large cost variability, AI-based or AI-assisted screening methods can be developed to significantly improve this clinical accuracy of mammography screening.

[0130]

[0152] Innovations in artificial intelligence and software can be leveraged to achieve significant improvements in health outcomes, including early and accurate cancer detection. Such improvements can impact one or more steps in patient behavior, from cost transparency, appointment scheduling, patient care, radiology workflow, diagnostic accuracy, and result communication to follow-up. AI-driven networks of imaging centers can be developed to achieve high-quality service, timeliness, accuracy, and cost-effectiveness. In such clinics, women can immediately schedule mammograms and receive a cancer diagnosis before they leave in a single visit. By using the disclosed "real-time radiology" methods and systems, AI-driven clinics can transform the traditional two-visit screening-diagnosis paradigm into a single visit. Artificial intelligence may also be used to customize clinical workflow for each patient using a triage engine and to adjust how screening tests are read to significantly improve radiologist accuracy (e.g., by reducing radiologist fatigue), thereby improving the accuracy of cancer detection. AI-based or AI-assisted approaches can be used to achieve additional improvements to the screening / diagnostic process, such as patient scheduling, improved adherence to screening guidelines through customer outreach, and timeliness of report delivery using patient-facing applications. Self-improving systems may use AI to create better clinics that generate data to improve AI-based systems.

[0131]

[0153] A key component of creating an AI-powered radiology network is driving growth through patient acquisition. While other components of the system can streamline radiology workflow processes and provide an improved and streamlined experience for patients, patient recruitment and enrollment is critical to collecting enough data to train the AI-powered system for high performance.

[0132]

[0154] Furthermore, AI-powered clinics may reduce barriers to screening mammography by improving the patient experience before they even arrive at the clinic. This may involve addressing two major barriers that limit uptake: (1) concerns about the cost of the consultation, and (2) not being aware of a conveniently located clinic. When pricing and availability are completely opaque, as in traditional clinics, there can be significant variation in prices and services, thereby creating barriers to patient scheduling appointments.

[0133]

[0155] AI-based user applications may be developed to streamline the scheduling process and provide transparency to patients. The application may be configured to provide users with a map of clinics that the user's insurance covers, as well as available appointment times. For those with health insurance, screening mammograms, both 2D and 3D, are free of co-pays. This may be made clear to patients at the time of scheduling, along with any potential costs they may incur. Assurances about the timeliness of results may also be provided to patients, addressing potential sources of patient anxiety that may discourage them from scheduling an appointment.

[0134]

[0156] The application may be configured to verify the patient's insurance and, if necessary, request work orders from the patient's primary care physician (PCP) during the scheduling process. The application may also be configured to receive user input of pre-appointment forms to more efficiently process patients during their clinic visit. If the patient has remaining forms to complete before their appointment, the patient may be given a device upon checking into the clinic to complete the remaining forms. The application may also be configured to facilitate electronic completion of such forms, reducing or eliminating the time-consuming and error-prone tasks of handwritten paper forms, as is currently the case in the standard of care. By facilitating user input of paperwork in advance of the appointment date, the application provides patients with a more streamlined experience, with less time and resources allocated to on-site operational tasks.

[0135]

[0157] The patient's previously obtained mammograms may also be obtained prior to the appointment. For images obtained at an affiliated clinic, this process may occur transparently to the patient. Obtaining previous images prior to the visit may eliminate a potential bottleneck for the rapid review of newly obtained images.

[0136]

[0158] After scheduling an appointment, the application may be configured to provide reminders about upcoming appointments to improve attendance. The application may also be configured to provide patients with information about the appointment procedure in advance to minimize anxiety and reduce time spent in the exam room explaining the procedure. Furthermore, to build relationships with primary care physicians (PCPs), referring physicians may check whether their patients have scheduled mammogram appointments. This allows physicians to assess compliance and encourage patients who have not timely scheduled appointments as recommended by their physicians.

[0137]

[0159] Real-time Radiology System

[0160] The traditional breast cancer screening paradigm can involve significant delays that cause anxiety for patients. This reduces the number of women who choose to receive this preventive care and potentially puts them at risk of finding their cancer later, when the treatment is more difficult and life-threatening. A typical patient visits a clinic for a screening mammogram and leaves after spending approximately 30 minutes in the clinic. The woman then waits 30 days for a phone call or letter informing her that there was a suspicious abnormality on her screening mammogram and that she should schedule a follow-up diagnostic appointment. The patient then waits another week for that appointment, during which time she may undergo additional imaging to determine whether a biopsy is necessary.

[0138]

[0161] The current paradigm is motivated by the volume of patients screened in large-scale practices (e.g., over 100 patients per day). Such imaging centers typically have a backlog of screening visits that must be read for at least 1–2 days before radiologists can process the screening mammograms performed on a given day. If any of these cases require diagnostic workup, the visit is often not available immediately due to the wide variability in the length of diagnostic visits (e.g., ranging from 20 to 120 minutes). Scheduling does not take this into account, resulting in long wait times for patients and poor workflow for technologists.

[0139]

[0162] Patients who receive an immediate, real-time reading of their screening mammogram may experience less anxiety than those who do not receive one until three weeks later. In contrast, women who receive a false-positive screening result (a normal case flagged as suspicious) but receive an immediate reading experience levels of anxiety similar to those of women who receive normal mammograms. Most of these women were not aware that they had an abnormal screen. However, women who are aware that they have an abnormal screen tend to seek further medical attention for breast-related concerns and other medical problems. Furthermore, women may be more satisfied with the screening process and may be more likely to comply with future screening recommendations if they know they will leave the mammography clinic with their mammogram results. Such increased patient satisfaction may improve member retention in health plans. Additionally, immediate reading of suspicious cases may decrease the time to breast cancer diagnosis, thereby improving patient care and outcomes.

[0140]

[0163] In some cases, clinics can provide real-time services by limiting volume. Such clinics may schedule only a few patients at a given time so that patients can immediately follow up the screening procedure with a diagnostic consultation if the need arises. This procedure can be expensive, time-consuming, and unacceptable on a large scale, which still means that most women must wait several weeks for potentially life-changing results. Roughly 4 million women may encounter such an unpleasant screening process each year.

[0141]

[0164] Using the methods and systems of the present disclosure, an AI-based triage system may be developed for screening mammography.

[0165] As screening images are received from clinical imaging systems, they may be processed by an AI-driven Triage Engine, which then stratifies patient cases into one of multiple workflows. For example, the workflows may include two categories (e.g., normal and suspicious). As another example, the workflows may include three categories (e.g., normal, equivocal, and suspicious). Each such category may then be handled by a different set of dedicated radiologists who specialize in performing their particular set of workflows.

[0142]

[0166] FIG. 9 shows an example schematic of a patient flow through a clinic using an AI-enabled real-time radiology system and a patient mobile application (app) according to disclosed embodiments. The patient begins by registering on the website or patient app. The patient then schedules a radiology screening appointment using the patient app. The patient then completes a pre-examination form using the patient app. The patient then arrives at the clinic for a screening visit. An AI-based radiology evaluation is then performed on the medical images obtained from the patient's screening visit. The patient's images and visit results are then provided to the patient through the patient app. The patient then reschedules the appointment using the patient app, if necessary or recommended. The screening visit process may proceed as before.

[0143]

[0167] FIG. 10 shows an example schematic of an AI-assisted radiology evaluation workflow according to disclosed embodiments. First, a dataset including a patient's electronic health record (EHR) and medical images is provided. Next, an AI-based triage engine processes the EHR and medical images to analyze the dataset and classify the dataset as likely normal, possibly suspicious, or likely suspicious. Next, a workflow distribution module distributes the patient's dataset to one of three workflows: a normal workflow, an uncertain workflow, and a suspicious workflow, based on the dataset's classification as likely normal, possibly suspicious, or likely suspicious, respectively. Each of the three workflows may include radiologist review or further AI-based analysis (e.g., by a trained algorithm).

[0144]

[0168] The majority of mammography screening visits may be classified as normal. By focusing a first set of radiologists exclusively on this workflow, the concept and value of “batch reading” and its associated productivity gains may be applied and expanded. Because the cases handled by this first set of radiologists may almost all be normal, there may be fewer context switches and penalties for handling significantly unusual cases. In an AI-based system, reports may be automatically pre-populated, allowing radiologists to spend significantly more time interpreting images rather than writing reports. In the rare cases where a radiologist disagrees with the AI-assessed normal case and considers the case suspicious, the case may be treated as normal and the patient may be scheduled for a diagnostic consultation. These normal cases may be further subdivided into even more homogeneous batches to realize productivity improvements by grouping cases that the AI-based system has identified as similar. For example, batching all AI-assessed dense breasts together, or batching cases that are visually similar based on AI-derived features.

[0145]

[0169] A small portion of mammography screening visits may fall into the uncertain case workflow. Such sessions may involve findings that the AI system does not classify as normal, but which do not meet the fully suspicious threshold. These can typically be highly complex cases requiring significantly more time per radiologist evaluation session than cases in the normal or suspicious case workflows. For this reason, it may be beneficial to focus a second set of distinct radiologists on this smaller number of tasks, which are less homogeneous and potentially have significantly more interpretation and reporting requirements. These radiologists, through years of experience or training, are more specialized in interpreting these difficult cases. This specialization may become even more specific based on the category or features identified by the AI. For example, one group of radiologists may perform better than others at correctly assessing AI-identified tumor masses. Therefore, visits so identified by the algorithm may be routed to this more suitable group of experts. In some cases, the second set of radiologists is the same as the first set of radiologists, but the radiological evaluation of different sets of cases is performed at different times based on case priority. In some cases, the second set of radiologists is a subset of the first set of radiologists.

[0146]

[0170] The fewest, but most important, portion of a mammography screening visit can be categorized as a workflow for suspicious cases. A third set of radiologists may be assigned to this role, effectively interpreting these cases as their "on-call" duty. Most of a radiologist's time may be spent performing scheduled diagnostic visits. However, during downtime between visits, radiologists may be alerted to any suspicious cases so that the diagnosis can be verified as soon as possible. Such cases may be crucial to handle efficiently so that patients can begin follow-up diagnostic visits as soon as possible. In some cases, the third set of radiologists is the same as the first or second set of radiologists, but radiological evaluations of different sets of cases occur at different times based on case priority. In some cases, the third set of radiologists is a subset of the first or second set of radiologists.

[0147]

[0171] In some cases, a workflow may include applying an AI-based algorithm to analyze medical images to determine the difficulty of performing a radiological evaluation of the medical images, and then prioritizing the medical images for radiological evaluation or assigning the medical images to a set of radiologists (e.g., among multiple different sets of radiologists) based on the determined degree of difficulty. For example, less difficult cases (e.g., more “routine” cases) may be assigned to a set of radiologists with a relatively lower degree of skill or experience, while more difficult cases (e.g., more questionable or unconventional cases) may be assigned to a different set of radiologists with a relatively higher degree of skill or experience (e.g., specialized radiologists). For example, less difficult cases (e.g., more “routine” cases) may be assigned to a first set of radiologists with a relatively lower level of schedule availability, while more difficult cases (e.g., more questionable or unconventional cases) may be assigned to a different set of radiologists with a relatively higher level of schedule availability.

[0148]

[0172] In some cases, the degree of difficulty may be measured by the estimated length of time required to fully evaluate the image (e.g., about 1 minute, about 2 minutes, about 3 minutes, about 4 minutes, about 5 minutes, about 6 minutes, about 7 minutes, about 8 minutes, about 9 minutes, about 10 minutes, about 15 minutes, about 20 minutes, about 25 minutes, about 30 minutes, about 40 minutes, about 50 minutes, about 60 minutes, or more than about 60 minutes). In some cases, the degree of difficulty may be measured by the estimated degree of agreement or concordance in radiological evaluations of medical images across multiple unrelated radiological evaluations (e.g., by different radiologists or by the same radiologist on different days). For example, the estimated degree of concordance or agreement in radiological assessments may be about 50%, about 55%, about 60%, about 65%, about 70%, about 75%, about 80%, about 85%, about 90%, about 95%, about 96%, about 97%, about 98%, about 99%, or greater than about 99%. In some cases, the degree of difficulty may be measured by the desired level of radiologist education, experience, or expertise (e.g., less than about 1 year, about 1 year, between 1 and 2 years, about 2 years, between 2 and 3 years, about 3 years, between 3 and 4 years, about 4 years, between 4 and 5 years, about 5 years, between 5 and 6 years, about 6 years, between 6 and 7 years, about 7 years, between 7 and 8 years, between 8 years, between 8 and 9 years, between 9 years, between 9 and 10 years, about 10 years, or greater than about 10 years). In some cases, the degree of difficulty may be measured by the estimated sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), or accuracy of the radiological assessment (e.g., about 50%, about 55%, about 60%, about 65%, about 70%, about 75%, about 80%, about 85%, about 90%, about 95%, about 96%, about 97%, about 98%, about 99%, or greater than about 99%).

[0149]

[0173] In some cases, the workflow may include applying an AI-based algorithm to analyze medical images to determine a categorization of the medical images, and then prioritizing the medical images for radiological evaluation or assigning the medical images to a set of radiologists (e.g., from among a plurality of different sets of radiologists) based on the determined categorization of the medical images. For example, a set of cases with similar characteristics may be categorized together and assigned to the same radiologist or set of radiologists, thereby achieving reduced context switches and improved efficiency and accuracy. The similar characteristics may be based, for example, on the region of the body where the ROI is located, tissue density, BIRADS score, etc. In some cases, the workflow may include applying an AI-based algorithm to analyze medical images to determine a lesion type in the medical images, and then prioritizing the medical images for radiological evaluation or assigning the medical images to a set of radiologists (e.g., from among a plurality of different sets of radiologists) based on the determined lesion type in the medical images.

[0150]

[0174] In some cases, the workflow may include having radiologists assign cases to themselves through a market-based system, whereby each case is evaluated by an AI-based algorithm to determine an appropriate price or cost for a radiological evaluation. Such a price or cost may be a defined relative unit of value that is reimbursed to each radiologist upon completion of the radiological evaluation. For example, each radiological evaluation of a case may be priced based on defined characteristics (e.g., difficulty, length of consultation). In such a workflow, cases may not be assigned to radiologists, thereby avoiding the problem of radiologists choosing relatively routine or easy cases to earn a high reimbursement rate per case.

[0151]

[0175] In some cases, workflows may include assigning cases to radiologists based on their assessed performance (e.g., their previous sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), accuracy, or efficiency in performing radiological assessments). Such performance may be determined or improved based on blinded assignment of control cases (e.g., positive or negative control cases) to radiologists to ensure quality control. For example, radiologists with better performance may be assigned high-volume cases or high-value or high-compensation cases. By defining such clear roles for a given radiologist (e.g., on any given day), each workflow can be individually optimized for task-specific needs. An AI-driven triage engine can enable the delivery of real-time radiology to patients at scale. The system may also enable dynamic case allocation based on expertise. For example, a fellowship-trained breast imager may be the most valuable imager in an uncertain case workflow, where their exceptional experience can be leveraged. Additionally, we are able to perform cross-clinic screen interpretation across a network of clinics, ensuring efficient use of radiologist time regardless of any individual clinic's staffing or patient base.

[0152]

[0176] Report delivery may occur as follows: The Mammography Quality Standards Act (MQSA) mandates that all patients receive a written lay summary of their mammography report in person. This report must be sent within 30 days of the mammogram. While verbal results are often used to guide care and ease anxiety, they must be supported by a written report. Reports can be mailed, sent electronically, or handed to the patient. Typically, clinics can deliver reports to patients using paper mail. AI-based clinics may deliver mammography reports electronically via a patient application. Source images may also be made available electronically so patients can easily retrieve and transfer information to other clinics. Patients in a real-time radiology workflow can receive their screening and diagnostic reports immediately before leaving the clinic.

[0153]

[0177] Timely reporting of screening results can be crucial to patient satisfaction. Waiting longer than two weeks for results and not being able to contact someone to have questions answered have been identified as major reasons contributing to patient dissatisfaction (which can also lead to lower screening rates in the future). This system can ensure that patients do not unexpectedly receive erroneous reports and that there is no uncertainty about when they will receive their results.

[0154]

[0178] AI-based systems may be continuously trained as follows: As clinical practice operates, new data is constantly collected and used to further train and refine the AI system, thereby further improving the quality of care and enabling new improvements to the patient experience. Each patient encounter provides the system with annotated, possibly live, examples to add to the dataset. In particular, the workflow of real-time radiology systems facilitates prioritizing the capture of high-value cases. Identifying false positives and false negatives (unflagged but suspicious cases) can be important for improving system performance by providing challenging examples with high educational value. Even cases that are correctly classified (e.g., as ground truth with respect to radiologist review) can provide useful feedback. Incorporating such examples into the training dataset can provide the system with a valuable source of information about uncertain calibrations, which ensures that the confidence values produced by AI-based systems are accurate. This can dramatically improve overall robustness and, therefore, trust in the system. By improving the end-to-end patient workflow and keeping radiologists in the loop, AI-based clinical systems can automatically discover the important information outlined above. The resulting system is constantly improving, providing consistently high quality patient care and radiologist support.

[0155]

[0179] AI-powered mammography screening clinics can provide patients with high-quality service and accuracy throughout the screening process. Patients can visit the clinic, be screened for cancer, receive any necessary follow-up care, and leave with their diagnosis, thereby completing the entire screening and diagnosis process in a single visit with rapid results. Patient applications are configured to enable price transparency, hassle-free scheduling, error-free form filling, and instant delivery of reports and images, thereby improving the ease, stress, and efficiency of the patient screening process.

[0156]

[0180] By adopting a specialized set of workflows for normal, equivocal, and suspicious cases (or alternative categorizations based on AI evaluation of images) orchestrated by an AI triage engine, radiologists can provide more accurate and more productive results. Clinicians may become more effective as the AI system learns and enhances their capabilities. AI-based or AI-assisted mammography may be performed on a large population scale with low cost and high efficiency, thereby improving the cancer screening process and patient outcomes.

[0157]

[0181] [Example]

[0158] Real-time radiology in breast cancer screening mammography when combined with artificial intelligence techniques

[0182] A software system configured to prioritize suspicious screening mammograms for immediate review by a radiologist has been developed, thereby reducing the time to diagnostic follow-up. By shortening the review time for suspicious mammography cases, the software system aims to significantly reduce patient anxiety and the overall time to treatment. Reducing the wait time, which is often up to approximately two to four weeks between the first and second evaluations, could be expected to increase the life expectancy of these patients who are actually breast cancer positive. A further potential benefit is that the software could reduce the likelihood of missing any cancer.

[0159]

[0183] Some studies have shown that women who screen with a false-positive result (a normal case flagged as suspicious, BIRADS 0) but receive immediate follow-up can experience levels of anxiety similar to those experienced by women who receive a normal diagnosis. Many of these women may not even be aware that they have an abnormal screening result. Therefore, immediate follow-up care can alleviate the potential anxiety caused by a false-positive screening result.

[0160]

[0184] On the other hand, women who receive false-positive screening results and are called back for a follow-up diagnostic visit days to weeks later tend to seek further medical attention for breast-related concerns and other medical problems. Thus, women who are able to receive their final mammography results during the same clinic visit as their mammography scan may be more satisfied with their screening experience and more likely to have high compliance with future screening recommendations.

[0161]

[0185] However, many breast imaging centers may be unable to communicate immediate follow-up visits. This can be attributed to several challenges, including scheduling constraints, the timeliness of receiving previous evaluations from other institutions, and the lost productivity of reading each visit immediately after acquisition. Perhaps most importantly, reading several breast screening cases together significantly improves reader accuracy. This requires waiting until a sufficiently large batch of cases has been collected before reading a visit, making it impossible to provide patients with immediate results and follow-up visits when indicated.

[0162]

[0186] Machine learning-based methods are employed to evaluate suspicious findings in mammography and tomosynthesis images. A triage software system is developed using machine learning for screening mammography to enable more timely report delivery and follow-up for suspicious cases (e.g., as done in a batch reading setting) (as shown in Figure 11). Medical images are provided to a real-time radiology system for processing. The real-time radiology system's AI-based triage engine processes the medical images and classifies them as suspicious or non-suspicious (e.g., normal or routine). If the image is classified as suspicious by the AI-based triage engine, it is sent for immediate radiologist review (e.g., during the same visit as the initial screening appointment or on the same day). The immediate radiologist review may confirm the suspicious case (leading to an immediate diagnostic consultation being ordered) or overturn the suspicious case (leading to the next scheduled routine annual screening). If the image is classified as not suspicious (e.g., normal or routine) by the AI-based triage engine, it is sent for routine radiologist review, which may assess the case as suspicious (leading to a routine diagnostic consultation being ordered) or may confirm the case as not suspicious (leading to the next scheduled routine annual screening being performed).

[0163]

[0187] This software enables large breast screening clinics to provide same-day or same-visit diagnostic follow-up imaging to patients with abnormal mammogram results. Leveraging such rapid diagnostic follow-up imaging can pave the way for breast imaging clinics to deliver the highest accuracy with the highest level of service and significantly reduce patient anxiety.

[0164]

[0188] Using such machine learning-based techniques, the time to true tumor treatment is reduced so that patients have a higher probability of living longer compared to patients not assessed by AI and patients who do not receive same-day follow-up diagnostic evaluations.

[0165]

[0189] Machine learning-based approaches for evaluating suspicious findings in mammography and tomosynthesis images contemplate several advantages and objectives: First, the time from the initial screening visit to the delivery of diagnostic imaging results for breast cancer screening may be reduced (possibly significantly), improving the likelihood of an accurate diagnosis. For example, such a diagnosis may be produced with increased sensitivity, specificity, positive predictive value, negative predictive value, area under the receiver operating characteristic (AUROC), or a combination thereof. Second, approaches combining radiologists with artificial intelligence may efficiently improve the speed and / or quality of the initial assessment. Third, more advanced diagnostic visits (e.g., additional X-ray-based imaging, ultrasound imaging, another type of medical imaging, or a combination thereof) may be completed within a short period (e.g., within 60 minutes) after the patient receives their screening results. Fourth, such methods may advantageously lead to improved patient satisfaction due to more timely delivery of results and follow-up imaging.

[0166]

[0190] method

[0191] Clinical workflows are optimized to provide higher levels of service to patients. As more patients and data are collected in the training dataset, machine learning algorithms continually improve the accuracy (or sensitivity, specificity, positive predictive value, negative predictive value, AUROC, or a combination thereof) of their computer-aided diagnoses.

[0167]

[0192] Computer algorithms and software are being developed to automatically classify breast screening images into possibly abnormal and normal categories with high accuracy. Such software could enable large-scale breast screening clinics to provide patients with abnormal initial screening results with same-day or same-visit diagnostic follow-up imaging. This would also require evaluating changes to clinical operations, particularly how screening cases are read and how a second diagnostic evaluation can be performed within the first 60 minutes of the exam.

[0168]

[0193] Rapid screening procedures are performed on all patients at breast screening clinics. Approximately 10% of patients who undergo screening have a suspicious result and are subsequently recommended for a diagnostic consultation on the same day or during the same visit. The rapid turnaround time for screening results and follow-up diagnostic consultations is made possible by careful coordination between radiologists, clinical staff, and patients in the clinical setting. As more information is collected, machine learning trained on increasingly large training datasets will provide even greater levels of accuracy in detecting suspicious mammography scans.

[0169]

[0194] Once the acquisition of a screening visit is complete, the images are sent to a router, received by software, and quickly classified (e.g., within about one minute). If the screening is marked as probably normal by the machine learning algorithm, the patient completes their visit and exits the clinic as usual. However, if the screening is flagged as probably abnormal by the machine learning algorithm, the patient is asked to wait up to about 10 minutes while the case is quickly reviewed by a radiologist (as shown in Figure 11).

[0170]

[0195] Assuming a given clinic screens approximately 30 patients per day and has a 10% chance of positivity, the machine learning algorithm will typically identify approximately 3 patients per day as positive and, after review by a radiologist, be designated suitable for real-time diagnostic follow-up (e.g., typically with additional tomosynthesis imaging and perhaps ultrasound consultation).

[0171]

[0196] Several metrics are used to demonstrate the effectiveness of real-time radiology methods and systems. First, the change in time between the patient's initial screening encounter and communication of diagnostic imaging results under the conventional and proposed real-time workflows can be measured to capture both changes in the delay in when a case's screening is reviewed, as well as changes in logistics such as mailing documents and scheduling appointments.

[0172]

[0197] Second, the real-time radiology model is continually evaluated (e.g., monthly) based on the most recent data collected. For example, the parameters of the computer vision algorithm are tweaked and changed to improve its accuracy for the upcoming screening period (e.g., one month). The effectiveness of changes to the computer program is evaluated against a blind test dataset of hundreds of representative consultations and from preliminary results from the subsequent screening period.

[0173]

[0198] Third, patient satisfaction surveys are reviewed periodically to help determine how operational processes can be improved to better enable follow-up diagnostic consultations within a short period of time (e.g., approximately 60 minutes).

[0174]

[0199] The following data may be collected for each patient undergoing mammography screening / diagnostic evaluation via the real-time radiology workflow: patient demographics (e.g., age, race, height, weight, socioeconomic background, smoking status, etc.), patient imaging data (e.g., obtained by mammography), patient results (e.g., BIRADS for screening and diagnostic visits and biopsy pathology results if applicable), patient visit event timestamps, patient callback rates for batch reading and real-time cases, and radiologist interpretation times for screening and diagnostic cases.

[0175]

[0200] The disclosed methods and systems can be used to perform real-time radiology with potential benefits including: detecting tumors that may otherwise go unrecognized (or only become recognized once the tumor has progressed); reducing time to treatment; improving patient longevity due to recognition and treatment compared to traditional evaluation processes; and reducing patient anxiety due to the elimination of waiting times between exams.

[0176]

[0201] [Example]

[0177] Multi-site investigation of a deep learning model of breast density for full-field digital mammography and digital breast tomosynthesis examinations

[0202] overview

[0203] Deep learning (DL) models show promise for mammographic breast density estimation, but performance can be hindered by limited training data or potential image variations across clinics. Digital breast tomosynthesis (DBT) examinations are increasingly standard for breast cancer screening and breast density assessment, but much more data is available for full-field digital mammography (FFDM) examinations. A breast density DL model was developed in a multisite setting using FFDM images and limited SM data for synthetic 2D mammography (SM) images derived from 3D DBT examinations. The DL model was trained to predict Breast Imaging Reporting and Data System (BI-RADS) breast density using FFDM images acquired from a retrospective study between 2008 and 2017 (Site 1: 57,492 patients, 750,752 images). The FFDM model was evaluated on SM datasets from two institutions (Site 1: 3,842 patients, 14,472 images; Site 2: 7,557 patients, 63,973 images). Adaptive methods were explored to improve performance on the SM datasets, taking into account the effect of dataset size for each adaptive method. Statistical significance was assessed through the use of confidence intervals and estimated by bootstrapping. Even without adaptive methods, the model showed close agreement with the original reporting radiologist for all three datasets (Site 1 FFDM: linearly weighted κw = 0.75, 95% confidence interval (CI): [0.74, 0.76]; Site 1 SM: κw = 0.71, CI: [0.64, 0.78]; Site 2 SM: κw = 0.72, CI: [0.70, 0.75]). When indicated, using only 500 SM images improved performance for Site 2 (Site 1: κw = 0.72, CI: [0.66, 0.79], Site 2: κw = 0.79, CI: [0.76, 0.81]).These results establish that the BI-RADS breast density DL model demonstrated a high level of performance on FFDM and SM images from two institutions using methods that required little or no SM images.

[0178]

[0204] Multisite studies have been conducted to develop breast density deep learning models for full-field digital mammography and synthetic mammography, such as those described by Matthews et al., "A Multisite Study of a Breast Density Deep Learning Model for Full-Field Digital Mammography and Synthetic Mammography," Radiology:Artificial Intelligence, doi.org / 10.1148 / ryai.2020200015. This document is incorporated herein by reference in its entirety.

[0179]

[0205] Introduction

[0206] Breast density is an important risk factor for breast cancer, and denser areas may mask findings in mammograms, further reducing sensitivity. In some situations, clinics are required to inform women of their density. Radiologists typically assess breast density using the Breast Imaging Reporting and Data System (BI-RADS) lexicon, which divides breast density into four categories: almost entirely fatty, scattered areas of fibroglandular density, heterogeneously dense, and extremely dense (as shown in Figures 12A–12D). Unfortunately, radiologists exhibit intra- and inter-reader variability in assessing BI-RADS breast density, which can translate into differences in clinical care and estimated risk.

[0180]

[0207] Figures 12A-12D show examples of composite 2D mammography (SM) images derived from digital breast tomosynthesis (DBT) examinations for each of four Breast Imaging Reporting and Data System (BI-RADS) breast density categories: (A) almost entirely fatty (Figure 12A), (B) scattered areas of fibroglandular density (Figure 12B), (C) heterogeneously dense (Figure 12C), and (D) extremely dense (Figure 12D). Images are normalized to fit the grayscale intensity window found in the Digital Imaging and Communications in Medicine (DICOM) header, ranging from 0.0 to 1.0.

[0181]

[0208] Deep learning (DL) may be employed to assess BI-RADS breast density on both film and full-field digital mammography (FFDM) images, with some models demonstrating closer agreement with consensus predictions than individual radiologists. To realize the promise of using such DL models in clinical practice, two key challenges must be met. First, as breast cancer screening increasingly shifts to digital breast tomosynthesis (DBT) due to improved reader performance, DL models may need to accommodate DBT consultations. Figures 13A–13D illustrate the differences in image characteristics between 2D images for FFDM and DBT consultations. However, the relatively recent adoption of DBT at many institutions means that the datasets available to train DL models are often quite limited for DBT consultations compared to FFDM consultations. Second, DL models may need to provide consistent performance across sites, where differences in imaging technology, patient demographics, or assessment practices may affect model performance. In practice, this may need to be achieved with little or no additional data required from each site.

[0182]

[0209] Figures 13A-13D show a comparison of a full-field digital mammography (FFDM) image (Figure 13A) and a synthetic 2D mammography (SM) image (Figure 13B) of the same breast under the same pressure in one subject, with zoomed-in areas to highlight possible texture and contrast differences between the two image types, with the original areas indicated by white boxes for both the FFDM image (Figure 13C) and the SM image (Figure 13D). The images are normalized to fit the grayscale intensity window found in the Digital Imaging and Communications in Medicine (DICOM) header, ranging from 0.0 to 1.0.

[0183]

[0210] A BI-RADS breast density DL model was developed that provided close agreement with the original reporting radiologist for both FFDM and DBT consultations at two institutions. The DL model was first trained to predict BI-RADS breast density using a large FFDM dataset from one institution. The model was then evaluated on a set of FFDM consultation exams from the same and separate institutions, as well as synthetic 2D mammography (SM) images (C-View, Hologic, Inc., Marlborough, MA) generated as part of a DBT consultation. To improve performance on the two SM datasets, adaptive techniques requiring fewer SM images were explored.

[0184]

[0211] Materials and Methods

[0212] Institutional review boards approved the retrospective study at each of the two sites where data were collected (Site 1: Internal Institutional Review Board, Site 2: Western Institutional Review Board). Informed consent was waived, and all data were handled in accordance with the Health Insurance Portability and Accountability Act.

[0185]

[0213] The dataset was collected from two sites: Site 1, an academic medical center in the Midwestern United States, and Site 2, an outpatient radiology clinic in Northern California. At Site 1, 191,493 mammography examinations were selected (FFDM: n = 187,627; SM: n = 3,866). Examinations were reviewed by one of 11 radiologists with breast imaging experience. At Site 2, 16,283 examinations were selected. Examinations were reviewed by one of 12 radiologists with breast imaging experience ranging from 9 to 41 years. Radiologists' BI-RADS breast density assessments were obtained from each site's mammography reporting software (Site 1: Magview version 7.1, Magview, Burtonsville, MD; Site 2: MRS version 7.2.0; MRS Systems Inc., Seattle, WA). To facilitate the development of our DL model, patients were randomly assigned to training (FFDM: 50,700, 88%; Site 1 SM: 3,169, 82%; Site 2 SM: 6,056, 80%), validation (FFDM: 1,832, 3%; Site 1 SM: 403, 10%; Site 2 SM: 757, 10%), or testing (FFDM: 4,960, 9%; Site 1 SM: 270, 7%; Site 2 SM: 744, 10%). All visits with BI-RADS breast density assessment were included. For the exam set, visits were required to have all four standard screening mammography images (medial and lateral oblique views and craniocaudal views of both breasts). The distribution of BI-RADS breast density assessments per set is shown in Table 1 (Site 1) and Table 2 (Site 2).

[0186] [Table 1]

[0187]

[0214] Table 1: Description of the training (Train), validation (Val), and test (Test) datasets for full-field digital mammography (FFDM) and synthetic 2D mammography (SM) at Site 1. The total number of patients, visits, and images is given for each dataset. The number of images in the four Breast Imaging Reporting and Data System (BI-RADS) breast density categories is also given.

[0188] [Table 2]

[0189]

[0215] Table 2: Description of the Site 2 synthetic 2D mammography (SM) training (Train), validation (Val), and test (Test) datasets. The total number of patients, visits, and images is given for each dataset. The number of images in the four Breast Imaging Reporting and Data System (BI-RADS) breast density categories is also given.

[0190]

[0216] The two sites represent different patient populations: the patient cohort from Site 1 was 59% Caucasian (34,192 / 58,397), 23% African American (13,201 / 58,397), 3% Asian (1,630 / 58,397), and 1% Hispanic (757 / 58,397), while Site 2 was 58% Caucasian (4,350 / 7557), 1% African American (110 / 7557), 21% Asian (1,594 / 7557), and 7% Hispanic (522 / 7557).

[0191]

[0217] Deep Learning Model

[0218] The DL model and training procedure were implemented using the Pytorch DL framework (pytorch.org, version 1.0), which includes a deep neural network model. The base model architecture included a pre-activation Resnet-34 in which batch normalization layers were replaced with group normalization layers. The model was configured to process as input a single image corresponding to one of the views from a mammography examination and produce estimated probabilities that the image was a breast image belonging to each of the BI-RADS breast density categories.

[0192]

[0219] Deep learning (DL) models have a learning speed of 10 -4 and weight decay is 10 -3 The model was trained using the full-field digital mammography (FFDM) dataset (shown in Table 1) using the Adam optimizer. Weight decay was not applied to parameters belonging to the normalization layer. The input was resized to 416 × 320 pixels, and pixel intensity values were normalized to fit the grayscale intensity window found in the Digital Imaging and Communications in Medicine (DICOM) header, ranging from 0.0 to 1.0. Training was performed using mixed precision and gradient checkpointing with a batch size of 256 distributed across two NVIDIA GTX 1080 Ti graphics processing units (Santa Clara, CA). Each batch was sampled such that the probability of selecting a BI-RADS B or BI-RADS C sample was four times higher than the probability of selecting a BI-RADS A or BI-RADS D sample, roughly corresponding to the density distribution found in the United States. Horizontal and vertical flipping were employed for data augmentation. To obtain more frequent information about the training progress, epochs were capped at 100,000 samples, for a total training set size of over 672,000 samples. The model was trained for 100 such epochs. Results are reported for the epoch with the smallest cross-entropy loss on the validation set, which occurred after 93 epochs.

[0193]

[0220] Parameters for the vector and matrix calibration methods were chosen by minimizing the cross-entropy loss function using the BFGS optimization method (scipy.org, version 1.1.0). Parameters were initialized so that the linear layers correspond to the identity transformation. Training was performed using a linear layer with the L2 norm of the gradient set to 10 -6 The training was stopped when the learning rate was less than 10 or when the number of iterations exceeded 500. Retraining the final fully connected layer for fine-tuning was -4 and weight decay is 10 -5 Fine-tuning was performed using the Adam optimizer in

[14] . The batch size was set to 64. The fully connected layers were trained for 100 epochs from random initialization, and results are reported for the epoch with the smallest validation cross-entropy loss. Training from scratch on a synthetic 2D mammography (SM) dataset was performed following the same procedure as for the base model. For fine-tuning and training from scratch, the epoch size was set to the number of training samples.

[0194]

[0221] Domain Adaptation

[0222] Domain adaptation is performed to use a model trained on a dataset from one domain (the source domain) to transfer that knowledge to a dataset from another domain (the target domain), which is usually of much smaller size. Features learned by DL models in previous layers can be general, i.e., domain and task agnostic. Depending on the similarity of the domains and tasks, it is possible to reuse deeper features learned from one domain for another domain or task. A model that can be directly applied to a new domain without modification is said to generalize.

[0195]

[0223] A method was developed to adapt a DL model trained on FFDM images (the source domain) to SM images (the target domain) by reusing all features learned from the FFDM domain. First, to calibrate the neural network, a small linear layer was added after the last fully connected layer. Two forms of linear layer were considered: (1) vector calibration, where the matrix is diagonal, and (2) matrix calibration, where the matrix is allowed to deform freely. Second, the last fully connected layer of the Resnet-34 model was retrained on samples from the target domain, a process called fine-tuning.

[0196]

[0224] To investigate the effect of the target domain dataset size, the adaptation technique was repeated on various SM training sets for a range of sizes. The adaptation process was repeated 10 times for each dataset size using different random samples of the training data. For each sample, training images were selected randomly without replacement from the full training set. As a baseline, a Resnet-34 model was trained from scratch, e.g., from random initialization, on the maximum number of training samples per SM dataset.

[0197]

[0225] Statistical analysis

[0226] To obtain a consultation-level assessment, each image in the consultation was processed using the DL model, and the resulting probabilities were averaged. Several performance metrics were calculated from these average probabilities for the four-class BI-RADS breast density task and the binary high-density (BI-RADS C+D) vs. non-high-density (BI-RADS A+B) task: (1) estimated accuracy based on agreement with the initial reporting radiologist, (2) area under the receiver operating characteristic curve (AUC), and (3) Cohen's kappa coefficient (scikit-learn.org, version 0.20.0). Confidence intervals were calculated using non-Studentized pivoted bootstrap of the test set for 8,000 random samples. For the four-class problem, macroAUC (the average of four AUC values from one task vs. the other) and linearly weighted Cohen's kappa coefficient were reported. For the binary density task, predicted high-density and non-high-density probabilities were calculated by adding the predicted probabilities for the corresponding BI-RADS density category.

[0198]

[0227] result

[0228] The performance of the deep learning model on FFDM consultations was evaluated as follows. First, the trained model was evaluated on a large, held-out set of FFDM consultations from Site 1 (4960 patients, 53,048 images, mean age: 56.9, age range: 23–97). In this case, the images were from the same institution and of the same image type employed to train the model. The BI-RADS breast density distribution predicted by the DL model (A: 8.5%, B: 52.2%, C: 36.1%, D: 3.2%) was similar to the distribution of the radiologist who originally reported the model (A: 9.3%, B: 52.0%, C: 34.6%, D: 4.0%). The DL model demonstrated close agreement with radiologists for the 4-class BI-RADS breast density task across a variety of performance measures (as shown in Table 3), including accuracy (82.2%, 95% confidence interval (CI): [81.6%, 82.9%]) and linearly weighted Cohen's kappa coefficient (κw = 0.75, CI: [0.74, 0.76]). A high level of agreement was also observed for the binary breast density task (accuracy = 91.1%, CI: [90.6%, 91.6%], AUC = 0.971, CI: [0.968, 0.973], κ = 0.81, CI: [0.80, 0.82]). As demonstrated by the confusion matrices shown in Figures 14A-14D, the DL model rarely deviated from multiple breast density categories (e.g., by calling extremely dense breast a scattered result; 0.03%, 4 / 13262), which is learned implicitly by the DL model without any explicit penalty for these types of large errors.

[0199]

[0229] Figures 14A-B show confusion matrices for the Breast Imaging Reporting and Data System (BI-RADS) breast density task (Figure 14A) and for the binary density task (BI-RADS C+D, high density, vs. BI-RADS A+B, non-high density) evaluated against the full-field digital mammography (FFDM) exam set. The number of exam samples (exams) in each bin is shown in parentheses.

[0200] [Table 3]

[0201]

[0230] Table 3: Performance of the disclosed deep learning model on the full-field digital mammography (FFDM) consultation test set for both the four-class Breast Imaging Reporting and Data System (BI-RADS) breast density task and the binary density task (BI-RADS C+D, which is dense, vs. BI-RADS A+B, which is non-dense). 95% confidence intervals are given in brackets. Results from other studies are evaluated for their respective test sets and are provided as a point of comparison.

[0202]

[0231] To place the results in the context of other studies, the performance of the deep learning model on the FFDM test set was compared with results evaluated on other large FFDM datasets obtained from academic centers and with commercially available breast density software, as shown in Table 3. The FFDM DL model appears to give comparable performance.

[0203]

[0232] The performance of the deep learning model for DBT consultations was evaluated as follows. Results were first reported for the Site 1 SM exam set (270 patients, 1080 images, mean age: 54.6, age range: 28–72) to avoid any possible differences between the two sites. As shown in Table 4, even when performed without adaptation, the model showed close agreement with the initial reporting radiologist for the BI-RADS breast density task (accuracy = 79%, CI: [74%, 84%]; κw = 0.71, CI: [0.64, 0.78]). The DL model slightly underestimated breast density on SM images (as shown in Figures 15A–15D), producing a BI-RADS breast density distribution with more non-dense cases and fewer dense cases (A: 10.4%, B: 57.8%, C: 28.9%, D: 3.0%) than radiologists (A: 8.9%, B: 49.6%, C: 35.9%, D: 5.6%). This bias may be due to the difference shown in Figure 13, where certain areas of the breast appear darker on SM images. Similar biases have been shown in other automated breast density estimation software

[33] . Agreement on the binary density task was also extremely high without adaptation (accuracy = 88%, CI: [84%, 92%]; κ = 0.75, CI: [0.67, 0.83]; AUC = 0.97, CI: [0.96, 0.99]).

[0204] [Table 4]

[0205]

[0233] Table 4: Performance of the disclosed method and system for adapting a deep learning (DL) model trained on one dataset to another with a set of 500 synthetic 2D mammography (SM) images. The datasets are denoted as "MM" for the full-field digital mammography (FFDM) dataset, "C1" for the SM dataset from Site 1, and "C2" for the SM dataset from Site 2. For reference, the performance of a model trained from scratch on the FFDM dataset (672,000 training samples) and evaluated on its test set is also shown. The 95% confidence intervals calculated by bootstrapping for the test set are given in brackets.

[0206]

[0234] After adaptation by matrix calibration using 500 SM images, the density distributions more closely resembled those of the radiologists (A: 5.9%, B: 53.7%, C: 35.9%, D: 4.4%), but overall agreement was similar (accuracy = 80%, CI: [76%, 85%]; κw = 0.72, CI: [0.66, 0.79]). Accuracy for the two dense classes improved at the expense of the two non-dense classes (as shown in Figures 15A-D). A significant improvement was observed in the binary density task, with Cohen's kappa coefficient increasing from 0.75 [0.67, 0.83] to 0.82 [0.76, 0.90] (accuracy = 91%, CI: [88%, 95%]; AUC = 0.97, CI: [0.96, 0.99]).

[0207]

[0235] Figures 15A-15D show confusion matrices for the Breast Imaging Reporting and Data System (BI-RADS) breast density task without adaptation (Figure 15A), the binary density task (BI-RADS C+D, which is high density, vs. BI-RADS A+B, which is non-high density) without adaptation (Figure 15B), the BI-RADS breast density task with adaptation using matrix calibration of 500 training samples (Figure 15C), and the binary density task (high density vs. non-high density) with adaptation using matrix calibration of 500 training samples (Figure 15B), evaluated against the SM test set from Site 1. The number of test samples (visits) in each bin is shown in parentheses.

[0208]

[0236] For the unadapted Site 2 SM exam set (744 patients, 6192 images, mean age: 55.2, age range: 30–92), we again observed a high degree of agreement between the DL model and the initial reporting radiologist (accuracy = 76%, CI: [74%, 78%]; κw = 0.72, CI: [0.70, 0.75], as shown in Table 4). The BI-RADS breast density distribution predicted by the DL model (A: 5.7%, B: 48.8%, C: 36.4%, D: 9.1%) was more similar to the distribution seen in the Site 1 dataset. The model may not be optimal for Site 2, where patient demographics differ, possibly due to prior learning from the Site 1 FFDM dataset. The predicted density distribution did not appear to be biased toward low density estimates as seen in Site 1 (as shown in Figures 16A–D). This may suggest some differences in the SM images or their interpretation between the two sites. Agreement was particularly strong for the binary density task (Accuracy = 92%, CI: [91%, 93%]; Kappa = 0.84, CI: [0.81, 0.87]; AUC = 0.980, CI: [0.976, 0.986]). The very good performance on the Site2 dataset without adaptation demonstrates that the DL model can generalize well across sites.

[0209]

[0237] Adaptation with matrix calibration on 500 training samples significantly improved performance on the BI-RADS breast density task for the SM dataset at Site 2 (precision = 80, CI: [78, 82]; κw = 0.79, CI: [0.76, 0.81]). After adaptation, the predicted BI-RADS breast density distribution (A: 16.9%, B: 43.3%, C: 29.4%, D: 10.4%) was more similar to the radiologist distribution (A: 15.3%, B: 42.2%, C: 30.2%, D: 12.3%). Adaptation may have helped adjust for the demographic distribution of breast density at this site. There was less improvement in the binary breast density task (accuracy = 92, CI: [91, 94]; kappa = 0.84, CI: [0.82, 0.87]; AUC = 0.983, CI: [0.978, 0.988]).

[0210]

[0238] Figures 16A-16D show confusion matrices for the Breast Imaging Reporting and Data System (BI-RADS) breast density task without adaptation (Figure 16A), the binary density task (BI-RADS C+D, which is high density, vs. BI-RADS A+B, which is non-high density) without adaptation (Figure 16B), the BI-RADS breast density task with adaptation using matrix calibration of 500 training samples (Figure 16C), and the binary density task (high density vs. non-high density) with adaptation using matrix calibration of 500 training samples (Figure 16B), evaluated against the Site 2 SM test set. The number of test samples (visits) in each bin is shown in parentheses.

[0211]

[0239] The relative performance of different adaptive methods may depend on the number of training samples available for adaptation, with more training samples being more beneficial for methods with more parameters. Figures 17A-17D show the effect of the amount of training data on the performance of adaptive methods, as measured by macroAUC for the Site 1 dataset (Figure 17A), the effect of the amount of training data on the performance of adaptive methods, as measured by linearly weighted Cohen's Kappa coefficient for the Site 1 dataset (Figure 17B), the effect of the amount of training data on the performance of adaptive methods, as measured by macroAUC for the Site 2 SM dataset (Figure 17C), and the effect of the amount of training data on the performance of adaptive methods, as measured by linearly weighted Cohen's Kappa coefficient for the Site 2 SM dataset (Figure 17D), according to disclosed embodiments. To explore uncertainties arising from the selection of training data rather than those arising from the limited size of the test set, as was done in calculating the 95% confidence intervals, results are reported for 10 random realizations of the training data for each dataset size (as described elsewhere herein). Each adaptation method has a range of sample numbers for which it provides the best performance, with the range corresponding to the number of parameters of the adaptation method (vector calibration: 4 + 4 = 8 parameters; matrix calibration: 4 × 4 + 4 = 20 parameters; fine-tuning: 512 × 4 + 4 = 2052 parameters). When the number of training samples was very small (e.g., less than 100 images), some adaptation methods negatively impacted performance. Even with the largest dataset size, the amount of training data was too limited for a Resnet-34 model trained from scratch on SM images to outperform a model adapted from FFDM.

[0212]

[0240] Figures 17A-17D show the effect of the number of training samples in the target domain on the performance of the adapted model, as measured by macroAUC (Figure 17A) and linearly weighted Cohen's kappa coefficient (Figure 17B) for the synthetic 2D mammography (SM) test set from Site 1, and by macroAUC (Figure 17C) and linearly weighted Cohen's kappa coefficient (Figure 17D) for the SM test set from Site 2. Results are shown for vector calibration, matrix calibration, and retraining the final fully connected layer (fine tuning). Error bars indicate the standard error of the mean calculated for 10 random samplings of the training data. Performance before adaptation (none) and training from scratch are shown as references. For the SM study from Site 1, full-field digital mammography (FFDM) performance served as an additional reference. Note that each graph is shown with its own full dynamic range to facilitate comparison of different adaptation methods for a given metric and dataset.

[0213]

[0241] Discussion

[0242] Breast Imaging Reporting and Data System (BI-RADS) breast density can be an important indicator of breast cancer risk and radiologist sensitivity, but intra- and inter-reader variability can limit the usefulness of this measure. Deep learning (DL) models for estimating breast density may be constructed to reduce this variability while still achieving accurate assessments. However, such DL models have been demonstrated to be applicable to digital breast tomosynthesis (DBT) examinations and can be generalized across institutions, thereby demonstrating their suitability as useful clinical tools. To overcome the limited training data available for DBT examinations, DL models are first trained on a large set of full-field digital mammography (FFDM) images. When evaluated on a held-out examination set of FFDM images, the model demonstrated close agreement with radiologist-reported BI-RADS breast density (κw = 0.75, 95% confidence interval (CI): [0.74, 0.76]). The model was then evaluated on two datasets of synthetic 2D mammography (SM) images generated as part of a DBT consultation. High levels of agreement were found for the FFDM data and the SM dataset from the same institution (Site 1: κw = 0.71, CI: [0.64, 0.78]) and for the SM dataset from a different institution (Site 2: κw = 0.72, CI: [0.70, 0.75]). The strong performance of the DL model demonstrates that it can be generalized to DBT consultations and data from different institutions. Further adaptation of the model to the SM dataset led to some improvement at Site 1 (κw = 0.72, CI: [0.66, 0.79]) and much larger improvement at Site 2 (κw = 0.79, CI: [0.76, 0.81]).

[0214]

[0243] Once the initial reporting radiologist's assessment is accepted as ground truth, the level of inter-reader variability between these radiologists has a significant impact on the performance that can be achieved for a given dataset. For example, after adaptation, the performance obtained for the Site2 SM dataset was higher than that obtained for the FFDM dataset used to train the model. This is likely a result of the limited inter-reader variability for the Site2 SM dataset, as over 80% of the examinations were read by only two readers.

[0215]

[0244] In contrast to other methods, the BI-RADS breast density DL model was evaluated on SM images from DBT examinations and on data from multiple institutions. Moreover, as discussed above, the DL model demonstrated comparable performance compared to other DL models and commercially available breast density software when evaluated on FFDM images (κw = 0.75, CI: [0.74, 0.76] vs. Lehman et al. 0.67, CI: [0.66, 0.68]; Volpara 0.57, CI: [0.55, 0.59]; Quantra 0.46, CI: [0.44, 0.47]) [19,3]. For each method, results are reported for its individual test set, similar to the way our own results are reported.

[0216]

[0245] Other measures of breast density, such as volumetric breast density, may be estimated by automated software for 3D tomosynthesis volumes or by inference from DBT examination. A threshold can be chosen to convert such measures to BI-RADS breast density, but this may result in a lower level of agreement than direct estimation of BI-RADS breast density (e.g., agreement between radiologist-assessed BI-RADS breast density and assessments derived from volumetric breast density was κw = 0.47). Here, BI-RADS breast density is estimated from 2D SM images instead of 3D tomosynthesis volumes, as this simplifies transfer learning from FFDM images and mirrors the way breast radiologists assess density.

[0217]

[0246] In some cases, when a deep learning (DL) model is adapted to a new institution, adjustments may be made for cross-institutional differences in image content, patient demographics, or interpreting radiologists. This final adjustment may result in some inter-reader variability between the initial DL model and the adapted DL model, but this may be lower than the inter-reader variability once the model learns the consensus of radiologists in each group. As a result, the improved DL model performance observed after adaptation on the Site 2 SM dataset may be due to differences in patient demographics or radiologist evaluation practices compared to the FFDM dataset. The weaker improvement on the Site 1 SM dataset may be due to similarities in these same factors. Regarding the comparison of domain adaptation techniques as a function of the number of training samples, adjusting the number of parameters in the model based on the number of training samples may yield better performance than training a DL model trained from scratch.

[0218]

[0247] These results confirm that widespread use of the Breast Imaging Reporting and Data System (BI-RADS) breast density deep learning (DL) model holds great potential for improving clinical care. The success of the DL model without adaptation indicates that the features learned by the model are broadly applicable to both full-field digital mammography (FFDM) and synthetic 2D mammography (SM) images from digital breast tomosynthesis (DBT) examinations, as well as across different readers and institutions. Therefore, the BI-RADS breast density DL model can be deployed to new sites and institutions without the additional effort of compiling large datasets and training models from scratch. A BI-RADS breast density DL model that can generalize across sites and image types can be used to provide faster, lower-cost, and more consistent estimates of women's breast density.

[0219]

[0248] [Example]

[0220] Real-time radiology for optimized radiology workflow

[0249] A machine learning-based classification system has been developed to sort, prioritize, enhance, or edit radiology interpretation tasks (e.g., among multiple different workflows) based on an analysis of a dataset including medical images of a patient. Sorting, prioritizing, enhancing, or editing cases for radiology evaluation may be performed based on medical image data (e.g., image data headers or database elements, instead of relying solely on metadata such as labels or annotation information). For example, the medical images may be processed by one or more image processing algorithms. The machine learning-based radiology system enables advanced radiology workflows that convey faster and more accurate diagnoses by enabling medical image datasets to be stratified into various radiology evaluations based on their suitability for such evaluations. For example, the multiple different workflows may include radiology evaluations by multiple different sets of radiologists. The radiologists may be on-site or remote from the clinic where the patient's medical images are obtained.

[0221]

[0250] In some embodiments, the machine learning-based classification system is configured to sort or prioritize radiology interpretation tasks among a plurality of different workflows based on an analysis of datasets including medical images of a subject. For example, one set of datasets including medical images may be prioritized for radiology evaluation over another set of datasets including medical images based on a determination by the AI triage engine that the first set of datasets has a higher priority or urgency than the second set of datasets.

[0222]

[0251] In some embodiments, the real-time radiology system uses an AI-enabled triage workflow to obtain medical images of a subject through a screening visit and then uses AI to communicate radiology results (e.g., screening results and / or diagnostic results) to the patient within minutes (e.g., within about 5 minutes, about 10 minutes, about 15 minutes, about 30 minutes, about 45 minutes, about 60 minutes, about 90 minutes, about 2 hours, about 3 hours, about 4 hours, about 5 hours, about 6 hours, about 7 hours, or about 8 hours) after the medical images are obtained.

[0223]

[0252] In some embodiments, the real-time radiology system includes a real-time notification system for interacting with clinic staff of AI-determined alert cases. The notification system is installed in various locations within the screening clinic (e.g., at clinic staff workstations). Users (e.g., physicians and clinic staff) are assigned roles, and each role receives different notifications. A notification is triggered when a trained algorithm determines an emergency situation for a patient's case. For example, a notification may include both advisory information as well as authorized users entering information that may affect the patient's clinical workflow in real time during the visit. A physician (e.g., a treating physician or radiologist) is notified of emergency cases as they occur via real-time alerts and uses the information from the notification to provide a better diagnosis.

[0224]

[0253] In some embodiments, the real-time radiology system includes a patient mobile application (app) for sending notifications to the patient, which may include the status of the patient's screening / diagnostic visit, radiology evaluations performed on the patient's medical images, presentations constructed from the radiology evaluations, etc.

[0225]

[0254] In some embodiments, the real-time radiology system includes a database configured to acquire, retrieve, and store for future retrieval datasets including medical images (e.g., radiology images), AI enrichment of the datasets (e.g., medical images labeled, annotated, or processed by AI, such as via image processing algorithms), screening results, diagnostic results, and presentations of the medical images and results. The real-time radiology system is configured to provide services to patients and their clinical caregivers (e.g., radiologists and clinic staff) to retrieve, access, and view the contents of the database. Services of the real-time radiology system may support building complex computational graphs from the stored datasets, including chaining several AI models.

[0226]

[0255] FIG. 18 shows an example schematic of a real-time radiology evaluation workflow. The real-time radiology evaluation workflow may include acquiring images from a subject (e.g., via mammography). The images may be processed using the systems and methods of the present disclosure (e.g., including AI algorithms) to detect that the images correspond to suspicious cases. A clinician may be alerted that the subject is suitable for real-time radiology evaluation. While the subject waits in the clinic, the images are sent to a radiologist for radiology evaluation, and the results of the radiology evaluation are provided to the clinician for further review.

[0227]

[0256] FIG. 19 shows another example of a schematic of a real-time radiology evaluation workflow. Using the systems and methods of the present disclosure (e.g., including AI algorithms), a subject's images are retrieved from a PACS database and analyzed. If the AI analysis indicates that a given subject (e.g., patient) does not have suspicious images, a patient coordinator is notified, who then informs the patient that the results will be received at home after a radiology evaluation is performed. If the AI analysis indicates that the patient has suspicious images, a technologist is notified, who then either (1) updates the medical history, notifies a radiologist to perform a radiology evaluation, and provides the results to the patient coordinator, or (2) notifies billing to process the patient's copay for a follow-up visit and notifies the patient coordinator. The patient coordinator may share the results with the patient and schedule a follow-up appointment, if necessary.

[0228]

[0257] In some embodiments, the real-time radiology evaluation workflow includes (i) sending the image or a derivative thereof to a first radiologist of a first set of radiologists for radiology evaluation and generating a screening result based at least in part on whether the image is classified as suspicious; (ii) sending the image or a derivative thereof to a second radiologist of a second set of radiologists for radiology evaluation and generating a screening result based at least in part on whether the image is classified as equivocal; or (iii) sending the image or a derivative thereof to a third radiologist of a third set of radiologists for radiology evaluation and generating a screening result based at least in part on whether the image is classified as normal.

[0229]

[0258] In some embodiments, the real-time radiology evaluation workflow includes, if at least one image is classified as suspicious, sending the image or a derivative thereof to a first radiologist of the first set of radiologists for radiological evaluation to produce a screening result. In some embodiments, the real-time radiology evaluation workflow includes, if the image is classified as equivocal, sending the image or a derivative thereof to a second radiologist of the second set of radiologists for radiological evaluation to produce a screening result. In some embodiments, the real-time radiology evaluation workflow includes, if the image is classified as normal, sending the image or a derivative thereof to a third radiologist of the third set of radiologists for radiological evaluation to produce a screening result.

[0230]

[0259] In some embodiments, the subject's screening results are produced in the same clinic visit as the step of acquiring the images or derivatives thereof, hi some embodiments, a first set of radiologists is located at an on-site clinic (e.g., the clinic where the images or derivatives thereof were acquired).

[0231]

[0260] In some embodiments, the second set of radiologists includes radiologists (e.g., radiologists trained to classify images, or derivatives thereof, as normal or suspicious with greater accuracy than a trained algorithm). In some embodiments, the third set of radiologists is located remotely from an on-site clinic (e.g., the clinic where the images were acquired). In some embodiments, a third radiologist in the third set of radiologists performs radiological evaluation of images, or derivatives thereof, of a batch containing multiple images (e.g., where the batch is selected to improve efficiency of radiological evaluation).

[0232]

[0261] In some embodiments, the real-time radiology evaluation workflow includes performing a diagnostic procedure for the subject based at least in part on the screening results to produce a diagnostic result for the subject. In some embodiments, the diagnostic result for the subject is produced in the same clinic visit as the step of acquiring the image. In some embodiments, the diagnostic result for the subject is produced within about one hour of the step of acquiring the image.

[0233]

[0262] In some embodiments, the image or a derivative thereof is sent to a first radiologist, a second radiologist, or a third radiologist based at least in part on additional characteristics of the part of the subject's body, in some embodiments, the additional characteristics include anatomy, tissue characteristics (e.g., tissue density or physical properties), the presence of foreign bodies (e.g., implants), a type of finding, a medical condition (e.g., predicted by an algorithm, such as a machine learning algorithm), or a combination thereof.

[0234]

[0263] In some embodiments, the image or a derivative thereof is sent to the first radiologist, the second radiologist, or the third radiologist based at least in part on additional characteristics of the first radiologist, the second radiologist, or the third radiologist (e.g., the personal ability of the first radiologist, the second radiologist, or the third radiologist to perform a radiological assessment of at least one image or a derivative thereof).

[0235]

[0264] In some embodiments, the real-time radiology assessment workflow includes generating an alert based at least in part on sending the image or a derivative thereof to a first radiologist or sending the image or a derivative thereof to a second radiologist. In some embodiments, the real-time radiology assessment workflow includes sending the alert to the subject or the subject's clinical caregiver. In some embodiments, the real-time radiology assessment workflow includes sending the alert to the subject through a patient mobile application. In some embodiments, the alert is generated in real time with (b) or near real time with (b).

[0236]

[0265] In some embodiments, the real-time radiology system includes an AI-driven remote imaging platform. The remote imaging platform includes an AI-based radiology work distributor that routes cases for physician review in real time, or substantially in real time, with the acquisition of medical images. The remote imaging platform may be configured to perform AI-based profiling of image types and physicians to assign each case to one physician from among multiple physicians based on the individual physician's aptitude for handling, evaluating, or interpreting a given case's dataset. Radiologists may belong to a network of radiologists, each with a distinct set of radiology skills, expertise, and experience. The remote imaging platform may assign cases to physicians based on searching the network for physicians with a desired combination of skills, expertise, experience, and cost. The radiologist may be on-site or remote from the clinic where the patient's medical images are acquired. In some embodiments, a radiologist's expertise may be determined by comparing the radiologist's performance to the performance of an AI model for various radiology tasks on an evaluative set of data. Radiologists may be paid for performing radiological assessments for each individual case they undertake and perform. In some embodiments, the real-time radiology system features dynamic pricing of radiological work based on the AI-determined difficulty, urgency, and value of the radiological work (e.g., radiological assessment, interpretation, or review).

[0237]

[0266] In some embodiments, the real-time radiology system is configured to organize, prioritize, or stratify multiple medical image cases into subgroups of medical image cases for radiological evaluation, interpretation, or review. Stratification of medical image cases may be performed by an AI algorithm based on image characteristics of the individual medical image cases to improve human efficiency in evaluating individual cases. For example, the algorithm may group visually similar or diagnostically similar cases together for human review, such as grouping identified cases with similar lesion types located in similar regions of the anatomy.

[0238]

[0267] FIG. 20 shows an example schematic of an AI-assisted radiology evaluation workflow in a remote imaging setting. Using the systems and methods (e.g., including AI algorithms) of the present disclosure, a subject's images are retrieved from a PACS database and analyzed using an AI algorithm to prioritize and exclude cases for radiology evaluation (e.g., based on the subject's breast density and / or breast cancer risk). The AI-assisted radiology evaluation workflow can optimize the routing of cases for radiology evaluation based on the radiologist's skill level. For example, a first radiologist may have an average reading time of 45 seconds, expert-level expertise, and the skill to evaluate highly dense breasts. As another example, a second radiologist may have an average reading time of 401 seconds and beginner-level expertise. As another example, a third radiologist may have an average reading time of 323 seconds and beginner-level expertise. As another example, a fourth radiologist may have an average reading time of 145 seconds and beginner-level expertise. For example, the fifth radiologist may have an average reading time of 60 seconds, expert-level expertise, and skill for evaluating benign masses. The AI-assisted radiology review workflow may route a given subject case to a radiologist selected from the first, second, third, fourth, or fifth radiologist based on the radiologists' average reading time, expertise level, and / or skill level for the given subject case.

[0239]

[0268] In some embodiments, the AI-assisted radiology evaluation workflow includes (i) sending the image or a derivative thereof to a first radiologist of a first set of radiologists for radiology evaluation and generating a screening result based at least in part on whether the image is classified as suspicious; (ii) sending the image or a derivative thereof to a second radiologist of a second set of radiologists for radiology evaluation and generating a screening result based at least in part on whether the image is classified as equivocal; or (iii) sending the image or a derivative thereof to a third radiologist of a third set of radiologists for radiology evaluation and generating a screening result based at least in part on whether the image is classified as normal.

[0240]

[0269] In some embodiments, the AI-assisted radiology evaluation workflow includes, if at least one image is classified as suspicious, sending the image or a derivative thereof to a first radiologist of the first set of radiologists for radiological evaluation to produce a screening result. In some embodiments, the AI-assisted radiology evaluation workflow includes, if the image is classified as equivocal, sending the image or a derivative thereof to a second radiologist of the second set of radiologists for radiological evaluation to produce a screening result. In some embodiments, the AI-assisted radiology evaluation workflow includes, if the image is classified as normal, sending the image or a derivative thereof to a third radiologist of the third set of radiologists for radiological evaluation to produce a screening result.

[0241]

[0270] In some embodiments, the subject's screening results are produced in the same clinic visit as the step of acquiring the images or derivatives thereof, hi some embodiments, a first set of radiologists is located at an on-site clinic (e.g., the clinic where the images or derivatives thereof were acquired).

[0242]

[0271] In some embodiments, the second set of radiologists includes radiologists (e.g., radiologists trained to classify images, or derivatives thereof, as normal or suspicious with greater accuracy than a trained algorithm). In some embodiments, the third set of radiologists is located remotely from an on-site clinic (e.g., the clinic where the images were acquired). In some embodiments, a third radiologist in the third set of radiologists performs radiological evaluation of images, or derivatives thereof, of a batch containing multiple images (e.g., where the batch is selected to improve efficiency of radiological evaluation).

[0243]

[0272] In some embodiments, the AI-assisted radiology evaluation workflow further includes performing a diagnostic procedure for the subject based at least in part on the screening results to produce a diagnostic result for the subject. In some embodiments, the diagnostic result for the subject is produced in the same clinic visit as the step of acquiring the images. In some embodiments, the diagnostic result for the subject is produced within about one hour of the step of acquiring the images.

[0244]

[0273] In some embodiments, the image or a derivative thereof is sent to a first radiologist, a second radiologist, or a third radiologist based at least in part on additional characteristics of the part of the subject's body, in some embodiments, the additional characteristics include anatomy, tissue characteristics (e.g., tissue density or physical properties), the presence of foreign bodies (e.g., implants), a type of finding, a medical condition (e.g., predicted by an algorithm, such as a machine learning algorithm), or a combination thereof.

[0245]

[0274] In some embodiments, the image or a derivative thereof is sent to the first radiologist, the second radiologist, or the third radiologist based at least in part on additional characteristics of the first radiologist, the second radiologist, or the third radiologist (e.g., the personal ability of the first radiologist, the second radiologist, or the third radiologist to perform a radiological assessment of at least one image or a derivative thereof).

[0246]

[0275] In some embodiments, the AI-assisted radiology evaluation workflow includes generating an alert based at least in part on sending the image or a derivative thereof to a first radiologist or sending the image or a derivative thereof to a second radiologist. In some embodiments, the AI-assisted radiology evaluation workflow includes sending the alert to the subject or the subject's clinical caregiver. In some embodiments, the AI-assisted radiology evaluation workflow includes sending the alert to the subject through a patient mobile application. In some embodiments, the alert is generated in real time with (b) or near real time with (b).

[0247]

[0276] While preferred embodiments of the present invention have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. The present invention is not intended to be limited by the specific examples given herein. While the present invention has been described with reference to the above-referenced specifications, the descriptions and illustrations of the embodiments herein are not meant to be construed in a limiting sense. Numerous variations, changes, and substitutions will occur to those skilled in the art without departing from the invention. Furthermore, it should be understood that all aspects of the present invention are not limited to the specific depictions, configurations, or relative proportions set forth herein, which depend upon a variety of conditions and variables. It should be understood that various alternatives to the embodiments of the invention described herein may be employed in practicing the invention. Accordingly, the present invention is also intended to encompass any such alternatives, modifications, variations, or equivalents. It is intended that the following claims define the scope of the invention, and that methods and structures within the scope of these claims and their equivalents be covered thereby.

Claims

1. A method for processing at least one image of a body part of a subject, comprising: (a) acquiring the at least one image of the region of the subject's body; (b) using a trained algorithm to classify the at least one image, or a derivative thereof, as being indicative of cancer, as normal, equivocal, or suspicious, wherein the classifying step comprises applying an image processing algorithm to the at least one image, or a derivative thereof; (c) classifying the at least one image or a derivative thereof in (b); (i) if the at least one image or a derivative thereof is classified as suspicious as being indicative of the cancer, sending the at least one image or a derivative thereof to a first radiologist for radiological evaluation to produce a screening or diagnostic result; (ii) if the at least one image or a derivative thereof is classified as equivocal as being indicative of the cancer, sending the at least one image or a derivative thereof to a second radiologist, different from the first radiologist, for radiological evaluation to produce a screening or diagnostic result; (iii) if the at least one image or a derivative thereof is classified as normal, sending the at least one image or a derivative thereof to a third radiologist for radiological evaluation to produce a screening or diagnostic result; wherein (c) is performed in real time or near real time with (b); (d) receiving a radiological assessment of the subject from the first radiologist, the second radiologist, or the third radiologist, wherein the radiological assessment of the subject is generated by the first radiologist, the second radiologist, or the third radiologist based at least in part on a radiological analysis of the at least one image or a derivative thereof; A method comprising:

2. The method of claim 1 , wherein the at least one image or derivative thereof is a medical image.

3. 10. The method of claim 1, wherein the trained algorithm is configured to classify the at least one image, or a derivative thereof, as normal, equivocal, or suspicious with at least about 80% sensitivity.

4. 10. The method of claim 1, wherein the trained algorithm is configured to classify the at least one image, or a derivative thereof, as normal, equivocal, or suspicious with at least about 80% specificity.

5. 10. The method of claim 1, wherein the trained algorithm is configured to classify the at least one image, or a derivative thereof, as normal, equivocal, or suspicious with a positive predictive value of at least about 80%.

6. 10. The method of claim 1, wherein the trained algorithm is configured to classify the at least one image, or a derivative thereof, as normal, equivocal, or suspicious with a negative predictive value of at least about 80%.

7. 10. The method of claim 1, wherein the trained algorithm is configured to identify at least one region of the at least one image or a derivative thereof that contains or is suspected of containing abnormal tissue.

8. 10. The method of claim 1, wherein the cancer is breast cancer.

9. 9. The method of claim 8, wherein the at least one image or a derivative thereof is a three-dimensional image of the subject's breast.

10. 10. The method of claim 1, wherein the trained algorithm is trained using at least about 100 independent training samples comprising images indicative of or suspected of being indicative of the cancer.

11. 10. The method of claim 1, wherein the trained algorithm is trained using a first plurality of independent training samples comprising positive images indicative of or suspected of being indicative of the cancer, and a second plurality of independent training samples comprising negative images not indicative of or not suspected of being indicative of the cancer.

12. The method of claim 1 , wherein the trained algorithm comprises a supervised machine learning algorithm.

13. 13. The method of claim 12, wherein the supervised machine learning algorithm comprises a deep learning algorithm, a support vector machine (SVM), a neural network, or a random forest.

14. 10. The method of claim 1, further comprising monitoring the subject, wherein the monitoring comprises evaluating images of the region of the body of the subject at a plurality of time points, the evaluating being based at least in part on the classification of the at least one image, or a derivative thereof, at each of the plurality of time points as normal, equivocal, or suspicious as indicative of cancer.

15. 15. The method of claim 14, wherein differences in the evaluation of the images of the body of the subject at the multiple time points indicate one or more clinical indicators selected from the group including: (i) a diagnosis of the subject, (ii) a prognosis of the subject, and (iii) the effectiveness or ineffectiveness of a course of treatment for the subject.

16. 16. The method of any one of claims 1 to 15, wherein the screening result for the subject is produced in the same clinic visit as the step of obtaining the at least one image.

17. 16. The method of any one of claims 1 to 15, wherein the first radiologist is located at an on-site clinic and the at least one image is acquired at the on-site clinic.

18. 16. The method of any one of claims 1 to 15, wherein the second radiologist is a radiologist trained to classify the at least one image or a derivative thereof as normal or suspicious with greater accuracy than the trained algorithm.

19. 16. The method of any one of claims 1 to 15, wherein the third radiologist is located remotely from an on-site clinic and the at least one image is acquired at the on-site clinic.

20. 16. The method of any one of claims 1 to 15, wherein the third radiologist performs the radiological evaluation of the at least one image or a derivative thereof from a batch comprising a plurality of images, the batch being selected to improve efficiency of the radiological evaluation.

21. 21. The method of any one of claims 1 to 20, further comprising the step of performing a diagnostic procedure for the subject based at least in part on the screening results to produce a diagnostic result for the subject.

22. 22. The method of claim 21, wherein the diagnosis for the subject is produced in the same clinic visit as the step of obtaining the at least one image.

23. 23. The method of claim 22, wherein the diagnosis for the subject is produced within about one hour of the step of acquiring the at least one image.

24. 24. The method of any one of claims 1 to 23, wherein the at least one image or a derivative thereof is sent to the first radiologist, the second radiologist, or the third radiologist based at least in part on additional characteristics of the part of the body of the subject.

25. 25. The method of claim 24, wherein the additional characteristics include anatomy, tissue characteristics, presence of foreign body, type of finding, pathology, or a combination thereof.

26. 26. The method of any one of claims 1 to 25, wherein the at least one image or a derivative thereof is sent to the first radiologist, the second radiologist, or the third radiologist based at least in part on additional characteristics of the first radiologist, the second radiologist, or the third radiologist.

27. 27. The method of any one of claims 1 to 26, wherein (c) further comprises generating an alert based at least in part on the step of sending the at least one image or a derivative thereof to the first radiologist, or the step of sending the at least one image or a derivative thereof to the second radiologist, or the step of sending the at least one image or a derivative thereof to the third radiologist.

28. 28. The method of claim 27, further comprising transmitting the alert to the subject or to the subject's clinical caregiver.

29. 30. The method of claim 28, further comprising sending the alert to the subject through a patient mobile application.

30. 28. The method of claim 27, wherein the alert is generated in real time or near real time with (b).

31. 2. The method of claim 1, wherein applying the image processing algorithm comprises identifying a region of interest within the at least one image or a derivative thereof, and labeling the region of interest to produce at least one labeled image.

32. 32. The method of claim 31, further comprising storing the at least one labeled image in a database.

33. 33. The method of any one of claims 1 to 32, further comprising storing one or more of said at least one image or derivatives thereof and said classification in a database.

34. 34. The method of any one of claims 1 to 33, further comprising generating a presentation of the at least one image or a derivative thereof based at least in part on one or more of the at least one image or a derivative thereof and the classification.

35. 35. The method of claim 34, further comprising storing the presentation in a database.

36. 10. The method of claim 1, wherein the at least one image comprises multiple images acquired of the subject, the multiple images being acquired using different modalities or at different points in time.

37. 10. The method of claim 1, wherein the step of classifying comprises processing clinical health data of the subject.

38. A computer system for processing at least one image of a body part of a subject, comprising: a database configured to store the at least one image of the region of the subject's body; one or more computer processors operably coupled to the database, the one or more computer processors individually or collectively: (a) using a trained algorithm to classify the at least one image, or a derivative thereof, as indicative of cancer, as normal, equivocal, or suspicious, wherein the classifying step comprises applying an image processing algorithm to the at least one image, or a derivative thereof; (b) classifying the at least one image or a derivative thereof in (b); (i) if the at least one image or a derivative thereof is classified as suspicious as being indicative of the cancer, sending the at least one image or a derivative thereof to a first radiologist for radiological evaluation to produce a screening or diagnostic result; (ii) if the at least one image or a derivative thereof is classified as equivocal as being indicative of the cancer, sending the at least one image or a derivative thereof to a second radiologist, different from the first radiologist, for radiological evaluation to produce a screening or diagnostic result; (iii) if the at least one image or a derivative thereof is classified as normal, sending the at least one image or a derivative thereof to a third radiologist for radiological evaluation to produce a screening or diagnostic result; (b) is performed in real time or near real time with (a); (c) receiving a radiological assessment of the subject from the first radiologist, the second radiologist, or the third radiologist, wherein the radiological assessment of the subject is generated by the first radiologist, the second radiologist, or the third radiologist based at least in part on a radiological analysis of the at least one image or a derivative thereof; one or more computer processors programmed to perform A computer system comprising:

39. 39. The computer system of claim 38, wherein the at least one image or derivative thereof is a medical image.

40. 39. The computer system of claim 38, wherein the trained algorithm is configured to classify the at least one image or a derivative thereof as normal, equivocal, or suspicious with at least about 80% sensitivity.

41. 39. The computer system of claim 38, wherein the trained algorithm is configured to classify the at least one image, or a derivative thereof, as normal, equivocal, or suspicious with at least about 80% specificity.

42. 39. The computer system of claim 38, wherein the trained algorithm is configured to classify the at least one image or a derivative thereof as normal, equivocal, or suspicious with a positive predictive value of at least about 80%.

43. 39. The computer system of claim 38, wherein the trained algorithm is configured to classify the at least one image or a derivative thereof as normal, equivocal, or suspicious with a negative predictive value of at least about 80%.

44. 39. The computer system of claim 38, wherein the trained algorithm is configured to identify at least one region of the at least one image or a derivative thereof that contains or is suspected of containing abnormal tissue.

45. 39. The computer system of claim 38, wherein the cancer is breast cancer.

46. 46. The computer system of claim 45, wherein the at least one image or derivative thereof is a three-dimensional image of the subject's breast.

47. 39. The computer system of claim 38, wherein the trained algorithm is trained using at least about 100 independent training samples comprising images indicative of or suspected of being indicative of the cancer.

48. 39. The computer system of claim 38, wherein the trained algorithm is trained using a first plurality of independent training samples comprising positive images indicative of or suspected of being indicative of the cancer, and a second plurality of independent training samples comprising negative images not indicative of or not suspected of being indicative of the cancer.

49. 39. The computer system of claim 38, wherein the trained algorithm comprises a supervised machine learning algorithm.

50. 50. The computer system of claim 49, wherein the supervised machine learning algorithm comprises a deep learning algorithm, a support vector machine (SVM), a neural network, or a random forest.

51. 39. The computer system of claim 38, wherein the one or more computer processors, individually or collectively, are further programmed to monitor the subject, the monitoring step comprising evaluating images of the region of the body of the subject at a plurality of time points, the evaluating step being based at least in part on the classification of the at least one image, or a derivative thereof, at each of the plurality of time points as normal, equivocal, or suspicious as indicative of cancer.

52. 52. The computer system of claim 51 , wherein differences in the evaluation of the images of the body of the subject at the multiple time points indicate one or more clinical indicators selected from the group including: (i) a diagnosis of the subject, (ii) a prognosis of the subject, and (iii) the effectiveness or ineffectiveness of a course of treatment for the subject.

53. 53. The computer system of any one of claims 38 to 52, wherein the screening results for the subject are produced in the same clinic visit as the step of obtaining the at least one image.

54. 53. The computer system of any one of claims 38 to 52, wherein the first radiologist is located at an on-site clinic and the at least one image is acquired at the on-site clinic.

55. 53. The computer system of any one of claims 38 to 52, wherein the second radiologist is a radiologist trained to classify the at least one image or a derivative thereof as normal or suspicious with greater accuracy than the trained algorithm.

56. 53. The computer system of any one of claims 38 to 52, wherein the third radiologist is located remotely from an on-site clinic and the at least one image is acquired at the on-site clinic.

57. 53. The computer system of any one of claims 38 to 52, wherein the third radiologist performs the radiological evaluation of the at least one image or a derivative thereof from a batch comprising a plurality of images, the batch being selected to improve efficiency of the radiological evaluation.

58. 58. The computer system of any one of claims 38 to 57, wherein the one or more computer processors are individually or collectively programmed to further obtain a diagnostic result for the subject from a diagnostic procedure performed on the subject based at least in part on the screening results.

59. 59. The computer system of claim 58, wherein the diagnosis for the subject is obtained at the same clinic visit as the step of obtaining the at least one image.

60. 60. The computer system of claim 59, wherein the diagnosis for the subject is obtained within about one hour of the step of acquiring the at least one image.

61. 61. The computer system of any one of claims 38 to 60, wherein the at least one image or a derivative thereof is sent to the first radiologist, the second radiologist, or the third radiologist based at least in part on additional characteristics of the part of the body of the subject.

62. 62. The computer system of claim 61, wherein the additional characteristics include anatomy, tissue characteristics, presence of foreign body, type of finding, pathology, or a combination thereof.

63. 63. The computer system of any one of claims 38 to 62, wherein the at least one image or a derivative thereof is sent to the first radiologist, the second radiologist, or the third radiologist based at least in part on additional characteristics of the first radiologist, the second radiologist, or the third radiologist.

64. 64. The computer system of any one of claims 38 to 63, wherein the one or more computer processors are further programmed to generate an alert based at least in part on the step of sending the at least one image or a derivative thereof to the first radiologist, or the step of sending the at least one image or a derivative thereof to the second radiologist, or the step of sending the at least one image or a derivative thereof to the third radiologist, individually or collectively.

65. 65. The computer system of claim 64, wherein the one or more computer processors, individually or collectively, are further programmed to transmit the alert to the subject or the subject's clinical caregiver.

66. 66. The computer system of claim 65, wherein the one or more computer processors, individually or collectively, are further programmed to transmit the alert to the subject through a patient mobile application.

67. 65. The computer system of claim 64, wherein the alert is generated in real time or near real time with (a).

68. 39. The computer system of claim 38, wherein applying the image processing algorithm comprises identifying regions of interest within the at least one image or a derivative thereof, and labeling the regions of interest to produce at least one labeled image.

69. 69. The computer system of claim 68, wherein the one or more computer processors are individually or collectively programmed to further store the at least one labeled image in the database.

70. 70. A computer system according to any one of claims 38 to 69, wherein the one or more computer processors are individually or collectively programmed to further store one or more of the at least one image or derivative thereof and the classification in the database.

71. 71. The computer system of any one of claims 38 to 70, wherein the one or more computer processors are further programmed to generate a presentation of the at least one image or a derivative thereof based at least in part on one or more of the at least one image and the classification, individually or collectively.

72. 72. The computer system of claim 71, wherein the one or more computer processors are individually or collectively programmed to further store the presentation in the database.

73. 40. The system of claim 38, further comprising an electronic display operably coupled to the one or more computer processors, the electronic display comprising a graphical user interface configured to display the recommendations.

74. 39. The method of claim 38, wherein the at least one image comprises multiple images acquired of the subject, the multiple images being acquired using different modalities or at different points in time.

75. 39. The method of claim 38, wherein the step of classifying comprises processing clinical health data of the subject.

76. 1. A non-transitory computer-readable medium containing machine-executable code that, when executed by one or more computer processors, implements a method for processing at least one image of a body part of a subject, the method comprising: (a) acquiring the at least one image of the region of the subject's body; (b) using a trained algorithm to classify the at least one image, or a derivative thereof, as being indicative of cancer, as normal, equivocal, or suspicious, wherein the classifying step comprises applying an image processing algorithm to the at least one image, or a derivative thereof; (c) classifying the at least one image or a derivative thereof in (b); (i) if the at least one image or a derivative thereof is classified as suspicious as being indicative of the cancer, sending the at least one image or a derivative thereof to a first radiologist for radiological evaluation to produce a screening or diagnostic result; (ii) if the at least one image or a derivative thereof is classified as equivocal as being indicative of the cancer, sending the at least one image or a derivative thereof to a second radiologist, different from the first radiologist, for radiological evaluation to produce a screening or diagnostic result; (iii) if the at least one image or a derivative thereof is classified as normal, sending the at least one image or a derivative thereof to a third radiologist for radiological evaluation to produce a screening or diagnostic result; wherein (c) is performed in real time or near real time with (b); (d) receiving a radiological assessment of the subject from the first radiologist, the second radiologist, or the third radiologist, wherein the radiological assessment of the subject is generated by the first radiologist, the second radiologist, or the third radiologist based at least in part on a radiological analysis of the at least one image or a derivative thereof; 1. A non-transitory computer-readable medium, comprising:

Images