Systems and methods for annotating pathology images to predict patient outcomes

Abstract

A method, system, apparatus, and computer program for predicting a clinical outcome for a patient. In one aspect, the method includes obtaining a pathology image associated with the patient, processing the pathology image, including determining a region of interest in the pathology image and dividing the pathology image into a plurality of tiles having uniform sizes, annotating the pathology image and providing the processed pathology image as an input to a machine learning model configured to predict a clinical outcome for the patient based on the annotated pathology image, obtaining an annotated pathology image associated with the patient, including annotations of different tissue classes including tumor regression, as an output of the machine learning model, and determining a metric indicative of the clinical outcome for the patient based on the annotated pathology image.

Description

[Technical field]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This application claims priority to U.S. Provisional Patent Application No. 63 / 313,548, filed February 24, 2022, which is incorporated by reference in its entirety.

[0002] The present disclosure is directed to prediction of patient outcomes, such as responsiveness to a treatment regimen, based on pathology image analysis. Summary of the Invention [Means for solving the problem]

[0003] According to one innovative aspect of the present disclosure, a computer-implemented method for predicting a clinical outcome for a patient includes obtaining a pathology image associated with the patient; processing the pathology image, the pathology image including determining a region of interest in the pathology image and dividing the pathology image into a plurality of tiles having uniform sizes; providing the processed pathology image as an input to a machine learning model configured to annotate the pathology image; obtaining an annotated pathology image associated with the patient as an output of the machine learning model, the annotated pathology image including annotations of different tissue classes including tumor regression; and determining a metric indicative of the clinical outcome of the patient based on the annotated pathology image.

[0004] Other variations include corresponding systems and computer programs for performing the operations of the methods defined by instructions encoded on a computer readable storage device.

[0005] These and other variations may optionally include one or more of the following features: For example, in some embodiments, patient clinical outcomes include responsiveness to one or more treatment regimens, survival, disease recurrence, and disease progression.

[0006] In some embodiments, the pathology images include histopathology slide images.

[0007] In some embodiments, the machine learning model is trained on a plurality of training data items, each training data item comprising an annotated pathology image having a tissue class.

[0008] In some embodiments, rendering data is generated that, when rendered by a user device, causes the user device to display a user interface displaying annotated pathology images associated with the patient and a predicted clinical outcome for the patient.

[0009] In some embodiments, the user interface includes a user-selectable element for uploading a pathology image for applying the machine learning model.

[0010] In some embodiments, the different tissue classes include tumor, necrosis, areas of tumor regression, and normal background, where normal background represents non-tumor tissue.

[0011] In some embodiments, the metric indicative of a patient's clinical outcome comprises a residual tumor percentage, which is calculated based on the areas of different tissue classes in the pathology images.

[0012] In some embodiments, the metrics indicative of a patient's clinical outcome include the rate of regression and necrosis in pathology images.

[0013] In some embodiments, the step of processing the pathology image further comprises normalizing a color of the pathology image based on the colors of the set of pathology images and applying an image transformation to the pathology image, the image transformation including rotation, shift, flip, zoom, affine transformation, and brightness and contrast adjustment.

[0014] These and other aspects of the present disclosure are described in more detail in the following detailed description, with reference to the accompanying drawings. [Brief description of the drawings]

[0015] [Figure 1]FIG. 1 is a block diagram of an example system for predicting patient outcomes based on annotated pathology images. [Diagram 2] FIG. 1 is a block diagram of an example system for training a machine learning model. [Diagram 3] Figure 3A shows the performance of the machine learning model. Figure 3B shows the performance of the machine learning model. [Figure 4A] 1 shows an example user interface for annotating pathology images and predicting patient outcomes. [Figure 4B] 13 shows examples of annotated pathology images and predicted patient outcomes. [Diagram 5] 1 is a flowchart of an example process of applying a machine learning model to predict patient outcomes based on annotated pathology images. [Figure 6] 1 is a flowchart of an example process for training a machine learning model to annotate pathology images for use in predicting patient outcomes. [Figure 7] FIG. 1 is an example block diagram of system components that may be used to implement a system for predicting patient outcomes based on annotated pathology images. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] According to an innovative aspect of the present disclosure, a system, method, and computer program are disclosed for predicting patient outcome(s) based on pathology images of cancer from a patient. Patient outcomes (also referred to as patient clinical outcomes) may include responsiveness to treatment (including pharmaceuticals, immunotherapy, and chemotherapy), survival, disease recurrence, and / or disease progression, etc. In some embodiments, the present disclosure can be used to annotate different tissue classes in a pathology image, such as tumor, necrosis, tumor regression, etc. However, the present disclosure need not be so limited and can be applied to non-tumor specimens and frozen specimens. Furthermore, the present disclosure need not be limited to human specimens. Instead, for example, specimens analyzed by the present disclosure can include human or animal specimens. Pathology images (e.g., images scanned by a slide imaging scanner) can include histopathological (tissue) images.

[0017] The systems and methods of the present disclosure solve certain problems known in the field of predicting patient outcomes, such as responsiveness to treatment regimens, e.g., anti-PD-1 immunotherapy. Current approaches include radiological assessment of response to treatment, which is inadequate for accurate prediction of patient outcomes. Interpretation and scoring of slides can be laborious and does not allow for fast enough data processing to provide useful insights for a significant number of patients in a timely manner. Furthermore, these current systems, such as visual scoring of slide(s) under a microscope or digital slide(s) by a pathologist, lend themselves to subjective predictions between interpreters, motivating standardization of assessment of pathological response to treatment regimens. To solve this problem of developing standardized pathological response criteria, the present disclosure computationally annotates different tissue classes or cells in pathology images and efficiently calculates metrics that predict patient outcomes. Metrics, such as percent residual tumor (also referred to as %RVT), percent necrosis, and percent regression, can be used to predict patient outcomes, including responsiveness to immunotherapy. Regression may refer to the area occupied by the tumor after the tumor has reduced in area. Areas of regression are also distinguished from non-tumor tissue that is no longer tumor but is "normal" background. The present disclosure also describes a user interface that allows a user to apply the system to view annotated pathology images and predicted patient outcomes.

[0018] 1 is a block diagram of an example system 100 for predicting patient outcomes based on annotated pathology images. The system 100 includes an input device 140, a network 120, and a computer 130. The computer 130 may include an input processing engine 104, a machine learning model 108, and a patient outcome prediction engine 112. In some embodiments, the computer 130 may be a server. In this disclosure, an "engine" may include one or more software modules, one or more hardware modules, or a combination of one or more software modules and one or more hardware modules.

[0019] The input device 140 may include a device configured to generate a pathology image 102 of one or more organs of the patient's body, a device configured to provide a generated pathology image 102 of one or more organs of the patient's body, or a combination thereof. As an example, the input device 140 may include a slide scanner 140a configured to scan one or more pathology slides (e.g., melanoma resection, tissue biopsy of one or more organs of the patient's body). The pathology slides may be tissue-level slides or cell-level slides. In some embodiments, the slide scanner may be a whole slide scanner. In some embodiments, the slide scanner 140a may generate the pathology image 102 and transmit the pathology image 102 to another device, such as the computer 130, via the network 120. Alternatively, the slide scanner 140a may generate a pathology image 102 accessible to one or more other input devices 140, such as a computer (e.g., desktop, laptop, tablet, etc.), a smartphone, or a server. In such a case, one or more other input devices may access the pathology images 102 generated by the slide scanner 140a and transmit the accessed pathology images to the computer 130 via the network 120.

[0020] Network 120 may include one or more of a wired Ethernet network, a wired optical network, a wireless WWiFi network, a LAN, a WAN, a Bluetooth network, a cellular network, the Internet, or any combination thereof.

[0021] Although the examples of the present disclosure described herein refer to pathology images 102 and slide scanners 140a, the present disclosure is not limited thereto. Instead, the present disclosure may include machine learning models (e.g., one or more neural networks) trained to predict patient outcomes based on any type of image. Such images may include two-dimensional images, three-dimensional images, image models generated based on other images, etc.

[0022] The computer 130 is configured to acquire one or more pathology images 102 generated by an input device 140, such as a slide scanner 140a. In some embodiments, the one or more pathology images may be images received over the network 120. In other embodiments, the computer 130 may access pathology images stored in a database 132 accessible to the computer 130. The database 132, which may be a local database or a cloud-based database, may include pathology images, a patient identifier, a patient history including current and past treatment plans, and predicted patient outcomes. The pathology images in the database may be raw or annotated pathology images. In some embodiments, the patient identifier may be encrypted. However, the present disclosure is not limited to the use of encrypted patient identifiers.

[0023] The computer 130 can provide acquired pathology images, such as pathology image 102, to the input processing engine 104. In some implementations, the computer 130 can provide the acquired pathology images for input to the input processing engine 104 in the same format in which the pathology image 102 was received. For example, the computer 130 can receive a two-dimensional image and provide the two-dimensional image as input to the input processing engine 104. In some implementations, the computer 130 can generate a three-dimensional pathology image from the two-dimensional pathology image and provide the three-dimensional pathology image for input to the input processing engine 104.

[0024] The input processing engine 104 is configured to receive the pathology image 102 and generate an advanced pathology image 106 for input to the machine learning model 108. In some implementations, the pathology image 102 is an image of approximately 50k x 50k pixels. With reference to FIG. 1 , the input processing engine 104 can perform one or more steps such as tissue filtering (e.g., using the tissue filter engine 104a), tile extraction (e.g., using the tile extraction engine 104b), normalization (e.g., using the normalization engine 104c), and dilation (e.g., using the dilation engine 104d). For example, the tissue filter engine 104a calculates a tissue mask(s) and extracts the portion of the pathology image 102 that corresponds to tissue. The tile extraction engine 104b divides the pathology image 102 before or after the tissue filtering into a set of tiles. In some implementations, each tile is an image of 96 x 96 pixels. The tile extraction engine 104b can determine the optimal size of the tiles. The normalization engine 104c normalizes the color of the pathology image 102 based on, for example, the average and distribution of colors across multiple pathology images. The augmentation engine 104d applies image transformations (also referred to as noise), such as rotation, shift, flip, zoom, affine transformation, brightness and / or contrast adjustment, to the pathology image 102. This image augmentation prevents overfitting of the machine learning model 108. The processed pathology image 106 generated by the input processing engine 104 may include a data structure representing a pathology image. The processed pathology image 106 output by the input processing engine 104 may be provided as an input to the machine learning model 108. In the example of FIG. 1, the processed pathology image 106 is a data structure representing the (e.g., two-dimensional) pathology image 102 processed by the engines 104a-104d.

[0025] The machine learning model 108 is configured to receive the processed pathology image 106 and generate the annotated pathology image 110. The machine learning model 108 is a model output from training a machine, e.g., a computer 130, to annotate the processed pathology image 106, the training being described in more detail below with reference to FIG. 2. The annotated pathology image 110 includes annotations for each tissue class. For example, a particular region of the pathology image is annotated as (predicted) tumor. Similarly, a particular region of the pathology image is annotated as necrosis. In some implementations, the annotations are graphic annotations, e.g., borders and / or regions. In some implementations, each region (e.g., tiles of uniform size) can be assigned a single tissue class, e.g., a particular tile cannot be annotated as both normal and tumor. The annotations are based on class membership probabilities predicted by the machine learning model 108. For example, a particular tile predicted to be 70% tumor, 10% regression, and 20% normal is annotated with the label tumor. In some implementations, each tile can have multiple labels based on probability thresholds for each membership. That is, an annotation for a particular tile that is predicted to be 40% tumor, 40% regression, and 20% normal (e.g., using a probability threshold of 33%) will be both tumor and regression. The annotated pathology images output from the machine learning model 108 may be provided as input to the patient outcome prediction engine 112. In some embodiments, a summary report is provided as an output of the machine learning model 108, the summary report including the area of ​​each of the tissue classes. The summary report may be provided in addition to or instead of the annotated pathology images. In some embodiments, prediction of patient outcomes is also performed by the machine learning model 108.

[0026] The patient outcome prediction engine 112 is configured to receive the annotated pathology images 110, the summary report, or both, and generate a predicted patient outcome 114. Patient outcomes (also referred to as clinical outcomes) include responsiveness to treatment regimens including pharmaceuticals, immunotherapy, and chemotherapy, patient survival, disease recurrence, and disease progression. In some embodiments, the patient outcome prediction engine 112 determines a metric indicative of the patient's clinical outcome. For example, in some embodiments, the metric may be a residual tumor fraction (RVT), which may be calculated based on the area of ​​each of the different tissue classes.

[0027] In some embodiments, for example, the RVT rate is:

number

[0028] In some embodiments, computer 130 may generate rendering data that, when rendered by a device having a display, such as output device 150, may cause the device to output data including the annotated pathology image 110 and the predicted patient outcome 114. Such rendering data may be transmitted by computer 130 over network 120 to output device 150 and processed to generate output data by output device 150, such as monitor 150a or smartphone 150b. In some embodiments, output device 150 may be coupled to computer 130. In such a case, the rendered data may be processed by computer 130 to cause computer 130 to output data including the annotated pathology image 110 and the predicted patient outcome 114.

[0029] FIG. 2 is a block diagram of an example of a system, machine learning training engine 200, for training the machine learning model 108. For example, the machine learning model may be trained to distinguish between tumors and non-tumors in pathology images. Such training may be accomplished by obtaining training data 202 and processing the training data 202. The training data 202 includes one or more pathology images for each tissue or cell class, such as normal, involution, tumor, and necrosis. However, the present disclosure need not be limited to these tissue types and may be applied to any tissue or cell type. In some embodiments, the training data 202 is processed using the input processing engine 104 as described with reference to FIG. 1. One or more steps, such as tissue filtering, tile extraction, normalization, and augmentation (discussed above), may be applied to the training data.

[0030] In some implementations, the machine learning training engine 200 includes a transfer learning engine 204. The transfer learning engine 204 processes the post-acquired training data 202 labeled with different tissue classes. The transfer learning engine 204 may include a convolutional layer 1606 and a fully connected layer 208. The transfer learning engine 204 trains the machine learning model 108 by performing the following steps: (i) freezing the convolutional layers, (ii) learning the weights of the fully connected layers, (iii) unfreezing the convolutional layers, and (iv) learning the weights of all layers, where the weights represent the learnable parameters of the machine learning model 108. In some implementations, the transfer learning engine 204 uses a DenseNet 169 that includes the following layers: convolutional + rectified linear unit (ReLU), max pooling (e.g., for downsampling), fully connected + ReLU, and softmax (e.g., to convert any real value into a probability). The convolutional layer 206 includes a convolutional + ReLU and max pooling layer. The fully connected layer 208 includes a fully connected+ReLU and a softmax layer. After training the weights through the transfer learning engine 204, the machine learning training engine 200 outputs the machine learning model 108. In some implementations, the trained machine learning model 108 can be stored in the database 132.

[0031] 3A and 3B show the performance of the machine learning model 108 in annotating pathology images. Referring to FIG. 3A, in some implementations, the machine learning training engine 200 can test the machine learning model 108 against test data. The test data can be hold-out validation data during training or independent test data. For example, FIG. 3A shows a receiver operating characteristic (ROC) curve for accurately predicting different tissue classes, such as involution, tumor, and necrosis, when compared against expert annotations. In some implementations, the prediction of different tissue classes is performed tile-by-tile for a set of tiles in the pathology image. In some implementations, the expert annotations represent the annotations by a pathologist per (digital) slide, i.e., the pathologist visual scoring. As shown in FIG. 3A, the machine learning model 108 outperforms significantly more than chance.

[0032] FIG. 3B shows the performance of the machine learning model in predicting patient outcomes. In some embodiments, the prediction of patient outcomes is based on annotated pathology images, e.g., the area of ​​each of the different tissue classes. In some embodiments, a portion of the pathology image is used to predict patient outcomes. Referring to FIG. 3B, the area under the ROC curve for accurate prediction of patient outcomes to immunotherapy is 0.98, where the patient outcomes are based on %RVT as described above. It can be seen that the predictions by the machine learning model 108 in 92 patients are highly correlated with the expert predictions, with a Pearson correlation coefficient (r) of 0.95 and a slope of the fitting line of 0.84.

[0033] Based on testing the performance of the machine learning model 108, the machine learning training engine 200 may determine that the machine learning model 108 meets predefined thresholds, e.g., regarding accuracy and computation speed. If the machine learning model 108 determines that it does not meet the predefined thresholds, the machine learning training engine 200 may adjust training parameters, such as the machine learning architecture shown in FIG. 2A, the training data 202, etc. (e.g., by obtaining more training data and / or validating labels) and retrain the machine learning model 108 based on the adjusted training parameters.

[0034] FIG. 4A illustrates an example of a user interface 400 for annotating pathology images and predicting patient outcomes. The user interface 400 is displayed on an output device 150, such as a monitor 150a or a smartphone 150b. The user interface 400, in some embodiments, may include a browse panel 402 and a case details panel 404. The browse panel 402 allows a user to browse different cases, for example, cases belonging to a particular disease or a particular patient. Upon selection of a case from the browse panel 402, the case details panel 404 displays the pathology images. For example, each column shown in FIG. 4A represents each pathology image of the case 103. In some embodiments, the case details panel 404 may display a pathology image identifier ("Slide ID"), image name, additional data, progress (e.g., whether the uploaded pathology image is annotated or not), and predicted patient outcome (e.g., "%RVT"). In some embodiments, the user can view final result categories at the slide or case level, such as "pathological response" or "pathological complete response" or other related terms associated with a medically meaningful classification of the elements detected by the system. Additionally, pending jobs can be viewed. In some embodiments, the user interface 400 includes a search panel 406 that accepts user input for a particular pathology image (e.g., a slide identifier) ​​and presents filtered results that satisfy the user input.

[0035] FIG. 4B illustrates an example of an annotated pathology image 450 and a predicted patient outcome 458. The annotated pathology image 450 is displayed in the user interface 400 in response to a user selection of an image associated with the patient. In some embodiments, the annotated pathology image 450 may include different tissue (or cell) classes. For example, predicted regression (area) 452, predicted tumor (area) 454, and predicted necrosis (area) 456 represent areas of different tissue classes in the annotated pathology image 450. The user interface 400 may display the different tissue classes with borders and / or different colors. In some embodiments, the annotated pathology image 450 includes information indicative of the predicted patient outcome 458, such as a dashboard or pop-up window summarizing the predicted patient outcome (e.g., "This patient's tumor is responding to anti-PD-1 immunotherapy"). In some embodiments, the dashboard includes a metric used to predict the patient outcome. For example, "Machine-computed residual tumor (RVT) rate = 9%". The predicted patient outcomes 458 are based on metric thresholds determined by experts and / or machine learning models 108, for example, RVT <10% for immunotherapy success.

[0036] 5 is a flow chart of an example process 500 for applying a machine learning model to predict a patient's clinical outcome. In general, the process 500 includes obtaining a pathology image associated with a patient (502), processing the pathology image, including determining a region of interest in the pathology image and dividing the pathology image into tiles (504), providing the processed pathology image as an input to a machine learning model configured to annotate the pathology image (506), obtaining an annotated pathology image associated with the patient, including annotations of different tissue classes, including tumor regression, as an output of the machine learning model (508), and determining a metric indicative of the patient's clinical outcome based on the annotated pathology image (510). Thus, the process 500 enables a machine learning model that facilitates prediction of a patient's clinical outcome based on the annotated pathology image generated by the machine learning model.

[0037] 6 is a flowchart of another example process 600 for training a machine learning model to annotate pathology images used to predict clinical outcomes of patients. In general, the process 600 includes steps of obtaining a plurality of training data items (e.g., each training data item includes pathology images annotated with tissue classes such as normal, involution, tumor, and necrosis) (602), processing each training data item, including determining a region of interest in the training data item and dividing the training data item into a plurality of tiles (604), determining a training set, a validation set, and a test set in the plurality of training data items (606), feeding the training set and the validation set to a plurality of fully connected layers to obtain weights for a plurality of fully connected layers (608), feeding the training set and the validation set to a plurality of convolutional layers and a plurality of fully connected layers to obtain weights for a plurality of convolutional layers and a plurality of fully connected layers (610), generating a trained model based on the obtained weights (612), and feeding the test set to the trained model (614).

[0038] FIG. 7 is an example block diagram of system components that can be used to implement a system for predicting patient outcomes based on annotated pathology images.

[0039] Computing device 700 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Computing device 750 is intended to represent various forms of mobile devices, such as personal digital assistants, mobile phones, smart phones, and other similar computing devices. Additionally, computing device 700 or 750 may include a Universal Serial Bus (USB) flash drive. A USB flash drive may store an operating system and other applications. A USB flash drive may include input / output components, such as a wireless transmitter or a USB connector that can be inserted into a USB port of another computing device. The illustrated components, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit the embodiments of the invention(s) described and / or claimed herein.

[0040] The computing device 700 includes a processor 702, a memory 704, a storage device 706, a high-speed controller 708 that connects to the memory 704 and a high-speed expansion port 710, and a low-speed controller 712 that connects to a low-speed bus 714 and the storage device 706. Each of the components 702, 704, 706, 708, 710, and 712 may be interconnected using various buses and mounted on a common motherboard or in other manners as appropriate. The processor 702 may process instructions that execute within the computing device 700, including instructions stored in the memory 704 or the storage device 706 for displaying graphical information of a GUI on an external input / output device, such as a display 716 coupled to the high-speed controller 708. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and memory types, as appropriate. Multiple computing devices 700 may also be connected such that each device provides a portion of the operations required, for example, as a server bank, a group of blade servers, or a multiprocessor system.

[0041] The memory 704 stores information within the computing device 700. In one implementation, the memory 704 is one or more volatile memory units. In another implementation, the memory 704 is one or more non-volatile memory units. The memory 704 may also be another form of computer readable medium, such as a magnetic or optical disk.

[0042] The storage device 706 can provide mass storage for the computing device 700. In one embodiment, the storage device 706 can be or contain a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or an array of devices including a tape device, a flash memory or other similar semiconductor memory device, or a storage area network or other configuration of devices. The computer program product can be tangibly embodied on an information carrier. The computer program product can also include instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as the memory 704, the storage device 706, or a memory on the processor 702.

[0043] The high-speed controller 708 manages the operations of the computing device 700 that consume large bandwidth, while the low-speed controller 712 manages the operations that consume less bandwidth. This allocation of functionality is merely exemplary. In one embodiment, the high-speed controller 708 is coupled to the memory 704, the display 716, e.g., via a graphics processor or accelerator, and to a high-speed expansion port 710 that can accept various expansion cards (not shown). In this embodiment, the low-speed controller 712 is coupled to the storage device 706 and the low-speed bus 714. The low-speed expansion port, which can include various communication ports, e.g., USB, Bluetooth, Ethernet, wireless Ethernet, can be coupled, e.g., via a network adapter, to one or more input / output devices, such as a keyboard, a pointing device, a microphone / speaker pair, a scanner, or a networking device, such as a switch or router. The computing device 700 can be implemented in several different forms as shown. For example, the computing device 700 can be implemented as a standard server 720, or multiple times in a cluster of such servers. Computing device 700 may also be implemented as part of a rack server system 724. Additionally, computing device 700 may be implemented in a personal computer, such as laptop computer 722. Alternatively, components from computing device 700 may be combined with other components in a mobile device (not shown), such as device 750. Each such device may include one or more of computing devices 700, 750, and the entire system may be composed of multiple computing devices 700, 750 in communication with each other.

[0044] The computing device 700 may be implemented in a number of different forms as shown. For example, the computing device 700 may be implemented as a standard server 720, or multiple times in a cluster of such servers. The computing device 700 may also be implemented as part of a rack server system 724. Additionally, the computing device 700 may be implemented in a personal computer, such as a laptop computer 722. Alternatively, components from the computing device 700 may be combined with other components in a mobile device (not shown), such as device 750. Each such device may include one or more of the computing devices 700, 750, and the entire system may be comprised of multiple computing devices 700, 750 in communication with each other.

[0045] Computing device 750 includes a processor 752, memory 764, input / output devices such as a display 754, a communication interface 766, and a transceiver 768, among other components. Device 750 may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of components 750, 752, 764, 754, 766, and 768 are interconnected using various buses, and some of the components may be mounted on a common motherboard or in other suitable manner.

[0046] The processor 752 can execute instructions within the computing device 750, including instructions stored in the memory 764. The processor may be implemented as a chipset of chips including separate analog and digital processors. Additionally, the processor may be implemented using any of a number of architectures. For example, the processor may be a CISC (Complex Instruction Set Computer) processor, a RISC (Reduced Instruction Set Computer) processor, or a MISC (Minimum Instruction Set Computer) processor. The processor may enable coordination of other components of the device 750, such as, for example, control of the user interface, applications executed by the device 750, and wireless communications by the device 750.

[0047] The processor 752 can communicate with a user via a control interface 758 and a display interface 756 coupled to a display 754. The display 754 can be, for example, a TFT (thin film transistor liquid crystal display) display or an OLED (organic light emitting diode) display, or other suitable display technology. The display interface 756 can include suitable circuitry for driving the display 754 to present graphical and other information to a user. The control interface 758 can receive commands from a user and convert them for sending to the processor 752. Additionally, an external interface 762 can be provided in communication with the processor 752 to enable proximity communication between the device 750 and other devices. The external interface 762 can be, for example, in some implementations for wired communication and in some implementations for wireless communication, and multiple interfaces can be used.

[0048] The memory 764 stores information within the computing device 750. The memory 764 may be implemented as one or more of one or more computer readable media, one or more volatile memory units, or one or more non-volatile memory units. An expansion memory 774 may also be provided and connected to the device 750 via an expansion interface 772, which may include, for example, a SIMM (single in-line memory module) card interface. The expansion memory 774 may provide extra storage space for the device 750 or may store applications or other information for the device 750. In particular, the expansion memory 774 may include instructions that perform or supplement the processes described above, and may also include secure information. Thus, for example, the expansion memory 774 may be provided as a security module for the device 750 and may be programmed with instructions that enable secure use of the device 750. Additionally, secure applications may be provided via a SIMM card along with additional information, such as placing identification information on the SIMM card in an unhackable manner.

[0049] The memory may include, for example, flash memory and / or NVRAM memory, as described below. In one embodiment, a computer program product may be tangibly embodied on an information carrier. The computer program product may also include instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as memory 764, expansion memory 774, or memory on processor 752, which may be received, for example, via transceiver 768 or external interface 762.

[0050] The device 750 may communicate wirelessly via a communication interface 766, which may include digital signal processing circuitry as required. The communication interface 766 may enable communication in various modes or protocols, such as GSM calling, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others. Such communication may take place, for example, via a (radio frequency) transceiver 768. Additionally, short-range communication may take place using Bluetooth, Wi-Fi, or other such transceivers (not shown), etc. Additionally, a GPS (Global Positioning System) receiver module 770 may provide additional navigation-related and location-related wireless data to the device 750, which may be used as appropriate by applications executing on the device 750.

[0051] Device 750 can also communicate audibly using an audio codec 760 that can receive verbal information from a user and convert it into usable digital information. Audio codec 760 can also generate audible sounds for the user, such as through a speaker in a handset of device 750. Such sounds can include sounds from a voice call, can include recorded sounds such as voice messages, music files, etc., and can also include sounds generated by applications running on device 750.

[0052] Computing device 750 may be implemented in a number of different forms as shown. For example, computing device 750 may be implemented as a mobile phone 780. Computing device 750 may also be implemented as part of a smartphone 782, personal digital assistant, or other similar mobile device.

[0053] Various implementations of the systems and methods described herein may be realized in digital electronic circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations of such implementations. These various implementations may include implementation in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor, which may be special purpose or general purpose, and which may be coupled to receive data and instructions from and send data and instructions to the storage system, at least one input device, and at least one output device.

[0054] These computer programs (also known as programs, software, software applications, or code) contain machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device used to provide machine instructions and / or data to a programmable processor, such as magnetic disks, optical disks, memories, programmable logic devices (PLDs), and include machine-readable media that receive machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0055] To allow for interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, for allowing the user to provide input to the computer. Other types of devices can also be used to allow for interaction with the user. For example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0056] The systems and techniques described herein may be implemented in a computing system that includes a back-end component, e.g., a data server, or includes a middleware component, e.g., an application server, or includes a front-end component, e.g., a client computer having a graphical user interface or a web browser that allows a user to interact with an embodiment of the systems and techniques described herein, or includes any combination of such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0057] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0058] Several embodiments have been described. However, it will be understood that various modifications can be made without departing from the spirit and scope of the invention. Additionally, the illustrated logic flows do not require the particular order or sequence shown to achieve the desired results. Additionally, other steps can be provided or steps can be eliminated from the described flows, and other components can be added or removed from the described systems. Additionally, other embodiments are within the scope of the following claims.

Claims

1. 1. A computer-implemented method for predicting a clinical outcome for a patient, comprising: obtaining a pathology image associated with the patient; processing the pathology image; determining a region of interest in the pathology image; and Dividing the pathology image into a plurality of tiles, each of the tiles having a uniform size. and providing the processed pathology image as an input to a machine learning model configured to annotate the pathology image; obtaining, as an output of the machine learning model, annotated pathology images associated with the patient, the annotated pathology images including annotations of different tissue classes including tumor regression; determining a metric indicative of the clinical outcome of the patient based on the annotated pathology images; 4. A computer-implemented method comprising:

2. 10. The computer-implemented method of claim 1, wherein the clinical outcomes of the patient include responsiveness to one or more treatment regimens, survival, disease recurrence, and disease progression.

3. 10. The computer-implemented method of claim 1, wherein the pathology images include histopathology slide images.

4. 10. The computer-implemented method of claim 1 further comprising: The computer-implemented method, wherein the machine learning model is trained on a plurality of training data items, each training data item comprising an annotated pathology image having a tissue class.

5. 2. The computer-implemented method of claim 1, generating rendering data that, when rendered by a user device, causes the user device to display a user interface that displays the annotated pathology image associated with the patient and the predicted clinical outcome for the patient. The computer-implemented method further comprising:

6. 6. The computer-implemented method of claim 5, wherein the user interface includes a user-selectable element for uploading the pathology image for applying the machine learning model.

7. 2. The computer-implemented method of claim 1, wherein the different tissue classes include tumor, necrosis, tumor regression areas, and normal background, the normal background being indicative of non-tumor tissue.

8. 2. The computer-implemented method of claim 1, wherein the metric indicative of the clinical outcome of the patient comprises a residual tumor rate, the residual tumor rate being calculated based on areas of the different tissue classes in the pathology images.

9. 2. The computer-implemented method of claim 1, wherein the metric indicative of the clinical outcome of the patient comprises a rate of regression and a rate of necrosis in the pathology images.

10. 2. The computer-implemented method of claim 1, wherein the step of processing the pathology image comprises: normalizing the color of the pathology image based on the colors of a set of pathology images; and applying image transformations to the pathology image, including rotation, shift, flip, zoom, affine transformation, and brightness and contrast adjustment; The computer-implemented method further comprising:

11. A system comprising one or more computers and one or more storage devices storing instructions operable, when executed by the one or more computers, to cause the one or more computers to perform a method according to any one of claims 1 to 10.

12. A computer storage medium encoded with a computer program comprising instructions operable, when executed by a data processing apparatus, to cause the data processing apparatus to perform a method according to any one of claims 1 to 10.

Images