System and method for annotating pathological images to predict patient outcome

Abstract

Methods, systems, devices, and computer programs for predicting a clinical outcome of a patient. In one aspect, a method includes obtaining a pathological image associated with a patient; processing the pathological image, including: determining a region of interest of the pathological image, and segmenting the pathological image into a plurality of tiles, where the plurality of tiles have a uniform size; providing the processed pathological image as input to a machine learning model configured to annotate the pathological image and predict a clinical outcome of the patient based on the annotated pathological image; obtaining an annotated pathological image associated with the patient as an output of the machine learning model, wherein the annotated pathological image includes annotations for different categories of tissue including tumor regression; and determining an indicator indicative of a clinical outcome of the patient based on the annotated pathological image.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 313,548, filed on February 24, 2022, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present disclosure relates to predicting patient outcomes, such as response to treatment regimens, based on pathology image analysis. Overview

[0003] According to one innovative aspect of the present disclosure, a computer-implemented method for predicting a clinical outcome of a patient comprises: obtaining a pathology image associated with the patient; processing the pathology image, comprising: determining a region of interest of the pathology image, and segmenting the pathology image into a plurality of tiles, wherein the plurality of tiles have uniform sizes; providing the processed pathology image as input to a machine learning model configured to annotate the pathology image; obtaining an annotated pathology image associated with the patient as an output of the machine learning model, wherein the annotated pathology image comprises annotations for different categories of tissue including tumor regression; and determining a metric indicating the clinical outcome of the patient based on the annotated pathology image.

[0004] Other versions include corresponding systems and computer programs to perform the actions of the methods defined by instructions encoded on computer-readable storage devices.

[0005] These and other versions may optionally include one or more of the following features. For example, in some embodiments, the patient's clinical outcome includes response to one or more treatment regimens, survival rate, disease recurrence, and disease progression.

[0006] In some embodiments, the pathology image comprises a histopathology section image.

[0007] In some embodiments, the machine learning model has been trained on a plurality of training data items. Each training data item includes annotated pathology images with tissue classes.

[0008] In some embodiments, the rendering data, when rendered by a user device, is generated such that the user device displays a user interface that displays an annotated pathology image associated with a patient and a predicted clinical outcome for the patient.

[0009] In some embodiments, the user interface includes a user-selectable element for uploading a pathology image for application of a machine learning model.

[0010] In some embodiments, the different categories of tissue include tumor, necrosis, tumor regression area, and normal background. Normal background indicates uninvolved tissue.

[0011] In some embodiments, the index indicating the clinical outcome of the patient includes percent residual viable tumor, which is calculated based on the area of ​​different types of tissue in the pathological image.

[0012] In some embodiments, the indicators indicative of the patient's clinical outcome include percent regression and percent necrosis in a pathology image.

[0013] In some embodiments, processing the pathology image further comprises: normalizing the color of the pathology image based on the color of a group of pathology images; and applying an image transformation to the pathology image. The image transformation comprises rotation, shift, flip, scaling, affine transformation, and adjusting brightness and contrast.

[0014] These and other aspects of the disclosure are discussed in more detail in the following detailed description with reference to the accompanying figures. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a block diagram of an example of a system for predicting patient outcomes based on annotated pathology images.

[0016] Figure 2 is a block diagram of an example of a system for training a machine learning model.

[0017] Figure 3A-3B The performance of the machine learning model is shown.

[0018] Figure 4A An example user interface for annotating pathology images and predicting patient outcomes is shown.

[0019] Figure 4B Examples of annotated pathology images and predicted patient outcomes are shown.

[0020] Figure 5 is a flow chart of an example of a process for applying a machine learning model to predict patient outcomes based on annotated pathology images.

[0021] Figure 6 is a flow chart of an example of a process for training a machine learning model to annotate pathology images for predicting patient outcomes.

[0022] Figure 7 is an example of a block diagram of system components that may be used to implement a system for predicting patient outcomes based on annotated pathology images. Detailed Description

[0023] According to an innovative aspect of the present disclosure, systems, methods, and computer programs for predicting patient outcomes based on pathological images of cancer from patients are disclosed. Patient outcomes (also referred to as clinical outcomes for patients) may include responses to treatments (including drug products, immunotherapy, and chemotherapy), survival rates, disease recurrence, and / or disease progression, among others. In some embodiments, the present disclosure may be used to annotate different tissue categories in pathological images, such as tumors, necrosis, and tumor regression. However, the present disclosure need not be so limited, but may be applied to non-tumor samples as well as frozen samples. In addition, the present disclosure need not be limited to human samples. On the contrary, for example, samples analyzed by the present disclosure may include human samples or animal samples. Pathological images (e.g., images scanned by a slice imaging scanner) may include histopathology (tissue) images.

[0024] The systems and methods of the present disclosure solve specific problems known in the art related to predicting patient outcomes, such as responses to treatment regimens (e.g., anti-PD-1 based immunotherapy). Current methods include radiological assessments of treatment response, however, these are not sufficient to correctly predict patient outcomes. Interpretation and scoring of slides can be tedious, and the speed of processing data may not be sufficient to provide useful insights for a meaningful number of patients in a timely manner. In addition, these current systems (such as pathologists visually scoring slides by viewing slides under a microscope or digitizing slides) lead to rather subjective predictions by the interpreter, which motivates the need for standardized assessments of pathological responses to treatment regimens. To address this problem of developing standardized pathological response criteria, the present disclosure computationally annotates different classes of tissues or cells in pathology images and efficiently calculates indicators that predict patient outcomes. Indicators (e.g., residual viable tumor percentage (also known as %RVT), necrosis, and regression) can be used to predict patient outcomes, including responses to immunotherapy. Regression can refer to an area once occupied by a tumor after the tumor area has decreased. The regressed area is no longer a tumor, but is also distinguished from the "normal" background, uninvolved tissue. The present disclosure also describes a user interface that allows a user to apply the system and view annotated pathology images and predicted patient outcomes.

[0025] Figure 11 is a block diagram of an example of a system 100 for predicting patient outcomes based on annotated pathology images. The system 100 includes an input device 140, a network 120, and a computer 130. The computer 130 may include an input processing engine 104, a machine learning model 108, and a patient outcome prediction engine 112. In some embodiments, the computer 130 may be a server. For purposes of this disclosure, an "engine" may include one or more software modules, one or more hardware modules, or a combination of one or more software modules and one or more hardware modules.

[0026] The input device 140 may include any device configured to generate a pathology image 102 of one or more organs of a patient's body, a device configured to provide a generated pathology image 102 of one or more organs of a patient's body, or any combination thereof. As an example, the input device 140 may include a slide scanner 140a configured to scan one or more pathology slides (e.g., melanoma resection, tissue biopsy of one or more organs of a patient's body). The pathology slide may be a tissue-level slide or a cell-level slide. In some embodiments, the slide scanner may be a whole slide scanner. In some embodiments, the slide scanner 140a may generate a pathology image 102 and transmit the pathology image 102 to another device, such as a computer 130, via a network 120. Alternatively, the slide scanner 140a may generate a pathology image 102 that may be accessed by one or more other input devices 140, such as a computer (e.g., a desktop computer, a laptop computer, a tablet computer, etc.), a smart phone, or a server. In this case, one or more other input devices may access the pathology image 102 generated by the slide scanner 140 a and transmit the accessed pathology image to the computer 130 via the network 120 .

[0027] The network 120 may include one or more of: a wired Ethernet network, a wired optical network, a wireless WiFi network, a LAN, a WAN, a Bluetooth network, a cellular network, the Internet, or any combination thereof.

[0028] Although the examples of the present disclosure described herein relate to pathology images 102 and slide scanners 140a, the present disclosure is not limited thereto. Rather, the present disclosure may include machine learning models (e.g., one or more neural networks) that have been trained to predict patient outcomes based on any type of image. Such images may include two-dimensional images, three-dimensional images, image models generated based on another image, and the like.

[0029] The computer 130 is configured to obtain one or more pathology images 102 generated by an input device 140 (such as a slide scanner 140a). In some embodiments, the one or more pathology images can be images received through the network 120. In other embodiments, the computer 130 can access pathology images stored in a database 132 accessible to the computer 130. The database 132, a local database or a cloud-based database, can include pathology images, patient identifiers, patient medical history including current and past treatment plans, and predicted patient outcomes. The pathology images in the database can be original or annotated pathology images. In some embodiments, the patient identifier can be encrypted. However, the present disclosure is not limited to the use of encrypted patient identifiers.

[0030] The computer 130 may provide an acquired pathology image, such as the pathology image 102, to the input processing engine 104. In some embodiments, the computer 130 may provide the acquired pathology image for input to the input processing engine 104 in the same format in which the pathology image 102 was received. For example, the computer 130 may receive a two-dimensional image and provide the two-dimensional image as input to the input processing engine 104. In some embodiments, the computer 130 may generate a three-dimensional pathology image from the two-dimensional pathology image and provide the three-dimensional pathology image for input to the input processing engine 104.

[0031] The input processing engine 104 is configured to receive the pathology image 102 and generate an advanced pathology image 106 for input to the machine learning model 108. In some embodiments, the pathology image 102 is an image having approximately 50k×50k pixels. Figure 1, the input processing engine 104 can perform one or more steps of the following items: tissue filtering (e.g., using the tissue filter engine 104a), tile extraction (e.g., using the tile extraction engine 104b), normalization (e.g., using the normalization engine 104c), and enhancement (e.g., using the enhancement engine 104d). For example, the tissue filter engine 104a calculates a tissue mask and extracts the parts of the pathology image 102 that constitute the tissue. The tile extraction engine 104b divides the pathology image 102 into a set of tiles before or after tissue filtering. In some embodiments, each tile is an image with 96×96 pixels. The tile extraction engine 104b can determine the optimal size of the tile. The normalization engine 104c normalizes the color of the pathology image 102, for example, based on the color mean and distribution across multiple pathology images. The enhancement engine 104d applies an image transformation (also referred to as one or more noises) to the pathology image 102, such as rotation, shift, flip, scaling, affine transformation, adjusting brightness and / or contrast. This image enhancement prevents the machine learning model 108 from overfitting. The processed pathology image 106 generated by the input processing engine 104 may include a data structure representing the pathology image. The processed pathology image 106 output by the input processing engine 104 may be provided as input to the machine learning model 108. Figure 1 In the example of FIG. 1 , the processed pathology image 106 is a data structure representing the (eg, two-dimensional) pathology image 102 processed by the engines 104 a - 104 d .

[0032] The machine learning model 108 is configured to receive the processed pathology image 106 and generate an annotated pathology image 110. The machine learning model 108 is a model output by a training machine (e.g., a computer 130) to annotate the processed pathology image 106; Figure 2Training is described in more detail. The annotated pathology image 110 includes annotations for each tissue category. For example, a specific area of ​​the pathology image is annotated as a (predicted) tumor. Similarly, a specific area of ​​the pathology image is annotated as necrosis. In some embodiments, the annotation is a graphical annotation, such as a boundary and / or region. In some embodiments, each region (e.g., a uniformly sized tile) can be assigned a single tissue category; for example, a specific tile cannot be annotated as both normal and tumor. The annotation is based on the class membership probability predicted by the machine learning model 108. For example, for a specific tile predicted to be 70% tumor, 10% regression, and 20% normal, the specific tile is annotated with the label tumor. In some embodiments, each tile can have multiple labels based on the probability threshold of each member; that is, a specific tile predicted to be 40% tumor, 40% regression, and 20% normal (e.g., using a probability threshold of 33%) is annotated with both tumor and regression. The annotated pathology image output from the machine learning model 108 can be provided as an input to the patient outcome prediction engine 112. In some embodiments, a summary report is provided as an output of the machine learning model 108, wherein the summary report includes corresponding regions for each tissue class. The summary report can be provided in addition to or in lieu of annotated pathology images. In some embodiments, predicting patient outcomes is also performed by the machine learning model 108.

[0033] The patient outcome prediction engine 112 is configured to receive the annotated pathology image 110, the summary report, or both, and generate a predicted patient outcome 114. Patient outcomes (also referred to as clinical outcomes) include response to treatment regimens (including drug products, immunotherapy, and chemotherapy), patient survival, disease recurrence, and disease progression. In some embodiments, the patient outcome prediction engine 112 determines an indicator that indicates the patient's clinical outcome. For example, in some embodiments, the indicator can be residual viable tumor percentage (RVT), where the residual viable tumor percentage can be calculated based on the corresponding areas of different tissue classes.

[0034] In some embodiments, for example, the percent RVT may be calculated using the following expression: , where the area is expressed in pixels 2 (pixel 2 ) as the unit. For example, for a 2 , 2,612 pixels 2 , 3,286 pixels 2For a pathology image of a patient with leukemia, the patient outcome prediction engine 112 calculates the %RVT as 9%. The patient outcome prediction engine 112 determines the clinical outcome of the patient based on the calculated indicator (such as %RVT). In some embodiments, the threshold of the indicator is determined by the patient outcome prediction engine 112. For example, less than 10% RVT indicates a high response to immunotherapy. In some embodiments, the threshold of the indicator is determined by a medical expert. In some embodiments, the machine learning model 108 determines the optimal threshold based on the patient outcomes of a group of patients (e.g., patients undergoing the same type of immunotherapy). In some embodiments, the indicators indicative of the patient's clinical outcome are the percentage of regression and the percentage of necrosis in the pathology image.

[0035] In some embodiments, the computer 130 can generate rendering data that, when rendered by a device having a display, such as an output device 150, can cause the device to output data including the annotated pathology image 110 and the predicted patient outcome 114. Such rendering data can be transmitted by the computer 130 to the output device 150 via the network 120 and processed by the output device 150, such as a monitor 150a or a smart phone 150b, to generate output data. In some embodiments, the output device 150 can be coupled to the computer 130. In this case, the rendered data can be processed by the computer 130 and cause the computer 130 to output data including the annotated pathology image 110 and the predicted patient outcome 114.

[0036] Figure 2 2 is a block diagram of an example of a system (machine learning training engine 200) for training a machine learning model 108. For example, a machine learning model can be trained to distinguish between tumors and non-tumors in pathology images. Such training can be achieved by obtaining training data 202 and processing the training data 202. The training data 202 includes one or more pathology images of each tissue or cell category (such as normal, regression, tumor, and necrosis). However, the present disclosure need not be so limited to these tissue types, but can be applied to any tissue or cell type. In some embodiments, using the method described in reference Figure 1 The described input processing engine 104 processes the training data 202. One or more of the steps of tissue filtering, tile extraction, normalization, and enhancement (as described above) may be applied to the training data 202.

[0037] In some embodiments, the machine learning training engine 200 includes a transfer learning engine 204. The transfer learning engine 204 processes the obtained training data 202 labeled with different tissue categories. The transfer learning engine 204 may include a convolution layer 206 and a fully connected layer 208. The transfer learning engine 204 trains the machine learning model 108 by performing the following steps: (i) freezing the convolution layer, (ii) learning the weights of the fully connected layer, (iii) unfreezing the convolution layer, and (iv) learning the weights of all layers, where the weights represent the learnable parameters of the machine learning model 108. In some embodiments, the transfer learning engine 204 uses a DenseNet169 architecture including the following layers: convolution + rectified linear unit (ReLU), maximum pooling (e.g., for downsampling), full connection + ReLU, and softmax (e.g., converting any real value into a probability). The convolution layer 206 includes a convolution + ReLU and a maximum pooling layer. The fully connected layer 208 includes a fully connected + ReLU and a softmax layer. After learning the weights through the transfer learning engine 204, the machine learning training engine 200 outputs the machine learning model 108. In some embodiments, the trained machine learning model 108 can be stored in the database 132.

[0038] Figure 3A-3B The performance of the machine learning model 108 in annotating pathology images is shown. Figure 3A In some embodiments, the machine learning training engine 200 can test the machine learning model 108 against the test data. The test data can be validation data retained during training or independent test data. For example, Figure 3A Receiver operating characteristic (ROC) curves for correctly predicting different tissue classes (such as regression, tumor, and necrosis) when compared to expert annotations are shown. In some embodiments, predicting different tissue classes is performed for each tile in a set of tiles in a pathology image. In some embodiments, the expert annotations represent pathologist annotations for each (digitized) slice (i.e., pathologist visual scores). Figure 3A As described in , the machine learning model 108 has significantly better performance than random chance.

[0039] Figure 3B The performance of a machine learning model in predicting patient outcomes is shown. In some embodiments, the predicted patient outcomes are based on annotated pathology images (e.g., corresponding regions of different tissue classes). In some embodiments, portions of pathology images are used to predict patient outcomes. Figure 3B, the area under the ROC curve for correctly predicting a patient's outcome on immunotherapy was 0.98. Here, patient outcomes were based on %RVT as described above. The correlation between the machine learning model 108 and the expert's predictions for the 92 patients was shown to be high, with a Pearson correlation coefficient (r) of 0.95 and a slope of the fitted line of 0.84.

[0040] Based on testing the performance of the machine learning model 108, the machine learning training engine 200 can determine whether the machine learning model 108 meets predefined thresholds, such as accuracy and computational speed. When it is determined that the machine learning model 108 does not meet the predefined thresholds, the machine learning training engine 200 can adjust the training parameters, such as Figure 2 The machine learning architecture shown, training data 202 (e.g., by obtaining more training data and / or by validating labels), and retraining the machine learning model 108 based on the adjusted training parameters.

[0041] Figure 4A An example user interface 400 for annotating pathology images and predicting patient outcomes is shown. The user interface 400 is displayed on an output device 150, such as a monitor 150a or a smart phone 150b. In some embodiments, the user interface 400 may include a browse panel 402 and a case detail panel 404. The browse panel 402 allows a user to browse different cases, for example, cases belonging to a specific disease or a specific patient. In response to selecting a case from the browse panel 402, the case detail panel 404 displays a pathology image. For example, Figure 4A Each row shown in represents each pathology image of case 103. In some embodiments, the case details panel 404 can display the pathology image identifier ("slice ID"), image name, date added, progress (e.g., whether the uploaded pathology image is annotated), and predicted patient outcome (e.g., "%RVT"). In some embodiments, the user can see the final result category at the slice level or case level, such as "major pathologic response" or "pathologic complete response" or other related terms associated with medically meaningful classification of elements detected by the system. In addition, unfinished work can also be seen. In some embodiments, the user interface 400 includes a search panel 406 that obtains user input for a specific pathology image (e.g., a slice identifier) ​​and presents filtered results that meet the user input.

[0042] Figure 4BAn example of an annotated pathology image 450 and a predicted patient outcome 458 is shown. In response to a user selection of an image associated with a patient, the annotated pathology image 450 is displayed on the user interface 400. In some embodiments, the annotated pathology image 450 may include different categories of tissue (or cells). For example, predicted regression (area) 452, predicted tumor (area) 454, and predicted necrosis (area) 456 represent areas distinguished by tissue categories in the annotated pathology image 450. The user interface 400 can display different tissue categories by boundaries and / or different colors. In some embodiments, the annotated pathology image 450 includes information indicating the predicted patient outcome 458, for example, a dashboard or pop-up window summarizing the predicted patient outcome (e.g., "This patient's tumor is responsive to anti-PD-1 immunotherapy"). In some embodiments, the dashboard includes indicators for predicting patient outcomes. For example, "Machine-calculated % residual viable tumor (RVT) = 9%". The predicted patient outcome 458 is based on a threshold for the metric determined by an expert and / or machine learning model 108, e.g., less than 10% RVT indicates a response to immunotherapy.

[0043] Figure 5 is a flow chart of an example of a process 500 for applying a machine learning model to predict a clinical outcome for a patient. In general, the process 500 includes obtaining a pathology image associated with a patient (502); processing the pathology image, including: determining a region of interest in the pathology image, and segmenting the pathology image into a plurality of tiles (504); providing the processed pathology image as input to a machine learning model configured to annotate the pathology image (506); obtaining an annotated pathology image associated with the patient as an output of the machine learning model, wherein the annotated pathology image includes annotations for different categories of tissue including tumor regression (508); and determining an indicator indicative of a clinical outcome for the patient based on the annotated pathology image (510). Thus, the process 500 implements a machine learning model that facilitates prediction of a clinical outcome for a patient based on the annotated pathology image generated by the machine learning model.

[0044] Figure 66 is a flowchart of an example of another process 600 for training a machine learning model to annotate pathology images for predicting clinical outcomes of patients. In general, the process 600 includes: obtaining a plurality of training data items (e.g., each training data item includes a pathology image annotated with a tissue class such as normal, regression, tumor, and necrosis) (602); processing each training data item, including: determining a region of interest of the training data item and segmenting the training data item into a plurality of tiles (604); determining a training set, a validation set, and a test set from the plurality of training data items (606); providing the training set and the validation set to a plurality of fully connected layers, thereby obtaining weights for the plurality of fully connected layers (608); providing the training set and the validation set to a plurality of convolutional layers and a plurality of fully connected layers, thereby obtaining weights for the plurality of convolutional layers and a plurality of fully connected layers (610); generating a trained model based on the obtained weights (612), and providing the test set to the trained model (614).

[0045] Figure 7 is an example of a block diagram of system components that may be used to implement a system for predicting patient outcomes based on annotated pathology images.

[0046] Computing device 700 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Computing device 750 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, and other similar computing devices. In addition, computing device 700 or 750 may include a universal serial bus (USB) flash drive. The USB flash drive may store operating systems and other applications. The USB flash drive may include input / output components, such as a wireless transmitter or USB connector that may be inserted into a USB port of another computing device. The components, their connections and relationships, and their functions shown here are merely meant to be exemplary and are not meant to limit the embodiments of the invention described and / or claimed in this document.

[0047] The computing device 700 includes a processor 702, a memory 704, a storage device 706, a high-speed controller 708 connected to the memory 704 and a high-speed expansion port 710, and a low-speed controller 712 connected to a low-speed bus 714 and the storage device 706. Each of the components 702, 704, 706, 708, 710, and 712 is interconnected using various buses and can be installed on a common motherboard or installed in other appropriate ways. The processor 702 can process instructions for execution within the computing device 700, including instructions stored in the memory 704 or on the storage device 706, to display graphical information of a GUI on an external input / output device (e.g., a display 716 coupled to the high-speed controller 708). In other embodiments, multiple processors and / or multiple buses, as well as multiple memories and memory types, can be used appropriately. In addition, multiple computing devices 700 can be connected, where each device provides part of the necessary operations, for example, as a server library, a group of blade servers, or a multi-processor system.

[0048] The memory 704 stores information within the computing device 700. In one implementation, the memory 704 is one or more volatile memory units. In another implementation, the memory 704 is one or more non-volatile memory units. The memory 704 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0049] The storage device 706 can provide mass storage for the computing device 700. In one embodiment, the storage device 706 can be or can include a computer-readable medium (such as a floppy disk device, a hard disk device, an optical disk device or a tape device, a flash memory or other similar solid-state storage device) or a device array (including a device in a storage area network or other configuration). A computer program product can be tangibly embodied in an information carrier. The computer program product can also include instructions that, when executed, perform one or more methods such as those described above. The information carrier is a computer or machine-readable medium, such as a memory 704, a storage device 706, or a memory on a processor 702.

[0050] The high-speed controller 708 manages bandwidth-intensive operations of the computing device 700, while the low-speed controller 712 manages less bandwidth-intensive operations. This allocation of functions is exemplary only. In one embodiment, the high-speed controller 708 is coupled to the memory 704, the display 716 (e.g., through a graphics processor or accelerator), and is coupled to a high-speed expansion port 710 that can accept various expansion cards (not shown). In an embodiment, the low-speed controller 712 is coupled to the storage device 706 and the low-speed bus 714. The low-speed expansion port, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices (such as keyboards, pointing devices, microphone / speaker combinations, scanners) or networking devices (such as switches or routers), for example, through a network adapter. As shown, the computing device 700 can be implemented in a variety of different forms. For example, it can be implemented as a standard server 720, or multiple implementations in a group of such servers. It can also be implemented as part of a rack server system 724. In addition, it can be implemented in the form of a personal computer (such as a laptop computer 722). Alternatively, components of computing device 700 may be combined with other components in a mobile device (not shown), such as device 750. Each of these devices may contain one or more of computing devices 700, 750, and the entire system may be composed of multiple computing devices 700, 750 in communication with each other.

[0051] As shown, computing device 700 can be implemented in a variety of different forms. For example, it can be implemented as a standard server 720, or multiple times in a group of such servers. It can also be implemented as part of a rack server system 724. In addition, it can be implemented in the form of a personal computer (such as a laptop computer 722). Alternatively, components of computing device 700 can be combined with other components in a mobile device (not shown) such as device 750. Each of these devices can contain one or more of computing devices 700, 750, and the entire system can be composed of multiple computing devices 700, 750 that communicate with each other.

[0052] The computing device 750 includes a processor 752, a memory 764, and input / output devices such as a display 754, a communication interface 766, and a transceiver 768, among other components. The device 750 may also be provided with a storage device (such as a micro drive or other device) to provide additional storage. Each of the components 750, 752, 764, 754, 766, and 768 are interconnected using various buses, and several components may be mounted on a common motherboard or in other suitable ways.

[0053] The processor 752 can execute instructions within the computing device 750, including instructions stored in the memory 764. The processor can be implemented as a chipset of chips, which includes a single and multiple analog processors and digital processors. In addition, the processor can be implemented using any of a number of architectures. For example, the processor can be a CISC (complex instruction set computer) processor, a RISC (reduced instruction set computer) processor, or a MISC (minimum instruction set computer) processor. The processor can provide, for example, coordination of other components of the computing device 750, such as control of a user interface, applications run by the device 750, and wireless communications performed by the device 750.

[0054] The processor 752 can communicate with the user through the control interface 758 and the display interface 756 coupled to the display 754. The display 754 can be, for example, a TFT (thin film transistor liquid crystal display) display or an OLED (organic light emitting diode) display or other appropriate display technology. The display interface 756 may include appropriate circuits for driving the display 754 to present graphics and other information to the user. The control interface 758 can receive commands from the user and convert them to submit to the processor 752. In addition, the external interface 762 can provide communication with the processor 752 to enable near-area communication of the device 750 with other devices. The external interface 762 can, for example, provide wired communication in some embodiments, or provide wireless communication in other embodiments, and multiple interfaces can also be used.

[0055] The memory 764 stores information within the computing device 750. The memory 764 may be implemented as one or more of the following: one or more computer-readable media, one or more volatile memory units, or one or more non-volatile memory units. An expansion memory 774 may also be provided and connected to the device 750 via an expansion interface 772, which may include, for example, a SIMM (single in-line memory module) card interface. The expansion memory 774 may provide additional storage space for the device 750, or may also store applications or other information for the device 750. Specifically, the expansion memory 774 may include instructions for executing or supplementing the above-mentioned processes, and may also include security information. Therefore, for example, the expansion memory 774 may be provided as a security module for the device 750, and may be programmed with instructions that allow the secure use of the device 750. In addition, security applications and additional information may be provided via a SIMM card, such as placing identification information on the SIMM card in an unbreakable manner.

[0056] As described below, the memory may include, for example, flash memory and / or NVRAM memory. In one implementation, a computer program product may be specifically embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods such as those described above. The information carrier is a computer-readable medium or machine-readable medium, such as memory 764, expansion memory 774, or memory on processor 752, which may be received, for example, via transceiver 768 or external interface 762.

[0057] The device 750 can communicate wirelessly via a communication interface 766, which may include digital signal processing circuitry when necessary. The communication interface 766 may provide communications under various modes or protocols, such as GSM voice calls, SMS, EMS or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000 or GPRS, etc. Such communications may occur, for example, via a (radio frequency) transceiver 768. In addition, short-range communications may occur, for example, using Bluetooth, Wi-Fi or other such transceivers (not shown). In addition, a GPS (global positioning system) receiver module 770 may provide additional navigation and location-related wireless data to the device 750, which may be used appropriately by applications running on the device 750.

[0058] Device 750 may also communicate audibly using audio codec 760, which may receive voice information from a user and convert it into usable digital information. Audio codec 760 may also generate audible sounds for the user, such as through a speaker (e.g., in a handset of device 750). Such sounds may include sounds from voice phone calls, may include recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications operating on device 750.

[0059] As shown, computing device 750 can be implemented in many different forms. For example, it can be implemented as a cellular phone 780. It can also be implemented as part of a smart phone 782, a personal digital assistant, or other similar mobile device.

[0060] Various embodiments of the systems and methods described herein can be implemented in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementation in the form of one or more computer programs executable and / or interpretable on a programmable system, the programmable system comprising at least one programmable processor, which may be special purpose or general purpose, coupled to receive data and instructions from a storage system and transmit data and instructions to the storage system.

[0061] These computer programs (also referred to as programs, software, software applications or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0062] To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, as well as a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0063] The systems and techniques described herein can be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0064] A computing system may include clients and servers. Generally speaking, clients and servers are remote from each other and usually interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0065] Several embodiments have been described. However, it will be appreciated that various changes may be made without departing from the spirit and scope of the invention. Furthermore, the logic flows depicted in the figures do not require the particular order or sequential order shown to achieve the desired results. Furthermore, other steps may be provided from the described flows, or steps may be eliminated from the described flows, and other components may be added to or removed from the described systems. Therefore, other embodiments are within the scope of the appended claims.

Claims

1. A computer-implemented method for predicting a clinical outcome of a patient, the method comprising: obtaining a pathology image associated with the patient; Processing the pathological image, comprising: determining a region of interest of the pathology image; and Segmenting the pathology image into a plurality of tiles, wherein each of the plurality of tiles has a uniform size; providing the processed pathology image as input to a machine learning model configured to annotate the pathology image; obtaining an annotated pathology image associated with the patient as an output of the machine learning model, wherein the annotated pathology image includes annotations for different classes of tissue including tumor regression; and An indicator indicative of the clinical outcome of the patient is determined based on the annotated pathology image.

2. The computer-implemented method of claim 1 , wherein: The clinical outcomes for the patients include response to one or more treatment regimens, survival rate, disease recurrence, and disease progression.

3. The computer-implemented method of claim 1 , wherein: The pathological image includes a histopathological section image.

4. The computer-implemented method of claim 1 , further comprising: in, The machine learning model has been trained on a plurality of training data items, wherein each training data item comprises annotated pathology images with tissue classes.

5. The computer-implemented method of claim 1 , further comprising: The rendering data is generated such that, when rendered by a user device, the user device displays a user interface displaying the annotated pathology image associated with the patient and a predicted clinical outcome for the patient.

6. The computer-implemented method of claim 5, wherein: The user interface includes a user-selectable element for uploading the pathology image to apply the machine learning model.

7. The computer-implemented method of claim 1 , wherein: The different categories of tissue include tumor, necrosis, tumor regression area and normal background, wherein the normal background indicates uninvolved tissue.

8. The computer-implemented method of claim 1 , wherein: The indicator indicative of the clinical outcome of the patient includes a residual viable tumor percentage, wherein the residual viable tumor percentage is calculated based on areas of the different categories of tissue in the pathology image.

9. The computer-implemented method of claim 1 , wherein: The indicators indicating the clinical outcome of the patient include a regression percentage and a necrosis percentage in the pathological image.

10. The computer-implemented method of claim 1, wherein: Processing the pathological image further includes: normalizing the color of the pathology image based on the color of a group of pathology images; and An image transformation is applied to the pathology image, wherein the image transformation includes rotation, shift, flipping, scaling, affine transformation, and adjusting brightness and contrast.

11. A system comprising: One or more computers and one or more storage devices storing instructions which, when executed by the one or more computers, are operable to cause the one or more computers to perform the method according to any one of claims 1 to 10.

12. A computer storage medium encoded with a computer program, the program comprising instructions operable when executed by a data processing apparatus to cause the data processing apparatus to perform the method according to any one of claims 1 to 10.

Images