Systems and methods for predicting patient responses - Patents.com

Abstract

Data processing systems for predicting one or more responses to chemicals based on biological images, at least some of the data processing systems including at least one processor configured to execute at least one artificial neural network trained to predict one or more responses to chemicals based on biological images, wherein when the at least one processor is executing the computer-executable instructions, the at least one processor is configured to perform operations including processing spatially arranged image tile data through one or more data structures that store one or more portions of executable logic included in the artificial neural network to determine one or more responses of the patient to the chemicals.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to European Patent Application No. 20315338.2, filed July 2, 2020, and U.S. Provisional Patent Application No. 62 / 971,630, filed February 7, 2020, the contents and disclosures of which are incorporated herein by reference in their entireties.

[0002] FIELD OF THE DISCLOSURE The present disclosure relates generally to systems and methods for predicting a patient's response to a compound, such as a pharmaceutical drug. [Background technology]

[0003] Clinical trials are typically conducted to gather data on the safety and efficacy of pharmaceuticals. These trials typically involve one or more phases that determine whether the drug can be sold in the consumer market. For example, clinical trials can include three phases. In the first phase, the drug is tested on a relatively small number of paid volunteers (e.g., 20-100 volunteers) to determine the drug's effects, including absorption, metabolism, and excretion. This phase takes several months to complete, with approximately 70% of experimental drugs passing the first phase. In the second phase, the experimental drug is tested on several hundred patients who meet one or more inclusion criteria. One group of patients receives the experimental drug, while another group receives a placebo or standard of care. Approximately one-third of experimental drugs complete both phases 1 and 2 of the trial. During the third phase, the drug is tested on hundreds to thousands of patients (or more). This phase tends to be the most expensive of all phases, with approximately 70% of drugs entering phase 3 successfully completing the phase. Summary of the Invention [Means for solving the problem]

[0004] In at least one embodiment of the present disclosure, a data processing system is provided. The data processing system includes a computer-readable memory including computer-executable instructions and at least one processor configured to execute at least one artificial neural network trained to predict one or more responses to chemicals based on biological images. When the at least one processor is executing the computer-executable instructions, the at least one processor is configured to perform one or more operations. The one or more operations include receiving spatially arranged image data representing a plurality of biological images of a patient. The one or more operations include, for each biological image of the plurality of biological images, generating spatially arranged image tile data representing a plurality of image tiles, where each image tile of the plurality of image tiles includes a distinct portion of the biological image. The one or more operations include processing the spatially arranged image tile data through one or more data structures that store one or more portions of executable logic included in the artificial neural network to: for each image of the plurality of images, classify the image as representing or not representing abnormal tissue by analyzing the plurality of image tiles corresponding to the image; for each image classified as representing abnormal tissue, classify the image as indicating or not indicating that the patient will react to the chemical by analyzing the plurality of image tiles corresponding to the image; and determine one or more responses of the patient to the chemical based on the images classified as indicating or not indicating that the patient will react to the chemical.

[0005] The artificial neural network can include at least one convolutional layer and at least one fully connected layer.

[0006] Classifying the image as being representative of or not representative of abnormal tissue may include assigning an abnormality weight to each image tile of the plurality of image tiles corresponding to the image. The abnormality weight may indicate a predictive power of the image tile corresponding to the abnormality weight for classifying the image corresponding to the image tile as being representative of or not representative of abnormal tissue. Classifying the image as being indicative of or not indicative of a patient responding to the chemical may include assigning a response weight to each image tile of the plurality of image tiles corresponding to the image. The response weight may indicate a predictive power of the image tile corresponding to the response weight for classifying the image corresponding to the image tile as being indicative of or not indicative of a patient responding to the chemical.

[0007] The plurality of images can include a plurality of immunohistochemistry images. The abnormal tissue can include cancerous tissue.

[0008] These and other aspects, configurations and implementations can be expressed as methods, apparatus, systems, components, program products, means or steps for performing functions, and in other ways.

[0009] Embodiments of the present disclosure may provide one or more of the following advantages: Image processing and machine learning techniques may be used to process image data and predict a patient's response to a drug in a manner that increases the accuracy of the prediction, increases computational efficiency, and / or reduces computing power requirements compared to conventional techniques; Predictions may take into account a greater number of variables, thereby increasing the accuracy of the prediction; Machine learning outputs are more interpretable and easier to converge compared to conventional techniques.

[0010] These and other aspects, configurations, and implementations will become apparent from the following description, including the claims. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 illustrates an example of a data processing system. [Figure 2] FIG. 1 is a flow diagram illustrating an exemplary architecture of a data processing system. [Figure 3] 3A-3B illustrate an example data processing system for labeling an image based on weights assigned to tiles of the image. [Figure 4] 1 is a flow chart illustrating an exemplary method for predicting a patient's response to a compound. [Figure 5] FIG. 1 illustrates an exemplary architecture of an artificial neural network. [Figure 6] FIG. 1 is a block diagram of an exemplary computer system that may be used to provide the computational functionality associated with the algorithms, methods, functions, processes, flows, and procedures described in this disclosure. [Figure 7] 1 shows an example of an image classified as a responder using the systems and methods described herein. [Figure 8] 1 shows an example of an image classified as a non-responder using the systems and methods described herein. DETAILED DESCRIPTION OF THE INVENTION

[0012] For clinical trials involving a given drug, selecting patients who can benefit from treatment with manageable side effects can be crucial, especially in the field of life-threatening diseases such as oncology. Due to recent advances in medical imaging techniques, medical or biomedical images (e.g., immunohistochemistry images) can be useful in predicting patient outcomes for investigational therapies. However, traditional patient outcome prediction techniques typically extract only a few features from biomedical images, such as proportional scores and histochemistry scores (sometimes referred to as "H scores"). As a result, the accuracy of patient response predictions can range from 20% to 45%. Furthermore, predicting patient response to a given drug using traditional machine learning techniques can be computationally infeasible because biomedical images can be 2 gigabytes (or more) in size, with dimensions of 50,000 pixels by 40,000 pixels (or more). Images of these sizes can require machine learning models to estimate billions (or more) of parameters.

[0013] Embodiments of the present disclosure provide systems and methods for predicting patient response that can be used to avoid some or all of the disadvantages described above. The systems and methods described herein can implement image processing and machine learning techniques, whereby image data representing biometric images can be processed in a computationally more efficient manner to predict a patient's response to a drug with a higher degree of accuracy compared to conventional techniques. In some embodiments, the machine learning technique includes a multi-instance learning technique that implements a two-step multi-classification method. In some embodiments, the systems and methods described herein can receive data representing multiple biometric images of a patient and generate image tiles (sometimes referred to herein as instances), each representing a distinct portion of the biometric image. The data representing each biometric image can be processed using an artificial neural network (ANN) that can classify, for each image, the image tile corresponding to the image as representing or not representing abnormal tissue (e.g., whether the image contains cancerous tumor cells) by analyzing the image tile corresponding to the image. This can include assigning an abnormality weight to each image tile corresponding to the image, where the abnormality weight reflects the predictive power of the image tile for predicting whether the image corresponding to the tile represents abnormal tissue. In some embodiments, image data representing images classified as representing abnormal tissue is further processed by an ANN to classify the images as indicative or not indicative of a patient responding to a chemical (e.g., whether the image indicates that the chemical reduces cancer tumors) by analyzing the corresponding image tiles. This can include assigning a response weight to each image tile corresponding to the image, where the response weight reflects the predictive power of the image tile for predicting whether the image corresponding to the tile indicates that the patient will respond to the chemical. This per-image prediction can then be aggregated to determine whether the patient will respond to the chemical.

[0014] The ANN can be trained to identify higher-level features from images that may influence patient response prediction. For example, the ANN can learn to associate a partially stained pattern of a target protein in the membrane of a tumor nest with a poor patient response because an active drug ingredient may not recognize the target protein to attack the tumor nest. Examples of patient response include efficacy responses (such as a reduction / change in size of a cancer tumor resulting from a patient receiving an oncology drug treatment regimen), safety responses (such as rejection, toxicity, and cardiovascular risk resulting from a patient receiving an oncology drug treatment regimen), or both.

[0015] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent that the present disclosure may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the present disclosure.

[0016] In the drawings, a particular arrangement or order of schematic elements, such as those representing devices, modules, instruction blocks, and data elements, is shown for ease of explanation. However, the particular order or arrangement of schematic elements in the drawings is not intended to imply that a particular order or sequence of operations, or separation of operations, is required. Furthermore, the inclusion of a schematic element in a drawing is not intended to imply that such element is required in all embodiments, nor that the structure represented by such element is not included in or combined with other elements in some embodiments.

[0017] Furthermore, when connecting elements, such as solid or dashed lines or arrows, are used in the drawings to illustrate a connection, relationship, or association between two or more other schematic elements, the absence of such connecting elements is not intended to imply that the connection, relationship, or association cannot exist. In other words, some connections, relationships, or associations between elements may not be shown in the drawings so as not to obscure the disclosure. Additionally, for ease of illustration, a single connecting element may be used to represent multiple connections, relationships, or associations between elements. For example, when a connecting element represents communication of signals, data, or instructions, it should be understood that such element represents one or more signal paths (e.g., buses) for affecting the communication, as appropriate.

[0018] Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments described. However, it will be apparent to those skilled in the art that the various embodiments described may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0019] Below are described several configurations, each of which can be used independently of one another or with any combination of the other configurations. However, any individual configuration may not address all of the problems discussed above, or may only address one of the problems discussed above. Some of the problems discussed above may not be completely solved by any of the configurations described herein. Although headings are provided, data related to a particular heading but not found in the section bearing that heading may be found elsewhere in this specification.

[0020] 1 illustrates an example of a data processing system 100. Typically, data processing system 100 is configured to process image data representing a patient's biological image to predict the patient's response (e.g., reduction in cancer tumor size) to a given chemical substance (e.g., a pharmaceutical agent). System 100 includes a computer processor 110. Computer processor 110 includes computer-readable memory 111 and computer-readable instructions 112. System 100 also includes a machine learning system 150. Machine learning system 150 includes a machine learning model 120. Machine learning model 120 can be separate from computer processor 110 or integrated with computer processor 110.

[0021] The computer readable medium 111 (or computer readable memory) comprises any form of data storage technology suitable for the local technology environment, including, but not limited to, semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, removable memory, disk memory, flash memory, dynamic random access memory (DRAM), static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), etc. In some implementations, the computer readable medium 111 includes code segments having executable instructions.

[0022] In some embodiments, computer processor 110 includes a general-purpose processor. In some embodiments, computer processor 110 includes a central processing unit (CPU). In some embodiments, computer processor 110 includes at least one application-specific integrated circuit (ASIC). Computer processor 110 may also include a general-purpose programmable microprocessor, a graphics processing unit, a special-purpose programmable microprocessor, a digital signal processor (DSP), a programmable logic array (PLA), a field programmable gate array (FPGA), special-purpose electronic circuitry, or the like, or a combination thereof. Computer processor 110 is configured to execute program code, such as computer-executable instructions 112, and is configured to execute executable logic, including machine learning model 120.

[0023] The computer processor 110 is configured to receive image data representing a plurality of medical images of a patient. For example, the patient's medical images may include one or more images of immunohistochemical staining results that describe the process of selectively identifying proteins (e.g., antigens) in cells of a biological tissue section by utilizing the principle of antibodies binding to antigens in the biological tissue. The image data may be obtained through any of a variety of techniques, such as wireless communication with a database, fiber optic communication, USB, CD-ROM, etc.

[0024] In some implementations, the computer processor 110 is configured to generate image tile data representing a plurality of image tiles, each image tile including a distinct portion of the biometric image. For example, if the biometric image includes dimensions of 4096x4096, the computer processor 110 may divide the image into 256 tiles, each having dimensions of 256x256.

[0025] The machine learning model 120 can process image data to predict a patient's response to a particular chemical (e.g., a pharmaceutical). For example, for a given oncology drug regimen, the machine learning model 120 can predict the amount of reduction in size of a cancer tumor by analyzing image tiles of a plurality of images. In some embodiments, predicting a patient's response includes assigning one or more weight values ​​to the tiles and labeling the images according to the weight values. Predicting a patient's response is discussed in more detail below with reference to FIGS. 2-5.

[0026] Machine learning system 150 can apply machine learning techniques to train machine learning model 120. As part of training machine learning model 120, machine learning system 150 forms a training set of input data by identifying a positive training set of input data items determined to have the characteristic of interest, and in some embodiments forms a negative training set of input data items that lack the characteristic of interest.

[0027] The machine learning system 150 extracts features from the input data of the training set, where features are variables considered to be potentially related to whether an input data item has an associated characteristic or characteristics. Herein, an ordered list of features of the input data is referred to as a feature vector of the input data. In one embodiment, the machine learning system 150 applies dimensionality reduction (e.g., via linear discriminant analysis (LDA), principal component analysis (PCA), etc.) to reduce the amount of data in the feature vector for the input data to a smaller, more representative set of data.

[0028] In some implementations, the machine learning system 150 uses supervised machine learning to train the machine learning model 120, with feature vectors from a positive training set and a negative training set serving as input. Different implementations can use different machine learning techniques, such as linear support vector machines (linear SVMs), boosting for other algorithms (e.g., AdaBoost), neural networks, logistic regression, naive Bayes, memory-based learning, random forests, bug trees, decision trees, boosted trees, or boosted stumps. When applied to feature vectors extracted from input data items, the machine learning model 120 outputs an indication of whether the input data items possess the property of interest, such as a Boolean yes / no inference or a scalar value representing a probability.

[0029] In some embodiments, the validation set is formed from additional input data other than the input data in the training set that have already been determined to have or lack the characteristic in question. The machine learning system 150 applies the trained machine learning model 120 to the data in the validation set to quantify the accuracy of the machine learning model 120. Common metrics applied in accuracy measurements include precision = TP / (TP+FP) and recall = TP / (TP+FN). Here, precision indicates the number of correct predictions (TP, i.e., true positives) made by the machine learning model out of the total number of predictions (TP+FP, i.e., false positives), and recall indicates the number of correct predictions (TP) made by the machine learning model out of the total number of input data items that had the characteristic in question (TP+FN, i.e., false negatives). The F-score (F-score = 2 × PR / (P+R)) unifies precision and recall into a single measure. In one embodiment, the machine learning module iteratively retrains the machine learning model until an accuracy measurement indication that the model is sufficiently accurate or a stopping condition occurs, such as the number of training rounds performed.

[0030] In some implementations, the machine learning model 120 is a convolutional neural network (CNN). The CNN can be constructed based on the assumption that inputs to the CNN correspond to image pixel data of an image or other data containing features at multiple spatial locations. For example, the set of inputs can form a multidimensional data structure, such as a tensor representing color features of an exemplary digital image (e.g., a biological image of biological tissue). In some implementations, the inputs to the CNN correspond to various other types of data, such as data acquired from different devices and sensors, point cloud data, audio data containing specific features, or raw audio at each of multiple time steps, or various types of one-dimensional or multidimensional data. The convolutional layers of the CNN can process the inputs to transform the image features represented by the inputs in the data structure. For example, the inputs are processed by performing a dot-product operation using the input data along a given dimension of the data structure and a set of parameters for the convolutional layer.

[0031] The computation of a convolutional layer can involve applying one or more sets of kernels to a portion of the input in the data structure. The manner in which a CNN performs computations can be based on the specific characteristics of each layer of an exemplary multilayer neural network or a deep neural network supporting a deep neural network load. A deep neural network can include one or more convolution towers (or layers) and other computational layers. In particular, for computer vision applications, these convolution towers often account for the majority of the inference computations performed. A convolutional layer of a CNN can have a set of artificial neurons arranged in three dimensions: width, height, and depth. The depth dimension corresponds to the third dimension of the input volume or activation volume and can represent each color channel of the image. For example, an input image can form an input volume of data (e.g., activation), with the volume having dimensions 32 x 32 x 3 (width, height, and depth, respectively). The depth dimension 3 can correspond to the RGB color channels of red (R), green (G), and blue (B).

[0032] Generally, a CNN layer is configured to convert a three-dimensional input volume (input) into a multidimensional output volume of neuron activations (activations). For example, a 32x32x3 3D ​​input structure holds the raw pixel values ​​of an example image, in this case an image 32 wide by 32 high and with three color channels R, G, and B. The convolutional layers of the CNN of the machine learning model 120 compute the outputs of neurons that can connect to local regions within the input volume. Each neuron in the convolutional layer can spatially connect only to a local region of the input volume, but can connect to the entire depth of the input volume (e.g., all color channels). For a set of neurons in the convolutional layer, the layer computes the dot product between the neuron's parameters (weights) and the specific region in the input volume to which the neuron is connected. This computation can result in a volume such as 32x32x12, where 12 corresponds to the number of kernels used in the computation. The neuron's connection to the region's input can have a spatial extent along the depth axis equal to the depth of the input volume. The spatial extent corresponds to the spatial dimensions of the kernel (eg, the x and y dimensions, etc.).

[0033] The set of kernels may have spatial characteristics that include width and height and span the depth of the input volume. Each set of kernels in a layer is applied to one or more sets of inputs provided to the layer. That is, for each kernel or set of kernels, the machine learning model 120 may overlay the kernel, which may be multidimensionally represented, on a first portion of the layer inputs (e.g., forming an input volume or input tensor). For example, the set of kernels in the first layer of a CNN may have a width of 5 pixels, a height of 5 pixels, a depth of 3 corresponding to the color channels of the input volume to which the kernel is applied, and a size of 5x5x3x16 corresponding to 16 output dimensions corresponding to the number of output channels. In this context, the set of kernels may include 16 kernels, and the resulting output of the convolution has a depth dimension of 16.

[0034] The machine learning model 120 can then calculate a dot product from the overlapped elements. For example, the machine learning model 120 can convolve (or slide) each kernel across the width and height of the input volume and calculate the dot product between the kernel's entry and the input at a location or region of the image. Each output value of the convolution output is the result of a dot product between the kernel and some set of inputs from the example input tensor. The dot product can result in a convolution output that corresponds to a single layer's input, e.g., an activation element with a top-left position in the overlapping multidimensional space. As described above, neurons in a convolution layer can connect to a region of the input volume that includes multiple inputs. The machine learning model 120 can convolve each kernel across each input of the input volume. The machine learning model 120 can perform this convolution operation, for example, by moving (or sliding) each kernel over each input within the region.

[0035] The machine learning model 120 can move each kernel over the input of a region based on the stride value of a given convolutional layer. For example, when the stride is set to 1, the machine learning model 120 can move the kernel over the region one pixel (or input) at a time. Similarly, when the stride is 2, the machine learning model 120 can move the kernel over the region two pixels at a time. Thus, the kernel can be shifted based on the stride value of the layer, and the machine learning model 120 can perform this process repeatedly until the inputs of the region have corresponding dot products. Related to the stride value is a skip value. The skip value can identify one or more sets (2x2) of inputs within the region of the input volume to be skipped when the inputs are loaded for processing in the neural network layer. In some implementations, the input volume of pixels of an image can be "padded" with zeros, for example, around the boundary regions of the image. This zero padding is used to control the spatial size of the output volume.

[0036] As discussed above, a convolutional layer of a CNN is configured to convert a three-dimensional input volume (region input) into a multidimensional output volume of neuron activations. For example, when a kernel is convolved across the width and height of the input volume, the machine learning model 120 can generate a multidimensional activation map that includes the results of convolving the kernel at one or more spatial locations based on a stride value. In some cases, increasing the stride value results in a spatially smaller output volume of activations. In some implementations, activations can be applied to the output of the convolution before the output is sent to subsequent layers of the CNN.

[0037] An exemplary convolutional layer can have one or more control parameters that describe the layer's characteristics. For example, the control parameters can include the number of kernels K, the spatial extent of the kernels F, the stride (or skip) S, and the amount of zero padding P. The numerical values ​​of these parameters, the inputs to the layer, and the parameter values ​​of the layer's kernels shape the computations that occur at the layer and the size of the layer's output volume. In some implementations, the spatial size of the output volume is calculated as a function of the input volume size W using the formula (W-F+2P) / S+1. For example, an input tensor can represent a pixel input volume of size [227 x 227 x 3]. A convolutional layer of a CNN can have a spatial extent value F=11, a stride value S=4, and no zero padding (P=0). Using the above formula and layer kernel amount K=96, the machine learning model 120 performs layer calculations that result in a convolutional layer output volume of size [55×55×96], where 55 is obtained from [(227−11+0) / 4+1=55].

[0038] The computation of a convolutional layer or other layer of a CNN (e.g., a dot product computation) involves performing mathematical operations, such as multiplication and addition, using computational units of the hardware circuit of the machine learning model 120. The design of the hardware circuit may limit the system's ability to fully utilize the circuit's computational cells when performing the computation of a layer of a neural network. A more detailed example of the architecture of the machine learning model 120 is discussed below with reference to FIG. 5.

[0039] Although this specification generally describes patients as human patients, embodiments are not so limited. For example, a patient can refer to a non-human animal, a plant, or a human replicating system.

[0040] 2 is a flow diagram illustrating the architecture of a data processing system 200. The data processing system 200 may be substantially similar to the data processing system 100 described above with reference to FIG. 1. The data processing system 200 includes a data labeling module 210, an instance generation module 220, a tumor classification module 230, a response classification module 240, and an interpretation module 250. Modules 210-250 may be executed, for example, by the computer processor 110 of the data processing system 100 discussed above with reference to FIG. 1.

[0041] The data labeling module 210 can receive data representing multiple biometric images of a patient. The data labeling module 210 facilitates labeling of tiles of the images. For example, the labeling module 210 can include a graphical user interface (GUI) that allows a user to manually label tiles to indicate whether the tiles contain images of cancerous tumors. Facilitating tile labeling (sometimes referred to as weak labeling compared to pixel-by-pixel labeling) can incorporate user (e.g., human medical professional) knowledge into the data, thereby enabling training of a machine learning model (such as the machine learning model 120 described above with reference to FIG. 1).

[0042] The instance generation module 220 is configured to generate image tile data by dividing each received image into multiple image tiles (instances). For example, if an image has dimensions of 4096 x 4096, the instance generation module 220 may divide the image into 256 tiles (instances) having dimensions of 256 x 256. Each of the multiple image tiles may include a distinct portion of the biological image. The number of generated image tiles may be selected based on considerations of computational efficiency, computational power, and computational accuracy. For example, the number of tiles per image may vary from a few tiles to thousands of tiles due to the heterogeneity of medical images (e.g., as seen in immunohistochemistry images of biological specimens from cancer patients).

[0043] The tumor classification module 230 can classify each image as "tumor" or "non-tumor" by analyzing the image tiles corresponding to the image. In some embodiments, a tumor classification indicates that at least one tile corresponding to the image contains an image of cancerous tumor cells. In some embodiments, a non-tumor classification indicates that none of the tiles corresponding to the image contain an image of cancerous tumor cells. In some embodiments, tiles can be weighted according to the predictive power associated with the tumor / non-tumor classification. For example, if an image tile of an image contains a stronger indication of cancerous cells, that image tile can be assigned a higher weight than other image tiles of the same image that have a weaker indication of cancerous cells (or fail to show an indication of cancerous cells).

[0044] FIG. 3A illustrates an example of a data processing system that labels image 300 as "tumor" and assigns weights to tiles of the image. As shown, image 300 is divided into four tiles. Image 300 was labeled as "tumor" because at least one image tile contains an image of cancerous tumor cells. Additionally, the tiles of image 300 were assigned weights relative to each other, where each weight indicates the amount of responsibility the corresponding tile had for the underlying classification. The sum of the weights is equal to 1. As explained below, ensuring that the sum of the weights is equal to 1 can reduce variability between images.

[0045] Referring back to FIG. 2 , the response classification module 240 receives each of the images labeled as “tumor” and classifies each of these images as a “responder” or a “non-responder.” In some embodiments, a “responder” classification reflects that the image includes at least one tile containing an image indicating that the patient is responding to the chemical (e.g., the chemical causes a reduction in tumor size). In some embodiments, a “non-responder” classification reflects that the image does not include at least one tile containing an image indicating that the patient is responding to the chemical (e.g., the chemical has little or no effect on tumor size). In some embodiments, tiles can be weighted according to the predictive power associated with the responder / non-responder classification. For example, if an image tile of an image includes a stronger indication that the patient is responding to the chemical (or fails to show an indication that the patient is responding to the chemical), that image tile can be assigned a higher weight than other image tiles of the same image that have a weaker indication that the patient is responding to the chemical (or fail to show an indication that the patient is responding to the chemical).

[0046] FIG. 3B shows an example of a data processing system that labels image 300 as a "responder" and assigns weights to tiles in image 300. As shown, image 300 was divided into four tiles. Image 300 was labeled a "responder" because at least one tile exhibits an image that indicates a patient responding to a chemical. Additionally, the tiles in image 300 were assigned weights relative to each other, where each weight indicates the amount of responsibility the corresponding tile had for the underlying classification. The sum of the weights is equal to 1. As explained below, ensuring that the sum of the weights is equal to 1 can reduce variability between images.

[0047] Referring back to FIG. 2 , the interpretation module 250 receives the weighted image tiles for each image and reconstructs each image. The interpretation module 250 can then determine the image portion that triggered the resulting classification for each image by multiplying the weight assigned to the tile with each original tile's data matrix. The interpretation module 250 can then process the image so that tiles with smaller weights appear darker than tiles with larger weights, and tiles with larger weights are more prominent in the image. The interpretation module 250 can then aggregate the classifications by multiplying each tile's weight with the prediction from each tile to predict whether a patient will respond to a chemical at the image level. In some embodiments, the weighting scheme serves as an attention mechanism that improves the interpretability of the output of the machine learning process.

[0048] Although certain modules, including a data labeling module 210, an instance generation module 220, a tumor classification module 230, a response classification module 240, and an interpretation module 250, are described as performing certain aspects of the techniques described herein, in some embodiments, some or all of the techniques may be performed by additional, fewer, or alternative modules.

[0049] 4 is a flowchart illustrating an exemplary method 400 for predicting a patient's response to a compound. Method 400 includes receiving spatially aligned image data (block 410). The spatially aligned image data may represent multiple biological images of the patient. For example, as discussed above with reference to FIG. 1, the biological images may be one or more immunohistochemical images of the patient's biological tissue.

[0050] The method 400 includes generating spatially arranged image tile data (block 420). For example, spatially arranged image tile data can be generated for each image, where the tile data represents image tiles of the image. Each image tile can include a separate portion of the corresponding image.

[0051] The method 400 includes processing the spatially arranged image tile data (block 430). For example, an artificial neural network can be used to process the spatially arranged image tile data and predict one or more responses of the patient by analyzing the image tiles. In some embodiments, this includes classifying each of the plurality of images as "tumor" or "non-tumor" by analyzing the plurality of image tiles corresponding to the images, as discussed above with reference to FIGS. 1-2. In some embodiments, this includes further classifying each of the images classified as "tumor" as a "responder" or a "non-responder," as discussed above with reference to FIGS. 1-2. The number of images classified as "responders" can be aggregated with the number of images classified as "non-responders" to determine whether the patient will respond to the chemical.

[0052] FIG. 5 illustrates an exemplary architecture 500 of an artificial neural network. Architecture 500 can be integrated with machine learning model 120 described above with reference to FIG. 1. As shown, architecture 500 includes one or more convolutional layers (“CNN”) and one or more fully connected layers (“FCN”). Architecture 500 may be referred to as an attention-based architecture that utilizes weighted averaging of instances (e.g., image tiles), where the weights are determined by the artificial neural network. In some implementations, the weights sum to 1 and are invariant to the size of the bag (e.g., the number of image tiles). For example, if H={h1,...,hk} is an image embedding with (k) instances, the architecture can use the following pooling scheme:

number

number

number

number

[0053] The attention-based architecture can facilitate the assignment of different weights to image tiles within an image, so that the final representation of the image can inform the image-level classification. That is, the architecture can derive "key" image tiles. By deriving these key instances, the architecture 500 can provide a final patient-level classification for the region of interest.

[0054] FIG. 6 is a block diagram of an exemplary computer system 600 that may be used to provide computational functionality associated with the described algorithms, methods, functions, processes, flows, and procedures described in this disclosure (such as method 200 described above with reference to FIG. 2 ), according to some embodiments of the present disclosure. The illustrated computer 602 is intended to encompass any computing device, such as a server, desktop computer, laptop / notebook computer, wireless data port, smartphone, personal data assistant (PDA), tablet computing device, or one or more processors (including physical instances, virtual instances, or both) within these devices. The computer 602 may include input devices, such as a keypad, keyboard, and touchscreen, that can accept user information. The computer 602 may also include output devices that can communicate information related to the operation of the computer 602. The information may include digital data, visual data, audio information, or a combination of information. The information may be presented in a graphical user interface (UI) (or GUI).

[0055] The computer 602 may act as a client, network component, server, database, persistency, or component of a computer system for implementing the subject matter described in this disclosure. The illustrated computer 602 is communicatively coupled to a network 630. In some implementations, one or more components of the computer 602 may be configured to operate in different environments, including a cloud computing-based environment, a local environment, a global environment, and combinations of environments.

[0056] At a high level, computer 602 is an electronic computing device operable to receive, transmit, process, store, and manage data and information related to the described subject matter. According to some embodiments, computer 602 may also include or be communicatively coupled to an application server, an email server, a web server, a caching server, a streaming data server, or a combination of servers.

[0057] The computer 602 can receive requests from client applications (e.g., running on another computer 602) over the network 630. The computer 602 can respond to the received requests by processing the received requests using a software application. Requests can also be sent to the computer 602 from internal users (e.g., command consoles), external (or third parties), automated applications, entities, individuals, systems, and computers.

[0058] Each component of the computer 602 can communicate using a system bus 603. In some implementations, any or all of the components of the computer 602, including hardware or software components, can interface with each other or with an interface 604 (or a combination of both) via the system bus 603. The interface can use an application programming interface (API) 612, a service layer 613, or a combination of an API 612 and a service layer 613. The API 612 can include specifications for routines, data structures, and object classes. The API 612 can be computer language independent or computer language dependent. The API 612 can refer to a complete interface, a single function, or a set of APIs.

[0059] The service layer 613 can provide software services to the computer 602 and other components (whether shown or not) communicatively coupled to the computer 602. The functionality of the computer 602 is accessible to all service consumers using this service layer. Software services such as those provided by the service layer 613 can provide reusable, defined functionality through defined interfaces. For example, the interfaces can be software written in JAVA, C++, or a language that provides data in Extensible Markup Language (XML) format. While illustrated as an integrated component of the computer 602, in alternative embodiments, the API 612 or the service layer 613 can be a standalone component relative to other components of the computer 602 and other components communicatively coupled to the computer 602. Furthermore, any or all portions of the API 612 or the service layer 613 can be implemented as a child module or sub-module of another software module, enterprise application, or hardware module without departing from the scope of this disclosure.

[0060] Computer 602 includes interface 604. While illustrated in FIG. 6 as a single interface 604, two or more interfaces 604 may be used depending on the particular needs, desires, or particular implementation of computer 602 and the described functionality. Interface 604 may be used by computer 602 to communicate with other systems (whether shown or not) connected to network 630 in a distributed environment. Generally, interface 604 may include or be implemented using logic encoded in software or hardware (or a combination of software and hardware) operable to communicate with network 630. More specifically, interface 604 may include software supporting one or more communication protocols associated with the communication. Thus, network 630 or interface hardware may be operable to communicate physical signals within and outside of the illustrated computer 602.

[0061] Computer 602 includes processor 605. While illustrated in Figure 6 as a single processor 605, two or more processors 605 may be used according to the particular needs, desires, or particular implementation and described functionality of computer 602. Generally, processor 605 can execute instructions and manipulate data to perform the operations of computer 602, including operations using algorithms, methods, functions, processes, flows, and procedures as described in this disclosure.

[0062] Computer 602 also includes a database 606 that can maintain data for computer 602 and other components (whether shown or not) connected to network 630. For example, database 606 can be an in-memory, traditional, or other database that stores data consistent with the present disclosure. In some embodiments, database 606 can be a combination of two or more different database types (e.g., a hybrid in-memory and traditional database) according to the particular needs, desires, or particular implementation of computer 602 and the described functionality. While depicted in FIG. 6 as a single database 606, two or more databases (of the same, different, or a combination of types) can be used according to the particular needs, desires, or particular implementation of computer 602 and the described functionality. While database 606 is depicted as an internal component of computer 602, in alternative embodiments, database 606 can be external to computer 602.

[0063] Computer 602 also includes memory 607, which can hold data for computer 602 or a combination of components (whether shown or not) connected to network 630. Memory 607 can store any data consistent with the present disclosure. In some implementations, memory 607 can be a combination of two or more different types of memory (e.g., a combination of semiconductor and magnetic storage) according to the particular needs, desires, or specific implementation of computer 602 and the described functionality. While illustrated in FIG. 6 as a single memory 607, two or more memories 607 (of the same, different, or a combination of types) can be used according to the particular needs, desires, or specific implementation of computer 602 and the described functionality. While memory 607 is illustrated as an internal component of computer 602, in alternative implementations, memory 607 can be external to computer 602.

[0064] Application 608 may be an algorithmic software engine that provides functionality according to the particular needs, desires, or particular implementation of computer 602 and the described functionality. For example, application 608 may function as one or more components, modules, or applications. Moreover, while illustrated as a single application 608, application 608 may be implemented as multiple applications 608 on computer 602. Moreover, while illustrated as internal to computer 602, application 608 may be external to computer 602 in alternative embodiments.

[0065] Computer 602 may also include a power supply 614. Power supply 614 may include a rechargeable or non-rechargeable battery, which may be configured to be either user-replaceable or non-user-replaceable. In some implementations, power supply 614 may include power conversion and management circuitry, including recharge, standby, and power management functions. In some implementations, power supply 614 may include a power plug that allows computer 602 to be plugged into a wall outlet or power source, for example, to power computer 602 or to recharge a rechargeable battery.

[0066] There can be any number of computers 602 associated with or external to the computer system including computer 602, with each computer 602 communicating via network 630. Furthermore, the terms "client," "user," and other appropriate terms may be used interchangeably where appropriate without departing from the scope of this disclosure. Furthermore, this disclosure contemplates that many users may use one computer 602, and that one user may use multiple computers 602.

[0067] Implementations of the subject matter and functional operations described herein can be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, in computer hardware containing the structures disclosed herein and their structural equivalents, or in one or more combinations thereof. Software implementations of the described subject matter can be implemented as one or more computer programs. Each computer program can comprise one or more modules of computer program instructions encoded on a tangible, non-transitory computer-readable computer storage medium for execution by or to control the operation of a data processing apparatus. Alternatively, or additionally, the program instructions can be encoded within or on an artificially generated propagated signal. For example, the signal can be a machine-generated electrical, optical, or electromagnetic signal generated to encode information for transmission to a receiver apparatus suitable for execution by a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of computer storage media.

[0068] The terms “data processing device,” “computer,” and “electronic computing device” (or equivalents as understood by those skilled in the art) refer to data processing hardware. For example, a data processing device can encompass any type of apparatus, device, and machine for processing data, including, by way of example, a programmable processor, computer, or multiple processors or computers. An apparatus can also include special-purpose logic circuitry, including, for example, a central processing unit (CPU), a field programmable gate array (FPGA), or an application-specific integrated circuit (ASIC). In some embodiments, a data processing device or special-purpose logic circuitry (or a combination of data processing devices or special-purpose logic circuitry) can be hardware-based or software-based (or a combination of both hardware- and software-based). The apparatus can optionally include code that creates an execution environment for a computer program, such as processor firmware, a protocol stack, a database management system, an operating system, or code that constitutes a combination of the execution environment. The present disclosure contemplates the use of a data processing device with or without a conventional operating system, such as LINUX, UNIX, WINDOWS, MACOS, ANDROID, or IOS.

[0069] A computer program, which may also be referred to or described as a program, software, software application, module, software module, script, or code, may be written in any form of programming language. Programming languages ​​may include, for example, compiled, interpreted, declarative, or procedural languages. A program may be arranged in any form, including as a stand-alone program, a module, a component, a subroutine, or a unit for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data, such as one or more scripts stored in a markup language document, in a single file dedicated to the program, or in multiple cooperating files that store one or more modules, subprograms, or portions of code. A computer program may be deployed to run on one computer or on multiple computers, for example, located at one site or distributed across multiple sites interconnected by a communications network. Although portions of the programs shown in the various figures may be depicted as individual modules that implement various structures and functions through various objects, methods, or processes, the programs may instead include several sub-modules, third-party services, components, and libraries. Conversely, the structures and functions of various components may be combined into a single component where appropriate. The thresholds used to make the computational determination may be determined statically, dynamically, or both statically and dynamically.

[0070] The methods, processes, or logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The methods, processes, or logic flows may also be performed by, and an apparatus may be implemented as, special purpose logic circuitry, such as a CPU, FPGA, or ASIC.

[0071] A computer suitable for running a computer program can be based on one or more general-purpose and special-purpose microprocessors and other types of CPUs. Elements of a computer are a CPU for executing or performing instructions and one or more memory devices for storing instructions and data. Generally, a CPU can receive instructions and data from (and write data to) memory. A computer can also include, or be operatively coupled to, one or more mass storage devices for storing data. In some embodiments, a computer can receive data from and transfer data to a mass storage device, including, for example, a magnetic disk, a magneto-optical disk, or an optical disk. Furthermore, a computer can be incorporated into another device, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive.

[0072] Computer-readable media (transient or non-transient, as appropriate) suitable for storing computer program instructions and data can include all forms of permanent / non-permanent and volatile / non-volatile memory, media, and memory devices. Computer-readable media can include, for example, semiconductor memory devices such as random access memory (RAM), read-only memory (ROM), phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory devices, etc. Computer-readable media can also include, for example, magnetic devices such as tapes, cartridges, cassettes, and internal / removable disks. Computer-readable media can also include magneto-optical disks and optical memory devices, as well as technologies including, for example, digital video disks (DVDs), CD-ROMs, DVD+ / -Rs, DVD-RAMs, DVD-ROMs, HD-DVDs, and Blu-ray discs. The memory may store a variety of objects or data, including caches, classes, frameworks, applications, modules, backup data, jobs, web pages, web page templates, data structures, database tables, repositories, and dynamic information. The types of objects and data stored in the memory may include parameters, variables, algorithms, instructions, rules, constraints, and references. Additionally, the memory may include logs, policies, security or access data, and report files. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0073] Implementations of the subject matter described in this disclosure can be implemented on a computer having a display device for providing user interaction, including displaying information to (and receiving input from) a user. Types of display devices can include, for example, cathode ray tubes (CRTs), liquid crystal displays (LCDs), light-emitting diodes (LEDs), and plasma monitors. Display devices can include keyboards and pointing devices, including, for example, mice, trackballs, or trackpads. User input can also be provided to a computer through the use of a touchscreen, such as a pressure-sensitive tablet computer surface or a multi-touch screen using capacitive or electrical sensing. Other types of devices can be used to provide user interaction, including receiving user feedback, including, for example, sensory feedback, including visual feedback, auditory feedback, or haptic feedback. Input from a user can be received in the form of acoustic, voice, or tactile input. Additionally, a computer can interact with a user by sending documents to and receiving documents from a device used by the user. For example, a computer can send a web page to a web browser on a user's client device in response to a request received from the web browser.

[0074] The term "graphical user interface" or "GUI," in the singular or plural, can be used to describe one or more graphical user interfaces and each display of a particular graphical user interface. Thus, a GUI can refer to any graphical user interface, including, but not limited to, a web browser, a touch screen, or a command line interface (CLI), that processes information and efficiently presents the information results to a user. Generally, a GUI can include multiple user interface (UI) elements, some or all of which are associated with a web browser, such as interactive fields, pull-down lists, buttons, etc. These and other UI elements can relate to or represent the functionality of a web browser.

[0075] Implementations of the subject matter described in this specification can be implemented in a computing system that includes a back-end component (e.g., as a data server) or includes a middleware component (e.g., an application server). Additionally, a computing system can include a front-end component, such as a client computer, that has a graphical user interface or a web browser, or both, through which a user can interact with the computer. The components of the system can be interconnected by any form or medium of wired or wireless digital data communication (or a combination of data communication) in a communications network. Examples of communications networks include a local area network (LAN), a radio access network (RAN), a metropolitan area network (MAN), a wide area network (WAN), WiMAX (Worldwide Interoperability for Microwave Access), a wireless local area network (WLAN) (e.g., using 802.11a / b / g / n or 802.20 or a combination of protocols), all or a portion of the Internet, or any other communications system or systems (or combinations of communications networks) in one or more locations. The network may communicate, for example, Internet Protocol (IP) packets, Frame Relay frames, Asynchronous Transfer Mode (ATM) cells, voice, video, data, or a combination of communication types between network addresses.

[0076] A computing system may include clients and servers. Clients and servers may generally be remote from each other and typically interact through a communication network. The relationship of client and server may arise by virtue of computer programs running on the respective computers having a client-server relationship.

[0077] A cluster file system can be any file system type that is accessible for reads and updates from multiple servers. Locking or consistency tracking may not be necessary for an exchange file system, as locking can be done at the application layer. Furthermore, Unicode data files can be different from non-Unicode data files.

[0078] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular implementations. Certain features described herein in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented in multiple implementations, separately, or in any suitable subcombination. Furthermore, while the foregoing features may be described as operating in a particular combination and may initially be claimed as such, one or more features from a claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to subcombinations or variations of the subcombination.

[0079] In the foregoing description, embodiments of the present invention have been described with reference to numerous specific details that may vary from embodiment to embodiment. Therefore, the description and drawings should be regarded in an illustrative, rather than a limiting sense. The sole and exclusive indication of the scope of the present invention, and what applicants intend to be the scope of the present invention, is the set of claims originating from this application, in the specific form from such claims, including any subsequent amendments, literal and equivalent scope. Any definitions expressly set forth herein for terms contained in such claims shall govern the meaning of such terms as used in the claims. Additionally, when the term "further comprising" is used in the foregoing description or in the claims that follow, what follows this phrase may be an additional step or entity, or a sub-step / sub-entity of a previously recited step or entity.

[0080] Test Results: Figure 7 shows an example of an image classified as a responder using the systems and methods described herein. As shown, the enhanced region contains an interpretable configuration of tumor cells bearing the drug's target antigen, facilitating the correct prediction of a responder.

[0081] Figure 8 shows an example of an image classified as a non-responder using the systems and methods described herein. As shown, the highlighted region contains tumor nests with antigen patterns associated with the non-responder classification. The use of a weighting (attention) mechanism allows a human medical professional to meaningfully interpret the results, thereby increasing the reliability of the machine learning model.

[0082] Specific embodiments of the subject matter have been described. Other implementations, modifications, and permutations of the described implementations are within the scope of the following claims, as will be apparent to those skilled in the art. Although operations are shown in a particular order in the drawings or in the claims, this should not be understood as requiring such operations to be performed in the particular order or sequential order shown to achieve desirable results, or as requiring that all illustrated operations be performed (although some operations may be considered optional). In certain situations, multitasking or parallel processing (or a combination of multitasking and parallel processing) may be advantageous and is performed where deemed appropriate.

[0083] Furthermore, the separation or integration of various system modules and components in the described implementations should not be understood as requiring such separation or integration in all implementations, and it should be understood that the described program components and systems may generally be integrated into a single software product or packaged into multiple software products.

[0084] Accordingly, the exemplary embodiments set forth above are not intended to be limiting or restrictive of this disclosure, and other changes, substitutions, and alterations are also possible without departing from the spirit and scope of this disclosure.

[0085] Furthermore, any claimed implementations are believed to be applicable to at least a computer-implemented method; a non-transitory computer-readable medium storing computer-readable instructions for performing the computer-implemented method; and a computer system including computer memory operably coupled to a hardware processor, the hardware processor configured to execute the computer-implemented method or instructions stored on the non-transitory computer-readable medium.

[0086] Although several embodiments of these systems and methods have been described, it will nevertheless be understood that various modifications can be made without departing from the spirit and scope of the disclosure.

Claims

1. A system comprising: The system comprises: one or more computers; one or more storage devices communicatively coupled to said one or more computers; Including, the one or more storage devices store instructions; The instructions, when executed by the one or more computers, cause the one or more computers to perform actions; The operations are: receiving spatially arranged image data representing a biometric image of a patient; generating spatially arranged image tile data representing a plurality of image tiles, each image tile of the plurality of image tiles including a distinct portion of the biometric image; processing the spatially arranged image tile data using an artificial neural network according to training set values ​​of parameters of the artificial neural network; Including, The artificial neural network: generating, with the artificial neural network, a response prediction for each of the plurality of image tiles, the response prediction characterizing whether the patient will respond to the chemical; generating, by the artificial neural network, for each of the plurality of image tiles, a response weight indicative of the predictive power of the image tile in predicting whether a patient will respond to a chemical; generating a predicted response of the patient to the chemical by combining the response predictions for the plurality of image tiles with the response weights for the plurality of image tiles; The system performs the operations including:

2. The artificial neural network comprises at least one convolutional layer and at least one The system of claim 1 , comprising two fully connected layers.

3. The system described in claim 1, wherein the biological image is an immunohistochemical image.

4. A method performed by one or more computers, comprising: The method comprises: receiving spatially arranged image data representing a biometric image of a patient; generating spatially arranged image tile data representing a plurality of image tiles, each image tile of the plurality of image tiles including a distinct portion of the biometric image; processing the spatially arranged image tile data using an artificial neural network according to training set values ​​of parameters of the artificial neural network; Including, The artificial neural network: generating, with the artificial neural network, a response prediction for each of the plurality of image tiles, the response prediction characterizing whether the patient will respond to the chemical; generating, by the artificial neural network, for each of the plurality of image tiles, a response weight indicative of the predictive power of the image tile in predicting whether a patient will respond to a chemical; generating a predicted response of the patient to the chemical by combining the response predictions for the plurality of image tiles with the response weights for the plurality of image tiles; identifying a subset of the plurality of image tiles having the highest response weights based on the response weights as key image tiles that contributed most significantly to generating the predicted response; outputting (i) a predicted response of the patient to said chemical substance, and (ii) a visual representation highlighting key image tiles in said biometric image; The method performs the operations including:

5. The method described in claim 4, wherein the biological image is an immunohistochemical image.

6. A non-transitory computer storage medium encoded with computer program instructions that, when executed by one or more computers, cause the one or more computers to: receiving spatially arranged image data representing a biometric image of a patient; generating spatially arranged image tile data representing a plurality of image tiles, each image tile of the plurality of image tiles including a distinct portion of the biometric image; processing the spatially arranged image tile data using an artificial neural network according to training set values ​​of parameters of the artificial neural network; Execute an operation including The artificial neural network comprises: generating, with the artificial neural network, a response prediction for each of the plurality of image tiles, the response prediction characterizing whether the patient will respond to the chemical; generating, by the artificial neural network, for each of the plurality of image tiles, a response weight indicative of the predictive power of the image tile in predicting whether a patient will respond to a chemical; generating a predicted response of the patient to the chemical by combining the response predictions for the plurality of image tiles with the response weights for the plurality of image tiles; identifying a subset of the plurality of image tiles having the highest response weights based on the response weights as key image tiles that contributed most significantly to generating the predicted response; outputting (i) a predicted response of the patient to said chemical substance, and (ii) a visual representation highlighting key image tiles in said biometric image; The non-transitory computer storage medium, which performs operations including:

7. The non-transitory computer storage medium of claim 6, wherein the biological image is an immunohistochemical image.

8. Based on the response weights, identifying a subset of the plurality of image tiles having the highest response weights as key image tiles that contributed most significantly to generating the predicted response; outputting (i) a predicted response of the patient to said chemical substance, and (ii) a visual representation highlighting key image tiles in said biometric image; The system of claim 1 further comprising:

9. Generating a predicted patient response to the chemical by combining the response predictions for the plurality of image tiles and response weights for the plurality of image tiles includes: generating, for an image of the plurality of image tiles, a tile-level response prediction for the image tile as a product of a response prediction for the image tile and a response weight for the image tile; generating the predicted response by aggregating tile-level response predictions for the plurality of image tiles; The system of claim 1 , comprising:

10. The system described in claim 8, wherein the predicted response defines the resulting cardiovascular risk from patients receiving tumor drug therapy including a chemical.

11. The system described in claim 8, wherein the predicted response defines a change in tumor size predicted to occur by administering a chemical to a patient.

12. The system described in claim 8, wherein the predicted response defines the predicted toxicity of the chemical to the patient.

13. The system described in claim 8, wherein the predicted response characterizes whether the patient will experience an adverse reaction to being administered a chemical substance.

14. The artificial neural network generates, for each of the plurality of image tiles, a response weight indicative of the predictive power of the image tile in predicting whether a patient will respond to a chemical; The system of claim 8 , further comprising generating the response weights under the constraint that a sum of the response weights across the plurality of image tiles is equal to a predetermined value.

15. The system of claim 14, wherein the predetermined value is 1.

16. The system of claim 8, wherein the chemical comprises a pharmaceutical.

17. Prior to generating the predicted response for a patient: receiving a plurality of input biometric images of a patient; classifying a subset of the plurality of input biological images as abnormal biological images displaying abnormal tissue using a second artificial neural network; further comprising the predicted response for the patient is generated based on only a subset of the plurality of input biometric images classified as the abnormal biometric images that display abnormal tissue. The system of claim 8.

18. The system described in claim 17, wherein the abnormal tissue includes cancerous tissue.

19. Classifying a subset of the plurality of input biometric images as being abnormal biometric images displaying abnormal biometric tissue using a second artificial neural network includes, for each of the plurality of input biometric images: generating, by a second artificial neural network, an abnormality feature for each of a plurality of input image tiles included in the input biometric image, the abnormality feature indicating whether the input image tile exhibits abnormal tissue; classifying the input biometric image as being the abnormal biometric image or not based on the abnormality characteristics; 20. The system of claim 17, comprising:

20. Classifying the input biometric image as being the abnormal biometric image or not based on the abnormality characteristics comprises:

20. The system of claim 19, further comprising: classifying the input biometric image as the abnormal biometric image if the abnormality signatures for the input image tiles of the input biometric image indicate that at least one input image tile exhibits abnormal tissue.

Images