Predicting patient response to a chemical

Abstract

In one aspect, a data processing system includes a computer-readable memory containing computer-executable instructions; and at least one processor configured to execute executable logic, the executable logic including at least one artificial neural network trained to predict one or more reactions to a chemical substance by identifying one or more discrete biological tissue components in a biological image. While the at least one processor is executing the computer-executable instructions, the at least one processor is configured to perform operations including: receiving spatially arranged image data representative of a biological image of a patient; generating spatially arranged tile data representative of a plurality of image tiles; processing the spatially arranged tile data through the one or more data structures storing one or more portions of executable logic included in the artificial neural network to predict one or more reactions of the patient.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims priority to Application No. EP 20305030.7 filed January 16, 2020 and U.S. Provisional Application No. 62 / 886,199 filed August 13, 2019. The disclosures of which are incorporated herein by reference. TECHNICAL FIELD

[0003] The present disclosure relates generally to systems and methods of predicting a patient’s response to a compound, such as a pharmaceutical. BACKGROUND

[0004] Clinical trials are generally conducted to collect data on the safety and effectiveness of a drug. Generally, these trials involve one or more phases to determine whether a drug can be sold on the consumer market. For example, a clinical trial can include three phases. In the first phase, a drug is tested on a relatively small number of paid volunteers (e.g., 20 to 100 volunteers) to determine the drug’s effects, including absorption, metabolism, excretion, etc. This phase can take several months to complete, and approximately 70% of experimental drugs pass the first phase. In the second phase, the experimental drug is tested on hundreds of patients who meet one or more inclusion criteria. One group of patients receives the experimental drug, while another group receives a placebo or standard treatment. Approximately one-third of experimental drugs that complete the first and second phases of testing. In the third phase, the drug is tested on hundreds to thousands (or more) of patients. This phase is often the most expensive of all, and approximately 70% of drugs that enter the third phase can successfully complete it. SUMMARY

[0005] In at least one aspect of the present disclosure, a data processing system is provided. The data processing system includes

[0006] A computer-readable storage medium includes computer-executable instructions; and at least one processor configured to execute executable logic, the executable logic including at least one artificial neural network trained to predict one or more responses to a chemical substance by recognizing one or more discrete biological tissue components in a biological image. While executing the computer-executable instructions, the at least one processor is configured to perform one or more operations. The one or more operations include receiving spatially arranged image data representing a biological image of a patient. The one or more operations include generating spatially arranged image patch data representing a plurality of image patches, each of which includes a discrete portion of the biological image. The one or more operations include processing the spatially arranged image patch data through one or more data structures to predict one or more responses of the patient by recognizing one or more pixels representing the location of one or more discrete biological tissue components in the image patch for each image patch, the one or more data structures storing one or more portions of the executable logic included in the artificial neural network.

[0007] The one or more operations may include generating preprocessed, spatially arranged image patch data, each representing a preprocessed image patch. For each image patch, generating the preprocessed, spatially arranged image patch data may include: identifying one or more pixels in the image patch that represent one or more biological tissue locations, and performing color normalization on the one or more biological tissue locations. The spatially arranged image patch data processed by the one or more data structures storing one or more portions of executable logic included in the artificial neural network may include the preprocessed, spatially arranged image patch data.

[0008] Artificial neural networks can include convolutional neural networks.

[0009] Predicting a patient's one or more responses may include assigning a weighted value to each image patch. The assigned weighted value for each image patch may be based on the predictive power of the discrete biological tissue components of that image patch.

[0010] In at least one aspect, a data processing system is provided. A data processing system includes a computer-readable memory containing computer-executable instructions. The data processing system includes at least one processor configured to execute executable logic, the executable logic including at least one artificial neural network trained to predict one or more reactions to a chemical substance by identifying one or more discrete biological tissue components in a biological image. The at least one processor is configured to perform one or more operations while the at least one processor is executing the computer-executable instructions. The one or more operations include receiving spatially arranged image data representative of a biological image of a patient. The one or more operations include processing the spatially arranged image data through the one or more data structures storing one or more portions of executable logic included in the artificial neural network to predict one or more reactions of the patient by identifying one or more pixels representative of one or more discrete biological tissue component locations of the patient. Processing the spatially arranged data includes selecting a first portion of the spatially arranged image data. Processing the spatially arranged data includes processing the first portion to identify one or more pixels of the first portion representative of one or more locations of a discrete biological tissue component corresponding to the first portion. Processing the spatially arranged data includes selecting at least one subsequent portion of the spatially arranged image data. Processing the spatially arranged data includes processing the at least one subsequent portion to identify one or more pixels in the at least one subsequent portion representative of one or more discrete biological tissue component locations corresponding to the at least one subsequent portion.

[0011] The biological image can include an immunohistochemistry image. The artificial neural network can include a deep recurrent attention model. The one or more reactions can include an amount of reduction in tumor size.

[0012] The one or more operations can include generating preprocessed spatially arranged image data representative of a preprocessed biological image. Generating the preprocessed spatially arranged image data can include identifying one or more pixels in the biological image representative of one or more biological tissue locations and color normalizing the one or more biological tissue locations. The spatially arranged image data processed through the one or more data structures storing one or more portions of executable logic included in the artificial neural network can include the preprocessed spatially arranged image data.

[0013] These and other aspects, features, and implementations can be expressed as methods, apparatus, systems, components, program products, means or steps for performing functions, and in other ways.

[0014] Implementations of the present disclosure can provide one or more of the following advantages. Image processing and machine learning techniques can be used to process image data to predict patient response to a drug, such that prediction accuracy is improved, computational efficiency is improved, and / or computational capacity requirements are reduced as compared to traditional techniques. More variables can be considered for the prediction as compared to traditional techniques, which can improve the accuracy of the prediction.

[0015] These and other aspects, features, and implementations can be expressed as methods, apparatus, systems, components, program products, means or steps for performing functions, and in other ways. BRIEF DESCRIPTION OF DRAWINGS

[0016] FIG. 1 is a diagram illustrating an example of a data processing system.

[0017] FIG. 2 is a flow diagram illustrating an example architecture of a data processing system.

[0018] FIG. 3 is a flow diagram illustrating an example architecture of a data processing system.

[0019] FIG. 4 is a flow diagram illustrating an example method for predicting patient response to a compound.

[0020] FIG. 5 is a flow diagram illustrating an example method for predicting patient response to a compound.

[0021] FIG. 6 is a block diagram of an example computer system for providing computational functionality associated with the algorithms, methods, functions, processes, flows, and steps described in the present disclosure. DETAILED DESCRIPTION

[0022] For clinical trials involving a given drug, it can be important to select patients who can benefit from a treatment with controllable side effects, especially in the field of life-threatening diseases, such as oncology. Due to recent advances in medical imaging technology, medical or biological images (e.g., immunohistochemistry images) can be used to predict patient outcomes for investigational treatments. However, traditional patient outcome prediction techniques typically extract only a few features from biological images, such as a proportion score and a histochemistry score (sometimes referred to as an “H-score”). As a result, the final patient response prediction accuracy can be between 20-45%. Moreover, using traditional machine learning techniques to predict patient response to a given drug is computationally infeasible because biological images can have dimensions of 50,000 pixels by 40,000 pixels (or larger), sizes of 2 terabytes (or larger). That is, machine learning models can be required to estimate billions (or more) of parameters for these sized images.

[0023] Implementations of the present disclosure provide systems and methods for predicting patient response that can be used to mitigate some or all of the foregoing shortcomings. The systems and methods described in the present disclosure can implement image processing techniques and machine learning techniques such that, when compared to traditional techniques, image data representative of biological images can be processed in a more computationally efficient manner to predict patient response to a drug with higher accuracy. In some implementations, the systems and methods described in the present disclosure can receive a biological image of a patient and generate image tiles, where each image tile is representative of a discrete portion of the biological image. The data representative of each image tile can then be pre-processed to, for example, identify locations of biological tissue captured in the biological image and color-normalize those identified locations. The pre-processed image data can then be processed with an artificial neural network (ANN) that can identify discrete tissue component locations for each image tile, the artificial neural network having learned to predict patient response to a given drug. That is, based on the identified discrete tissue components, the ANN can identify higher-level features from the image that can impact patient response prediction. For example, the ANN can learn to associate a partial staining pattern of a targeted protein on a tumor nest membrane with a poor patient response because the active pharmaceutical ingredient can not recognize the targeted protein to attack the tumor nest. Examples of patient response can include efficacy response (e.g., a reduction / change in size of a cancerous tumor resulting from a patient receiving a tumor drug treatment regimen), safety response (e.g., adverse reactions, toxicity, and cardiovascular risks resulting from a patient receiving a tumor drug treatment regimen), or both.

[0024] Each discrete tissue component can be assigned a value based on learned predictive power (e.g., their efficacy in predicting patient response through the ANN) and the predicted response can include aggregating the values corresponding to all of the image tiles. In some implementations, the ANN can be used to process image data representative of the entire image, where the ANN is configured to process one discrete portion of the biological image at a time (sometimes referred to as “chunking”).

[0025] By color-normalizing the locations of biological tissue in the biological image, computational efficiency in processing the biological image data can be facilitated because, for example, the ANN can more easily identify the locations of biological tissue as compared to traditional techniques. Further, by processing the medical image data one discrete portion at a time (e.g., one chunk or one tile at a time) using the ANN, computational demand issues can be mitigated as compared to traditional techniques.

[0026] In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. It will be apparent, however, that the present disclosure can be practiced without some or all of these specific details. In other instances, well known structures and devices are shown in block diagram form in order to avoid unnecessarily occluding the present disclosure.

[0027] In the drawings, specific arrangements or orderings of illustrative elements can be shown as being present in the drawings or in example scenarios. However, the specific ordering of elements shown or described in any figure or example is not meant to imply that such orderings are strictly followed, unless claimed by the claimant. Further, any given element depicted as being present in one figure can be absent in another figure or example, and this issuance does not imply that a particular element must be present in any given situation or implementation.

[0028] Further, in the drawings, connection elements, such as lines or arrows, are used to illustrate connections, relationships or associations between elements or other items. The absence of any such connection elements does not imply that no connections, relationships or associations can exist.

[0029] Reference will now be made in detail to implementations, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various described implementations. However, it will be apparent to one of ordinary skill in the art that the various implementations described can be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the implementations.

[0030] Several features are described below, each of which can be used independently of the other features, or in any combination with other features. However, any single feature can not address any of the problems discussed above, or can only address one of the problems discussed above. Any feature described in this specification can not fully solve some of the problems discussed above. Although provided with headings, data related to a specific heading but not found in the section having that heading can also be found elsewhere in the specification.

[0031] FIG. 1 illustrates an example of a data processing system 100. Generally, the data processing system is configured to process image data representative of a patient biological image to predict a patient response (e.g., a reduction in size of a cancerous tumor) to a given chemical substance (e.g., a drug). The system 100 includes a computer processor 110. The computer processor 110 includes a computer readable memory 111 and computer readable instructions 112. The system 100 also includes a machine learning system 150. The machine learning system 150 includes a machine learning model 120. The machine learning model 120 can be separate from or integrated with the computer processor 110.

[0032] The computer readable medium 111 (or computer readable memory) can include any data storage technology type which is suitable for a local technical environment, including but not limited to semiconductor based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, removable memory, disc memory, flash memory, dynamic random access memory (DRAM), static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), and the like. In some implementations, the computer readable medium 111 includes a code segment having executable instructions.

[0033] In some implementations, the computer processor 110 includes a general purpose processor. In some implementations, the computer processor 110 includes a central processing unit (CPU). In some implementations, the computer processor 110 includes at least one application specific integrated circuit (ASIC). The computer processor 110 can also include a general purpose programmable microprocessor, a graphics processing unit, a special purpose programmable microprocessor, a digital signal processor (DSP), a programmable logic array (PLA), a field programmable gate array (FPGA), a special purpose electronic circuit, and the like, or a combination thereof. The computer processor 110 is configured to execute program code, such as the computer executable instructions 112, and is configured to execute executable logic including the machine learning model 120.

[0034] The computer processor 110 is configured to receive image data representative of a patient medical image. For example, the patient’s medical image can be an image of an immunohistochemical staining result, which describes a process of selectively recognizing a protein (e.g., an antigen) in cells of a biological tissue section by utilizing the principle of antibodies that specifically bind to antigens in the biological tissue. The image data can be obtained by various techniques, such as wireless communication with a database, fiber optic communication, USB, CD-ROM, and the like.

[0035] In some implementations, the computer processor 110 is configured to generate image patch data representative of a plurality of image patches, where each image patch includes a discrete portion of a biological image. A more detailed example of generating image patch data is discussed later with reference to FIG. 2. In some implementations, the computer processor 110 is configured to pre-process the image data prior to transmitting the image data to the machine learning model 120. In some implementations, pre-processing the image data includes identifying one or more pixel locations of the image data that correspond to biological tissue, and color normalizing those identified locations. Color normalizing can refer to a process of normalizing different color schemes to a standard color scheme, and can increase contrast between the captured biological tissue / tumor and the image background for more effective signal identification. For example, the computer processor 110 can associate certain pixel locations in the image data that have values (e.g., color values, intensity values, etc.) that correspond to biological tissue, and color normalize those pixel locations. The association can be pre-programmed or learned through one or more machine learning techniques (e.g., Bayesian techniques, neural network techniques, etc.).

[0036] The machine learning model 120 is capable of processing the image data (after it has been pre-processed by the computer processor 110, after it has been converted into image patch data, or both, in some implementations) to predict a patient response corresponding to a particular drug. For example, for a given tumor treatment drug regimen, the machine learning model 120 can predict an amount of reduction in a cancerous tumor size based on identifying and analyzing one or more pixel locations of the image data representative of discrete biological tissue components. In some implementations, predicting a patient response includes assigning a value to the identified and analyzed one or more pixel locations representative of discrete biological tissue components based on learned associations of the discrete biological tissue components to patient responses. Predicting a patient response is discussed in more detail later with reference to FIGS. 2-5.

[0037] The machine learning system 150 is capable of applying machine learning techniques to train the machine learning model 120. As part of the training of the machine learning model 120, the machine learning system 150 forms a training set of input data by identifying a positive training set of input data items that have been determined to have the attribute, and in some embodiments, a negative training set of input data items that lack the attribute.

[0038] The machine learning system 150 extracts feature values from the input data of the training set, which are variables that are believed to be potentially relevant to whether an input data item has the relevant attribute. An ordered list of features of the input data is referred to herein as a feature vector of the input data. In one embodiment, the machine learning system 150 applies dimensionality reduction (e.g., by linear discriminant analysis (LDA), principal component analysis (PCA), etc.) to reduce the amount of data in the feature vector of the input data to a smaller, more representative set of data.

[0039] In some implementations, the machine learning system 150 trains the machine learning model 120 using supervised machine learning with the feature vectors of the positive training set and the negative training set as input. Different machine learning techniques can be used in different embodiments, such as linear support vector machines (linear SVM), boosting against other algorithms (e.g., AdaBoost), neural networks, logistic regression, Naive Bayes, memory-based learning, random forests, bagged trees, decision trees, boosted trees, or boosted stumps. When applied to a feature vector extracted from an input data item, the machine learning model 120 outputs an indication of whether the input data item has the attribute in question, such as a Boolean yes / no estimate, or a scalar value representing a probability.

[0040] In some embodiments, a validation set is formed from additional input data other than those in the training set, which have been determined to have or lack the attribute in question. The machine learning system 150 applies the trained machine learning model 120 to the data of the validation set to quantify the accuracy of the machine learning model 120. Common metrics applied in accuracy metrics include Precision = TP / (TP+FP) and Recall = TP / (TP+FN), where Precision is how many of the total that the machine learning model predicted correctly (TP or true positives) out of the total that it predicted (TP+FP or false positives), and Recall is how many of the total that did have the attribute in question (TP) that the machine learning model predicted correctly out of the total that did have the attribute in question (TP+FN or false negatives). The F-score (F-score = 2*PR / (P+R)) unifies Precision and Recall into a single measure. In one embodiment, the machine learning module iteratively re-trains the machine learning model until the occurrence of a stopping condition, such as an accuracy measure indicating that the model is sufficiently accurate, or a number of rounds of training has been performed.

[0041] In some implementations, the machine learning model 120 is a convolutional neural network (CNN). The CNN can be configured based on the assumption that the input to the CNN corresponds to image pixel data of an image or other data that includes features of a plurality of spatial locations. For example, the plurality of groups of input can form a multi-dimensional data structure, such as a tensor, that represents color features of an example digital image (e.g., a biological image of biological tissue). In some implementations, the input to the CNN corresponds to various other types of data, such as data obtained from different devices and sensors of a vehicle, point cloud data, audio data that includes particular features or raw audio for each of a plurality of time steps, or various types of one- or multi-dimensional data. Convolutional layers of the CNN can process the input to transform image features represented by the input of the data structure. For example, the input is processed by performing dot product operations using input data along a given dimension of the data structure and a set of parameters of the convolutional layer.

[0042] Performing computations of a convolutional layer can include applying one or more groups of kernels to input portions in the data structure. The manner in which the CNN performs computations can be based on particular properties of each layer of an example multi-layer neural network or deep neural network that supports deep neural network workloads. The deep neural network can include one or more convolutional towers (or layers) as well as other computational layers. In particular, these convolutional towers often account for a majority of the inference computations performed, such as for computer vision applications. A convolutional layer of the CNN can have a three-dimensional arrangement of artificial neurons, including a width dimension, a height dimension, and a depth dimension. The depth dimension corresponds to a third dimension of an input or activation volume and can represent respective color channels of an image. For example, an input image can form an input volume (e.g., activation) of data and the volume has dimensions of 32x32x3 (width, height, depth, respectively). The depth dimension of 3 can correspond to RGB color channels of red (R), green (G), and blue (B).

[0043] Generally, the layers of a CNN are configured to convert a three-dimensional input volume (input) into a multi-dimensional output volume of neuron activations (activation). For example, a 32x32x3 3D input structure holds raw pixel values of an example image, in this case, the example image has a width of 32, a height of 32, and has three color channels, i.e., R, G, and B. The convolutional layer computation of the CNN of the machine learning model 120 can connect the output of a neuron to a local region in the input volume. Each neuron in the convolutional layer can be connected to a local region in the input volume only spatially, but to the entire depth of the input volume (e.g., all color channels). For a set of neurons of the convolutional layer, the layer computes the dot product between the parameters (weights) of the neurons and the particular region in the input volume to which the neurons are connected. This computation can result in a volume such as 32x32x12, where 12 corresponds to the number of kernels used for the computation. The connection of the neuron to the region input can have a spatial extent along the depth axis equal to the depth of the input volume. The spatial extent corresponds to the spatial dimensions of the kernel (e.g., x and y dimensions).

[0044] A set of kernels can have a spatial characteristic that includes a width and a height and extends through the depth of the input volume. Each set of kernels of the layer is applied to a set or sets of inputs provided to the layer. That is, for each kernel or set of kernels, the machine learning model 120 can overlay the kernel, which can be represented multi-dimensionally, on a first portion of the layer input, which can be represented multi-dimensionally (e.g., forming an input volume or input tensor). For example, a set of kernels for a first layer of a CNN can have a size of 5x5x3x16, corresponding to a width of 5 pixels, a height of 5 pixels, a depth of 3 corresponding to the color channels of the input volume to which the kernel is applied, and an output dimension of 16 corresponding to a number of output channels. In this case, the set of kernels includes 16 kernels, such that the output of the convolution has a depth dimension of 16.

[0045] The machine learning model 120 can then compute the dot product from the overlapping elements. For example, the machine learning model 120 can convolve (or slide) each kernel over the width and height of the input volume and compute the dot product between the entries of the kernel and the input of the location or region of the image. Each output value in the convolution output is the result of a dot product between the kernel and some set of inputs from the example input tensor. The dot product can result in a convolution output that corresponds to a single layer input, e.g., an activation element having a top-left position in the overlapping multi-dimensional space. As described above, the neurons of a convolutional layer can be connected to a region of an input volume that includes multiple inputs. The machine learning model 120 can convolve each kernel over each input of the input volume. The machine learning model 120 can perform this convolution operation by, for example, moving (or sliding) each kernel over each input in the region.

[0046] The machine learning model 120 can move each kernel over the input of a region based on the stride value for a given convolutional layer. For example, when the stride is set to 1, the machine learning model 120 can move the kernel over a region of one pixel (or input) at a time. Similarly, when the stride is 2, the machine learning model 120 can move the kernel over a region of two pixels at a time. Thus, the kernel can be moved based on the stride value for the layer, and the machine learning model 120 can repeat this process until the input of the region has a corresponding dot product. Related to the stride value is the skip value. The skip value can identify one or more groups of inputs (2x2) in the input volume region that are skipped when the inputs are loaded for processing at a neural network layer. In some implementations, the input pixel volume of an image can be “padded” with zeros, for example, around the border regions of the image. This zero padding is used to control the spatial size of the output volume.

[0047] As previously described, the convolutional layers of a CNN are configured to convert a three-dimensional input volume (input of a region) into a multi-dimensional output volume of neuron activations. For example, when a kernel is convolved over the width and height of an input volume, the machine learning model 120 can produce a multi-dimensional activation map that includes the results of the convolution kernel at one or more spatial locations based on the stride value. In some cases, increasing the stride value results in a smaller amount of activation output in space. In some implementations, activations can be applied to the output of the convolution before the output is sent to a subsequent layer of the CNN.

[0048] An example convolutional layer can have one or more control parameters that represent properties of the layer. For example, the control parameters can include the number of kernels K, the spatial extent of the kernels F, the stride (or skip) S, and the amount of zero padding P. The values of these parameters, the input to the layer, and the parameter values of the kernels of the layer determine the computations that occur on the layer and the size of the output volume of the layer. In some implementations, the spatial size of the output volume is computed as a function of the input volume size W using the formula (W - F + 2P) / S + 1. For example, an input tensor can represent a pixel input volume of size [227x227x3]. A convolutional layer of a CNN can have a spatial extent value of F = 11, a stride value of S = 4, and no zero padding (P = 0). Using the above formula and a layer kernel amount of K = 96, the machine learning model 120 performs computations on the layer resulting in a convolutional layer output volume of size [55x55x96], where 55 is derived from [(227 - 11 + 0) / 4 + 1 = 55].

[0049] Computations (e.g., dot product computations) of convolutional layers or other layers of a CNN include performing mathematical operations, such as multiplication and addition, using computational units of hardware circuitry of the machine learning model 120. When performing computations for layers of a neural network, the design of the hardware circuitry can cause the system to be limited in taking full advantage of the capabilities of the computational units of the circuitry. A more detailed example of an architecture of the system 100 with a machine learning model 120 including a CNN will be discussed later with reference to FIG. 2.

[0050] In some implementations, the machine learning model 120 includes a recurrent attention model (RAM). The RAM can process biological image data in a sequential manner, building a dynamic representation of the biological image. For example, at each step of a time step (t), the RAM can selectively focus on a given location in a patch of the image, which refers to a discrete portion of the image. The RAM can then extract features from the patch, update its internal state, and select the next patch to focus on. This process can repeat for a fixed number of steps, during which the RAM is able to incrementally combine the extracted features in a consistent manner. The general structure of the RAM can be defined by a number of multi-layer neural networks, where each multi-layer neural network is able to map some input vectors into output vectors. A more detailed example of an architecture of the system 100 with a machine learning model 120 including a RAM will be discussed later with reference to FIG. 3.

[0051] While this specification generally describes patients as human patients, implementations are not so limited. For example, a patient can refer to a non-human animal, a plant, or a human reproductive system.

[0052] FIG. 2 is a flowchart illustrating an architecture of a data processing system 200. The data processing system 200 can be substantially similar to the data processing system 100 described earlier with reference to FIG. 1. The data processing system 200 includes an image patch generation module 220, a pre-processing module 230, a feedback module 240, and a machine learning system 250. The modules 220, 230, 240 can be executed by, for example, the computer processor 110 of the data processing system 100 discussed earlier with reference to FIG. 1.

[0053] The image patch generation module 220 can receive image data representative of the biological image 210 and generate image patch data representative of a plurality of image patches 210a of the biological image 210. As shown, each of the plurality of image patches 210a includes a discrete portion of the biological image 210. Although the illustrated implementation shows six image patches 210a, the number of image patches can be more or less than six and can be selected based on computational efficiency, computational power, and computational accuracy considerations. For example, due to the heterogeneity of medical images (e.g., as seen in immunohistochemistry images of biopsy samples from cancer patients), the number of patches per image can range from a few patches to several thousand patches. The image patch data for each image patch 210a is transmitted to the pre-processing module 230. For each image patch, the pre-processing module 230 can generate pre-processed image patch data by identifying one or more pixel locations of the image patch data that correspond to biological tissue and color normalizing the identified locations, as previously discussed with reference to FIG. 1.

[0054] The pre-processed image patch data is transmitted to the machine learning system 250. As shown, the pre-processed image patch data is sequentially transmitted to the machine learning system 250, where pre-processed image patch data corresponding to a first image patch is transmitted to the machine learning system 250 at a first time, pre-processed image patch data corresponding to a second image patch is transmitted to the machine learning system 250 at a second time, and so on until pre-processed image patch data corresponding to all (or a portion of) the image patches has been received by the machine learning system 250.

[0055] In the illustrated implementation, the machine learning system 250 includes a CNN. The machine learning system 250 can identify, for each image patch, one or more pixel locations in the pre-processed image patch data that are representative of one or more discrete tissue components that predict a patient outcome for a given drug. The machine learning system 250 can assign a value to the one or more pixel locations, and can weight the assigned values based on learned predictive efficacy of the identified discrete tissue components. The machine learning system 250 can aggregate the weighted values (e.g., sum, average, etc.) across all image patches 210a to generate an aggregated weight value, and predict a patient response (e.g., amount of reduction in cancerous tumor) based on the aggregated weight value. For example, the machine learning system 250 can predict a patient outcome based on a learned association between the total weight value and the patient response. The predicted patient response can be transmitted to the feedback module 240, which can compare the predicted patient response to an observed patient response (e.g., observed experimental result), and generate an error value based on the comparison. The error value can be transmitted to the machine learning system 150, and the machine learning system can update its weights and biases in accordance with the error value. In some implementations, the feedback module 240 uses cross-validation techniques to validate the predicted outcome. For example, in the early stages of drug development, there can be medical images of a small number of patients. To assess the robustness of a model fitted on such a small dataset, a statistical method, cross-validation, can be utilized. For example, in a k-fold cross-validation process, the entire dataset can be randomly split into k approximately equal-sized subsets. Each time a subset is taken as the validation dataset, the rest is used as the training set, a model can be fitted, and its performance on the test set is recorded. Then, a different subset is taken as the validation set, and a new model is trained on the rest of the subsets. As the final result, each subset can have served as a validation set once, and the predicted outcomes of all k-fold validations are aggregated to obtain Precision, Recall, and so on. These aggregated results can more accurately measure the robustness of the method.

[0056] Although certain modules including the image patch generation module 220, the pre-processing module 230, and the feedback module 240 are described as performing certain aspects of the techniques described in this specification, in some implementations, some or all of the techniques can be performed by additional, fewer, or alternative modules.

[0057] FIG. 3 is a flowchart illustrating an example architecture of a data processing system 300. The data processing system 300 includes a pre-processing module 330, a feedback module 340, and a machine learning system 350.

[0058] The pre-processing module 330 is configured to receive image data representative of the biological image 310. In some implementations, the pre-processing module 330 is substantially similar to the pre-processing module 230 of the data processing system 200 discussed previously with reference to FIG. 2. Thus, the pre-processing module 330 is capable of identifying one or more pixel locations in the image data that represent biological tissue and color normalizing the identified one or more pixel locations to generate pre-processed image data. The pre-processed image data can then be transmitted to the machine learning system 350.

[0059] The machine learning system 350 is capable of processing the pre-processed image data to predict patient response to a given drug. In the illustrated implementation, the machine learning system 350 includes a RAM 350. The RAM 350 includes a patching module 351, a feature extraction module 352, a location module 353, and a prediction module 354.

[0060] In some implementations, the patching module 351 includes one or more convolutional layers (e.g., 3 convolutional layers).

[0061] In some implementations, the patching module 351 includes one or more max-pooling layers, which refer to layers that can perform a sample-based discretization process. In some implementations, the patching module 351 includes one or more fully connected layers. In some implementations, the patching module 351 receives location data for a respective patch in each of a series of consecutive time steps. As previously discussed, a patch corresponds to a discrete portion of the image 310. The patching module 351 generates patch data representative of each patch corresponding to the location data. For example, as illustrated, the patching module 351 generates a first patch 310a for a first time step. In some implementations, the patching module 351 randomly selects a first location for the first patch 310a. In some implementations, the patching module 351 selects a center of the image 310 as the first location for the first patch 310a.

[0062] The feature extraction module 352 receives patch data corresponding to the first patch 310a. The feature extraction module 352 is capable of identifying one or more pixel locations of the first patch 310a that correspond to one or more discrete tissue components of a patient and assigning a value to each of the one or more discrete tissue components based on learned associations between the one or more discrete tissue components and patient response. In some implementations, the feature extraction module 352 includes two or more stacked long short-term memory units, which describe a neural network architecture that includes feedback connections that process individual data points (e.g., images) and entire sequences of data (e.g., speech or video).

[0063] The position module 353 receives the assigned value and determines the next chunk position based on the assigned value. The position of the next chunk can be determined based on an optimization protocol learned by the position module 353 through a reinforcement learning process, where different protocols can be evaluated and the optimal protocol for assigning a position to a given next chunk can be decided: the position of the current chunk, the information extracted from the previous chunk, and maximizing a reward function, a portion of which can correspond to prediction accuracy. The position module 353 generates position data indicating the position of the next chunk to be processed.

[0064] The chunk module 353 receives the position data generated by the position module 353 and generates chunk data representing the second chunk 310b to be processed. The aforementioned process of processing chunks and selecting the next position continues until the final chunk 310c is processed by the feature extraction module 352. Each assigned value is transmitted to the prediction module 354. Once the prediction module 354 receives the assigned values corresponding to all chunks, the prediction module 354 predicts the patient response to a given drug by generating an aggregate value and predicting the patient response based on the learned association between the aggregate value and the patient response. Data representing the predicted patient response is transmitted to the feedback module 340, which compares the predicted patient outcome to the observed outcome and produces an error value. Data representing the error value is transmitted to the machine learning system 350, which updates its weights and biases accordingly. In some implementations, the feedback module 340 uses cross-validation techniques to validate the prediction results, as previously discussed with reference to FIG. 2.

[0065] Although specific modules including the pre-processing module 330, the chunk module 351, the feature extraction module 352, the position module 353, the prediction module 354, and the feedback module 340 are described as performing certain aspects of the techniques described in this specification, in some implementations, some or all of the techniques can be performed by additional, fewer, or alternative modules.

[0066] FIG. 4 is a flowchart illustrating an example method 400 for predicting a patient’s response to a compound. For purposes of illustration, the method 400 will be described as being performed by the data processing system 100 previously discussed with reference to FIG. 1. The method 400 includes receiving spatially arranged image data (block 410), generating spatially arranged chunk data (block 420), and processing the spatially arranged chunk data (block 430).

[0067] At block 410, the computer processor 110 receives spatially arranged image data representing a biological image of a patient. For example, as previously discussed with reference to FIG. 1, the biological image can be an immunohistochemistry image of a patient’s biological tissue.

[0068] At block 420, the computer processor 110 generates spatially arranged patch data representative of the image patches of the image. For example, as previously discussed with reference to FIGS. 1-2, each image patch can include a discrete portion of the image.

[0069] At block 430, the machine learning model 120 processes the spatially arranged patch data by one or more data structures storing one or more portions of executable logic included in the machine learning model 120 to predict one or more responses of the patient by identifying, for each image patch, one or more pixels in the image patch representative of a location of one or more discrete biological tissue components of the patient. In some implementations, the machine learning model 120 can assign a value to each identified location of a discrete biological tissue component. These values can be weighted based on, for example, the learned predictive power of the identified discrete biological tissue component. The values corresponding to all image patches can be aggregated (e.g., summed, averaged, etc.) to generate an aggregated value, and this value can be used to predict a response. Such a response can be a response to a particular compound, such as a drug. For example, the response can be an amount of reduction in size of a cancerous tumor due to the patient receiving a tumor drug treatment regimen.

[0070] As previously indicated with reference to FIGS. 1-2, prior to being processed by the machine learning model 120, the computer processor 110 can generate pre-processed patch data in which, for each image patch, the computer processor 110 can identify one or more pixels of the respective image patch data representative of biological tissue and color normalize the identified one or more pixels.

[0071] FIG. 5 is a flowchart illustrating an example method 500 for predicting a response of a patient to a compound. For purposes of illustration, the method 500 will be described as being performed by the system 100 previously discussed with reference to FIG. 1. The method 500 includes receiving spatially arranged image data (block 510) and processing the spatially arranged image data (block 520).

[0072] At block 510, the computer processor 110 receives spatially arranged image data representative of a biological image of a patient. For example, as previously discussed with reference to FIG. 1, the biological image can be an immunohistochemistry image of a biological tissue of the patient.

[0073] At block 520, the machine learning model 120 processes the spatially arranged image data by one or more data structures storing one or more portions of executable logic included in the machine learning model 120 to predict one or more responses of the patient by identifying one or more pixels representing one or more locations of discrete biological tissue components of the patient. In some implementations, processing the spatially arranged image data includes selecting a first portion of the spatially arranged image data (e.g., a first patch as described previously with reference to FIG. 3), processing the first portion to identify one or more pixels of the first portion representing one or more locations of discrete biological tissue components corresponding to the first portion as described previously with reference to FIG. 3. In some implementations, processing the spatially arranged image data includes selecting at least one subsequent portion of the spatially arranged image data (e.g., a second patch as described previously with reference to FIG. 3), and processing the at least one subsequent portion to identify one or more pixels in the at least one subsequent portion representing one or more locations of discrete biological tissue components corresponding to the at least one subsequent portion as described previously with reference to FIG. 3.

[0074] In some implementations, the machine learning model 120 can assign a value to each identified location of a discrete biological tissue component for each patch. These values can be weighted based on, for example, the learned predictive power of the discrete biological tissue component. The values corresponding to all patches can be aggregated (e.g., summed, averaged, etc.) to generate an aggregated value, and this value can be used to predict a response. Such a response can be a response to a particular compound, such as a drug. For example, the response can be an amount of reduction in size of a cancerous tumor due to the patient receiving a tumor drug treatment regimen.

[0075] FIG. 6 is a block diagram of an example computer system 600 for providing computational functionality associated with the algorithms, methods, functions, processes, flows, and steps described in the present disclosure (e.g., the method 200 described previously with reference to FIG. 2 and the method 300 described previously with reference to FIG. 3), according to some implementations of the present disclosure. The illustrated computer 602 is intended to encompass any computing device such as a server, desktop computer, laptop / notebook computer, wireless data port, smart phone, personal data assistant (PDA), tablet computing device, or one or more processors of any of these devices, including physical instances, virtual instances, or both. The computer 602 can include an input device to receive

[0076] The computer 602 can act as a client, a network component, a server, a database, a persistence, or a component of a computer system for performing the subject matter described in the present disclosure. The illustrated computer 602 is communicably coupled with the network 630. In some implementations, one or more components of the computer 602 can be configured to operate in different environments, including cloud-computing-based environments, local environments, global environments, and combinations of environments.

[0077] At a high level, the computer 602 is an electronic computing device operable to receive, transmit, process, store, and manage data and information associated with the described subject matter. According to some implementations, the computer 602 can also include, be

[0078] The computer 602 can receive requests from client applications (e.g., executing on another computer 602) over the network 630. The computer 602 can respond to the received requests by processing them using software applications. Requests can also be sent to the computer 602 from internal users (e.g., from a command console), external (or third party), automated applications, entities, individuals, systems, and computers.

[0079] Each component of the computer 602 can communicate using the system bus 603. In some implementations, any or all of the components of the computer 602, including hardware or software components, can interface or be interfaced with through the system bus 603 or interface 604 (or a combination of both). The interface can use an application programming interface (API) 612, a service layer 613, or a combination of the API 612 and the service layer 613. The API 612 can include specifications for routines, data structures, and object classes. The API 612 can be language dependent or language independent. The API 612 can refer to complete interfaces, single functions, or a group of APIs.

[0080] The service layer 613 can provide software services to the computer 602 and other components communicably coupled to the computer 602 (whether directly connected to the computer 602 or not). All of the services provided by the service layer 613 can be accessible through defined interfaces. Software services provided by the service layer 613 can be provided via defined APIs 612. These APIs 612 can be implemented utilizing languages such as JAVA, C++, or languages providing data in extensible markup language (XML) format. Although illustrated as an integrated component of the computer 602, in alternative implementations, the APIs 612 or the service layer 613 can be stand-alone components in relation to other components of the computer 602 and other components communicably coupled to the computer 602. Further, any or all parts of the APIs 612 or the service layer 613 can be implemented as child or sub-modules of another software module, enterprise application, or hardware module without departing from the scope of the present disclosure.

[0081] The computer 602 includes an interface 604. Although illustrated as a single interface 604 in FIG. 6, two or more interfaces 604 can be used according to the needs, desires, or particular implementations of the computer 602 and the described functionality. The interface 604 can be used with, or as, an input device, output device, or both, in conjunction with the network 630 (whether illustrated or not). Generally, the interface 604 can include or

[0082] The computer 602 includes a processor 605. Although illustrated as a single processor 605 in FIG. 6, two or more processors 605 can be used according to the needs, desires, or particular implementations of the computer 602 and the described functionality. Generally, the processor 605 can execute instructions and can manipulate data to perform the operations of the computer 602, including the algorithms, methods, functions, processes, flows, and steps as described in the present disclosure.

[0083] The computer 602 also includes a database 606 that can hold data for the computer 602 and other components connected to the network 630, whether illustrated or not. For example, the database 606 can be an in-memory, conventional, or database that stores data consistent with the present disclosure. In some implementations, the database 606 can be a combination of two or more different database types (e.g., a hybrid in-memory database and conventional database) depending on the particular needs, desires, or particular implementation of the computer 602 and the described functionality. Although illustrated as a single database 606 in FIG. 6, two or more databases (of the same type, different types, or a combination of types) can be used depending on the particular needs, desires, or particular implementation of the computer 602 and the described functionality. Although the database 606 is illustrated as an internal component of the computer 602 in, in alternative implementations, the database 606 can be external to the computer 602.

[0084] The computer 602 also includes a memory 607 that can hold data for the computer 602 or a combination of components connected to the network 630, whether illustrated or not. The memory 607 can store any data consistent with the present disclosure. In some implementations, the memory 607 can be a combination of two or more different types of memory (e.g., a combination of semiconductor and magnetic memory) depending on the particular needs, desires, or particular implementation of the computer 602 and the described functionality. Although illustrated as a single memory 607 in FIG. 6, two or more memories 607 (of the same, different, or a combination of types) can be used depending on the particular needs, desires, or particular implementation of the computer 602 and the described functionality. Although the memory 607 is illustrated as an internal component of the computer 602 in, in alternative implementations, the memory 607 can be external to the computer 602.

[0085] The applications 608 can be algorithmic software engines that provide functionality depending on the particular needs, desires, or particular implementation of the computer 602 and the described functionality. For example, the applications 608 can function as one or more components, modules, or applications. Further, although illustrated as a single application 608, the applications 608 can be implemented as multiple applications 608 on the computer 602. Further, although illustrated as internal to the computer 602, in alternative implementations, the applications 608 can be external to the computer 602.

[0086] The computer 602 can also include a power supply 614. The power supply 614 can include a rechargeable or non-rechargeable battery that can be configured to be either user- or non-user-replaceable. In some implementations, the power supply 614 can include power-conversion and management circuits including recharging, standby, and power- management functionality. In some implementations, the power supply 614 can include a power plug to allow the computer 602 to be plugged into a wall socket or power supply, for example, to power the computer 602 or to recharge a rechargeable battery.

[0087] There can be any number of computers 602 associated with or external to a computer system that includes the computer 602, each computer 602 communicating over the network 630. Furthermore, the terms "client," "user," and other appropriate terminology can be used interchangeably as appropriate, without departing from the scope of the present disclosure. Moreover, the present disclosure contemplates that many users can use one computer 602, and that one user can use multiple computers 602.

[0088] Implementations of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Software implementations of the described subject matter can be implemented as one or more computer programs. Each computer program can include one or more modules of computer program instructions encoded on a tangible, non-transitory, computer-readable computer storage medium for execution by, or to control the operation of, data processing apparatus. Alternatively or additionally, the program instructions can be encoded in / on an artificially generated propagated signal, for example, a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of computer storage mediums.

[0089] The terms“data processing apparatus,”“computer,” and“electronic computer device” (or equivalent as understood by one of ordinary skill in the art) refer to data processing hardware. For example, a data processing apparatus can include all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also include special purpose logic, including, for example, a central processing unit (CPU), a field programmable gate array (FPGA), or an application-specific integrated circuit (ASIC). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of

[0090] A computer program, which can also be referred to or described as a program, software, a software application, a module, a software module, a script, or code, can be written in any form of programming language. The programming language can include for example a compiled language, an interpreted language, a declarative language, or a procedural language. The program can be deployed in any form, including as a stand-alone program, a module, a component, a subroutine, or a

[0091] The methods, procedures, or logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The methods, procedures, or logic flows can also be performed by special purpose logic circuitry, and the apparatus can also be implemented as special purpose logic circuitry, e.g., a CPU, a FPGA, or an ASIC.

[0092] Computers suitable for the execution of a computer program can be based on one or more general and special purpose microprocessors and other types of CPUs. The essential elements of a computer are a CPU for performing or executing instructions and one or more memory devices for storing instructions and data. Generally, a CPU can receive instructions and data from memory (and write data to memory). A computer can also include, or be operatively coupled to, one or more mass storage devices for storing data. In some implementations, a computer can receive data from and transmit data to mass storage devices, such as magnetic, magneto-optical drives or optical drives. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive).

[0093] Computer-readable media (transitory or non-transitory, as appropriate) suitable for storing computer program instructions and data include all forms of permanent / non-permanent and volatile / non-volatile memory, media and memory devices. Computer-readable media can include, for example, semiconductor memory devices such as random access memory (RAM), read only memory (ROM), phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), and flash memory devices. Computer-readable media can also include, for example, magnetic devices such as tape, cartridges, cassettes, and internal / removable disks. Computer-readable media can also include magneto-optical disks and optical storage such as digital video disk (DVD), CD ROM, DVD + / - R, DVD-RAM, DVD-ROM, HD-DVD, and BLURAY. Memory can store various objects or data, including caches, classes, frameworks, applications, modules, backup data, jobs, web pages, web page templates, data structures, database tables, repositories, and dynamic information. Types of objects and data that can be stored in memory include parameters, variables, algorithms, instructions, rules, constraints, and references. Moreover, memory can include log files, policy files, security or access files, and reporting files. Processors and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0094] Implementations of the subject matter described in this disclosure can be implemented on a computer having a display device, e.g., a cathode ray tube (CRT), a liquid crystal display (LCD), a light emitting diode (LED), and a plasma display, for providing a user interface with a user, including displaying information to the user and receiving input from the user. The display device can include a keyboard and a pointing device, including, for example, a mouse, a trackball, or a trackpad. User input can also be provided to the computer through the use of a touchscreen, e.g., a tablet computer surface with pressure sensitivity or a multi-touch touchscreen using capacitive or electric sensing. Other kinds of devices can be used to provide interaction with a user as well, including receiving user feedback, including sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback. Input from a user can be received in the form of acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device used by the user. For example, a computer can send web pages to and receive web pages from a web browser running on a user's client device in response to requests received from the web browser.

[0095] The term "graphical user interface" or "GUI" can be used in the singular or the plural to describe one or more graphical user interface and each of the displays of a particular graphical user interface. Therefore, a GUI can represent any graphical user interface, including, but not limited to, a web browser, a touch screen, or a command line interface (CLI), that processes information and efficiently presents information results to a user. Generally, a GUI can include a number of user interface (UI) elements, some or all of which can be associated with a web browser, such as interactive fields, drop-down lists, and buttons. These and other UI elements can be related to or represent functionality of the web browser.

[0096] Implementations of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having one or both of a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or a combination of one or more such computing systems. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN), a radio access network (RAN), a metropolitan area network (MAN), a wide area network (WAN), Worldwide Interoperability Microwave Access (WIMAX), a wireless local area network (WLAN) (e.g., using 802.11a / b / g / n or 802.20 or combinations of protocols), all or a portion of the Internet, or any other communication system or systems at one or more locations (or combinations of communication systems). The network can communicate information using either or both digital or analog communications techniques. Examples of digital communication techniques include communication among open systems interconnection (OSI) model- compliant devices; examples of analog communication techniques include communication using modulated electromagnetic signals.

[0097] The computing system can include clients and servers. A client and server can generally be remote from each other and typically interact through a communication network. The relationship of client and server can arise by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0098] The clustered file system can be any type of file system accessible from multiple servers for reading and updating. Locking or consistency tracking can not be necessary as the locking of the swap file system can be done at the application layer. In addition, the Unicode data files can be different from the non-Unicode data files.

[0099] Although this specification contains many specific implementation details, these should not be construed as limitations on the scope of what can be claimed, but as descriptions of features that can be specific to particular implementations. Certain features that are described in this specification in the context of separate implementations can also be implemented in combination with each other. Conversely, various features that are described in the context of a single implementation can also be implemented separately from that single implementation or in any suitable subcombination. Moreover, although features can be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination and the claimed combination can be directed to a subcombination or variation of a subcombination.

[0100] In the foregoing description, embodiments of the application have been described with reference to numerous specific details that can vary with implementation. As such, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. The sole and exclusive indicator of the scope of the application, and what is intended by the applicants to be the scope of the application, is the literal and equivalent scope of the claims that issue from this application (including any subsequent amendments or modifications). Any definitions of terms in this detailed description are for purposes of interpreting the claims, and are not limiting on the terms as interpreted by one of ordinary skill in the art. Furthermore, recitation of multiple steps or entities in a claim is not intended to imply that the associated steps or entities were required to be performed in that order, or that the associated steps or entities were required to be performed at all, unless precedence to such order or requirements is expressly given in the claim.

[0101] Particular implementations of the subject matter have been described. Other implementations, alterations, and permutations of the described implementations are within the scope of the following claims as interpreted in accordance with the principles of patent law, including the Patent Act, and 35 U.S.C. § 101. While operations can be described as following a certain order, this is not meant to be limiting. Rather, certain operations can be reordered, added, or omitted without departing from the spirit of the described implementations. Moreover, where appropriate, elements, components, and / or modules from one described implementation can be used in a different implementation, and the examples given with respect to one described implementation can be replaced with examples given with respect to another described implementation, as will be apparent to those of skill in the art having benefit of the present disclosure. It is intended that the following claims be interpreted as broadly as reasonably permitted by the law.

[0102] In addition, the separation or integration of various system modules and components in the previously described implementations should not be understood as requiring such separation or integration in all implementations, and it should be understood that the described program components and systems can typically be integrated in a single software product or packaged into multiple software products.

[0103] Thus, the previously described example implementations do not limit or restrict the present disclosure. Other changes, substitutions, and alterations are also possible without departing from the spirit and scope of the present disclosure.

[0104] In addition, any claimed implementation is considered to be at least applicable to: a computer-implemented method; a non-transitory computer-readable medium storing computer-readable instructions to perform a computer-implemented method; and a computer system comprising a computer memory operatively interconnected with a hardware processor configured to perform a computer-implemented method or instructions stored on a non-transitory computer-readable medium.

[0105] A number of embodiments of these systems and methods have been described. Nevertheless, it will be understood that various modifications can be made without departing from the spirit and scope of the present disclosure.

Claims

1. A data processing system, comprising: A computer-readable storage medium, the computer-readable storage medium including computer-executable instructions; and At least one processor is configured to execute executable logic, the executable logic including at least one artificial neural network trained to predict one or more patient responses to a chemical substance by recognizing one or more discrete biological tissue components in a biological image, wherein while the at least one processor is executing the computer-executable instructions, the at least one processor is configured to perform operations including: Receive image data representing the patient's bioimage; Generate image patch data representing multiple image patches, wherein each of the multiple image patches includes a discrete portion of the biological image; Predicting a patient's response to one or more chemicals by processing the image patch data using one or more data structures, said one or more data structures storing one or more portions of executable logic included in the artificial neural network, wherein processing the image patch data includes: For each image patch, identify one or more pixels in that image patch that represent the location of one or more discrete biological tissue components of the patient; For each image patch, a value is assigned to one or more pixels that are identified for that image patch; For each image patch, a weighting value is determined for that image patch, wherein the weighting value for each image patch is determined based on the predictive power of discrete biological tissue components in that image patch; the predictive power of discrete biological tissue components in the image patch represents the utility of the artificial neural network in predicting patient responses by learning from the discrete biological tissue components. For each image block, the assigned values ​​of the one or more identified pixels in the image block are weighted by using the weighting value for that image block; Aggregate the weighted values ​​on the image patch to generate an aggregation weight value; and The clustering weight values ​​are used to predict one or more of the patient's responses to the chemical substance.

2. The data processing system according to claim 1, wherein the operation further comprises: Generate preprocessed image patch data, which represents a preprocessed image patch for each image patch; For each image patch, generating preprocessed image patch data includes: identifying one or more pixels in the image patch that represent one or more biological tissue locations, and normalizing the color of the one or more biological tissue locations; as well as The image block data processed by the one or more data structures storing one or more portions of executable logic included in the artificial neural network includes the preprocessed image block data.

3. A method implemented by at least one processor executing executable logic, the executable logic including at least one artificial neural network trained to predict one or more patient responses to a chemical substance by recognizing one or more discrete biological tissue components in a biological image, the method comprising: Receive image data representing the patient's bioimage; Generate image patch data representing multiple image patches, wherein each of the multiple image patches includes a discrete portion of the biological image; Predicting a patient's response to one or more chemicals by processing the image patch data using one or more data structures, said one or more data structures storing one or more portions of executable logic included in the artificial neural network, wherein processing the image patch data includes: For each image patch, identify one or more pixels in that image patch that represent the location of one or more discrete biological tissue components of the patient; For each image patch, a value is assigned to one or more pixels that are identified for that image patch; For each image patch, a weighting value is determined for that image patch, wherein the weighting value for each image patch is determined based on the predictive power of discrete biological tissue components in that image patch; the predictive power of discrete biological tissue components in the image patch represents the utility of the artificial neural network in predicting patient responses by learning from the discrete biological tissue components. For each image block, the assigned values ​​of the one or more identified pixels in the image block are weighted by using the weighting value for that image block; Aggregate the weighted values ​​on the image patch to generate an aggregation weight value; and The clustering weight values ​​are used to predict one or more of the patient's responses to the chemical substance.

4. The method of claim 3, further comprising: Generate preprocessed image patch data, which represents a preprocessed image patch for each image patch; For each image patch, the generated preprocessed image patch data includes: Identify one or more pixels in the image block that represent one or more biological tissue locations, and normalize the color of the one or more biological tissue locations; as well as The image block data processed by the one or more data structures storing one or more portions of executable logic included in the artificial neural network includes the preprocessed image block data.

5. The data processing system according to any one of claims 1-2 or the method according to any one of claims 3-4, wherein the artificial neural network comprises a convolutional neural network.

6. A data processing system, comprising: A computer-readable storage medium, the computer-readable storage medium including computer-executable instructions; and At least one processor, configured to execute executable logic including at least one artificial neural network trained to predict a patient's response to a chemical substance, wherein while executing computer-executable instructions, the at least one processor is configured to perform operations including: Receive image data representing the patient's bioimage; The image data is processed by the one or more data structures storing one or more portions of executable logic included in the artificial neural network to predict the patient's response to the chemical substance by identifying one or more pixels representing one or more discrete biological tissue components of the patient. The processing of the image data includes, at each of a series of time steps: Obtain the current location data for this time step, where the current location data defines the position of the current block of the biological image; The current block of the biological image is processed using the feature extraction module of an artificial neural network to generate a feature representation of the current block of the biological image. The feature representation characterizes the position of discrete biological tissue components in the current block of the biological image. The feature representation of the current block of the biological image is processed by the localization module of the artificial neural network to generate the next position data for the next time step. The next position data defines the position of the next block of the biological image to be processed in the next time step. Provide the next position data for processing the next time step in the series of time steps; and The processing further includes generating a response score based on the feature representation of the segments of the biological image, the response score defining the patient's predicted response to the chemical substance.

7. The data processing system according to claim 6, wherein the biological image includes an immunohistochemical image.

8. The data processing system according to claim 6 or 7, wherein the artificial neural network comprises a deep recurrent attention model.

9. The data processing system according to any one of claims 6-8, wherein the operation further comprises: Generate preprocessed image data representing the preprocessed biological image; The generation of preprocessed image block data includes identifying one or more pixels in the biological image that represent one or more biological tissue locations, and normalizing the color of the one or more biological tissue locations. as well as The image data processed by the one or more data structures storing one or more portions of executable logic included in the artificial neural network includes the preprocessed image data.

10. The data processing system according to any one of claims 6-9, wherein the patient's response to the chemical substance includes a reduction in the size of the patient's tumor.

11. A method implemented by at least one processor executing executable logic, the executable logic including at least one artificial neural network trained to predict a patient's response to a chemical substance, the method comprising: Receive image data representing the patient's bioimage; The image data is processed by the one or more data structures storing one or more portions of executable logic included in the artificial neural network to predict the patient's response to the chemical substance by identifying one or more pixels representing one or more discrete biological tissue components of the patient. The data processing includes, at each of a series of time steps: Obtain the current location data for this time step, where the current location data defines the position of the current block of the biological image; The current block of the biological image is processed using the feature extraction module of an artificial neural network to generate a feature representation of the current block of the biological image. The feature representation characterizes the position of discrete biological tissue components in the current block of the biological image. The feature representation of the current block of the biological image is processed by the localization module of the artificial neural network to generate the next position data for the next time step. The next position data defines the position of the next block of the biological image to be processed in the next time step. Provide the next position data for processing the next time step in the series of time steps; and The processing further includes generating a response score based on the feature representation of the segments of the biological image, the response score defining the patient's predicted response to the chemical substance.

12. The method of claim 11, wherein the biological image comprises an immunohistochemical image, and / or wherein the artificial neural network comprises a deep recurrent attention model.

13. The method according to claim 11 or 12, further comprising: Generate preprocessed image data representing the preprocessed biological image; The generation of preprocessed image block data includes identifying one or more pixels in the biological image that represent one or more biological tissue locations, and normalizing the color of the one or more biological tissue locations. as well as The image data processed by the one or more data structures storing one or more portions of executable logic included in the artificial neural network includes the preprocessed image data.

14. The method according to any one of claims 11-13, wherein the patient's response to the chemical substance includes a reduction in the size of the patient's tumor.

Images