Direct Medical Treatment Prediction Using Artificial Intelligence

Abstract

Disclosed herein is a device that receives image data corresponding to a patient's anatomy. The device applies the image data to one or more feature models trained using training data that pairs anatomical images with anatomical feature markers, and receives as output from the one or more feature models a score for each of a plurality of anatomical features corresponding to the image data. The device applies the scores as input to a treatment model, which is trained to output a prediction of a measure of effectiveness of a particular treatment based on the features of the patient's anatomy. The device receives data representing a predicted measure of effectiveness of a particular treatment as output from the treatment model.

Description

[Background technology]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Utility Patent Application No. 17 / 541,936, filed December 3, 2021, which is incorporated by reference in its entirety herein.

[0002] (background) The present invention generally relates to the use of artificial intelligence (AI) to directly determine a patient-specific treatment or management of a disease that will benefit the patient, rather than a manual selection based on a diagnosis. Currently, an autonomous AI system uses machine learning or other optimization techniques to determine a patient's diagnosis, and a clinician then uses this AI diagnosis, in addition to other relevant patient and population information, to subjectively determine a patient-specific management or prescribe a patient-specific treatment (also referred to herein as an "intervention"). Thus, while an autonomous AI diagnosis can determine a diagnosis without human oversight, it still relies on the clinician to interpret the diagnosis and then decide on an intervention in the context of the patient's entire case. However, this intermediate subjective decision step suffers from high inter- and intra-clinician variability, i.e., temporal and other variations. Furthermore, the interaction between artificial intelligence and clinicians is variable, often with unforeseen risks, and is known to worsen rather than improve outcomes. Furthermore, obtaining the highest quality reference standard ("ground truth") and training an autonomous AI model to generate a diagnosis can be ethically problematic and expensive. If the reference standard relies on clinician expertise, such as subjective reading of images, the reference standard can be noisy. When clinical outcomes (which combine the effects of both the accuracy of the diagnostic process as well as the precision of the treatment or management selection process) rather than intermediate diagnoses can instead be used as the reference standard to train the AI, this subjective noisy step is eliminated, with the potential for higher performance. Summary of the Invention [Means for solving the problem]

[0003] (summary) Disclosed herein are systems and methods that use machine learning to output an indication of whether a patient will benefit from a specific intervention (e.g., in terms of clinical outcomes) without first requiring a prediction of a diagnosis for the patient. Advantageously, the disclosed systems and methods eliminate the need for subjective and noisy clinician interpretation of the diagnostic output, since the disclosed machine learning models directly output whether a specific patient will benefit from a specific intervention, which may be a binary recommendation or prescription or a likelihood or other probabilistic output. Still further, verifying a diagnosis from an artificial intelligence typically involves expensive specialist, radiation, or other potentially harmful processes for both cases and normals, and the effort in combining those decisions, whereas determining the extent to which a patient will benefit is a low-cost investigation, so fewer resources are required for validation of the intervention, and the patient risk of harm is lower. Moreover, obtaining truth data regarding whether a patient will benefit can be much more accurate than doing so for a diagnosis, since clinical outcomes are typically much easier to observe than accurately diagnosing a condition.

[0004] In one embodiment, a device receives sensor data from an electronic device that monitors a patient. The device accesses a machine learning model that is trained using training data that pairs information about the patient with indicators that explain whether a particular treatment has produced a positive outcome. The device applies the sensor data to the machine learning model and receives as output from the machine learning model data that represents the likelihood that the patient will benefit from one or more treatments.

[0005] In an embodiment, a device receives image data corresponding to a patient's anatomy. The device applies the image data to a feature extraction model, trained using training data that pairs anatomical images with anatomical feature markers, and receives as output from the feature extraction model a score for each of a plurality of anatomical features corresponding to the image data. The device applies the scores as input to a treatment model, which is trained to output a prediction of the likelihood of effectiveness of a particular treatment based on the features of the patient's anatomy as described in the data structure. The device then receives data from the treatment model representing the predicted likelihood of effectiveness of a particular treatment. Advantageously, the treatment model does not have access to the image data, thus ensuring that the treatment model does not use information from the image (e.g., skin color, gender information, etc.) that may bias the decision. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 illustrates one embodiment of a system environment for implementing a treatment decision tool.

[0007] [Diagram 2] FIG. 2 illustrates one embodiment of exemplary modules and databases used by the treatment decision tool in directing treatment output using artificial intelligence.

[0008] [Diagram 3] FIG. 3 illustrates one embodiment of a multi-step model used to output a treatment.

[0009] [Figure 4] FIG. 4 illustrates one embodiment of a single-stage model that is trained to output a treatment based on input sensor data.

[0010] [Diagram 5]FIG. 5 illustrates one embodiment of a multitask model with branches that are tailored to output different treatments, as well as a shared layer that is used to adapt patient data.

[0011] [Figure 6] FIG. 6 is a flowchart of an exemplary process for using a multi-step model to directly output a treatment, according to an embodiment.

[0012] [Figure 7] FIG. 7 is a flowchart of an exemplary process for using a single-stage model to directly output a treatment, according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] The figures depict various embodiments of the present invention for illustrative purposes only. Those skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles of the present invention as described herein.

[0014] (Detailed Description) (a) Overview Figure 1 illustrates one embodiment of a system environment for implementing a treatment decision tool. As depicted in Figure 1, environment 100 includes a client device 110, a network 120, a treatment decision tool 130, and patient data 140. The elements of environment 100 are merely exemplary, and fewer or more elements may be incorporated into environment 100 to achieve the functionality disclosed herein.

[0015] The client device 100 is a device in which input of patient data may be provided and one or more recommended treatments for the patient may be output whereby the treatment is determined by the treatment determination tool 130. The term "patient data," as used herein, may refer to any data describing a patient, including images of the patient's anatomy, biometric sensor data, physician's notes, and the like. The client device 110 may launch applications installed thereon or may have a browser installed thereon through which applications are accessed, the applications implementing some or all of the functionality of the treatment determination tool 130 and / or communicating information to and from the treatment determination tool 130. The applications may include a user interface through which a user may input data into the client device 110. The user interface may be graphical and the user may manually input patient data (e.g., through a keyboard or touch screen). The user interface may additionally or alternatively be a biometric sensor and the patient data is automatically sensed and transmitted by the application to the treatment determination tool 130. The user interface may be used to access existing patient data stored in patient data 140, which may then be communicated to the treatment decision tool 130 for processing.

[0016] Client device 110 may be any device capable of transmitting data communications over network 120. In an embodiment, client device 110 is a consumer electronics device such as a laptop, a smart phone, a personal computer, a tablet, a personal computer, etc. In an embodiment, client device 110 may be any device that senses, is a sensor, or incorporates patient data (e.g., motion data, blood saturation data, respiration data, or any other biometric data).

[0017] The network 120 may be any data network capable of carrying data communications between the client device 110 and the treatment decision tool 130. The network 120 may be, for example, the Internet, a local area network, a wide area network, or any other network.

[0018] The treatment decision tool 130 receives patient data from the client device 110 and outputs therefrom a treatment that the patient is likely to improve upon. Further details of the operation of the treatment decision tool 130 are discussed below with reference to FIGURE 2. The operation of the treatment decision tool 130 may be instantiated, in whole or in part, on the client device 111 (e.g., through an application installed on the client device 110 or accessed by the client device 110 through a browser).

[0019] Patient data 140 is a database that stores records of data for one or more patients. Patient data 140 may be hospital records, patient personal records, physician notes, etc. Patient data 140 may be co-located with client device 110 and / or treatment decision tool 130.

[0020] 2 illustrates one embodiment of exemplary modules and databases used by the treatment decision tool in directly outputting a treatment using artificial intelligence. As depicted in FIG. 2, the treatment decision tool 130 includes a patient data readout module 231, a treatment decision module 232, a feature model selection module 233, a feature vector module 234, patient data 240, and a feature model 241. The modules and databases depicted in FIG. 2 are merely exemplary, and the treatment decision tool 130 may include more or fewer modules and / or databases and still achieve the functionality described herein. Additionally, the modules and / or databases of the treatment decision tool 130 may be instantiated, in whole or in part, on the client device 110 and / or one or more servers.

[0021] The patient data readout module 231 receives patient data (e.g., from the client device 110 and / or the patient data 140). In two exemplary embodiments, the patient data may be image data corresponding to the patient's anatomy or sensor data from an electronic device monitoring the patient. These exemplary embodiments will be directed to detailed treatment in this disclosure, although any other form of patient data may be received by the patient data readout module 231.

[0022] The treatment decision module 232 accesses one or more machine learning models, applies the received patient data to the one or more machine learning models, and receives as output from the one or more machine learning models data representing predicted measures of efficacy of one or more treatments. Focusing on an embodiment in which the received patient data includes image data, FIG. 3 illustrates one embodiment of a multi-stage model used to output a treatment.

[0023] As depicted in FIG. 3, image data 310 is received (e.g., from client device 110 or patient data 240). The term "image data," as used herein, may refer to an image captured using an image sensor. The image may be any type of image, including, for example, a grayscale image, a red-green-blue image, an infrared image, an X-ray image, an optical coherence tomography image, an ultrasound image, or any other type of image. The image data corresponds to the patient's anatomy. That is, the image includes, at least in part, a depiction of a human. The features may be external (e.g., skin, eyes, etc.) or internal (e.g., an image of an organ such as the retina or liver).

[0024] The treatment decision module 232 applies the image data to one or more feature models 320 (e.g., accessed from feature model 241). The image data may include data from one or more images. The term "feature model" (or "feature extraction model"), as used herein, may refer to a model that is trained to identify or extract one or more features in an image. The terms "feature model" and "feature extraction model" are used synonymously herein. When used in the singular, "feature model" or "feature extraction model" may refer to a single model or an ensemble of two or more models. The term "feature" as used herein may refer to an object in an image. The objects may include anatomical objects (e.g., blood vessels, organs (e.g., optic nerves), etc. The objects may also include biomarkers, which may be abnormalities such as lesions, fissures, dark spots, and any other abnormality to normal human anatomy. The feature model may be trained using labeled training images, which show at least a portion of human anatomy and are labeled with at least a score (e.g., likelihood or probability) of whether the image contains a biomarker (e.g., feature). The labels are also referred to herein as anatomical feature labels. The training images are also referred to herein as anatomical images. In an embodiment, the markers may include identification of one or more specific biomarkers in the image. The markers may include additional information, such as other objects in the image and one or more body parts that the training images depict. Further discussion of the construction, training, and use of feature models is disclosed in commonly owned U.S. Pat. No. 10,115,194, filed April 6, 2016, and issued October 30, 2018, the disclosure of which is incorporated herein by reference in its entirety.

[0025] In an embodiment, additional data to the image data is input into the feature model 320. The additional data may be any patient data. In such an embodiment, the training data for the feature model 320 may include pairings of images with other patient data. For example, blood pressure data, oxygen saturation data, etc. may be included along with the training images. The feature model 320 outputs a probability that one or more features are present in the image data, or directly outputs a binary decision of whether a feature is present in the image data based on a probability exceeding a corresponding threshold. In an embodiment in which the feature model 320 is trained to receive additional patient data as input, the probability may be more accurately assessed, thus allowing for a more accurate assessment of whether a feature is present in the input image data.

[0026] In one embodiment, the image data 310 is applied by the treatment decision module 232 to a single feature model 320, which outputs data representing one or more corresponding features. As noted above, this data may include a probability that the image data includes one or more features, or may include a binary decision that a feature is included in the image data.

[0027] In an embodiment, the image data 310 is applied by the treatment decision module 232 to a number of feature models 320. Returning momentarily to FIG. 2, the feature model selection module 233 may select one or more of the feature models 320 to which a given image of image data should be applied. In an embodiment, the feature selection module 223 may determine the body part to which the input image data corresponds. The feature models 320 may be any machine learning model, such as a deep learning model, a convolutional neural network (CNN), an ensemble model, a biomarker-based system of multiple machine learning models, etc. The feature models 320 may each correspond to one or more individual body parts. The feature model selection module 223 may then input the image data into one or more feature models that correspond to the body part to which the image data corresponds. For example, if images are taken of the retina, where one image includes the optic disc, one image includes the fovea, and one image includes both, these images may be applied to the feature model for the optic disc, the feature model for the fovea, and both models, respectively. More granular data may be used, such as a particular body part within a body part or the location of a body part or object within an image. For example, images centered around the optic disc may be applied to one feature model, and images centered around the fovea may be applied to another feature model, even though some of the images may include both the optic disc and the fovea. A feature model 320 may be selected by feature selection module 223 based on other than the body part depicted in the image, such as any other characteristics of the patient or disease (e.g., a specific age range, a disease that requires analysis of both the ear and the bladder, etc.). In an embodiment, other patient data applied to feature model 320 may also be used by feature model selection module 223 to select a feature model to apply to the image (e.g., a scenario in which multiple feature models are trained for a given body part but tuned using different patient data).

[0028] As discussed above, the treatment decision module 232 determines the features 330 based on the output from the feature model 320. The feature vector module 234 aggregates the features 330 into a feature vector 340. The term "feature vector," as used herein, may refer to a data structure that includes each of the different features. The feature vector may map the different features to auxiliary information. For example, if the image data includes images that correspond to different locations of a body part, the feature vector may map features identified from those images to those respective different locations of the body part. As an example, if the images are retinal images, with one image taken for each quadrant of the retina, the feature vector may include four data points that include the respective features identified in the images for each of the four quadrants.

[0029] The treatment decision module 232 applies the feature vector 340 to a treatment model 350. The treatment model 350 is trained to output a prediction of a measure of efficacy of one or more specific treatments for a patient based on the input feature vector and on the characteristics of the patient's anatomy as described in the data structure. The treatment model 350 may be any machine learning model (e.g., deep learning model, convolutional neural network (CNN), etc.). The training data may include data manually labeled by a patient indicating that the patient was in good or bad shape after the treatment, and the manual labels are paired to the image feature vector, and optionally with other patient data, to a label corresponding to what the patient indicated. The labels may alternatively or additionally be manually labeled by a physician. In an embodiment, patient data such as vitals or descriptors of patient health may be monitored from the time of the treatment. The patient data may be compared to data from a time prior to the treatment. Monitoring both before and after the time of the treatment may be bounded by a maximum threshold amount of time. In response to determining that the patient data indicates improvement, the feature vector may be labeled with an indicator of the treatment and that the patient has improved. Similarly, images may be labeled to indicate no improvement or a worsening condition based on the monitored patient data. In an embodiment, the degree to which the patient has improved or worsened may be labeled for the images and / or feature vectors. In some embodiments, the labeling of the training data may be automatic, and the patient data and / or disease data and / or treatment data are labeled with an outcome. These labels may alternatively be applied manually (e.g., by a clinician monitoring the patient response to a treatment).

[0030] The treatment decision module 232 receives as output from the treatment model 350 a prediction of a measure of effectiveness of one or more particular treatments for the patient based on the features of the patient's anatomy as described in the data structure (e.g., the likelihood that the treatment will be effective for the patient according to a predetermined criterion therefor). As output of the feature model 320, the treatment model 350 may output probabilities corresponding to the predicted effectiveness of multiple candidate treatments, or alternatively may directly output an identifier of a treatment that has a probability that exceeds a threshold. The treatment decision module 232 determines one or more treatments that are likely to be effective for the patient based on the output of the treatment model 350, and outputs the determined one or more treatments to a user (e.g., a physician, a patient, or another medical clinician).

[0031] In some embodiments, rather than using a two-stage model as depicted in Figure 3, a single-stage model may be used to directly predict an effective treatment for a patient based on underlying patient data. Exemplary single-stage models are discussed with reference to Figures 4 and 5. However, these are merely representative, and any form of machine learning model that is trained to directly output a treatment based on patient data may be used to determine one or more effective treatments for a patient.

[0032] 4 illustrates one embodiment of a single-stage model trained to output a therapy based on input sensor data. As depicted in FIG. 4, sensor data 405 is received from a patient. The patient data readout module 231 may receive the sensor data. The term "sensor data," as used herein, may refer to data obtained from sensors that monitor physical attributes of a patient. Exemplary sensors may include sensors including electroencephalogram (EEG) sensors, motion sensors, retinal or ear canal imaging, respiration sensors, image sensors, video sensors, acceleration data, etc.

[0033] The treatment decision module 232 applies the sensor data 405 into a treatment model 410, which may be any machine-learned model (e.g., CNN) described herein. The treatment model 410 may be trained using the historical sensor data as labeled with a treatment outcome (alone and / or in combination with other historical sensor data). In an embodiment, the label may be applied manually. In another embodiment, the patient's condition may be monitored before and after treatment (e.g., either unilaterally or bilaterally, as bounded by a threshold time period), and the label may be automatically determined from the indications of the condition improving, staying the same, or worsening following treatment. Similar to the model 350, the treatment model 406 outputs data representing the effectiveness of treatments 415, from which the treatment decision module 232 determines treatments that are likely to be effective in treating the patient and provides an identification of those determined treatments to the user. In some embodiments, the model 410 may be trained to output multiple treatments. In some embodiments, different ones of the model 410 are used to determine different treatments.

[0034] 5 illustrates an embodiment of a multitask model with branches tuned to output different treatments and a shared layer used to fit patient data. In general, the treatment decision tool 130 receives sensor data 505 and applies it to a treatment model 506, resulting in an output treatment 550, in the same manner described with respect to like reference numbers in FIG. 4. The treatment model 506 is a multitask model with a shared layer 510 and branches 520, 530, and 540, as depicted. The shared layer 510 is trained using training data for each branch 520, 530, and 540. Each of the branches 520, 530, and 540 is trained to output a probability of effectiveness of a different one of the treatments 550. For example, if a patient is having difficulty breathing, branch 520 may output the probability that a CPAP machine will assist the patient, branch 530 may output the probability that taking in more oxygen will assist the patient, and branch 540 may output the probability that a lung biopsy would benefit the patient. As in Figure 4, three branches are depicted, but any number of branches may be used.

[0035] The model 506 is trained using training data that indicates input sensor data paired with an indicator indicating whether a patient associated with that sensor data improved or worsened after receiving a treatment associated with one of the branches of the model 506. Thus, the training data includes different sets of training data, each set with the same input (e.g., sensor data) but different indicators corresponding to the effectiveness of different treatments. During training, a training example from one of the sets of training data is selected, and standard backpropagation is performed through the branch (520, 530, or 540) of the model 506 that corresponds to the treatment of the selected example, and through the shared layer 510 of the model 506. This process is repeated, eventually selecting a training example from each set, thereby training each of the branches 520, 530, and 540 of the model 506. In an embodiment, backpropagation occurs from clinical outcomes, through treatment options, and back to diagnostic options. Backpropagation may occur through any layer or branch, or layers or branches may be selectively omitted. In some embodiments, the different training sets used to train different branches of the multitask model may differ substantially in terms of size. This may be a result of some treatments being used at high frequency among patients and other treatments being used at much lower frequency, thus resulting in less training data for a given label in a given training set. The treatment decision tool 130 may sample from the different training sets according to a proportion of the training data available across the different training sets. In this way, a treatment for which there is relatively little training data benefits from a treatment for which there is relatively more training data, because the training data for all of the sets is used for the shared layer 510. This may allow a model to be trained for a treatment that would otherwise not have enough training data to train its own model, because the branch for that treatment is aided by the work done by the shared layer 510.

[0036] Technical benefits are thus achieved from using a multi-task model (e.g., neural network) as shown in FIG. 5, as opposed to using a separate neural network for each treatment. In addition to reducing the processing power required by distributing redundant processing across multiple neural networks, accuracy is improved for treatments for which there are only a small number of training instances. By training a shared layer (e.g., layer 510) with training data for multiple treatments, the shared layer approximates the overall model 506 for a given treatment corresponding to a given branch, and does so using relatively less training data than is available for the other branches. Further training of that given branch fine-tunes the prediction for that treatment.

[0037] 6 is a flow chart of an exemplary process for using a multi-phase model to directly output a treatment, according to an embodiment. The process 600 begins with the treatment decision tool 130 receiving 602 image data corresponding to a patient's anatomy (e.g., using the patient data read module 231). The treatment decision tool 130 applies 604 the image data to a feature extraction model that is trained using training data that pairs anatomical images with anatomical feature indicators (e.g., applying the image data to one or more of the feature models 241 using the treatment decision module 232). The treatment decision tool 130 receives 606 a score (e.g., probability or likelihood) for each of a plurality of anatomical features corresponding to the anatomy as output from the feature extraction model, and optionally combines the scores for the plurality of anatomical features into a data structure (e.g., using the feature vector module 234). The treatment decision tool 130 applies 608 the scores (e.g., via a data structure, if used) as inputs to a treatment model (e.g., treatment model 350), which is trained to output a prediction of a measure of the effectiveness of a particular treatment based on the characteristics of the patient's anatomy (e.g., as described in the data structure). The treatment decision tool 130 receives 610 data (e.g., representing treatment 360) as output from the treatment model, which represents the predicted measure of the effectiveness of a particular treatment.

[0038] 7 is a flow chart of an exemplary process for using a single-stage model to directly output a treatment, according to an embodiment. The process 700 begins with the treatment decision tool 130 receiving 702 sensor data from an electronic device monitoring a patient (e.g., using the patient data readout module 231). The treatment decision tool 130 accesses 704 a machine learning model (e.g., model 506), configured to output the likelihood that a particular treatment will result in a positive outcome, the machine learning model being trained using training data that pairs previously acquired sensor data for a plurality of patients with an indicator that describes whether a particular treatment resulted in a positive outcome for each of the plurality of patients. The treatment decision tool 130 applies 706 the received sensor data to the machine learning model, and receives 708 data as output from the machine learning model that represents the likelihood that a patient will benefit from a particular treatment (e.g., representing treatment 550).

[0039] The above disclosure translates to patient benefits if the patient suffers from any number of conditions. For example, if the patient has a sleep apnea condition, the condition may be diagnosed autonomously, and a model may be trained using sleep study data such as EEG data, motion data, respiratory data, and the like, and the model may output a sleep apnea diagnosis. This allows the clinician to prescribe treatment (e.g., the patient should use a CPAP machine). Advantageously, the systems and methods disclosed herein eliminate the need for diagnosis, i.e., a prediction that CPAP will help the patient is output directly and without a diagnosis. The benefit from CPAP is determined with fewer computational resources and without the need for a physician, i.e., if the patient's saturation increases, CPAP will benefit the patient, and thus ground truth data regarding using CPAP, which benefits in this example, is easily obtained without the need for the clinician to manually inspect the input data in the first place and determine whether the patient has sleep apnea. The systems and methods disclosed herein also have the advantage of process integrity in that they lead to maximizing the population benefits of CPAP machines.

[0040] As another example, if a patient has an ear infection, rather than outputting that the patient has an "otitis media" diagnosis (requiring verification by a physician, followed by a prescription), or a specific prescription for "Amoxicillin 200 mg", the system may directly output "Patient would benefit from Amoxicillin". This may be based on training data indicating whether ear infections by patients with similar patient data are flagged based on whether those patients would not benefit from Amoxicillin at all. Again, obtaining ground truth training data for such a model requires much less cost, since the benefit to the patient is self-validating. In some embodiments, for this reason, the model output may be binary (i.e., whether the patient would benefit from a particular treatment), rather than granular (e.g., specific dosage or course of treatment), for which ground truth data is much harder to obtain.

[0041] (b) Summary The foregoing description of embodiments of the invention has been presented for purposes of illustration and is not intended to be exhaustive or to limit the invention to the precise form disclosed. Those skilled in the art will appreciate that numerous modifications and variations are possible in light of the above disclosure.

[0042] Some portions of this description describe embodiments of the invention in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to effectively convey the substance of their work to others skilled in the art. While these operations are described functionally, computationally, or logically, it will be understood that they may be implemented by computer programs or equivalent electrical circuits, microcode, or the like. Further, and without loss of generality, it has proven convenient at times to refer to these arrangements of operations as modules. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combination thereof.

[0043] Any of the steps, operations, or processes described herein may be performed or implemented using one or more hardware or software modules alone or in combination with other devices. In one embodiment, the software modules are implemented using a computer program product comprising a computer-readable medium containing computer program code, which can be executed by a computer processor to perform any steps, operations, or processes described.

[0044] The embodiments of the present invention may also relate to an apparatus for performing the operations herein. The apparatus may be specially constructed for the required purposes and / or may comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a non-transitory tangible computer-readable storage medium or any type of medium suitable for storing electronic instructions, which may be coupled to a computer system bus. Furthermore, any computing system referred to herein may include a single processor or may be an architecture employing a multiple processor design for increased computing power.

[0045] Embodiments of the invention may also relate to products produced by the computing processes described herein. Such products may comprise information resulting from the computing processes, which information may be stored on a non-transitory, tangible computer-readable storage medium and may include any of the embodiments of the computer program products or other data combinations described herein.

[0046] Finally, the language used in the specification has been selected primarily for readability and instructional purposes, and may not have been selected to bound or limit the subject matter of the invention. Accordingly, it is intended that the scope of the invention be limited not by this detailed description, but rather by any claims that issue on an application based thereon. Thus, the disclosure of embodiments of the invention is intended to be illustrative, but not limiting, of the scope of the invention, which is set forth in the following claims.

Claims

1. 1. A method for autonomously predicting efficacy of a treatment for a patient, the method being executed on a computer comprising one or more processors, the method comprising: The method comprises: receiving image data of a patient's anatomy, by the one or more processors; applying, by the one or more processors, the image data to a feature extraction model that pairs anatomical images with anatomical feature markers, the feature extraction model being trained using training data; receiving, by the one or more processors, a score for each of a plurality of anatomical features corresponding to the patient's anatomy as output from the feature extraction model; the one or more processors applying the scores as inputs to a treatment model, the treatment model being trained to output a prediction of a measure of efficacy of a particular treatment based on characteristics of the patient's anatomy; and receiving, by the one or more processors, data representing the predicted measure of effectiveness of the particular treatment as output from the treatment model; A method comprising:

2. The one or more processors determining, based on the image data, a body part to which the image data corresponds; the one or more processors selecting one or more feature models from a plurality of candidate feature models based on a match between each candidate feature model and a given body part; The method of claim 1 further comprising:

3. 2. The method of claim 1, wherein applying the identifications as inputs to the treatment model comprises generating a feature vector that stores, for each anatomical feature of the plurality of anatomical features, its individual identification, and applying the feature vector as inputs to the treatment model.

4. The one or more processors determining whether, with respect to the particular treatment, its predicted measure of effectiveness exceeds a threshold; outputting a recommendation to a user regarding the particular treatment in response to the one or more processors determining that the predicted measure of effectiveness exceeds the threshold; and The method of claim 1 further comprising:

5. 10. The method of claim 1, wherein receiving data representing the predicted measure of effectiveness of the particular treatment as output from the treatment model comprises receiving data representing a separate measure of effectiveness for each of a plurality of candidate treatments as output from the treatment model.

6. 1. A non-transitory computer-readable medium having instructions encoded thereon that, when executed, cause one or more processors to perform operations for autonomously determining a treatment for a patient, the instructions comprising: receiving image data of a patient's anatomy; applying the image data to a feature extraction model that pairs anatomical images with anatomical feature labels, the feature extraction model being trained using training data; receiving, as output from the feature extraction model, a score for each of a plurality of anatomical features corresponding to the patient's anatomy; applying the scores as inputs to a treatment model, the treatment model being trained to output a prediction of a measure of efficacy of a particular treatment based on characteristics of the patient's anatomy; receiving, as output from the treatment model, data representing the predicted measure of effectiveness of the particular treatment; 1. A non-transitory computer-readable medium comprising instructions for performing

7. The instructions further include: determining, based on the image data, a body part to which the image data corresponds; selecting one or more feature models from the plurality of candidate feature models based on a match between each candidate feature model and the given body part; 7. The non-transitory computer-readable medium of claim 6, comprising instructions for:

8. The non-transitory computer-readable medium of claim 6, wherein applying the scores as inputs to the treatment model includes generating a feature vector that stores, for each anatomical feature of the plurality of anatomical features, its individual score, and applying the feature vector as inputs to the treatment model.

9. The instructions further include: determining whether the predicted measure of efficacy for the particular treatment exceeds a threshold; In response to determining that the predicted measure of effectiveness exceeds the threshold, outputting a recommendation to a user regarding the particular treatment.

7. The non-transitory computer-readable medium of claim 6, comprising instructions for:

10. 7. The non-transitory computer-readable medium of claim 6, wherein receiving data representing the predicted measure of effectiveness of the particular treatment as output from the therapeutic model comprises receiving data representing a separate measure of effectiveness for each of a plurality of candidate treatments as output from the therapeutic model.

11. 1. A method for autonomously determining a therapy for a patient, the method being executed on a computer comprising one or more processors, the method comprising: The method comprises: the one or more processors receiving sensor data from an electronic device monitoring a patient; the one or more processors access a machine learning model, the machine learning model configured to output a likelihood that a particular treatment will result in a positive outcome, the machine learning model being trained using training data that pairs previously acquired sensor data for a plurality of patients with an indicator that explains, for each of the plurality of patients, whether the particular treatment resulted in a positive outcome; the one or more processors applying the received sensor data to the machine learning model; receiving, by the one or more processors, data as output from the machine learning model, the data representing the likelihood that the patient will benefit from the particular treatment; A method comprising:

12. The data representing one or more treatments comprises a probability that each of the one or more treatments will benefit the patient, and the method further comprises: the one or more processors determining, for each of the one or more treatments, whether its corresponding probability exceeds a threshold; the one or more processors outputting to a user the one or more treatment recommendations having corresponding probabilities that exceed the threshold; The method of claim 11 , comprising:

13. The method of claim 11 , wherein the machine learning model is a convolutional neural network.

14. 12. The method of claim 11, wherein the machine learning model is a multi-task model comprising a shared layer and branches, the shared layer being trained to determine one or more candidate diagnoses based on the sensor data, each branch corresponding to a different treatment, and each branch being trained to output a likelihood that the different treatment to which the branch corresponds will be effective.

15. 15. The method of claim 14, wherein an amount of training data for a branch corresponding to a given treatment is below a threshold, and the multi-task model enriches the amount of training data by backpropagating the training data with information from different branches.

16. 1. A non-transitory computer-readable medium having instructions encoded thereon that, when executed, cause one or more processors to perform operations for autonomously determining a treatment for a patient, the instructions comprising: receiving sensor data from an electronic device monitoring a patient; accessing a machine learning model, the machine learning model configured to output a likelihood that a particular treatment will result in a positive outcome, the machine learning model being trained using training data that pairs previously acquired sensor data for a plurality of patients with an indicator that explains, for each of the plurality of patients, whether the particular treatment resulted in a positive outcome; applying the received sensor data to the machine learning model; receiving, as output from the machine learning model, data representing the likelihood that the patient will benefit from the particular treatment; 1. A non-transitory computer-readable medium comprising instructions for performing

17. The data representing one or more treatments comprises a probability that each of the one or more treatments will benefit the patient, and the instructions further include: determining, for each of the one or more treatments, whether its corresponding probability exceeds a threshold; outputting to the user a recommendation for each of the one or more treatments having a corresponding probability of exceeding the threshold; 20. The non-transitory computer-readable medium of claim 16 comprising instructions for:

18. 17. The non-transitory computer-readable medium of claim 16, wherein the machine learning model is a convolutional neural network.

19. 17. The non-transitory computer-readable medium of claim 16, wherein the machine learning model is a multi-task model comprising a shared layer and branches, the shared layer being trained to determine one or more candidate diagnoses based on the sensor data, each branch corresponding to a different treatment, and each branch being trained to output a likelihood that the different treatment to which the branch corresponds will be effective.

20. 20. The non-transitory computer-readable medium of claim 19, wherein an amount of training data for a branch corresponding to a given treatment falls below a threshold, and the multi-task model enriches the amount of training data by backpropagating the training data with information from different branches.

Images